Machine Translation: Decoding, Evaluation, and Scale

This builds on Machine Translation, which developed the transformer encoder-decoder that computes $P(\mathbf{y}\mid \mathbf{x})$ for a source $\mathbf{x}$ and a candidate target $\mathbf{y}$. That lesson built the model; this one runs it. Two questions remain: given a trained model, how do we turn its per-step word distributions into a single output sentence, and once we have an output, how do we tell whether it is any good? The first is decoding, the second evaluation, and the lesson closes on what changes when translation has to cover more than one well-resourced language pair.

Decoding with beam search

At generation time the decoder gives a distribution over the next word at every step, and we must turn those distributions into a single output string. The obvious rule — take the single most probable word at each step — is greedy decoding:

Greedy decoding is locally optimal and globally wrong. The word that looks best now can force worse words later, and a high-probability first word can strand the model on a low-probability continuation. But the space of full translations is $|V{|}^T$ strings, far too many to enumerate, and the long-distance dependencies between output words rule out the dynamic-programming shortcut that Viterbi uses for tagging.¹

The practical middle ground keeps a few candidates alive instead of one. Beam search is the standard compromise: keep not one but the $k$ best partial translations at every step, where $k$ (the beam width) is a small constant, and prune back to $k$ after each extension.²

Scoring is done in log probability, so the score of a hypothesis is a running sum that extends by one term per word — turning the product in the chain rule into a sum that never underflows:

One catch follows from the scoring. Because completed hypotheses have different lengths and a model assigns any longer string a lower joint probability, raw log-probability systematically favors short translations. The standard remedy is length normalization: divide the score by the number of words $T$ before comparing complete hypotheses,

Evaluation: BLEU and beyond

We can now produce a translation; the next question is whether it is good, and that needs a definition of good a machine can compute. The two dimensions are adequacy (does the output carry the source's meaning?) and fluency (is it natural in the target language?). Human raters are the gold standard, but they are slow and expensive, so automatic metrics are used to compare systems and to guide development. The oldest and still most cited automatic metric is BLEU.³

BLEU (bilingual evaluation understudy) rests on one intuition: a good candidate translation shares many word n-grams with a human reference translation. It is a precision metric — it asks what fraction of the candidate's n-grams appear in the reference — computed for several n-gram sizes and multiplied by a brevity penalty that punishes translations shorter than the reference (precision alone could be gamed by outputting a single sure word).⁴

For example, take a reference and a candidate:

Reference: the green witch arrived at the house Candidate: the green witch arrived

Unigram precision $p_1$: the candidate has $4$ unigrams (the, green, witch, arrived), and all $4$ appear in the reference, so $p_1=4/4=1$. Bigram precision $p_2$: the candidate's $3$ bigrams are the green, green witch, witch arrived, and all $3$ appear in the reference, so $p_2=3/3=1$. So far precision is perfect — which exposes exactly why the brevity penalty is needed. The candidate has length $c=4$ against reference length $r=7$, so

and even with perfect precision the score is held down to about $0.47$, correctly docking the candidate for dropping half the sentence. The penalty is what stops a short, cherry-picked fragment from scoring as a full translation.

The first example was chosen to isolate the brevity penalty; the second shows the clipping inside modified precision. Take a candidate that repeats a good word to game a naive precision:

Reference: the cat is on the mat Candidate: the the the the

A raw unigram precision would be $4/4=1$ — every word the is in the reference. BLEU forbids this by clipping each candidate n-gram's count to the maximum number of times it appears in any single reference. The reference contains the twice, so the candidate's four the tokens contribute at most $2$ matches:

Clipping is the modified in modified n-gram precision: without it, precision could be pushed to $1$ by any word that occurs in the reference at all. The bigram precision finishes the job — the candidate's three bigrams are all the the, which never occurs in the reference, so $p_2=0/3=0$, and because BLEU is a geometric mean a single zero $p_n$ zeros the whole score. Repetition is penalized from two directions at once.

BLEU has real limits. It is word-based and so sensitive to tokenization, which makes scores hard to compare across setups and weakens it for morphologically rich languages. It is purely local — moving a whole phrase barely changes the score — and it cannot credit a correct paraphrase that happens to use different words. Newer metrics attack these gaps.⁵

• chrF scores overlap of character n-grams instead of word n-grams and combines precision and recall into an F-score. Being sub-word it degrades gracefully under morphology, and it correlates better with human judgments than BLEU across many languages, which is why J&M take it as the default.⁶
• BERTScore replaces exact string matching with embedding similarity: it runs reference and candidate through BERT, greedily matches each token to its most similar counterpart by cosine, and reports precision, recall, and F1 — so a paraphrase with different surface words can still score well.⁷
• COMET (and BLEURT) go further and learn the metric: they train a model on datasets of human quality ratings $(\mathbf{x},\tilde{\mathbf{x}},r)$ to predict the human score $r$ directly from source and candidate, and their outputs correlate most closely with human judgment.⁸

Whatever the metric, comparing two systems needs a significance test — the paired bootstrap — since a small score gap can be noise; and automatic metrics are most trustworthy when measuring changes to a single system, least trustworthy when comparing very different kinds of systems.⁵

Multilingual and low-resource MT

Everything so far assumes a large parallel corpus for the language pair in hand. For most of the world's languages no such corpus exists, and the methods have to change. The Europarl and UN corpora cover a handful of official languages; the long tail of low-resource languages has little or no parallel text, and the field's habit of routing everything through English makes the gap worse.⁹

Backtranslation is the standard way to manufacture training data when parallel text is scarce but monolingual target text is plentiful. Train a rough model in the reverse direction on whatever small bitext exists, run it on the abundant monolingual target text to produce synthetic source sentences, and add these synthetic pairs to the training set for the forward model.¹⁰

The synthetic pairs are noisy on the source side but pristine on the target side, which is the side the model must learn to generate. In practice backtranslation works well — one estimate is that synthetic data yields about two-thirds the gain of the same volume of natural parallel text — and it scales any low-resource language that has raw monolingual text, which nearly all of them do.¹⁰

Bias and ethical issues in MT

The divergences that open the first lesson return here as an ethical problem. When a system translates from a language that leaves gender unmarked into one that forces it, it must guess — and it guesses according to its training data. Translating the Hungarian gender-neutral pronoun ő into English, current systems render ő is a nurse as she is a nurse but ő is a CEO as he is a CEO, and ő is an engineer, a scientist, a baker as he.¹¹ The default is not merely to guess the majority gender; the bias is amplified. Prates and colleagues found MT systems map pronouns to male or female more skewedly than the actual labor statistics would justify, so the output is more stereotyped than the world it was trained on. The WinoMT challenge set makes the same point from the other side: systems translate worse on sentences that describe people in non-stereotypical roles (the doctor asked the nurse to help her), where getting the coreference right cuts against the prior.

The other standing issue is inequity of coverage. Neural MT needs parallel text, most of the world's languages have little or none, and the field's habit of routing translation through English deepens the gap — a low-resource pair is often served only by pivoting through a high-resource one. One response is participatory design: ∀ and colleagues built online groups, mentoring, and infrastructure so speakers of low-resource African languages help develop the systems that serve them, rather than being modeled from the outside.¹² A subtler open problem is knowing what the system does not know. MT is used in medical and legal settings where a human translator may be unavailable — a patient and doctor without a shared language, a judge and a witness. To do no harm there, a system needs calibrated confidence so it can abstain from a translation it is likely to botch, rather than emit a fluent, confident, wrong sentence.¹¹

From Transformer NMT to massively multilingual models

Jurafsky and Martin present the transformer encoder-decoder as the modern MT architecture; the public record fills in where that architecture came from and how far it has since been pushed.

The Transformer was introduced for translation. Vaswani and colleagues' 2017 paper, Attention Is All You Need (NeurIPS 2017), proposed the transformer as a sequence-to-sequence model and evaluated it first on machine translation. On the WMT 2014 English-to-German task it reported $28.4$ BLEU, and on English-to-French $41.8$ BLEU, above the previous best published results including ensembles, while training in a fraction of the time because the model dispenses with recurrence and computes all positions in parallel. The paper's own framing is that attention alone, with no RNN or convolution, suffices — its contribution is the cross-attention and self-attention the first lesson describes.¹³

Subword tokenization has a citable origin too. The BPE scheme MT uses was adapted to NMT by Sennrich, Haddow, and Birch, Neural Machine Translation of Rare Words with Subword Units (ACL 2016). They showed that segmenting into subword units lets a fixed-size vocabulary translate rare and unseen words by composing them from pieces, improving BLEU over a back-off dictionary on English-German and English-Russian. This is the concrete reason MT prefers subwords to words.¹⁴

Backtranslation was measured, not assumed. The two-thirds-of-natural-data figure this lesson cites is from Edunov, Ott, Auli, and Grangier, Understanding Back-Translation at Scale (EMNLP 2018). They generated synthetic source sentences with a reverse model and found that adding them raised BLEU substantially, that sampling or noised beam output made better synthetic data than plain beam search, and that hundreds of millions of backtranslated sentences kept helping — establishing backtranslation as the standard way to exploit monolingual data.¹⁵

One model for two hundred languages. Jurafsky & Martin note the low-resource gap as an open problem; the largest public attack on it is Meta AI's No Language Left Behind effort, No Language Left Behind: Scaling Human-Centered Machine Translation (2022). The NLLB-200 model translates directly among $200$ languages, including many with almost no parallel text, without pivoting through English, and was trained with a mixture-of-experts architecture and heavily mined and backtranslated bitext. Meta reported an average gain of roughly $44\%$ in chrF++ over the prior state of the art on the associated benchmark, and released the model and the FLORES-200 evaluation set publicly. NLLB is the direct realization of the route everything through English critique this lesson raises: it translates low-resource pairs end to end.¹⁶

Where MT is heading: general LLMs as translators. The most recent shift is that general-purpose large language models translate competently with no MT-specific training. GPT-style models prompted with a source sentence and an instruction produce fluent translations, and controlled evaluations (for instance the WMT shared-task findings from 2023 onward) report that they are strong on high-resource pairs and document-level context while still trailing dedicated systems on many low-resource pairs. The practical consequence is that the encoder-decoder specialist and the decoder-only generalist now overlap: the same architecture the first lesson traces from translation is used to translate without being trained for it. The dividing line has moved from which architecture to how much data the target language has, which returns the field to the coverage problem the ethics section names.¹⁷

The through-line is worth restating. Translation set the specification — map one sequence to another of different length and order — that the RNN encoder-decoder was built to meet, that attention was built to fix, and that the transformer inherited wholesale. Every one of those inventions now lives in the general transformer and drives question answering, dialogue, and the large language models that followed. Machine translation is where the modern architecture was born.