---
title: "Machine Translation: Decoding, Evaluation, and Scale"
module: Applications
moduleNumber: 7
lessonNumber: 2
order: 702
summary: >
  Having built the transformer translation model, we now decode from it and
  measure the output. Beam search turns the decoder's per-step distributions into a
  single output string; length normalization keeps it from favoring short
  translations. We then score translations automatically — BLEU with its
  n-gram precision, clipping, and brevity penalty, worked through by hand, then
  its successors chrF, BERTScore, and COMET — and close on the parts of MT that
  scale beyond one language pair: multilingual and low-resource translation,
  backtranslation, gender bias, and the lineage from the Transformer to
  massively multilingual models like NLLB-200.
topics: [Applications]
sources:
  - book: Jurafsky
    ref: "§10.5 Beam Search; §10.8 MT Evaluation"
  - book: Jurafsky
    ref: "§10.7 Practical Details (Tokenization, Corpora, Backtranslation); §10.9 Bias and Ethical Issues"
---

This builds on [Machine Translation](/natural-language-processing/applications/machine-translation),
which developed the transformer encoder-decoder that computes
$P(\mathbf{y}\mid\mathbf{x})$ for a source $\mathbf{x}$ and a candidate target
$\mathbf{y}$. That lesson built the model; this one runs it. Two questions remain:
given a trained model, how do we turn its per-step word distributions into a single
output sentence, and once we have an output, how do we tell whether it is any good?
The first is **decoding**, the second **evaluation**, and the lesson closes on what
changes when translation has to cover more than one well-resourced language pair.

## Decoding with beam search

At generation time the decoder gives a distribution over the next word at every step,
and we must turn those distributions into a single output string. The obvious rule —
take the single most probable word at each step — is **greedy decoding**:

$$
\hat{y}_t = \argmax_{w \in V} P(w \mid \mathbf{x}, y_1, \ldots, y_{t-1}).
$$

Greedy decoding is locally optimal and globally wrong. The word that looks best now
can force worse words later, and a high-probability first word can strand the model
on a low-probability continuation. But the space of full translations is $|V|^T$
strings, far too many to enumerate, and the long-distance dependencies between output
words rule out the dynamic-programming shortcut that
[Viterbi](/natural-language-processing/sequences/sequence-labeling) uses for
tagging.[^jm-greedy]

The practical middle ground keeps a few candidates alive instead of one. **Beam
search** is the standard compromise: keep not one but the $k$ best partial
translations at every step, where $k$ (the **beam width**) is a small constant, and
prune back to $k$ after each extension.[^jm-beam]

> **Definition (Beam search).** A decoding algorithm that maintains a frontier of
> the $k$ best **hypotheses** (partial translations with their scores). At each
> step it extends every hypothesis by every vocabulary word, giving $k \times |V|$
> candidates, scores each, and keeps the top $k$. A hypothesis ending in the
> end-of-sequence token is removed to the completed set and the beam shrinks by one;
> search ends when the beam is empty. Production MT uses $k$ between $5$ and $10$.

Scoring is done in **log** probability, so the score of a hypothesis is a running
sum that extends by one term per word — turning the product in the chain rule into a
sum that never underflows:

$$
\mathrm{score}(\mathbf{y}) = \log P(\mathbf{y}\mid\mathbf{x})
= \sum_{i=1}^{t} \log P(y_i \mid y_1, \ldots, y_{i-1}, \mathbf{x}).
$$

$$
% caption: Beam search with width k = 2 translating "the green witch arrived". At
% each step the two surviving hypotheses (bold) are each extended over the whole
% vocabulary, all extensions are scored by summed log-probability, and only the
% best two are kept. "the witch" and "the green" survive step 2; the globally best
% path is recovered that greedy search, locked to the single best first word, would
% miss.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  keep/.style={draw=acc, text=acc, fill=acc!10, minimum width=17mm, minimum height=6mm, inner sep=1pt},
  drop/.style={draw, text=black, minimum width=17mm, minimum height=6mm, inner sep=1pt}]
  \definecolor{acc}{HTML}{2348F2}
  \node (st) at (0,0) {start};
  % step 1: keep "arrived" and "the"
  \node[keep] (s1a) at (2.6,1.1)  {arrived};
  \node[keep] (s1b) at (2.6,-1.1) {the};
  \draw[->, acc] (st) -- (s1a);
  \draw[->, acc] (st) -- (s1b);
  % step 2: extend each; keep "the green" and "the witch"
  \node[drop] (s2a) at (5.6,2.0)   {arrived the};
  \node[drop] (s2b) at (5.6,0.6)   {arrived who};
  \node[keep] (s2c) at (5.6,-0.6)  {the green};
  \node[keep] (s2d) at (5.6,-2.0)  {the witch};
  \draw[->, black] (s1a) -- (s2a);
  \draw[->, black] (s1a) -- (s2b);
  \draw[->, acc] (s1b) -- (s2c);
  \draw[->, acc] (s1b) -- (s2d);
  % step 3: extend survivors
  \node[drop] (s3a) at (8.8,-0.1) {the green witch};
  \node[drop] (s3b) at (8.8,-1.5) {the witch came};
  \draw[->, acc] (s2c) -- (s3a);
  \draw[->, acc] (s2d) -- (s3b);
\end{tikzpicture}
$$

One catch follows from the scoring. Because completed hypotheses have different
lengths and a model assigns any longer string a lower joint probability, raw
log-probability systematically favors short translations. The standard remedy is **length
normalization**: divide the score by the number of words $T$ before comparing
complete hypotheses,

$$
\mathrm{score}(\mathbf{y}) = \frac{1}{T}\sum_{i=1}^{T} \log P(y_i \mid y_1, \ldots, y_{i-1}, \mathbf{x}).
$$

## Evaluation: BLEU and beyond

We can now produce a translation; the next question is whether it is good, and that
needs a definition of "good" a machine can compute. The
two dimensions are **adequacy** (does the output carry the source's meaning?) and
**fluency** (is it natural in the target language?). Human raters are the gold
standard, but they are slow and expensive, so automatic metrics are used to compare
systems and to guide development. The oldest and still most cited automatic metric is
**BLEU**.[^jm-eval]

BLEU (bilingual evaluation understudy) rests on one intuition: a good candidate
translation shares many word **n-grams** with a human reference translation. It is a
**precision** metric — it asks what fraction of the candidate's n-grams appear in the
reference — computed for several n-gram sizes and multiplied by a **brevity
penalty** that punishes translations shorter than the reference (precision alone
could be gamed by outputting a single sure word).[^jm-bleu]

> **Definition (BLEU).** For n-gram sizes $n = 1, \ldots, N$ (typically $N = 4$),
> let $p_n$ be the modified n-gram precision — the fraction of candidate n-grams
> that occur in the reference, each reference n-gram counted at most as often as it
> appears. With a brevity penalty $\mathrm{BP}$ that is $1$ when the candidate is at
> least as long as the reference and decays exponentially when it is shorter,
> $$
> \mathrm{BLEU} = \mathrm{BP}\cdot\exp\!\Big(\sum_{n=1}^{N} w_n \log p_n\Big),
> \qquad
> \mathrm{BP} = \begin{cases} 1 & c > r \\ e^{\,1 - r/c} & c \le r \end{cases}
> $$
> where $c$ is the candidate length, $r$ the reference length, and $w_n = 1/N$ are
> uniform weights. BLEU combines the $p_n$ as a geometric mean, so a zero at any $n$
> zeros the score.

For example, take a
reference and a candidate:

> Reference: _the green witch arrived at the house_
> Candidate: _the green witch arrived_

**Unigram precision** $p_1$: the candidate has $4$ unigrams (_the, green, witch,
arrived_), and all $4$ appear in the reference, so $p_1 = 4/4 = 1$. **Bigram
precision** $p_2$: the candidate's $3$ bigrams are _the green_, _green witch_,
_witch arrived_, and all $3$ appear in the reference, so $p_2 = 3/3 = 1$. So far
precision is perfect — which exposes exactly why the brevity penalty is needed. The
candidate has length $c = 4$ against reference length $r = 7$, so

$$
\mathrm{BP} = e^{\,1 - 7/4} = e^{-0.75} \approx 0.47,
$$

and even with perfect precision the score is held down to about $0.47$, correctly
docking the candidate for dropping half the sentence. The penalty is what stops a
short, cherry-picked fragment from scoring as a full translation.

The first example was chosen to isolate the brevity penalty; the second shows the
**clipping** inside modified precision.
Take a candidate that repeats a good word to game a naive precision:

> Reference: _the cat is on the mat_
> Candidate: _the the the the_

A raw unigram precision would be $4/4 = 1$ — every word _the_ is "in" the reference.
BLEU forbids this by **clipping** each candidate n-gram's count to the maximum number
of times it appears in any single reference. The reference contains _the_ twice, so
the candidate's four _the_ tokens contribute at most $2$ matches:

$$
p_1 = \frac{\min(\mathrm{count}_{\text{cand}}, \max\text{-ref-count})}{\text{candidate unigrams}}
    = \frac{\min(4, 2)}{4} = \frac{2}{4} = 0.5.
$$

Clipping is the "modified" in _modified n-gram precision_: without it, precision could
be pushed to $1$ by any word that occurs in the reference at all. The bigram precision
finishes the job — the candidate's three bigrams are all _the the_, which never occurs
in the reference, so $p_2 = 0/3 = 0$, and because BLEU is a geometric mean a single
zero $p_n$ zeros the whole score. Repetition is penalized from two directions at once.

$$
% caption: Clipping in modified n-gram precision. The candidate "the the the the"
% would score unigram precision 1 under a raw count, but each candidate n-gram is
% clipped to how often it appears in the reference. The reference has "the" twice, so
% only 2 of the 4 candidate tokens count; p1 falls to 0.5, and the bigram "the the"
% (absent from the reference) makes p2 = 0.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  tokc/.style={draw, minimum width=11mm, minimum height=6.5mm, font=\scriptsize, align=center},
  tokm/.style={draw=acc, text=acc, minimum width=11mm, minimum height=6.5mm, font=\scriptsize, align=center},
  tokx/.style={draw=red, text=red, minimum width=11mm, minimum height=6.5mm, font=\scriptsize, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[anchor=east, font=\scriptsize, text=black] at (-0.4,0) {candidate};
  \node[tokm] (c1) at (0.6,0)  {the};
  \node[tokm] (c2) at (1.9,0)  {the};
  \node[tokx] (c3) at (3.2,0)  {the};
  \node[tokx] (c4) at (4.5,0)  {the};
  \node[anchor=west, font=\scriptsize, text=acc]   at (5.3,0.3)  {2 match (clip)};
  \node[anchor=west, font=\scriptsize, text=red]   at (5.3,-0.3) {2 over the cap};
  \node[anchor=east, font=\scriptsize, text=black] at (-0.4,-1.3) {reference};
  \node[tokc] (r1) at (0.6,-1.3)  {the};
  \node[tokc] (r2) at (1.9,-1.3)  {cat};
  \node[tokc] (r3) at (3.2,-1.3)  {is};
  \node[tokc] (r4) at (4.5,-1.3)  {on};
  \node[tokc] (r5) at (5.8,-1.3)  {the};
  \node[tokc] (r6) at (7.1,-1.3)  {mat};
  \node[anchor=west, font=\scriptsize, text=black] at (0.1,-2.2) {cap = 2 occurrences of "the"; p1 = min(4,2)/4 = 0.5, p2 = 0};
\end{tikzpicture}
$$

$$
% caption: BLEU on the worked example. Every candidate n-gram is present in the
% reference, so the precisions are 1, but the candidate is shorter (c = 4 words vs.
% r = 7), and the brevity penalty exp(1 - r/c) pulls the final score down to about
% 0.47. Precision measures what is present; the brevity penalty measures what is
% missing.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  b/.style={draw, minimum width=20mm, minimum height=8mm, align=center, font=\scriptsize},
  r/.style={draw, minimum width=26mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[b] (p1) at (0,1.4)  {unigram P\\4/4 = 1.0};
  \node[b] (p2) at (0,0)    {bigram P\\3/3 = 1.0};
  \node[b, draw=red, text=red] (bp) at (0,-1.4) {brevity BP\\c=4, r=7};
  \node[r, draw=acc, text=acc] (out) at (4.6,0) {BLEU = BP x exp(mean log P)\\about 0.47};
  \draw[->, black] (p1) -- (out);
  \draw[->, black] (p2) -- (out);
  \draw[->, red!70!black, thick] (bp) -- (out);
\end{tikzpicture}
$$

BLEU has real limits. It is word-based and so sensitive to tokenization, which makes
scores hard to compare across setups and weakens it for morphologically rich
languages. It is purely local — moving a whole phrase barely changes the score — and
it cannot credit a correct paraphrase that happens to use different words. Newer
metrics attack these gaps.[^jm-limits]

- **chrF** scores overlap of **character** n-grams instead of word n-grams and
  combines precision **and** recall into an F-score. Being sub-word it degrades
  gracefully under morphology, and it correlates better with human judgments than
  BLEU across many languages, which is why J&M take it as the default.[^jm-chrf]
- **BERTScore** replaces exact string matching with **embedding** similarity: it
  runs reference and candidate through [BERT](/natural-language-processing/transformers/large-language-models),
  greedily matches each token to its most similar counterpart by cosine, and reports
  precision, recall, and F1 — so a paraphrase with different surface words can still
  score well.[^jm-bertscore]
- **COMET** (and **BLEURT**) go further and **learn** the metric: they train a model
  on datasets of human quality ratings $(\mathbf{x}, \tilde{\mathbf{x}}, r)$ to
  predict the human score $r$ directly from source and candidate, and their outputs
  correlate most closely with human judgment.[^jm-comet]

Whatever the metric, comparing two systems needs a significance test — the **paired
bootstrap** — since a small score gap can be noise; and automatic metrics are most
trustworthy when measuring changes to a single system, least trustworthy when
comparing very different kinds of systems.[^jm-limits]

## Multilingual and low-resource MT

Everything so far assumes a large **parallel corpus** for the language pair in hand.
For most of the world's languages no such corpus exists, and the methods have to
change. The Europarl and UN corpora cover a handful of official
languages; the long tail of **low-resource** languages has little or no parallel
text, and the field's habit of routing everything through English makes the gap
worse.[^jm-corpora]

**Backtranslation** is the standard way to manufacture training data when parallel text is
scarce but **monolingual** target text is plentiful. Train a rough model in the
**reverse** direction on whatever small bitext exists, run it on the abundant
monolingual target text to produce synthetic source sentences, and add these
synthetic pairs to the training set for the forward model.[^jm-backtr]

> **Algorithm (Backtranslation).** To build a source-to-target model with little
> parallel data but much monolingual target text: (1) train an intermediate
> **target-to-source** model on the small parallel corpus; (2) run it on the
> monolingual target sentences to generate synthetic source sentences; (3) pair
> each natural target sentence with its synthetic source and add these pairs to the
> parallel data; (4) retrain the source-to-target model on the combined data.

The synthetic pairs are noisy on the source side but pristine on the target side,
which is the side the model must learn to generate. In practice backtranslation
works well — one estimate is that synthetic data yields about two-thirds the gain of
the same volume of natural parallel text — and it scales any low-resource language
that has raw monolingual text, which nearly all of them do.[^jm-backtr]

## Bias and ethical issues in MT

The divergences that open the [first lesson](/natural-language-processing/applications/machine-translation)
return here as an ethical problem. When a system translates from a language that
leaves gender unmarked into one that forces it, it must _guess_ — and it guesses
according to its training data. Translating the Hungarian gender-neutral
pronoun _ő_ into English, current systems render _ő is a nurse_ as _she is a nurse_
but _ő is a CEO_ as _he is a CEO_, and _ő is an engineer_, _a scientist_, _a baker_
as _he_.[^jm-bias] The default is not merely to guess the majority gender; the bias
is **amplified**. Prates and colleagues found MT systems map pronouns to male or
female more skewedly than the actual labor statistics would justify, so the output is
more stereotyped than the world it was trained on. The WinoMT challenge set makes the
same point from the other side: systems translate _worse_ on sentences that describe
people in **non-stereotypical** roles ("the doctor asked the nurse to help _her_"),
where getting the coreference right cuts against the prior.

$$
% caption: Gender-default errors translating the Hungarian gender-neutral pronoun o
% into English. The system supplies she for traditionally female occupations and he
% for traditionally male ones, amplifying a stereotype the source never expressed.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  src/.style={draw, minimum width=30mm, minimum height=6.5mm, align=center, font=\scriptsize},
  she/.style={draw=acc, text=acc, minimum width=26mm, minimum height=6.5mm, align=center, font=\scriptsize},
  he/.style={draw=red, text=red, minimum width=26mm, minimum height=6.5mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[anchor=south, font=\scriptsize, text=black] at (0,1.2) {source (gender-neutral)};
  \node[anchor=south, font=\scriptsize, text=black] at (5.0,1.2) {MT output};
  \node[src] (n1) at (0,0.6)  {o is a nurse};
  \node[she] (o1) at (5.0,0.6) {she is a nurse};
  \node[src] (n2) at (0,-0.3) {o is a teacher};
  \node[she] (o2) at (5.0,-0.3) {she is a teacher};
  \node[src] (n3) at (0,-1.2) {o is an engineer};
  \node[he]  (o3) at (5.0,-1.2) {he is an engineer};
  \node[src] (n4) at (0,-2.1) {o is a CEO};
  \node[he]  (o4) at (5.0,-2.1) {he is a CEO};
  \foreach \a/\b in {n1/o1,n2/o2,n3/o3,n4/o4} \draw[->, black] (\a) -- (\b);
\end{tikzpicture}
$$

The other standing issue is **inequity of coverage**. Neural MT needs parallel text,
most of the world's languages have little or none, and the field's habit of routing
translation through English deepens the gap — a low-resource pair is often served
only by pivoting through a high-resource one. One response is participatory design:
∀ and colleagues built online groups, mentoring, and infrastructure so speakers of
low-resource African languages help develop the systems that serve them, rather than
being modeled from the outside.[^jm-lowres] A subtler open problem is **knowing what
the system does not know**. MT is used in medical and legal settings where a human
translator may be unavailable — a patient and doctor without a shared language, a
judge and a witness. To _do no harm_ there, a system needs calibrated **confidence**
so it can abstain from a translation it is likely to botch, rather than emit a fluent,
confident, wrong sentence.[^jm-bias]

## From Transformer NMT to massively multilingual models

Jurafsky and Martin present the transformer encoder-decoder as the modern MT
architecture; the public record fills in where that architecture came from and how
far it has since been pushed.

**The Transformer was introduced for translation.** Vaswani and colleagues' 2017
paper, "Attention Is All You Need" (NeurIPS 2017), proposed the transformer as a
sequence-to-sequence model and evaluated it first on machine translation. On the
WMT 2014 English-to-German task it reported $28.4$ BLEU, and on English-to-French
$41.8$ BLEU, above the previous best published results including ensembles, while
training in a fraction of the time because the model dispenses with recurrence and
computes all positions in parallel. The paper's own framing is that attention alone,
with no RNN or convolution, suffices — its contribution is the cross-attention and
self-attention the first lesson describes.[^bb-transformer]

**Subword tokenization has a citable origin too.** The BPE scheme MT uses was adapted
to NMT by Sennrich, Haddow, and Birch, "Neural Machine Translation of Rare Words with
Subword Units" (ACL 2016). They showed that segmenting into subword units lets a
fixed-size vocabulary translate rare and unseen words by composing them from pieces,
improving BLEU over a back-off dictionary on English-German and English-Russian. This
is the concrete reason MT prefers subwords to words.[^bb-bpe]

**Backtranslation was measured, not assumed.** The two-thirds-of-natural-data figure
this lesson cites is from Edunov, Ott, Auli, and Grangier, "Understanding Back-Translation
at Scale" (EMNLP 2018). They generated synthetic source sentences with a reverse model
and found that adding them raised BLEU substantially, that sampling or noised beam output
made better synthetic data than plain beam search, and that hundreds of millions of
backtranslated sentences kept helping — establishing backtranslation as the standard way
to exploit monolingual data.[^bb-backtrans]

**One model for two hundred languages.** Jurafsky & Martin note the low-resource gap as an open
problem; the largest public attack on it is Meta AI's **No Language Left Behind** effort,
"No Language Left Behind: Scaling Human-Centered Machine Translation" (2022). The NLLB-200
model translates directly among $200$ languages, including many with almost no parallel
text, without pivoting through English, and was trained with a mixture-of-experts
architecture and heavily mined and backtranslated bitext. Meta reported an average gain of
roughly $44\%$ in chrF++ over the prior state of the art on the associated benchmark, and
released the model and the FLORES-200 evaluation set publicly. NLLB is the direct
realization of the "route everything through English" critique this lesson raises: it
translates low-resource pairs end to end.[^bb-nllb]

$$
% caption: The public MT lineage this lesson sits inside. The RNN encoder-decoder with
% attention (2014-15) gave way to the Transformer (2017), which is the architecture the
% "Transformer-based NMT" section describes; subword tokenization (2016) and
% backtranslation-at-scale (2018) are the training-side pieces; massively multilingual
% direct models like NLLB-200 (2022) push a single network across 200 languages.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  stg/.style={draw, minimum width=27mm, minimum height=11mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[stg] (rnn) at (0,0)   {RNN enc-dec\\+ attention\\2014-2015};
  \node[stg, draw=acc, text=acc] (tfmr) at (3.6,0) {Transformer\\2017};
  \node[stg] (nllb) at (7.6,0) {NLLB-200\\200 languages\\2022};
  \node[stg] (bpe)  at (3.6,-2.0) {subword BPE\\2016};
  \node[stg] (bt)   at (7.6,-2.0) {backtranslation\\at scale, 2018};
  \draw[->, acc, thick] (rnn) -- (tfmr);
  \draw[->, acc, thick] (tfmr) -- (nllb);
  \draw[->, black] (bpe) -- (tfmr);
  \draw[->, black] (bt) -- (nllb);
\end{tikzpicture}
$$

**Where MT is heading: general LLMs as translators.** The most recent shift is that
general-purpose large language models translate competently with no MT-specific training.
GPT-style models prompted with a source sentence and an instruction produce fluent
translations, and controlled evaluations (for instance the WMT shared-task findings from
2023 onward) report that they are strong on high-resource pairs and document-level
context while still trailing dedicated systems on many low-resource pairs. The practical
consequence is that the encoder-decoder specialist and the decoder-only generalist now
overlap: the same architecture the first lesson traces from translation is used to
translate without being trained for it. The dividing line has moved from "which
architecture" to "how much data the target language has," which returns the field to the
coverage problem the ethics section names.[^bb-llmmt]

The through-line is worth restating. Translation set the specification — map one
sequence to another of different length and order — that the RNN encoder-decoder was
built to meet, that attention was built to fix, and that the transformer inherited
wholesale. Every one of those inventions now lives in the general
[transformer](/natural-language-processing/transformers/transformers-and-attention)
and drives [question answering](/natural-language-processing/applications/question-answering),
[dialogue](/natural-language-processing/applications/dialogue-and-chatbots), and the
[large language models](/natural-language-processing/transformers/large-language-models)
that followed. Machine translation is where the modern architecture was born.

[^jm-greedy]: **Jurafsky & Martin**, _Speech and Language Processing_ (3rd ed.), §10.5 (Eq. 10.18; Fig. 10.11) — greedy decoding as locally optimal argmax and its failure to find the globally best sequence; the $|V|^T$ search space and why dynamic programming (Viterbi) does not apply to generation with long-distance dependencies.
[^jm-beam]: **Jurafsky & Martin**, §10.5 (Figs. 10.12–10.14; Eqs. 10.19–10.20) — beam search: keep the $k$ best hypotheses, extend and prune to $k$ each step, score by summed log-probability, apply length normalization; beam widths $k$ of 5–10 in production.
[^jm-eval]: **Jurafsky & Martin**, §10.8 — MT Evaluation along adequacy and fluency; human raters as the gold standard; automatic metrics for convenience in comparing and developing systems.
[^jm-bleu]: **Jurafsky & Martin**, §10.8.2 — BLEU (Papineni et al., 2002) as an n-gram word-precision metric combined with a corpus-level brevity penalty; word-based and precision-only, in contrast to the recall-inclusive chrF.
[^jm-limits]: **Jurafsky & Martin**, §10.8.2, "chrF: Limitations" and "Statistical Significance Testing" — overlap metrics are local, tokenization-sensitive, cannot credit paraphrase, and are best for changes to a single system; the paired bootstrap test compares two systems.
[^jm-chrf]: **Jurafsky & Martin**, §10.8.2 (Eq. 10.24) — chrF (Popović, 2015): character n-gram precision and recall combined into an F-score (commonly $\beta = 2$), robust across morphology and well correlated with human judgment.
[^jm-bertscore]: **Jurafsky & Martin**, §10.8.3 (Eq. 10.25; Fig. 10.18) — BERTScore (Zhang et al., 2020): BERT embeddings and greedy cosine token matching to compute precision, recall, and F1, allowing paraphrase and synonym matches beyond exact overlap.
[^jm-comet]: **Jurafsky & Martin**, §10.8.3 — COMET (Rei et al., 2020) and BLEURT (Sellam et al., 2020): metrics trained on human quality ratings $(\mathbf{x},\tilde{\mathbf{x}},r)$ to predict the human score directly, correlating most highly with human judgment.
[^jm-corpora]: **Jurafsky & Martin**, §10.7.2, §10.9 — MT corpora (Europarl, UN, ParaCrawl) as parallel bitexts; the shortage of parallel text for low-resource languages and the field's English-centric focus as an open problem.
[^jm-backtr]: **Jurafsky & Martin**, §10.7.3 — Backtranslation: use a reverse-direction model to translate monolingual target text into synthetic source sentences, add the synthetic bitext to training; about two-thirds the gain of natural bitext (Edunov et al., 2018).
[^jm-bias]: **Jurafsky & Martin**, §10.9 — Bias and Ethical Issues: gender-default errors translating gender-neutral pronouns (Hungarian _ő_) into English with grammatical gender, amplified beyond labor statistics (Prates et al., 2019); the WinoMT non-stereotypical-role challenge set (Stanovsky et al., 2019); and the need for confidence estimates so systems can abstain in safety-critical (medical, legal) use.
[^jm-lowres]: **Jurafsky & Martin**, §10.9 — low-resource languages: most of the world's languages lack large parallel corpora, a gap worsened by the field's English-centric focus; ∀ et al. (2020) propose a participatory design process for low-resource African languages.
[^bb-transformer]: **Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin**, "Attention Is All You Need," _NeurIPS_ 2017. Introduced the transformer as a sequence-to-sequence model evaluated first on machine translation, reporting 28.4 BLEU on WMT 2014 English-German and 41.8 BLEU on English-French, above prior published results at lower training cost; the source of the cross-attention/self-attention architecture in this lesson.
[^bb-bpe]: **Sennrich, Haddow, and Birch**, "Neural Machine Translation of Rare Words with Subword Units," _ACL_ 2016. Adapted byte-pair encoding to NMT, showing a fixed-size subword vocabulary translates rare and unseen words by composing them from pieces and improves BLEU over a back-off dictionary on English-German and English-Russian.
[^bb-backtrans]: **Edunov, Ott, Auli, and Grangier**, "Understanding Back-Translation at Scale," _EMNLP_ 2018. Showed that synthetic parallel data from a reverse model raises BLEU substantially, that sampled or noised generation yields better synthetic sources than plain beam search, and that backtranslation scales to hundreds of millions of sentences; the source of the "about two-thirds the gain of natural bitext" figure.
[^bb-nllb]: **NLLB Team (Costa-jussà et al.)**, "No Language Left Behind: Scaling Human-Centered Machine Translation," Meta AI technical report, 2022. A single mixture-of-experts model (NLLB-200) translating directly among 200 languages without English pivoting, reporting an average improvement of about 44% in chrF++ over the prior state of the art on its benchmark, with the model and the FLORES-200 evaluation set released publicly.
[^bb-llmmt]: General-purpose large language models translate competently under instruction prompting with no MT-specific training; controlled evaluations (e.g. the WMT shared-task human evaluations from 2023 onward, and studies such as Hendy et al. 2023 on GPT models for translation) find them strong on high-resource pairs and document-level context while still trailing dedicated systems on many low-resource pairs.