---
title: Machine Translation
module: Applications
moduleNumber: 7
lessonNumber: 1
order: 701
summary: >
  Machine translation is the task that built the modern toolkit: the
  encoder-decoder was invented for it, attention was invented to fix its
  fixed-context bottleneck, and both were later folded into the general
  transformer. We work through why translation is hard (word order, morphology,
  lexical and structural divergences), the sequence-to-sequence model and its
  attention mechanism, transformer-based NMT with cross-attention, subword
  tokenization with a shared vocabulary, beam-search decoding, and evaluation by
  BLEU and its successors chrF, BERTScore, and COMET — closing on multilingual
  and low-resource translation and backtranslation.
topics: [Applications]
sources:
  - book: Jurafsky
    ref: "Ch. 10 — Machine Translation and Encoder-Decoder Models; §10.1 Language Divergences and Typology"
  - book: Jurafsky
    ref: "§10.2 The Encoder-Decoder Model; §10.3 Encoder-Decoder with RNNs; §10.4 Attention; §10.5 Beam Search"
  - book: Jurafsky
    ref: "§10.6 Encoder-Decoder with Transformers; §10.7 Practical Details (Tokenization, Corpora, Backtranslation); §10.8 MT Evaluation; §10.9 Bias and Ethical Issues"
---

**Machine translation** is the use of a computer to render text from one language
into another: an English source $\mathbf{x} = x_1, \ldots, x_n$ becomes a target
$\mathbf{y} = y_1, \ldots, y_m$ in, say, Spanish. Stated as a probability model it
is short — pick the target string that is most likely given the source,

$$
\hat{\mathbf{y}} = \argmax_{\mathbf{y}} P(\mathbf{y} \mid \mathbf{x}),
$$

— but that single line hides everything hard about the task, and everything the
last decade of NLP was built to solve. The standard model for computing
$P(\mathbf{y}\mid\mathbf{x})$ is the **encoder-decoder** (or **sequence-to-sequence**)
network, and it did not exist before translation demanded it. The
[encoder-decoder shape and the attention mechanism](/natural-language-processing/transformers/transformers-and-attention)
now central to all of modern NLP were both invented here, to make one sequence map
onto another of a different length and a different order.[^jm-intro] This lesson
follows that history forward: the problem, the model, attention, and the modern form.

## Language divergences and typology

Translation is not a word-for-word substitution, because languages do not line up
word for word. Consider an English sentence and its Japanese translation:

> English:  He wrote a letter to a friend
> Japanese: tomodachi ni tegami-o kaita _(friend to letter wrote)_

The verb _wrote_ sits in the middle of the English clause and at the very end of
the Japanese one; the English subject pronoun _he_ has no counterpart in the
Japanese at all. A model that mapped input word $x_t$ to output word $y_t$ position
by position — the way a [part-of-speech tagger](/natural-language-processing/sequences/sequence-labeling)
maps each word to its tag — cannot express this, because the target is a **complex
function of the entire source**, not a per-word relabeling.[^jm-hard] The
systematic ways languages fail to line up are called **translation divergences**,
and they come in several kinds.

> **Definition (Translation divergence).** A systematic difference between two
> languages that prevents word-by-word correspondence: differences in word order,
> in how meaning is packaged into words (morphology, lexical granularity), and in
> what may be left unsaid (dropped pronouns). Divergences are what force
> translation to reorder, split, merge, insert, and delete rather than substitute.

Some aspects of language are **universal** — every language has words for people,
for eating and drinking, for being polite; every language has nouns and verbs, ways
to ask questions and issue commands. Others are statistical **universals**, holding
for most languages. What is left over is variation, and it comes in two flavors. Some
differences are **idiosyncratic and lexical** and must be handled one at a time (the
word for "dog" is unpredictable from language to language). Others are **systematic**
and can be modeled in general — many languages put the verb before the object, many
put it after. The study of these systematic cross-linguistic similarities and
differences is **linguistic typology**, and how hard a language is to translate
into or out of comes down to the typological facts it sits on top of.[^jm-typology]

$$
% caption: A word-order divergence: English is SVO (subject, verb, object) and
% Japanese is SOV, so the verb crosses to the far end and the alignment lines
% cross. A position-for-position model cannot produce this reordering.
\begin{tikzpicture}[>=stealth, font=\small,
  s/.style={draw, minimum width=16mm, minimum height=6mm, inner sep=1pt},
  t/.style={draw, minimum width=16mm, minimum height=6mm, inner sep=1pt, draw=acc, text=acc}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % source row (English, SVO)
  \node[s] (he)  at (0,2)   {he};
  \node[s] (wr)  at (2.2,2) {wrote};
  \node[s] (le)  at (4.4,2) {a letter};
  \node[s] (fr)  at (6.6,2) {a friend};
  \node[anchor=east, font=\scriptsize, text=black] at (-0.7,2) {source (SVO)};
  % target row (Japanese gloss, SOV)
  \node[t] (tom) at (0,0)   {friend};
  \node[t] (teg) at (2.2,0) {letter};
  \node[t] (kai) at (6.6,0) {wrote};
  \node[anchor=east, font=\scriptsize, text=black] at (-0.7,0) {target (SOV)};
  % crossing alignment lines
  \draw[red, thick] (fr.south) -- (tom.north);
  \draw[red, thick] (le.south) -- (teg.north);
  \draw[red, thick] (wr.south) -- (kai.north);
  \node[anchor=west, font=\scriptsize, text=black] at (7.3,1) {he: dropped};
\end{tikzpicture}
$$

**Word order.** Languages differ in the basic order of subject, verb, and object.
English, French, German, and Mandarin are **SVO** (the verb sits between subject
and object); Japanese and Hindi are **SOV** (the verb comes last); Irish and Arabic
are **VSO**. Two languages of the same type tend to share other traits — VO
languages generally have prepositions, OV languages postpositions — but crossing
between types forces the system to move whole phrases across the sentence as it
generates.[^jm-order]

**Lexical divergence.** One word rarely has one translation. English _bass_ is
Spanish _lubina_ (the fish) or _bajo_ (the instrument); the mapping is one-to-many
and must be disambiguated by context, which ties translation tightly to
[word-sense disambiguation](/natural-language-processing/semantics/vector-semantics-and-embeddings).
Worse, the carve-up of conceptual space can be many-to-many. English _leg_ becomes
French _jambe_ for a person, _patte_ for an animal, _pied_ for a chair, and _étape_
for a leg of a journey — one English word fanning out across four French ones, none
of them a clean synonym of the others.[^jm-lex] Sometimes the target language forces
a distinction the source never made: German splits _wall_ into _Wand_ (inside a
building) and _Mauer_ (outside), and Mandarin distinguishes older brother _gege_
from younger brother _didi_ where English has only _brother_. Sometimes a language
imposes a **grammatical** constraint the source lacks: French and Spanish mark
grammatical gender on adjectives, so translating into them requires choosing a gender
English left unspecified.

At the extreme, one language may have a **lexical gap** — no word or short phrase
that expresses a concept another language lexicalizes. English has no clean
equivalent of Mandarin _xiao_ or Japanese _oyakoko_, and must fall back on awkward
paraphrases like _filial piety_. Languages also differ systematically in where they
package the components of an event. In a **satellite-framed** language like English,
the _direction_ of motion rides on a particle — _the bottle floated **out**_ — while
the verb carries the manner; in a **verb-framed** language like Spanish, direction
rides on the verb — _la botella **salió** flotando_ ("the bottle exited floating") —
leaving manner to a satellite. A translator must repackage motion from verb to
satellite or back.[^jm-lex2]

**Morphology.** Languages differ in how much meaning is packed into a single word,
and typology places this variation on two axes. The first is the **number of
morphemes per word**, running from **isolating** languages like Vietnamese and
Cantonese (roughly one morpheme per word) to **polysynthetic** languages like
Siberian Yupik, where a single word may hold enough morphemes to translate a whole
English sentence. The second is the **degree to which morphemes are segmentable**,
running from **agglutinative** languages like Turkish, where morphemes have clean
boundaries, to **fusional** languages like Russian, where one affix conflates several
categories (_-om_ in _stolom_ fuses instrumental, singular, and first declension into
a single suffix). Rich morphology explodes the vocabulary — every case and tense is a
distinct surface form — which is the direct reason modern systems translate
**subword units** rather than words.[^jm-morph]

$$
% caption: The two axes of morphological typology. The horizontal axis is the number
% of morphemes per word (isolating to polysynthetic); the vertical axis is how
% cleanly those morphemes segment (agglutinative to fusional). A language's position
% predicts how badly a word vocabulary blows up, and thus how much subword
% tokenization matters.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  lang/.style={draw, minimum width=23mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  % axes
  \draw[->, black, thick] (-0.4,0) -- (10.2,0);
  \draw[->, black, thick] (0,-0.4) -- (0,5.6);
  \node[anchor=north east, black, font=\scriptsize] at (10.2,-0.55) {morphemes per word (more)};
  \node[anchor=south west, black, font=\scriptsize] at (0.1,5.2) {clean boundaries};
  % x-axis end labels, kept clear of the axis-title row
  \node[anchor=north, font=\scriptsize] at (1.9,-0.15) {isolating};
  \node[anchor=north, font=\scriptsize] at (5.6,-0.15) {polysynthetic};
  % y ticks: low = fusional, high = agglutinative
  \node[anchor=east, font=\scriptsize] at (-0.15,1.1) {fusional};
  \node[anchor=east, font=\scriptsize] at (-0.15,4.2) {agglutinative};
  % language examples placed by axes, spaced to avoid box overlap
  \node[lang] (viet) at (1.9,1.1)  {Vietnamese\\one morpheme};
  \node[lang] (rus)  at (5.0,2.1)  {Russian\\fused af\/f\/ixes};
  \node[lang, draw=acc, text=acc] (turk) at (3.4,4.5) {Turkish\\clean glue};
  \node[lang] (yup)  at (8.0,4.2)  {Yupik\\word = clause};
\end{tikzpicture}
$$

**Referential density.** Some languages require an explicit pronoun for a referent
already in the discourse; others may drop it. **Pro-drop** languages like Spanish,
Chinese, and Japanese routinely omit pronouns a non-pro-drop language like English
must supply, and even among pro-drop languages the rate differs — Japanese and
Chinese omit far more than Spanish. Languages that lean on more pronouns are
**referentially dense**; those that rely on the hearer to recover dropped referents
are referentially sparse, and are sometimes called **cold** languages (they leave the
reader more inferential work, by analogy with cold media) against the more explicit
**hot** languages. Translating _out_ of a cold, pro-drop language means the model has
to recover a pronoun (and, into English, its gender) that was never written down — a
guess it often gets wrong, and a source of the gender bias MT systems are known
for.[^jm-refdens]

$$
% caption: A typological profile places two languages on the same axes MT cares
% about. English and Japanese diverge on every one — word order, morpheme count,
% pronoun dropping, motion framing — and every divergence is a reordering, splitting,
% or insertion the model must learn.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  hd/.style={font=\scriptsize\bfseries},
  cell/.style={font=\scriptsize, anchor=center}]
  \definecolor{acc}{HTML}{2348F2}
  \draw[draw=black] (-0.2,0.55) rectangle (10.4,-4.15);
  \draw[black] (-0.2,0.05) -- (10.4,0.05);
  \draw[black] (4.2,0.55) -- (4.2,-4.15);
  \draw[black] (7.3,0.55) -- (7.3,-4.15);
  \node[hd, anchor=west] at (0.0,0.3) {typological axis};
  \node[hd] at (5.75,0.3) {English};
  \node[hd, text=acc] at (8.85,0.3) {Japanese};
  \foreach \y/\ax/\en/\jp in {
    -0.5/{word order}/{SVO}/{SOV},
    -1.2/{adposition}/{prepositions}/{postpositions},
    -1.9/{morphemes per word}/{low-moderate}/{moderate},
    -2.6/{referential density}/{hot (explicit)}/{cold (pro-drop)},
    -3.3/{motion framing}/{satellite-framed}/{verb-framed},
    -4.0/{gender on pronoun}/{yes}/{often absent}} {
    \node[cell, anchor=west] at (0.0,\y) {\ax};
    \node[cell] at (5.75,\y) {\en};
    \node[cell, text=acc] at (8.85,\y) {\jp};
  }
\end{tikzpicture}
$$

These divergences are the specification for the model: it must map a variable-length
source to a variable-length target, reorder freely, split and merge words, and
insert or delete material — all learned from examples, none of it hand-coded.

## The encoder-decoder model

The idea that meets that specification is to split the network in two. An **encoder**
reads the whole source and compresses it into a representation; a **decoder** reads
that representation and generates the target one token at a time. The two halves are
joined by a **context** — a summary of the source that the decoder consults as it
writes.[^jm-encdec]

> **Definition (Encoder-decoder).** A model in two parts. The **encoder** maps a
> source sequence $\mathbf{x} = x_1, \ldots, x_n$ to a sequence of contextual
> representations $\mathbf{h}^e_1, \ldots, \mathbf{h}^e_n$. A **context** vector
> $\mathbf{c}$, a function of those representations, conveys the essence of the
> source to the **decoder**, which generates the target $\mathbf{y} = y_1, \ldots,
> y_m$ autoregressively — each token conditioned on the context and on the tokens
> already produced. Encoder and decoder can each be an RNN, an LSTM, or a
> transformer.

$$
% caption: The encoder-decoder architecture. The encoder turns the source into
% hidden states; a context summarizing them is handed to the decoder, which
% generates the target one token at a time, feeding each output back as the next
% input.
\begin{tikzpicture}[>=stealth, font=\small,
  box/.style={draw, minimum width=24mm, minimum height=9mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box] (enc) at (0,0)   {encoder};
  \node[box, draw=acc, text=acc] (ctx) at (4.0,0) {context c};
  \node[box] (dec) at (8.0,0) {decoder};
  \node[anchor=north, font=\scriptsize, text=black] at (0,-0.75) {source x1 . . . xn};
  \node[anchor=north, font=\scriptsize, text=acc]     at (8.0,-0.75) {target y1 . . . ym};
  \draw[->, black] (enc) -- (ctx);
  \draw[->, acc, thick] (ctx) -- (dec);
  % autoregressive feedback on the decoder
  \draw[->, black] (dec.north) to[out=60, in=120, looseness=6]
    node[above, font=\scriptsize] {feed each output back} (dec.north);
\end{tikzpicture}
$$

In the recurrent form, the decoder is just a **conditional language model**. An
ordinary [RNN language model](/natural-language-processing/sequences/rnns-and-lstms)
factors the probability of a string by the chain rule, one word at a time. To make
it translate, condition every one of those factors on the source:

$$
P(\mathbf{y}\mid\mathbf{x}) =
P(y_1\mid\mathbf{x})\,P(y_2\mid y_1, \mathbf{x})\,P(y_3\mid y_1, y_2, \mathbf{x})
\cdots P(y_m\mid y_1, \ldots, y_{m-1}, \mathbf{x}).
$$

Mechanically, the encoder RNN reads the source and its final hidden state
$\mathbf{h}^e_n$ becomes the context $\mathbf{c}$; the decoder RNN starts from
$\mathbf{c}$ and generates. At each decoder step, a hidden state $\mathbf{h}^d_t$ is
computed from the previous output, the previous hidden state, and the context, and a
softmax over the vocabulary turns it into a distribution over the next word:

$$
\mathbf{c} = \mathbf{h}^e_n,
\qquad
\mathbf{h}^d_0 = \mathbf{c},
\qquad
\mathbf{h}^d_t = g(\hat{y}_{t-1}, \mathbf{h}^d_{t-1}, \mathbf{c}),
\qquad
y_t = \mathrm{softmax}(f(\mathbf{h}^d_t)).
$$

Training is end-to-end on a **parallel corpus** of source-target sentence pairs.
The two sides are concatenated with a separator token, and the decoder is trained to
predict each next target word by cross-entropy, averaged over the sentence. As with
any autoregressive generator, training uses **teacher forcing**: the decoder is fed
the true previous target word rather than its own (possibly wrong) previous
prediction, which keeps training stable and fast.[^jm-train]

## Attention: fixing the context bottleneck

The clean split has a flaw. The entire source, however long, is squeezed into the
single fixed-length vector $\mathbf{c} = \mathbf{h}^e_n$ — one final hidden state
that must carry every noun, every clause, every dependency the decoder will ever
need. This is the **bottleneck**: information from the start of a long sentence has
to survive being copied through every intervening encoder step to reach that last
state, and much of it does not.[^jm-bottleneck]

$$
% caption: The bottleneck. Requiring the context to be only the encoder's final
% hidden state forces every word of the source to pass through one fixed-length
% vector; early words fade before the decoder ever sees them.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  h/.style={draw, minimum width=11mm, minimum height=6mm, inner sep=1pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \foreach \i in {1,...,5} \node[h] (e\i) at (\i*1.35,0) {he\i};
  \node[h, draw=red, text=red] (c) at (8.1,0) {c};
  \node[h] (d1) at (9.9,0) {hd1};
  \node[h] (d2) at (11.2,0) {hd2};
  \foreach \i in {1,...,4} \draw[->, black] (e\i) -- (e\the\numexpr\i+1\relax);
  \draw[->, red, thick] (e5) -- (c);
  \draw[->, acc] (c) -- (d1);
  \draw[->, acc] (d1) -- (d2);
  \node[anchor=north, font=\scriptsize, text=red] at (8.1,-0.65) {everything squeezes through here};
\end{tikzpicture}
$$

**Attention** removes the bottleneck. Instead of handing the decoder one frozen
context, it lets the decoder look back at **all** the encoder hidden states and, at
each step of generation, build a **fresh** context tailored to the word it is about
to produce. The context stops being static and becomes a weighted average of the
encoder states, with the weights concentrated on the source words most relevant
right now.[^jm-attn]

The construction has three steps. At decoding step $i$, with the previous decoder
state $\mathbf{h}^d_{i-1}$ in hand:

1. **Score** each encoder state for relevance to the current decoder state. The
   simplest score is a dot product — relevance as similarity:
   $\mathrm{score}(\mathbf{h}^d_{i-1}, \mathbf{h}^e_j) = \mathbf{h}^d_{i-1} \cdot \mathbf{h}^e_j$.
2. **Normalize** the scores across the source with a softmax into weights that sum
   to one:
   $\alpha_{ij} = \mathrm{softmax}\big(\mathrm{score}(\mathbf{h}^d_{i-1}, \mathbf{h}^e_j)\big)$.
3. **Blend** the encoder states by those weights to form this step's context:
   $\mathbf{c}_i = \sum_j \alpha_{ij}\,\mathbf{h}^e_j$.

The decoder then computes its state from this dynamic context,
$\mathbf{h}^d_i = g(\hat{y}_{i-1}, \mathbf{h}^d_{i-1}, \mathbf{c}_i)$, and emits the
next word. A richer, learned score parameterizes the comparison with its own weight
matrix, $\mathrm{score}(\mathbf{h}^d_{i-1}, \mathbf{h}^e_j) = \mathbf{h}^d_{i-1}
\mathbf{W}_s \mathbf{h}^e_j$, so the model learns which aspects of similarity matter
and can even relate encoder and decoder vectors of different dimensions.[^jm-attn]

The weights $\alpha_{ij}$ have a direct reading: $\alpha_{ij}$ is how much target
word $i$ **attends to** source word $j$. Laid out as a matrix over one sentence pair
they form a soft **alignment** — the model's learned, distributed answer to the old
question of which source words each target word came from.

$$
% caption: An attention alignment heatmap for translating "the green witch
% arrived" into Spanish "llego la bruja verde". Each row is a target word, each
% column a source word; a darker cell means a larger weight. The bright
% off-diagonal cells (witch-bruja, green-verde) show the model reordering the
% adjective, the divergence a fixed context could not handle.
\begin{tikzpicture}[>=stealth, font=\scriptsize]
  \definecolor{acc}{HTML}{2348F2}
  \def\s{1.05}
  % source column labels (top): the green witch arrived
  \foreach \c/\lab in {1/the, 2/green, 3/witch, 4/arrived}
    \node[anchor=south, font=\scriptsize] at (\c*\s, 0.15) {\lab};
  % target row labels (left): llego la bruja verde
  \foreach \r/\lab in {1/llego, 2/la, 3/bruja, 4/verde}
    \node[anchor=east, font=\scriptsize] at (0.35, -\r*\s) {\lab};
  % weight grid: rows = target, cols = source; value in {0..9} -> tint
  % llego<-arrived; la<-the; bruja<-witch; verde<-green (reordered)
  \foreach \r/\c/\w in {
    1/1/1, 1/2/0, 1/3/1, 1/4/8,
    2/1/8, 2/2/1, 2/3/1, 2/4/0,
    3/1/1, 3/2/1, 3/3/8, 3/4/1,
    4/1/0, 4/2/8, 4/3/1, 4/4/1} {
    \fill[acc!\the\numexpr\w*11\relax] (\c*\s-0.5*\s,-\r*\s-0.5*\s) rectangle (\c*\s+0.5*\s,-\r*\s+0.5*\s);
    \draw[black] (\c*\s-0.5*\s,-\r*\s-0.5*\s) rectangle (\c*\s+0.5*\s,-\r*\s+0.5*\s);
  }
  \node[anchor=north, font=\scriptsize, text=black] at (2.5*\s, -4.7*\s) {source (columns) vs. target (rows)};
\end{tikzpicture}
$$

This is the mechanism that Bahdanau and colleagues introduced and named
"attention," applying it to translation. Generalized — let a word attend to the
words of its **own** sentence rather than a separate source — the same operation
becomes the [self-attention](/natural-language-processing/transformers/transformers-and-attention)
at the center of the transformer. Attention started as a patch on the
encoder-decoder bottleneck for MT and became the primitive the whole field now
runs on.[^jm-attn-hist]

## Transformer-based NMT

The modern system replaces both RNNs with [transformers](/natural-language-processing/transformers/transformers-and-attention),
but keeps the encoder-decoder skeleton exactly. A stack of $N$ (typically $6$)
bidirectional encoder blocks maps the source $\mathbf{x}$ to representations
$\mathbf{H}^{enc} = \mathbf{h}_1, \ldots, \mathbf{h}_n$; a stack of decoder blocks
generates the target autoregressively, conditioning on the source and on the target
words already produced.[^jm-tfmr]

The one new part is in the decoder block. An encoder block has self-attention, then
a feedforward layer, each wrapped in a residual add and layer norm. A **decoder**
block inserts a third sublayer between them: **cross-attention**.

> **Definition (Cross-attention).** A multi-head attention sublayer in each decoder
> block whose **queries** come from the previous decoder layer but whose **keys**
> and **values** come from the encoder's final output $\mathbf{H}^{enc}$. It is the
> transformer's version of the attention mechanism above: it lets every generated
> word attend over the whole source. The decoder's other attention sublayer is
> ordinary **causal** self-attention over the target produced so far.

Concretely, cross-attention projects the encoder output into keys and values and the
previous decoder layer into queries, then runs the same scaled dot-product attention
used everywhere else in the transformer:

$$
\mathbf{Q} = \mathbf{W}^Q \mathbf{H}^{dec[i-1]},
\quad
\mathbf{K} = \mathbf{W}^K \mathbf{H}^{enc},
\quad
\mathbf{V} = \mathbf{W}^V \mathbf{H}^{enc},
$$

$$
\mathrm{CrossAttention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) =
\mathrm{softmax}\!\left(\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d_k}}\right)\mathbf{V}.
$$

$$
% caption: A transformer decoder block for translation. Causal self-attention over
% the target so far, then cross-attention whose keys and values come from the
% encoder output (red), then a feedforward layer; each sublayer wrapped in a
% residual add and layer norm. The cross-attention layer is the only addition to
% the encoder block.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  sub/.style={draw, fill=black!4, minimum width=52mm, minimum height=8mm, align=center, font=\scriptsize},
  cross/.style={draw, fill=acc!12, minimum width=52mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[draw, minimum width=52mm, minimum height=6mm] (in) at (0,0) {target so far};
  \node[sub]   (sa) at (0,1.3) {causal self-attention};
  \node[cross] (ca) at (0,2.9) {cross-attention (K, V from encoder)};
  \node[sub]   (ff) at (0,4.5) {feedforward layer};
  \node[draw, draw=acc, text=acc, minimum width=52mm, minimum height=6mm] (out) at (0,5.8) {to next block};
  \draw[->, black] (in) -- (sa);
  \draw[->, black] (sa) -- (ca);
  \draw[->, black] (ca) -- (ff);
  \draw[->, acc, thick] (ff) -- (out);
  % encoder output feeds cross-attention
  \node[draw, draw=red, text=red, minimum width=24mm, minimum height=6mm] (henc) at (5.6,2.9) {encoder output};
  \draw[->, red!70!black, thick] (henc) -- (ca);
\end{tikzpicture}
$$

Training is unchanged: teacher forcing, cross-entropy on the next target token,
end-to-end. The encoder's self-attention is bidirectional (the whole source is
available at once); the decoder's self-attention is causal (a word may not see the
future words it is meant to predict); cross-attention bridges them. This is the
architecture behind essentially every production translation system today, and the
same encoder-decoder shape does summarization, semantic parsing, and dialogue as
well.[^jm-tfmr]

## Tokenization for MT

Rich morphology and proper names make a fixed word vocabulary hopeless: it would
have to enumerate every inflected form and would still miss the next unseen name.
MT systems instead break text into **subword** units with an algorithm like **BPE**
(byte-pair encoding) or **wordpiece**, so a rare or unseen word decomposes into
pieces the model has seen.[^jm-tok]

Wordpiece builds its lexicon by a greedy, likelihood-driven merge:

```algorithm
caption: $\textsc{Wordpiece}$ — build a subword lexicon of target size $V$
input: training corpus, target vocabulary size $V$
initialize the lexicon with the individual characters
repeat
  train an n-gram language model on the corpus using the current lexicon
  for each pair of current wordpieces do
    form the candidate wordpiece by concatenating the pair
  add the candidate that most increases the corpus likelihood to the lexicon
until the lexicon has $V$ wordpieces
```

Two choices matter for translation specifically. First, the vocabulary size is
usually $8\text{K}$ to $32\text{K}$ wordpieces — small enough to keep the softmax
tractable, large enough to keep common words whole. Second, MT uses a **shared
vocabulary** built on a corpus of **both** languages, so a token like a proper name
that appears identically in source and target has one shared id and can be copied
straight across.[^jm-tok]
The model is now complete: it encodes a source, attends over it, generates with a
transformer decoder, and tokenizes into shared subwords. What remains is to run it —
to turn its per-step distributions into a sentence — and to measure the result. This
continues in [Machine Translation: Decoding, Evaluation, and Scale](/natural-language-processing/applications/machine-translation-decoding-and-evaluation).

[^jm-intro]: **Jurafsky & Martin**, _Speech and Language Processing_ (3rd ed.), Ch. 10 — Machine Translation and Encoder-Decoder Models: MT as the practical task (information access, computer-aided translation, in-the-moment communication) whose standard algorithm is the encoder-decoder / sequence-to-sequence network.
[^jm-hard]: **Jurafsky & Martin**, Ch. 10 intro — the target of translation is a complex function of the entire source, not a per-word relabeling, as the English/Japanese and English/Chinese examples show; this is what distinguishes encoder-decoder modeling from sequence labeling.
[^jm-typology]: **Jurafsky & Martin**, §10.1 — Language Divergences and Typology: linguistic universals and statistical universals; the split between idiosyncratic/lexical differences (handled one by one) and systematic differences (modeled generally); linguistic typology and the World Atlas of Language Structures (WALS).
[^jm-order]: **Jurafsky & Martin**, §10.1.1 — Word Order Typology: SVO, SOV, and VSO basic orders; correlated properties like prepositions in VO vs. postpositions in OV languages; reordering as a source of translation difficulty.
[^jm-lex2]: **Jurafsky & Martin**, §10.1.2 — Lexical Divergences: target-forced distinctions (German _Wand_/_Mauer_, Mandarin _gege_/_didi_), grammatical constraints on word choice (adjective gender in French/Spanish), lexical gaps (Mandarin _xiao_, Japanese _oyakoko_), and Talmy's verb-framed vs. satellite-framed motion typology.
[^jm-lex]: **Jurafsky & Martin**, §10.1.2 — Lexical Divergences: one-to-many mappings (English _bass_ to Spanish _lubina_/_bajo_), many-to-many mappings (English _leg_ to French _jambe_/_patte_/_pied_/_étape_), lexical gaps, and the link to word-sense disambiguation.
[^jm-morph]: **Jurafsky & Martin**, §10.1.3 — Morphological Typology: isolating vs. polysynthetic (morphemes per word) and agglutinative vs. fusional (segmentability); rich morphology motivates subword (BPE/wordpiece) models.
[^jm-refdens]: **Jurafsky & Martin**, §10.1.4 — Referential Density: pro-drop languages omit pronouns; translating out of them requires recovering the dropped referent and, into English, its gender, a source of MT gender bias (§10.9).
[^jm-encdec]: **Jurafsky & Martin**, §10.2 — The Encoder-Decoder Model (Fig. 10.3): an encoder producing contextual representations, a context vector conveying the input's essence, and a decoder generating an arbitrary-length output; realizable with RNNs, LSTMs, or transformers.
[^jm-train]: **Jurafsky & Martin**, §10.3, §10.3.1 (Eqs. 10.10, 10.12; Fig. 10.7) — the RNN encoder-decoder as a conditional language model $P(\mathbf{y}\mid\mathbf{x})$, the context as the encoder's final hidden state made available at each decoder step, end-to-end training on a parallel corpus with teacher forcing and per-word cross-entropy.
[^jm-bottleneck]: **Jurafsky & Martin**, §10.4 (Fig. 10.8) — the bottleneck: forcing the context to be only the encoder's final hidden state makes all source information pass through one fixed-length vector, poorly representing the start of long sentences.
[^jm-attn]: **Jurafsky & Martin**, §10.4 (Eqs. 10.15–10.17) — the attention mechanism: a per-decoding-step context $\mathbf{c}_i = \sum_j \alpha_{ij}\mathbf{h}^e_j$ built from dot-product (or learned bilinear $\mathbf{W}_s$) scores softmaxed into weights $\alpha_{ij}$, giving a dynamic context in place of the static bottleneck.
[^jm-attn-hist]: **Jurafsky & Martin**, Ch. 10 Bibliographical and Historical Notes — attention as a soft weighting of inputs (Graves 2013), extended, named "attention," and applied to MT by Bahdanau et al. (2015); the transformer encoder-decoder by Vaswani et al. (2017).
[^jm-tfmr]: **Jurafsky & Martin**, §10.6 (Figs. 10.15–10.16; Eqs. 10.21–10.22) — Encoder-Decoder with Transformers: $N=6$ stacked encoder/decoder blocks; the decoder block's extra cross-attention sublayer with queries from the decoder and keys/values from the encoder output; bidirectional encoder self-attention and causal decoder self-attention; teacher-forced cross-entropy training.
[^jm-tok]: **Jurafsky & Martin**, §10.7.1 — Tokenization: BPE and wordpiece subword vocabularies; the wordpiece merge that greedily adds the pair most increasing corpus likelihood; a shared source-target vocabulary of 8K–32K wordpieces to ease copying tokens like names.