Machine Translation

Machine translation is the use of a computer to render text from one language into another: an English source $\mathbf{x}=x_1,\dots ,x_n$ becomes a target $\mathbf{y}=y_1,\dots ,y_m$ in, say, Spanish. Stated as a probability model it is short — pick the target string that is most likely given the source,

— but that single line hides everything hard about the task, and everything the last decade of NLP was built to solve. The standard model for computing $P(\mathbf{y}\mid \mathbf{x})$ is the encoder-decoder (or sequence-to-sequence) network, and it did not exist before translation demanded it. The encoder-decoder shape and the attention mechanism now central to all of modern NLP were both invented here, to make one sequence map onto another of a different length and a different order.¹ This lesson follows that history forward: the problem, the model, attention, and the modern form.

Language divergences and typology

Translation is not a word-for-word substitution, because languages do not line up word for word. Consider an English sentence and its Japanese translation:

English: He wrote a letter to a friend Japanese: tomodachi ni tegami-o kaita (friend to letter wrote)

The verb wrote sits in the middle of the English clause and at the very end of the Japanese one; the English subject pronoun he has no counterpart in the Japanese at all. A model that mapped input word $x_t$ to output word $y_t$ position by position — the way a part-of-speech tagger maps each word to its tag — cannot express this, because the target is a complex function of the entire source, not a per-word relabeling.² The systematic ways languages fail to line up are called translation divergences, and they come in several kinds.

Some aspects of language are universal — every language has words for people, for eating and drinking, for being polite; every language has nouns and verbs, ways to ask questions and issue commands. Others are statistical universals, holding for most languages. What is left over is variation, and it comes in two flavors. Some differences are idiosyncratic and lexical and must be handled one at a time (the word for dog is unpredictable from language to language). Others are systematic and can be modeled in general — many languages put the verb before the object, many put it after. The study of these systematic cross-linguistic similarities and differences is linguistic typology, and how hard a language is to translate into or out of comes down to the typological facts it sits on top of.³

Word order. Languages differ in the basic order of subject, verb, and object. English, French, German, and Mandarin are SVO (the verb sits between subject and object); Japanese and Hindi are SOV (the verb comes last); Irish and Arabic are VSO. Two languages of the same type tend to share other traits — VO languages generally have prepositions, OV languages postpositions — but crossing between types forces the system to move whole phrases across the sentence as it generates.⁴

Lexical divergence. One word rarely has one translation. English bass is Spanish lubina (the fish) or bajo (the instrument); the mapping is one-to-many and must be disambiguated by context, which ties translation tightly to word-sense disambiguation. Worse, the carve-up of conceptual space can be many-to-many. English leg becomes French jambe for a person, patte for an animal, pied for a chair, and étape for a leg of a journey — one English word fanning out across four French ones, none of them a clean synonym of the others.⁵ Sometimes the target language forces a distinction the source never made: German splits wall into Wand (inside a building) and Mauer (outside), and Mandarin distinguishes older brother gege from younger brother didi where English has only brother. Sometimes a language imposes a grammatical constraint the source lacks: French and Spanish mark grammatical gender on adjectives, so translating into them requires choosing a gender English left unspecified.

At the extreme, one language may have a lexical gap — no word or short phrase that expresses a concept another language lexicalizes. English has no clean equivalent of Mandarin xiao or Japanese oyakoko, and must fall back on awkward paraphrases like filial piety. Languages also differ systematically in where they package the components of an event. In a satellite-framed language like English, the direction of motion rides on a particle — the bottle floated out — while the verb carries the manner; in a verb-framed language like Spanish, direction rides on the verb — la botella salió flotando (the bottle exited floating) — leaving manner to a satellite. A translator must repackage motion from verb to satellite or back.⁶

Morphology. Languages differ in how much meaning is packed into a single word, and typology places this variation on two axes. The first is the number of morphemes per word, running from isolating languages like Vietnamese and Cantonese (roughly one morpheme per word) to polysynthetic languages like Siberian Yupik, where a single word may hold enough morphemes to translate a whole English sentence. The second is the degree to which morphemes are segmentable, running from agglutinative languages like Turkish, where morphemes have clean boundaries, to fusional languages like Russian, where one affix conflates several categories (-om in stolom fuses instrumental, singular, and first declension into a single suffix). Rich morphology explodes the vocabulary — every case and tense is a distinct surface form — which is the direct reason modern systems translate subword units rather than words.⁷

Referential density. Some languages require an explicit pronoun for a referent already in the discourse; others may drop it. Pro-drop languages like Spanish, Chinese, and Japanese routinely omit pronouns a non-pro-drop language like English must supply, and even among pro-drop languages the rate differs — Japanese and Chinese omit far more than Spanish. Languages that lean on more pronouns are referentially dense; those that rely on the hearer to recover dropped referents are referentially sparse, and are sometimes called cold languages (they leave the reader more inferential work, by analogy with cold media) against the more explicit hot languages. Translating out of a cold, pro-drop language means the model has to recover a pronoun (and, into English, its gender) that was never written down — a guess it often gets wrong, and a source of the gender bias MT systems are known for.⁸

These divergences are the specification for the model: it must map a variable-length source to a variable-length target, reorder freely, split and merge words, and insert or delete material — all learned from examples, none of it hand-coded.

The encoder-decoder model

The idea that meets that specification is to split the network in two. An encoder reads the whole source and compresses it into a representation; a decoder reads that representation and generates the target one token at a time. The two halves are joined by a context — a summary of the source that the decoder consults as it writes.⁹

In the recurrent form, the decoder is just a conditional language model. An ordinary RNN language model factors the probability of a string by the chain rule, one word at a time. To make it translate, condition every one of those factors on the source:

Mechanically, the encoder RNN reads the source and its final hidden state ${\mathbf{h}}_n^e$ becomes the context $\mathbf{c}$; the decoder RNN starts from $\mathbf{c}$ and generates. At each decoder step, a hidden state ${\mathbf{h}}_t^d$ is computed from the previous output, the previous hidden state, and the context, and a softmax over the vocabulary turns it into a distribution over the next word:

Training is end-to-end on a parallel corpus of source-target sentence pairs. The two sides are concatenated with a separator token, and the decoder is trained to predict each next target word by cross-entropy, averaged over the sentence. As with any autoregressive generator, training uses teacher forcing: the decoder is fed the true previous target word rather than its own (possibly wrong) previous prediction, which keeps training stable and fast.¹⁰

Attention: fixing the context bottleneck

The clean split has a flaw. The entire source, however long, is squeezed into the single fixed-length vector $\mathbf{c}={\mathbf{h}}_n^e$ — one final hidden state that must carry every noun, every clause, every dependency the decoder will ever need. This is the bottleneck: information from the start of a long sentence has to survive being copied through every intervening encoder step to reach that last state, and much of it does not.¹¹

Attention removes the bottleneck. Instead of handing the decoder one frozen context, it lets the decoder look back at all the encoder hidden states and, at each step of generation, build a fresh context tailored to the word it is about to produce. The context stops being static and becomes a weighted average of the encoder states, with the weights concentrated on the source words most relevant right now.¹²

The construction has three steps. At decoding step $i$, with the previous decoder state ${\mathbf{h}}_{i-1}^d$ in hand:

1. Score each encoder state for relevance to the current decoder state. The simplest score is a dot product — relevance as similarity: $\mathrm{score}({\mathbf{h}}_{i-1}^d,{\mathbf{h}}_j^e)={\mathbf{h}}_{i-1}^d\cdot {\mathbf{h}}_j^e$.
2. Normalize the scores across the source with a softmax into weights that sum to one: ${\alpha }_{ij}=\mathrm{softmax}(\mathrm{score}({\mathbf{h}}_{i-1}^d,{\mathbf{h}}_j^e))$.
3. Blend the encoder states by those weights to form this step's context: ${\mathbf{c}}_i={\sum }_j{\alpha }_{ij}{\mathbf{h}}_j^e$.

The decoder then computes its state from this dynamic context, ${\mathbf{h}}_i^d=g({\hat{y}}_{i-1},{\mathbf{h}}_{i-1}^d,{\mathbf{c}}_i)$, and emits the next word. A richer, learned score parameterizes the comparison with its own weight matrix, $\mathrm{score}({\mathbf{h}}_{i-1}^d,{\mathbf{h}}_j^e)={\mathbf{h}}_{i-1}^d{\mathbf{W}}_s{\mathbf{h}}_j^e$, so the model learns which aspects of similarity matter and can even relate encoder and decoder vectors of different dimensions.¹²

The weights ${\alpha }_{ij}$ have a direct reading: ${\alpha }_{ij}$ is how much target word $i$ attends to source word $j$. Laid out as a matrix over one sentence pair they form a soft alignment — the model's learned, distributed answer to the old question of which source words each target word came from.

This is the mechanism that Bahdanau and colleagues introduced and named attention, applying it to translation. Generalized — let a word attend to the words of its own sentence rather than a separate source — the same operation becomes the self-attention at the center of the transformer. Attention started as a patch on the encoder-decoder bottleneck for MT and became the primitive the whole field now runs on.¹³

Transformer-based NMT

The modern system replaces both RNNs with transformers, but keeps the encoder-decoder skeleton exactly. A stack of $N$ (typically $6$) bidirectional encoder blocks maps the source $\mathbf{x}$ to representations ${\mathbf{H}}^{enc}={\mathbf{h}}_1,\dots ,{\mathbf{h}}_n$; a stack of decoder blocks generates the target autoregressively, conditioning on the source and on the target words already produced.¹⁴

The one new part is in the decoder block. An encoder block has self-attention, then a feedforward layer, each wrapped in a residual add and layer norm. A decoder block inserts a third sublayer between them: cross-attention.

Concretely, cross-attention projects the encoder output into keys and values and the previous decoder layer into queries, then runs the same scaled dot-product attention used everywhere else in the transformer:

Training is unchanged: teacher forcing, cross-entropy on the next target token, end-to-end. The encoder's self-attention is bidirectional (the whole source is available at once); the decoder's self-attention is causal (a word may not see the future words it is meant to predict); cross-attention bridges them. This is the architecture behind essentially every production translation system today, and the same encoder-decoder shape does summarization, semantic parsing, and dialogue as well.¹⁴

Tokenization for MT

Rich morphology and proper names make a fixed word vocabulary hopeless: it would have to enumerate every inflected form and would still miss the next unseen name. MT systems instead break text into subword units with an algorithm like BPE (byte-pair encoding) or wordpiece, so a rare or unseen word decomposes into pieces the model has seen.¹⁵

Wordpiece builds its lexicon by a greedy, likelihood-driven merge:

Two choices matter for translation specifically. First, the vocabulary size is usually $8\text{K}$ to $32\text{K}$ wordpieces — small enough to keep the softmax tractable, large enough to keep common words whole. Second, MT uses a shared vocabulary built on a corpus of both languages, so a token like a proper name that appears identically in source and target has one shared id and can be copied straight across.¹⁵ The model is now complete: it encodes a source, attends over it, generates with a transformer decoder, and tokenizes into shared subwords. What remains is to run it — to turn its per-step distributions into a sentence — and to measure the result. This continues in Machine Translation: Decoding, Evaluation, and Scale.