Large Language Models

The Transformer gave us the architecture; this lesson is about what happens when one of its three families is scaled by four orders of magnitude. A large language model is a decoder-only Transformer trained on a single objective, next-token prediction, and then made very large.¹ The architecture barely changes; the new design axis is scale, and the new phenomena, in-context learning, emergent abilities, and the entire alignment problem, are consequences of it.

From Transformer to LLM: scale as the design axis

The three families diverge by which stack they keep and how they mask attention. Decoder-only models came to dominate generative use because causal masking realizes the autoregressive factorization in a single parallel forward pass, and because that one objective places no ceiling on the data: every token of raw text is a free training label.

Axis	Classical Transformer	Large language model
Primary design lever	architecture (depth, heads, $d_{ff}$ )	scale ( $N$ , $D$ , compute $C$ )
Training signal	task labels (translation pairs, etc.)	next token from raw text
Adaptation	retrain per task	prompt, or light fine-tune
Dominant family	encoder--decoder	decoder-only

The loss is the same negative log-likelihood the decoder always used; only its scale of application changed.

L (θ) = - \frac{1}{T} t = 1 \sum T log p_{θ} (x_{t} ∣ x_{< t}) .

What makes this objective so productive is that minimizing it forces the model to compress the corpus: an accurate $p_{θ} (x_{t} ∣ x_{< t})$ is, by Shannon's source-coding theorem, a near-optimal code for the text, so next-token prediction is equivalent to lossless compression.²

Tokenization

A model over raw characters has a tiny vocabulary but very long sequences; a model over whole words has short sequences but an unbounded vocabulary and no way to spell a word it has never seen. Subword tokenization sits between: a fixed vocabulary of frequent fragments, with rare words spelled out of common pieces.

Byte-pair encoding

Byte-pair encoding (BPE) starts from single characters and greedily merges the most frequent adjacent pair into a new token, repeating until the vocabulary reaches a target size.³ Each merge replaces the most common bigram in the current segmentation with a single symbol.

Algorithm:

\textsc{LearnBPE}(\text{corpus}, k)

— learn

k

merge rules

1
$V \gets$ set of all characters in corpus
2
segment each word as a sequence of characters
3
for $i \gets 1$ to $k$ do
4
count all adjacent token pairs $(a,b)$ across the corpus
5
$(a^{\ast}, b^{\ast}) \gets \arg\max_{(a,b)} \count(a,b)$
most frequent pair
6
add merged token $\underline{a^{\ast}b^{\ast}}$ to $V$
7
replace every adjacent pair $a^{\ast} \to b^{\ast}$ with $\underline{a^{\ast}b^{\ast}}$ in the corpus
8
return $V$ and the ordered list of merges

The merges are applied at inference time in the same order they were learned, which makes the segmentation deterministic. The figure traces the rule learned from a toy corpus.

One BPE step on a toy corpus. The most frequent adjacent pair, "e"+"s", is merged into a single token "es", shrinking every occurrence by one symbol.

Here it is on a real sentence. Split the fat cat sat on the mat at the gate into characters — 29 symbols. The most common adjacent pair, $a t$ , appears six times (in fat, cat, sat, mat, at, and gate), so it merges first and leaves 23. The next two pairs each appear three times and spell out a word: $t h$ merges to $th$ , then $th e$ to $the$ , leaving 17 — a 41% cut from three merges. A merge removes one symbol for every place its pair occurs, so the most frequent pairs save the most and go first.

BPE on "the fat cat sat on the mat at the gate" — 29 characters once the words are split. Each row merges the most frequent adjacent pair (shown at left with its count); the running token count is at right. Three merges form "at" and "the", taking the sentence to 17 tokens — 41% fewer.

WordPiece and Unigram

WordPiece (used by BERT) is BPE with a likelihood criterion instead of raw frequency: it merges the pair that most increases the language-model likelihood of the corpus, scoring a candidate $(a, b)$ by $count (ab) / (count (a) count (b))$ rather than $count (ab)$ alone.⁴ The Unigram model takes the opposite direction: it starts from a large vocabulary and prunes it, modeling each sentence as a bag of independent subword probabilities and removing tokens whose loss of likelihood is smallest.⁵

SentencePiece is the implementation that makes Unigram (or BPE) operate directly on raw text including the space character, encoded as a meta-symbol, so it needs no language-specific pre-tokenizer and is fully reversible.⁵

Method	Direction	Merge / prune criterion	Probabilistic	Used by
BPE	bottom-up (merge)	most frequent adjacent pair	no	GPT-2, GPT-3, LLaMA
WordPiece	bottom-up (merge)	max likelihood gain of the merge	partly	BERT
Unigram	top-down (prune)	min likelihood loss on removal	yes	T5, multilingual models
SentencePiece	wraps BPE / Unigram	(whichever it wraps)	yes	T5, LLaMA

The choice matters less than the fact of subwording: all four keep $∣ V ∣$ near $30, 000$ to $50, 000$ while spelling any string, which is what lets one model read code, prose, and rare proper nouns through a single embedding table.

Pretraining objectives

The four families correspond to four ways of factoring the corpus likelihood, each realized by a different attention mask.

Span corruption, the T5 objective, masks contiguous spans rather than isolated tokens and asks an encoder--decoder to emit the missing spans in order, replacing each by a sentinel token.⁷ Prefix LM splits the sequence: a bidirectional prefix conditions a causal continuation, a hybrid that keeps generation while letting the conditioning context attend in both directions.

Objective	Model type	Context for target	Mask	Example
Causal LM	decoder-only	left only	causal	GPT
Masked LM	encoder-only	both sides	full	BERT
Span corruption	encoder--decoder	full (encoder)	full + causal	T5
Prefix LM	decoder-only	bi-dir prefix, causal suffix	block	UniLM, PaLM

The attention mask is the whole difference, and it is worth seeing the three patterns side by side: which key columns each query row is permitted to read.

Attention masks. Bidirectional (left) reads all positions; causal (middle) reads only the past; prefix (right) is bidirectional in the prefix, causal after it. Filled cells are attended.

The three families in depth

The transformer lesson sketched the split; here we work out parameter counts, masks, and use.

Encoder-only (BERT, RoBERTa). A bidirectional encoder stack trained by masked LM, read out by a pooled [CLS] representation or per-token heads. BERT-base is $110$ M parameters ( $12$ layers, $d = 768$ , $12$ heads); BERT-large is $340$ M. RoBERTa keeps the architecture and removes the next-sentence objective, trains on ten times the data with larger batches, and shows the original BERT was badly under-trained.⁴ These models classify, tag, and retrieve; they cannot generate, because no position is causal.

Decoder-only (GPT-2/3, LLaMA). A causal decoder with the cross-attention removed, so each layer is masked self-attention plus an FFN. This is the family that scaled.⁶⁸ GPT-2 is $1.5$ B parameters; GPT-3 is $175$ B; LLaMA ranges $7$ B to $65$ B and matches GPT-3 quality at a fraction of the size by training longer on more tokens, a direct application of compute-optimal scaling.⁹

Encoder--decoder (T5). Both stacks with cross-attention, trained by span corruption, casting every task as text-to-text. T5 ranges from $60$ M (small) to $11$ B parameters.⁷ It is built for transduction: translation, summarization, question answering.

Family	Mask	Objective	Parameters (examples)	Built for
Encoder-only	full	masked LM	BERT $110$ M / $340$ M; RoBERTa $355$ M	understanding
Decoder-only	causal	causal LM	GPT-2 $1.5$ B; GPT-3 $175$ B; LLaMA $7$ -- $65$ B	generation
Encoder--decoder	full + causal	span corruption	T5 $60$ M-- $11$ B	transduction

The accounting from the transformer lesson carries over: per layer, attention costs $4 d^{2}$ and the FFN $8 d^{2}$ , so two-thirds of the weights live in the FFN, and total parameters scale as $N \approx 12 n_{layers} d^{2}$ plus the embedding table $∣ V ∣ d$ .

Tokenization is not neutral

Standard treatments present tokenization as a preprocessing step, but the choice of vocabulary leaks into every downstream behavior, and three consequences are worth naming because they surface constantly in practice.

Number handling. BPE merges frequent strings, and digit sequences are not uniformly frequent, so a naive tokenizer splits 1234 into whatever fragments its merges happened to learn (12, 34, or 1, 234), with no relation to place value. Because arithmetic then operates on inconsistent chunks, early GPT models were erratic at multi-digit addition for a tokenization reason, not a reasoning one. LLaMA fixes this by forcing every digit to its own token, so 1234 is always four tokens 1 2 3 4 and the model sees a consistent positional structure.⁹ The lesson is that an emergent-looking weakness can be an artifact of the vocabulary.

Fertility and the multilingual tax. The fertility of a tokenizer on a language is its average tokens per word. A vocabulary fit mostly on English spends few tokens per English word but shatters Hindi or Thai into many bytes, so the same sentence costs several times more tokens in an underrepresented language. Since context length, latency, and price are all counted in tokens, high fertility is a direct, measurable tax paid by exactly the languages with the least training data, and it compounds the data imbalance rather than correcting it.

Glitch tokens. A vocabulary is frozen after the tokenizer is fit, but the corpus the model then trains on may barely contain some tokens (a scraping artifact, a username, a stray Unicode string). Such a token has an embedding row that receives almost no gradient and stays near its random initialization, so prompting the model with it produces bizarre, off-distribution output. These glitch tokens are a direct illustration of the point that the embedding table $∣ V ∣ d$ is only as good as the coverage each row received during training.

Tokenizer fertility. The same meaning costs a different number of tokens per language under an English-fit vocabulary: low fertility for English, high for a script the merges never saw, so token cost tracks training-data coverage, not sentence length.

This continues in Scaling, Inference, and Alignment of Language Models, which takes the object built here and asks what happens when it is scaled, how it is decoded and served, and how the raw next-token predictor is turned into an assistant.

Takeaways

A large language model is a decoder-only Transformer trained on next-token prediction $L = - \sum_{t} log p_{θ} (x_{t} ∣ x_{< t})$ , then scaled; the architecture is fixed and scale is the design axis. Minimizing that loss is equivalent to lossless compression of the corpus.
Subword tokenization (BPE merges, WordPiece by likelihood, Unigram by pruning, SentencePiece wrapping either) keeps $∣ V ∣ \approx 30$ -- $50$ k while spelling any string; a real sentence collapses by $41%$ under three greedy BPE merges.
The four objectives are causal LM (GPT), masked LM (BERT), span corruption (T5), and prefix LM, distinguished entirely by the attention mask.
The three families: encoder-only (BERT/RoBERTa, understanding), decoder-only (GPT/LLaMA, generation), encoder--decoder (T5, transduction); two-thirds of the parameters live in the FFN, and $N \approx 12 n_{layers} d^{2}$ plus the embedding table.
Tokenization is not neutral: digit splitting can fake an arithmetic weakness (LLaMA forces one token per digit), tokenizer fertility taxes low-resource languages, and undertrained glitch tokens leave near-random embedding rows.

Vaswani et al., Attention Is All You Need, NeurIPS 2017 — introduces the Transformer; scaled dot-product and multi-head attention replace recurrence, the substrate every LLM is built on. ↩
Chollet, Deep Learning with Python, Ch. 11 — Deep Learning for Text: next-token language modeling as sequence prediction, and why large pretrained Transformers transfer across text tasks. ↩
Sennrich, Haddow & Birch, Neural Machine Translation of Rare Words with Subword Units, ACL 2016 — adapts byte-pair encoding to open-vocabulary NMT, the origin of subword tokenization for neural models. ↩
Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers, NAACL 2019 — masked language modeling and the bidirectional encoder; WordPiece tokenization and the [CLS]/[MASK] scheme. ↩ ↩² ↩³
Kudo & Richardson, SentencePiece (EMNLP 2018) and Kudo, Subword Regularization (ACL 2018) — the Unigram language-model tokenizer and a language-agnostic, reversible tokenization toolkit. ↩ ↩²
Radford et al., Language Models are Unsupervised Multitask Learners (GPT-2), 2019 — a large decoder-only causal LM performs many tasks zero-shot from scale alone. ↩ ↩²
Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5), JMLR 2020 — span-corruption pretraining and the text-to-text framing of every NLP task in one encoder--decoder. ↩ ↩²
Brown et al., Language Models are Few-Shot Learners (GPT-3), NeurIPS 2020 — the $175$ B decoder-only model that demonstrated in-context (few-shot) learning without weight updates. ↩
Touvron et al., LLaMA: Open and Efficient Foundation Language Models, 2023 — compute-optimal training of $7$ -- $65$ B decoder-only models that match far larger ones by training longer on more tokens. ↩ ↩²