--- title: Large Language Models module: Large Models & Agents moduleNumber: 10 lessonNumber: 1 order: 1001 summary: > A large language model is a decoder-only Transformer trained on one objective, next-token prediction, then scaled until new behavior appears. This first part builds the object itself: the equivalence between next-token prediction and lossless compression, subword tokenization (BPE, WordPiece, Unigram, SentencePiece) worked on a real sentence, the four pretraining objectives and the attention masks that distinguish them, and the three model families (encoder-only, decoder-only, encoder--decoder) with their parameter budgets. Scaling, decoding, the KV cache, and alignment continue in part two. topics: [Large Models & Agents] sources: - book: Chollet ref: "Ch. 11 — Deep Learning for Text; large-scale language models" - book: Goodfellow ref: "Ch. 12 — Applications; large-scale language modeling postdates the 2016 text" --- [The Transformer](/deep-learning/architectures/the-transformer-architecture) gave us the architecture; this lesson is about what happens when one of its three families is scaled by four orders of magnitude. A **large language model** is a decoder-only Transformer trained on a single objective, next-token prediction, and then made very large.[^vaswani] The architecture barely changes; the new design axis is scale, and the new phenomena, in-context learning, emergent abilities, and the entire alignment problem, are consequences of it. > **Definition (Large language model).** A neural sequence model > $p_\theta(x_t \mid x_{ [causal attention](/deep-learning/architectures/the-transformer-architecture), > trained by maximum likelihood on a corpus of $10^{11}$ to $10^{13}$ tokens, > with parameter count $N$ ranging from $10^{8}$ to $10^{12}$. "Large" is not a > qualitative class but a region of a scaling curve where capabilities appear > that are absent at small $N$. ## From Transformer to LLM: scale as the design axis The [three families](/deep-learning/architectures/the-transformer-architecture) diverge by which stack they keep and how they mask attention. Decoder-only models came to dominate generative use because causal masking realizes the autoregressive factorization in a single parallel forward pass, and because that one objective places no ceiling on the data: every token of raw text is a free training label. | Axis | Classical Transformer | Large language model | | --- | --- | --- | | Primary design lever | architecture (depth, heads, $d_{\text{ff}}$) | scale ($N$, $D$, compute $C$) | | Training signal | task labels (translation pairs, etc.) | next token from raw text | | Adaptation | retrain per task | prompt, or light fine-tune | | Dominant family | encoder--decoder | decoder-only | The loss is the same negative log-likelihood the decoder always used; only its scale of application changed. $$ \mathcal{L}(\theta) = -\frac{1}{T}\sum_{t=1}^{T} \log p_\theta\!\parens{x_t \mid x_{ **Definition (Subword vocabulary).** A finite set $V$ of tokens, where each > token is a character or a string of characters, such that every string in the > corpus can be segmented into a sequence of tokens in $V$. The aim is a small > $\abs{V}$ that still keeps the average tokens-per-word low. ### Byte-pair encoding **Byte-pair encoding (BPE)** starts from single characters and greedily merges the most frequent adjacent pair into a new token, repeating until the vocabulary reaches a target size.[^sennrich] Each merge replaces the most common bigram in the current segmentation with a single symbol. ```algorithm caption: $\textsc{LearnBPE}(\text{corpus}, k)$ — learn $k$ merge rules $V \gets$ set of all characters in corpus segment each word as a sequence of characters for $i \gets 1$ to $k$ do count all adjacent token pairs $(a,b)$ across the corpus $(a^{\ast}, b^{\ast}) \gets \arg\max_{(a,b)} \count(a,b)$ // most frequent pair add merged token $\underline{a^{\ast}b^{\ast}}$ to $V$ replace every adjacent pair $a^{\ast} \to b^{\ast}$ with $\underline{a^{\ast}b^{\ast}}$ in the corpus return $V$ and the ordered list of merges ``` The merges are applied at inference time in the same order they were learned, which makes the segmentation deterministic. The figure traces the rule learned from a toy corpus. $$ % caption: One BPE step on a toy corpus. The most frequent adjacent pair, "e"+"s", % is merged into a single token "es", shrinking every occurrence by one symbol. \begin{tikzpicture}[>=stealth, font=\footnotesize, tk/.style={draw, black, minimum width=7mm, minimum height=7mm, inner sep=1pt, align=center}, mg/.style={draw=acc, text=acc, thick, minimum width=10mm, minimum height=7mm, inner sep=1pt, align=center, fill=acc!15}] \definecolor{acc}{HTML}{2348F2} % before row \node[anchor=east, black] at (-0.3,0) {before}; \node[tk] (a0) at (0.4,0) {l}; \node[tk] (a1) at (1.2,0) {o}; \node[tk] (a2) at (2.0,0) {w}; \node[tk] (a3) at (2.8,0) {e}; \node[tk] (a4) at (3.6,0) {s}; \node[tk] (a5) at (4.4,0) {t}; % highlight the chosen pair e + s \draw[acc, thick] (2.55,-0.45) rectangle (3.95,0.45); \node[acc, anchor=south] at (3.25,0.55) {top pair}; % after row \node[anchor=east, black] at (-0.3,-1.8) {after}; \node[tk] (b0) at (0.4,-1.8) {l}; \node[tk] (b1) at (1.2,-1.8) {o}; \node[tk] (b2) at (2.0,-1.8) {w}; \node[mg] (b3) at (2.95,-1.8) {es}; \node[tk] (b4) at (3.9,-1.8) {t}; % merge arrow \draw[->, acc, thick] (3.25,-0.55) -- (2.95,-1.35) node[midway, right, text=acc] {merge}; \end{tikzpicture} $$ Here it is on a real sentence. Split **the fat cat sat on the mat at the gate** into characters — 29 symbols. The most common adjacent pair, $\mathtt{a}\,\mathtt{t}$, appears six times (in _fat_, _cat_, _sat_, _mat_, _at_, and _gate_), so it merges first and leaves 23. The next two pairs each appear three times and spell out a word: $\mathtt{t}\,\mathtt{h}$ merges to $\mathtt{th}$, then $\mathtt{th}\,\mathtt{e}$ to $\mathtt{the}$, leaving 17 — a 41% cut from three merges. A merge removes one symbol for every place its pair occurs, so the most frequent pairs save the most and go first. $$ % caption: BPE on "the fat cat sat on the mat at the gate" — 29 characters once the % words are split. Each row merges the most frequent adjacent pair (shown at left with % its count); the running token count is at right. Three merges form "at" and "the", % taking the sentence to 17 tokens — 41% fewer. \begin{tikzpicture}[x=1cm, y=1cm, font=\footnotesize, tk/.style={draw=black!70, minimum width=0.4cm, minimum height=0.5cm, inner sep=0}, mg/.style={draw=acc, fill=acc!16, text=acc, minimum width=0.4cm, minimum height=0.5cm, inner sep=0}] \definecolor{acc}{HTML}{2348F2} % left labels: the merge applied and how often the pair occurs \node[anchor=east] at (-0.35,0) {start}; \node[anchor=east] at (-0.35,-1) {\texttt{a+t} (6x)}; \node[anchor=east] at (-0.35,-2) {\texttt{t+h} (3x)}; \node[anchor=east] at (-0.35,-3) {\texttt{th+e} (3x)}; % right: running token count \node[anchor=west, text=black] at (14.0,0) {\texttt{29 tokens}}; \node[anchor=west, text=black] at (14.0,-1) {23}; \node[anchor=west, text=black] at (14.0,-2) {20}; \node[anchor=west, text=black] at (14.0,-3) {17}; % row 0 — characters \node[tk] at (0.2,0){t};\node[tk] at (0.6,0){h};\node[tk] at (1.0,0){e}; \node[tk] at (1.64,0){f};\node[tk] at (2.04,0){a};\node[tk] at (2.44,0){t}; \node[tk] at (3.08,0){c};\node[tk] at (3.48,0){a};\node[tk] at (3.88,0){t}; \node[tk] at (4.52,0){s};\node[tk] at (4.92,0){a};\node[tk] at (5.32,0){t}; \node[tk] at (5.96,0){o};\node[tk] at (6.36,0){n}; \node[tk] at (7.0,0){t};\node[tk] at (7.4,0){h};\node[tk] at (7.8,0){e}; \node[tk] at (8.44,0){m};\node[tk] at (8.84,0){a};\node[tk] at (9.24,0){t}; \node[tk] at (9.88,0){a};\node[tk] at (10.28,0){t}; \node[tk] at (10.92,0){t};\node[tk] at (11.32,0){h};\node[tk] at (11.72,0){e}; \node[tk] at (12.36,0){g};\node[tk] at (12.76,0){a};\node[tk] at (13.16,0){t};\node[tk] at (13.56,0){e}; % row 1 — merge a+t -> at (in fat, cat, sat, mat, at, gate) \node[tk] at (0.2,-1){t};\node[tk] at (0.6,-1){h};\node[tk] at (1.0,-1){e}; \node[tk] at (1.64,-1){f};\node[mg,minimum width=0.8cm] at (2.24,-1){at}; \node[tk] at (3.08,-1){c};\node[mg,minimum width=0.8cm] at (3.68,-1){at}; \node[tk] at (4.52,-1){s};\node[mg,minimum width=0.8cm] at (5.12,-1){at}; \node[tk] at (5.96,-1){o};\node[tk] at (6.36,-1){n}; \node[tk] at (7.0,-1){t};\node[tk] at (7.4,-1){h};\node[tk] at (7.8,-1){e}; \node[tk] at (8.44,-1){m};\node[mg,minimum width=0.8cm] at (9.04,-1){at}; \node[mg,minimum width=0.8cm] at (10.08,-1){at}; \node[tk] at (10.92,-1){t};\node[tk] at (11.32,-1){h};\node[tk] at (11.72,-1){e}; \node[tk] at (12.36,-1){g};\node[mg,minimum width=0.8cm] at (12.96,-1){at};\node[tk] at (13.56,-1){e}; % row 2 — merge t+h -> th (in the x3) \node[mg,minimum width=0.8cm] at (0.4,-2){th};\node[tk] at (1.0,-2){e}; \node[tk] at (1.64,-2){f};\node[mg,minimum width=0.8cm] at (2.24,-2){at}; \node[tk] at (3.08,-2){c};\node[mg,minimum width=0.8cm] at (3.68,-2){at}; \node[tk] at (4.52,-2){s};\node[mg,minimum width=0.8cm] at (5.12,-2){at}; \node[tk] at (5.96,-2){o};\node[tk] at (6.36,-2){n}; \node[mg,minimum width=0.8cm] at (7.2,-2){th};\node[tk] at (7.8,-2){e}; \node[tk] at (8.44,-2){m};\node[mg,minimum width=0.8cm] at (9.04,-2){at}; \node[mg,minimum width=0.8cm] at (10.08,-2){at}; \node[mg,minimum width=0.8cm] at (11.12,-2){th};\node[tk] at (11.72,-2){e}; \node[tk] at (12.36,-2){g};\node[mg,minimum width=0.8cm] at (12.96,-2){at};\node[tk] at (13.56,-2){e}; % row 3 — merge th+e -> the \node[mg,minimum width=1.2cm] at (0.6,-3){the}; \node[tk] at (1.64,-3){f};\node[mg,minimum width=0.8cm] at (2.24,-3){at}; \node[tk] at (3.08,-3){c};\node[mg,minimum width=0.8cm] at (3.68,-3){at}; \node[tk] at (4.52,-3){s};\node[mg,minimum width=0.8cm] at (5.12,-3){at}; \node[tk] at (5.96,-3){o};\node[tk] at (6.36,-3){n}; \node[mg,minimum width=1.2cm] at (7.4,-3){the}; \node[tk] at (8.44,-3){m};\node[mg,minimum width=0.8cm] at (9.04,-3){at}; \node[mg,minimum width=0.8cm] at (10.08,-3){at}; \node[mg,minimum width=1.2cm] at (11.32,-3){the}; \node[tk] at (12.36,-3){g};\node[mg,minimum width=0.8cm] at (12.96,-3){at};\node[tk] at (13.56,-3){e}; \end{tikzpicture} $$ ### WordPiece and Unigram **WordPiece** (used by BERT) is BPE with a likelihood criterion instead of raw frequency: it merges the pair that most increases the language-model likelihood of the corpus, scoring a candidate $(a,b)$ by $\count(ab) / (\count(a)\count(b))$ rather than $\count(ab)$ alone.[^devlin] The **Unigram** model takes the opposite direction: it starts from a large vocabulary and prunes it, modeling each sentence as a bag of independent subword probabilities and removing tokens whose loss of likelihood is smallest.[^kudo] > **Definition (Unigram language model).** A segmentation $S = (s_1, \dots, s_m)$ > of a string is scored as a product of independent token probabilities, > $P(S) = \prod_{j=1}^{m} p(s_j)$, with $\sum_{s \in V} p(s) = 1$. The most > probable segmentation is recovered by the Viterbi algorithm, and the token > probabilities are fit by expectation--maximization. **SentencePiece** is the implementation that makes Unigram (or BPE) operate directly on raw text including the space character, encoded as a meta-symbol, so it needs no language-specific pre-tokenizer and is fully reversible.[^kudo] | Method | Direction | Merge / prune criterion | Probabilistic | Used by | | --- | --- | --- | --- | --- | | BPE | bottom-up (merge) | most frequent adjacent pair | no | GPT-2, GPT-3, LLaMA | | WordPiece | bottom-up (merge) | max likelihood gain of the merge | partly | BERT | | Unigram | top-down (prune) | min likelihood loss on removal | yes | T5, multilingual models | | SentencePiece | wraps BPE / Unigram | (whichever it wraps) | yes | T5, LLaMA | The choice matters less than the fact of subwording: all four keep $\abs{V}$ near $30{,}000$ to $50{,}000$ while spelling any string, which is what lets one model read code, prose, and rare proper nouns through a single embedding table. ## Pretraining objectives The four families correspond to four ways of factoring the corpus likelihood, each realized by a different attention mask. > **Definition (Causal language modeling).** Predict each token from its left > context only; train to maximize the autoregressive log-likelihood > $$ > \mathcal{L}_{\text{CLM}} = \sum_{t=1}^{T} \log p_\theta\!\parens{x_t \mid x_{ $$ > realized by a lower-triangular (causal) attention mask. This is the GPT > objective and the only one that generates left to right.[^radford] > **Definition (Masked language modeling).** Corrupt a random subset > $\mathcal{M}$ of positions (replace with a `[MASK]` token) and predict them from > the full bidirectional context, > $$ > \mathcal{L}_{\text{MLM}} = \sum_{t \in \mathcal{M}} \log p_\theta\!\parens{x_t \mid x_{\setminus \mathcal{M}}}. > $$ > Because each masked position sees both sides, the attention mask is full > (unrestricted). This is the BERT objective and is built for understanding, not > generation.[^devlin] **Span corruption**, the T5 objective, masks contiguous spans rather than isolated tokens and asks an encoder--decoder to emit the missing spans in order, replacing each by a sentinel token.[^raffel] **Prefix LM** splits the sequence: a bidirectional prefix conditions a causal continuation, a hybrid that keeps generation while letting the conditioning context attend in both directions. | Objective | Model type | Context for target | Mask | Example | | --- | --- | --- | --- | --- | | Causal LM | decoder-only | left only | causal | GPT | | Masked LM | encoder-only | both sides | full | BERT | | Span corruption | encoder--decoder | full (encoder) | full + causal | T5 | | Prefix LM | decoder-only | bi-dir prefix, causal suffix | block | UniLM, PaLM | The attention mask is the whole difference, and it is worth seeing the three patterns side by side: which key columns each query row is permitted to read. $$ % caption: Attention masks. Bidirectional (left) reads all positions; causal % (middle) reads only the past; prefix (right) is bidirectional in the prefix, % causal after it. Filled cells are attended. \begin{tikzpicture}[font=\scriptsize, x=0.46cm, y=0.46cm] \definecolor{acc}{HTML}{2348F2} \def\n{5} % ---------- bidirectional ---------- \begin{scope} \foreach \r in {1,...,5}{ \foreach \c in {1,...,5}{ \fill[acc!18] (\c-1,\n-\r) rectangle (\c,\n-\r+1); }} \draw[black] (0,0) grid (\n,\n); \node[anchor=south, black] at (\n/2,\n+0.3) {bidirectional}; \end{scope} % ---------- causal ---------- \begin{scope}[xshift=3.6cm] \foreach \r in {1,...,5}{ \foreach \c in {1,...,5}{ \pgfmathparse{\c<=\r ? 1 : 0} \ifnum\pgfmathresult=1 \fill[acc!18] (\c-1,\n-\r) rectangle (\c,\n-\r+1); \else \fill[black] (\c-1,\n-\r) rectangle (\c,\n-\r+1); \fi }} \draw[black] (0,0) grid (\n,\n); \node[anchor=south, black] at (\n/2,\n+0.3) {causal}; \end{scope} % ---------- prefix (prefix length 2) ---------- \begin{scope}[xshift=7.2cm] \foreach \r in {1,...,5}{ \foreach \c in {1,...,5}{ \pgfmathparse{(\r<=2) ? 1 : (\c<=\r ? 1 : 0)} \ifnum\pgfmathresult=1 \fill[acc!18] (\c-1,\n-\r) rectangle (\c,\n-\r+1); \else \fill[black] (\c-1,\n-\r) rectangle (\c,\n-\r+1); \fi }} \draw[black] (0,0) grid (\n,\n); \draw[acc, thick] (0,\n-2) rectangle (\n,\n); % prefix band outline \node[anchor=south, black] at (\n/2,\n+0.3) {\texttt{prefix}}; \end{scope} \end{tikzpicture} $$ ## The three families in depth The transformer lesson sketched the split; here we work out parameter counts, masks, and use. **Encoder-only (BERT, RoBERTa).** A bidirectional encoder stack trained by masked LM, read out by a pooled `[CLS]` representation or per-token heads. BERT-base is $110$M parameters ($12$ layers, $d = 768$, $12$ heads); BERT-large is $340$M. RoBERTa keeps the architecture and removes the next-sentence objective, trains on ten times the data with larger batches, and shows the original BERT was badly under-trained.[^devlin] These models classify, tag, and retrieve; they cannot generate, because no position is causal. **Decoder-only (GPT-2/3, LLaMA).** A causal decoder with the cross-attention removed, so each layer is masked self-attention plus an FFN. This is the family that scaled.[^radford][^brown] GPT-2 is $1.5$B parameters; GPT-3 is $175$B; LLaMA ranges $7$B to $65$B and matches GPT-3 quality at a fraction of the size by training longer on more tokens, a direct application of compute-optimal scaling.[^touvron] **Encoder--decoder (T5).** Both stacks with cross-attention, trained by span corruption, casting every task as text-to-text. T5 ranges from $60$M (small) to $11$B parameters.[^raffel] It is built for transduction: translation, summarization, question answering. | Family | Mask | Objective | Parameters (examples) | Built for | | --- | --- | --- | --- | --- | | Encoder-only | full | masked LM | BERT $110$M / $340$M; RoBERTa $355$M | understanding | | Decoder-only | causal | causal LM | GPT-2 $1.5$B; GPT-3 $175$B; LLaMA $7$--$65$B | generation | | Encoder--decoder | full + causal | span corruption | T5 $60$M--$11$B | transduction | The accounting from the transformer lesson carries over: per layer, attention costs $4d^2$ and the FFN $8d^2$, so two-thirds of the weights live in the FFN, and total parameters scale as $N \approx 12\,n_{\text{layers}}\,d^2$ plus the embedding table $\abs{V}\,d$. ## Tokenization is not neutral Standard treatments present tokenization as a preprocessing step, but the choice of vocabulary leaks into every downstream behavior, and three consequences are worth naming because they surface constantly in practice. **Number handling.** BPE merges frequent strings, and digit sequences are not uniformly frequent, so a naive tokenizer splits `1234` into whatever fragments its merges happened to learn (`12`, `34`, or `1`, `234`), with no relation to place value. Because arithmetic then operates on inconsistent chunks, early GPT models were erratic at multi-digit addition for a tokenization reason, not a reasoning one. LLaMA fixes this by forcing every digit to its own token, so `1234` is always four tokens `1 2 3 4` and the model sees a consistent positional structure.[^touvron] The lesson is that an emergent-looking weakness can be an artifact of the vocabulary. **Fertility and the multilingual tax.** The **fertility** of a tokenizer on a language is its average tokens per word. A vocabulary fit mostly on English spends few tokens per English word but shatters Hindi or Thai into many bytes, so the same sentence costs several times more tokens in an underrepresented language. Since context length, latency, and price are all counted in tokens, high fertility is a direct, measurable tax paid by exactly the languages with the least training data, and it compounds the data imbalance rather than correcting it. **Glitch tokens.** A vocabulary is frozen after the tokenizer is fit, but the corpus the model then trains on may barely contain some tokens (a scraping artifact, a username, a stray Unicode string). Such a token has an embedding row that receives almost no gradient and stays near its random initialization, so prompting the model with it produces bizarre, off-distribution output. These "glitch tokens" are a direct illustration of the point that the embedding table $\abs{V}\,d$ is only as good as the coverage each row received during training. $$ % caption: Tokenizer fertility. The same meaning costs a different number of tokens per % language under an English-fit vocabulary: low fertility for English, high for a script % the merges never saw, so token cost tracks training-data coverage, not sentence length. \begin{tikzpicture}[>=stealth, font=\footnotesize, x=1cm, y=1cm] \definecolor{acc}{HTML}{2348F2} \draw[->, thick] (0,0) -- (0,3.4) node[above, font=\footnotesize] {\texttt{tokens per word}}; \draw[->, thick] (0,0) -- (6.4,0) node[right, font=\footnotesize] {\texttt{language}}; \draw[draw=acc, thick, fill=acc!15] (0.5,0) rectangle (1.4,0.9); \draw[draw=acc, thick, fill=acc!15] (2.0,0) rectangle (2.9,1.7); \draw[draw=acc, thick, fill=acc!15] (3.5,0) rectangle (4.4,2.6); \draw[draw=acc, thick, fill=acc!15] (5.0,0) rectangle (5.9,3.1); \node[black, anchor=north, font=\footnotesize] at (0.95,-0.15) {\texttt{English}}; \node[black, anchor=north, font=\footnotesize] at (2.45,-0.15) {\texttt{Spanish}}; \node[black, anchor=north, font=\footnotesize] at (3.95,-0.15) {\texttt{Hindi}}; \node[black, anchor=north, font=\footnotesize] at (5.45,-0.15) {\texttt{Thai}}; \end{tikzpicture} $$ This continues in [Scaling, Inference, and Alignment of Language Models](/deep-learning/large-models-and-agents/scaling-inference-and-alignment), which takes the object built here and asks what happens when it is scaled, how it is decoded and served, and how the raw next-token predictor is turned into an assistant. ## Takeaways - A **large language model** is a decoder-only Transformer trained on next-token prediction $\mathcal{L} = -\sum_t \log p_\theta(x_t \mid x_{