--- title: Transformers in Practice module: Architectures moduleNumber: 5 lessonNumber: 7 order: 507 summary: > The Transformer makes no assumption about what a token represents. This part follows the architecture out of language: image patches feed a plain encoder (the Vision Transformer), the decoder-only half scales into the GPT line of large language models, and one substrate covers translation, retrieval, and multimodal grounding. We work the ViT patch arithmetic and a GPT parameter count by hand, then close on the empirical scaling laws — power-law loss, the Chinchilla compute-optimal balance, and emergent behavior — that made scale the dominant lever. topics: [Architectures] sources: - book: Chollet ref: "Ch. 11 — Deep Learning for Text; The Transformer Architecture" - book: Goodfellow ref: "Ch. 9 & §10.11 — convolution beyond grids and attention (ViT / LLMs postdate the 2016 text)" --- This builds on [The Transformer Architecture](/deep-learning/architectures/the-transformer-architecture), which assembled the encoder–decoder, worked through causal masking, and accounted for where the parameters ($12d^2$ per layer, two-thirds in the FFN) and the compute ($O(n^2 d)$ attention) live. That lesson stayed inside language. This one follows the same architecture out of it: the encoder classifies images, the decoder scales into large language models, and the design turns out to be a substrate rather than a single model. We work two accounting examples by hand — ViT patch geometry and a GPT parameter budget — and finish on the scaling laws that make growing the model pay off so predictably. ## The Vision Transformer Nothing in the architecture is specific to language. A Transformer consumes a sequence of vectors; supply image patches instead of word embeddings and the same encoder classifies pictures. The **Vision Transformer (ViT)** does exactly this, discarding the convolution entirely.[^chollet-vit] > **Definition (Patch embedding).** Split an image > $x \in \mathbb{R}^{H \times W \times C}$ into $N = HW / P^2$ non-overlapping > $P \times P$ patches, flatten each to a vector of length $P^2 C$, and project it > linearly to the model width $d$: > $$ > z_i = x_p^{(i)} E, \qquad E \in \mathbb{R}^{P^2 C \times d}, \quad i = 1, \dots, N. > $$ > A learnable class token $z_{\text{cls}}$ is prepended, and learned positional > embeddings $E_{\text{pos}}$ are added, giving the input sequence > $z_0 = [\,z_{\text{cls}};\, z_1; \dots; z_N\,] + E_{\text{pos}}$. The sequence $z_0$ runs through a standard encoder stack. Classification reads the final state of the class token alone, $\hat{y} = \softmax(z_{\text{cls}}^{(L)} W)$: the `[CLS]` token aggregates evidence from every patch through self-attention and serves as the pooled image representation. $$ % caption: ViT. The image is cut into fixed patches, each linearly embedded to a token; a CLS token % (blue) is prepended and positions added, then a Transformer encoder classifies from CLS. \begin{tikzpicture}[>=stealth, font=\scriptsize, px/.style={draw, black, minimum size=5mm, inner sep=0pt}, tok/.style={draw, minimum width=7mm, minimum height=5mm, align=center, fill=black!8}, cls/.style={draw=acc, text=acc, thick, minimum width=7mm, minimum height=5mm, align=center, fill=acc!15}, enc/.style={draw=acc, text=acc, thick, minimum width=46mm, minimum height=8mm, align=center}] \definecolor{acc}{HTML}{2348F2} % image grid \foreach \r in {0,1,2} \foreach \c in {0,1,2} \node[px] at (\c*0.55, 1.9-\r*0.55) {}; \node[anchor=north, font=\footnotesize] at (0.55,0.05) {\texttt{image patches}}; % arrow to embedding \draw[->, thick] (1.9,1.35) -- (2.9,1.35) node[midway, above, font=\footnotesize] {\texttt{embed}}; % token row \node[cls] (c) at (3.7,1.35) {\texttt{CLS}}; \node[tok] (t1) at (4.55,1.35) {$z_1$}; \node[tok] (t2) at (5.4,1.35) {$z_2$}; \node[tok] (t3) at (6.25,1.35) {...}; \node[tok] (t4) at (7.1,1.35) {$z_N$}; % position embeddings (added) \foreach \x in {3.7,4.55,5.4,6.25,7.1} \draw[->, black, thick] (\x,0.55) -- (\x,1.05) ; \node[anchor=north, font=\footnotesize] at (5.4,0.55) {\texttt{+ position embeddings}}; % encoder \node[enc] (enc) at (5.4,3.0) {\texttt{Transformer encoder}}; \foreach \x in {3.7,4.55,5.4,6.25,7.1} \draw[->, thick] (\x,1.6) -- (\x,2.6); % readout \draw[->, acc, thick] (c) .. controls (2.2,2.2) and (2.2,3.9) .. (4.4,3.9) node[pos=0.42, left, text=acc, font=\footnotesize] {\texttt{CLS out}}; \node[draw=acc, text=acc, thick, minimum width=20mm, minimum height=6mm] (head) at (5.4,3.9) {\texttt{class head}}; \draw[->, thick] (head) -- (6.6,3.9) node[right, font=\footnotesize] {\texttt{label}}; \end{tikzpicture} $$ **Worked example: patch geometry.** Take the base ViT configuration on a standard input: $H = W = 224$, $C = 3$ color channels, patch size $P = 16$, model width $d = 768$. The sequence length is $$ N = \frac{HW}{P^2} = \frac{224 \times 224}{16 \times 16} = \frac{50{,}176}{256} = 196 $$ patches, so with the prepended class token the encoder sees $197$ tokens. Each patch flattens to $P^2 C = 16 \times 16 \times 3 = 768$ raw values, which the patch-embedding matrix $E \in \mathbb{R}^{768 \times 768}$ maps to the model width — here the input and output widths coincide, but only by coincidence of this configuration. That embedding matrix alone holds $768 \times 768 \approx 590\text{k}$ weights, and the learned position table adds $197 \times 768 \approx 151\text{k}$ more. Halving the patch to $P = 8$ quarters nothing about the weights but _quadruples_ the token count to $N = 784$, and since attention is $O(N^2)$, that is a $16\times$ jump in attention compute. Patch size is the single knob that trades spatial resolution against sequence length, and through the $N^2$ term it is the dominant cost lever in a ViT. A convolution hard-wires **locality** and **translation equivariance**: a small kernel slides over the image, so nearby pixels interact first and the same filter applies everywhere. ViT bakes in none of this. Self-attention is global from layer one and the patch order is known only through learned position embeddings, so the network must _learn_ that nearby patches are related rather than assume it. This weaker inductive bias is the central trade-off. | Property | CNN | Vision Transformer | | --- | --- | --- | | Inductive bias | strong: locality, translation equivariance | weak: learned from data | | Receptive field | grows with depth | global at every layer | | Data efficiency | trains well on mid-size sets | needs large-scale pretraining | | Compute vs. resolution | $O(HW)$ per layer | $O(N^2)$ in patch count | | Long-range mixing | indirect (deep/dilated) | direct (any patch to any patch) | The cost of dropping the prior is data. On mid-size datasets a CNN of comparable size wins, because its built-in assumptions substitute for examples. Pretrained on hundreds of millions of images, ViT matches or beats CNNs: with enough data the learned bias overtakes the hand-coded one. Scale, again, is the lever.[^chollet-vit] ## The GPT family and decoder-only LLMs The decoder-only branch is the one that scaled into large language models. Strip the cross-attention from a decoder and you have a pure autoregressive model: stacked masked-self-attention layers trained on a single objective, predicting the next token. > **Definition (Causal language modeling).** Factor the joint probability of a > token sequence by the chain rule and train to maximize its log-likelihood: > $$ > p(x) = \prod_{t=1}^{n} p(x_t \mid x_{ \qquad > \mathcal{L} = -\sum_{t=1}^{n} \log p_\theta(x_t \mid x_{ $$ > Causal masking lets one forward pass score every factor in parallel; the same > network samples text by feeding its own output back as the next input. The striking fact about the **GPT** line is that the architecture barely changed across generations. GPT-1, GPT-2, GPT-3, and GPT-4 are the _same_ decoder-only Transformer scaled up: more layers, wider $d$, more heads, and vastly more data and compute. The qualitative jumps in capability came from scale, not from a new mechanism.[^chollet-gpt] | Model | Parameters (order) | Headline change | | --- | --- | --- | | GPT-1 | $10^8$ | decoder-only LM + supervised fine-tuning | | GPT-2 | $10^9$ | zero-shot tasks from scale alone | | GPT-3 | $10^{11}$ | in-context / few-shot learning | | GPT-4 | undisclosed | multimodal input, stronger reasoning | **Worked example: counting a GPT's parameters.** The per-layer count from the previous lesson was $12d^2$ (four $d\times d$ attention projections plus a $4d$-wide FFN). GPT-2 small uses $L = 12$ layers, width $d = 768$, and vocabulary $|V| = 50{,}257$. The blocks hold $$ L \cdot 12 d^2 = 12 \times 12 \times 768^2 \approx 12 \times 12 \times 590\text{k} \approx 85\text{M} $$ weights. The token embedding is $|V| \cdot d = 50{,}257 \times 768 \approx 38.6\text{M}$, and GPT-2 _ties_ the output projection to this same matrix, so it is not counted twice. The learned position table ($1024 \times 768 \approx 0.8\text{M}$) and the LayerNorm parameters are negligible. Summing $85\text{M} + 38.6\text{M} \approx 124\text{M}$ recovers the published 124M-parameter count almost exactly. Two facts fall out of the arithmetic: at this modest width the embedding table is nearly a third of the model, and scaling $d$ grows the blocks _quadratically_ but the embedding only _linearly_ — which is why the block term dominates completely by the time you reach GPT-3's $d = 12{,}288$, where $12 d^2$ per layer is already $\approx 1.8\text{B}$ weights. Two behaviors emerge at the top of this trajectory and postdate Goodfellow's 2016 text entirely. - **In-context learning.** A sufficiently large model performs a new task from a handful of examples placed _in the prompt_, with **no weight updates**. The conditioning examples steer the forward pass; the gradient never runs. Few-shot prompting is inference, not training. - **Instruction tuning and RLHF.** A pretrained LM predicts likely text, not helpful text. Supervised fine-tuning on instruction–response pairs, followed by **reinforcement learning from human feedback** (optimizing a reward model fit to human preference comparisons), aligns the raw next-token predictor with what a user actually wants. The three architectural families map cleanly onto three pretraining objectives and three uses. | Family | Example | Objective | Built for | | --- | --- | --- | --- | | Encoder-only | BERT | masked LM (predict held-out tokens) | understanding: classification, retrieval, tagging | | Decoder-only | GPT | autoregressive LM (next token) | generation: open-ended text, chat, code | | Encoder–decoder | T5 | span-corruption seq2seq | transduction: translation, summarization | ## Other derivatives The Transformer turned out to be a substrate, not a single model. Three directions matter here. - **T5 (text-to-text).** Casts _every_ NLP task — translation, classification, summarization, question answering — as mapping an input string to an output string, so one encoder–decoder with one objective covers them all. Task identity lives in a text prefix (`"translate English to German: …"`), not in the architecture. - **Multimodal (CLIP-style).** A vision encoder (often a ViT) and a text encoder are trained so that matching image–caption pairs land near each other in a shared embedding space, via a contrastive objective. This grounds language in images and underpins text-conditioned generation and zero-shot classification. - **Transformer everywhere.** Audio, protein sequences, time series, reinforcement learning trajectories, and code all became Transformer problems once they were expressed as token sequences. The architecture's indifference to _what_ a token is, shown already by ViT, is why it spread across so many fields. ## Scaling Why does simply making the model bigger work so reliably? Because the test loss follows a smooth, predictable **scaling law** as a function of model size, dataset size, and compute. These empirical laws postdate Goodfellow's 2016 text and are the quantitative reason scale became the dominant research lever.[^chollet-scaling] > **Definition (Scaling laws).** Over many orders of magnitude, the test > cross-entropy loss falls as a power law in each of parameters $N$, dataset size > $D$, and compute $C$, when the other two are not bottlenecks: > $$ > L(N) \approx L_\infty + \parens{\frac{N_c}{N}}^{\alpha_N}, > \qquad > L(D) \approx L_\infty + \parens{\frac{D_c}{D}}^{\alpha_D}, > \qquad > L(C) \approx \parens{\frac{C_c}{C}}^{\alpha_C}. > $$ > $L_\infty$ is the irreducible loss (the entropy of the data); the exponents > $\alpha$ are small positive constants, so each tenfold increase yields a fixed, > diminishing decrement of loss. A power law is a straight line on log–log axes. Plotting loss against compute, the points fall on a line of slope $-\alpha_C$ across many decades: the predictability is what lets large training runs be planned in advance. $$ % caption: Scaling law. On log–log axes the test loss falls along a straight line of slope % $-\alpha_C$ in compute (blue), flattening toward the irreducible loss $L_\infty$ (black). \begin{tikzpicture}[>=stealth, font=\footnotesize, x=1.0cm, y=1.0cm] \definecolor{acc}{HTML}{2348F2} \draw[->, thick] (0,0) -- (7.0,0) node[right, font=\scriptsize] {log compute C}; \draw[->, thick] (0,0) -- (0,4.6) node[above, font=\scriptsize] {log loss}; % power-law line \draw[acc, very thick] (0.4,4.0) -- (6.2,1.0) node[pos=0.62, above=4pt, sloped, text=acc, font=\scriptsize] {power law}; % irreducible-loss asymptote \draw[black, dashed, thick] (0.4,0.7) -- (6.0,0.7) node[right, text=black, font=\scriptsize] {irreducible loss}; % a couple of run markers on the line \fill[acc] (1.5,3.43) circle (1.8pt); \fill[acc] (3.4,2.45) circle (1.8pt); \fill[acc] (5.3,1.47) circle (1.8pt); \end{tikzpicture} $$ Given a fixed compute budget $C$, the laws also dictate _how to spend it_: parameters and data should grow together. The **Chinchilla** result corrects an earlier bias toward huge models trained on too little data — many large models were badly _under-trained_. > **Theorem (Compute-optimal balance).** For a fixed compute budget $C$, the loss > is minimized when parameters $N$ and training tokens $D$ are scaled in roughly > equal proportion, $N \propto C^{1/2}$ and $D \propto C^{1/2}$, so that > $D / N$ stays near a constant of order $20$ tokens per parameter. > **Proof.** With $C \approx 6 N D$ (the forward/backward FLOP count for a > Transformer), substitute $D = C / 6N$ into $L(N, D) = L_\infty + (N_c/N)^{\alpha_N} + > (D_c/D)^{\alpha_D}$ and minimize over $N$. Setting $\partial L / \partial N = 0$ > gives a unique optimum at which both terms shrink at matched rates, yielding > $N \propto C^{a}$, $D \propto C^{b}$ with $a, b \approx 1/2$ when > $\alpha_N \approx \alpha_D$. Holding $C$ fixed while pushing $N$ past this point > starves the model of data and raises the loss. $\qed$ **Worked example: reading the Chinchilla ratio.** GPT-3 has $N = 175\text{B}$ parameters and was trained on $D = 300\text{B}$ tokens, a ratio of only $D/N \approx 1.7$ tokens per parameter — far below the compute-optimal $\approx 20$. The Chinchilla model reallocated the _same_ compute budget the other way: $N = 70\text{B}$ parameters (four times smaller) trained on $D = 1.4\text{T}$ tokens (four times more data), giving $D/N \approx 20$. With $C \approx 6ND$, both runs cost $6 \times 175 \times 300 \approx 3.2 \times 10^{23}$ FLOPs versus $6 \times 70 \times 1400 \approx 5.9 \times 10^{23}$ — the same order of magnitude — yet the smaller model trained on more data wins on loss. The lesson is blunt: at a fixed budget, a model can be too big, because every parameter spent past the optimum is a token of data not seen. Smoothness at the level of loss coexists with sharp jumps at the level of behavior. Some capabilities are **emergent**: absent at small scale and present, sometimes abruptly, past a threshold. > **Definition (Emergent capability).** A task ability that is near-random for > small models and rises sharply once model scale crosses some threshold, so it is > not visible by extrapolating small-model performance even though the underlying > loss curve is smooth. | Lever | Symbol | Effect on loss | Practical note | | --- | --- | --- | --- | | Parameters | $N$ | $\propto N^{-\alpha_N}$ | width/depth/heads; two-thirds in the FFN | | Data | $D$ | $\propto D^{-\alpha_D}$ | balance with $N$ (Chinchilla, $\sim 20D/N$) | | Compute | $C \approx 6ND$ | $\propto C^{-\alpha_C}$ | the planned budget; sets $N$ and $D$ jointly | Architecture has been stable while scale moved by many orders of magnitude. The lesson of the last decade is that the same decoder-only Transformer, made larger and trained on more data, keeps improving along a predictable curve, which is why scaling, not redesign, has been the dominant lever.[^chollet-scaling] ## The primary sources Goodfellow, Chollet, and Stevens predate most of what this lesson covers — the Transformer itself postdates Goodfellow (2016), and ViT, the GPT scaling story, and the scaling laws postdate all three. The canonical primary sources fill the gap. - **The original Transformer.** Vaswani et al., "Attention Is All You Need" (NeurIPS 2017), is the source for every equation in the previous lesson: scaled dot-product attention, multi-head projection, sinusoidal positional encoding, and the post-norm encoder–decoder. Everything here is a descendant of that one paper. - **Vision Transformer.** Dosovitskiy et al., "An Image Is Worth $16\times16$ Words" (ICLR 2021), introduced ViT and made the data-efficiency trade-off concrete: below roughly ImageNet scale a ResNet wins, but pretrained on the $300\text{M}$-image JFT set, ViT overtakes it. The $P = 16$ patch and the `[CLS]` read-out in the worked example above are theirs. - **The GPT line.** Radford et al.'s GPT-2 report ("Language Models Are Unsupervised Multitask Learners," 2019) established zero-shot ability from scale; Brown et al., "Language Models Are Few-Shot Learners" (NeurIPS 2020), introduced GPT-3 and _in-context learning_ — a $175\text{B}$-parameter model solving new tasks from prompt examples alone, no gradient step. Ouyang et al., "Training Language Models to Follow Instructions with Human Feedback" (NeurIPS 2022), is the RLHF recipe (InstructGPT) that turns a next-token predictor into an assistant. - **Scaling laws and Chinchilla.** Kaplan et al., "Scaling Laws for Neural Language Models" (2020), fit the power laws $L(N), L(D), L(C)$. Hoffmann et al., "Training Compute-Optimal Large Language Models" (NeurIPS 2022) — the Chinchilla paper — corrected the parameter/data balance to $\approx 20$ tokens per parameter, the ratio traced above. Wei et al., "Emergent Abilities of Large Language Models" (TMLR 2022), catalogued the sharp capability jumps. - **Efficient attention.** The $O(n^2)$ cost drove two lines of work: rotary position embeddings (Su et al., RoFormer, 2021), which replace the additive table with a query/key rotation and are now standard in decoder-only LLMs, and FlashAttention (Dao et al., NeurIPS 2022), an IO-aware exact-attention kernel that never materializes the full $n\times n$ matrix, cutting memory from $O(n^2)$ to $O(n)$ without changing the math. The pattern across these sources: the architecture froze while the surrounding practice — scale, position encoding, attention kernels, and alignment — kept changing. ## Takeaways - A Transformer consumes a **sequence of vectors** and is indifferent to what they represent: the **Vision Transformer** tokenizes an image into $P \times P$ patches, embeds each linearly, prepends a `[CLS]` token, and classifies with a plain encoder — no convolution. - ViT trades the CNN's **locality prior** for data: it must learn spatial relations from scratch, so it needs large-scale pretraining before it matches or beats a CNN. Patch size sets the sequence length $N = HW/P^2$ and, through the $O(N^2)$ attention term, dominates the compute. - **Decoder-only LLMs** (the GPT line) are causal language models trained on $p(x) = \prod_t p(x_t \mid x_{