Transformers in Practice

This builds on The Transformer Architecture, which assembled the encoder–decoder, worked through causal masking, and accounted for where the parameters ($12d^2$ per layer, two-thirds in the FFN) and the compute ($O(n^{2}d)$ attention) live. That lesson stayed inside language. This one follows the same architecture out of it: the encoder classifies images, the decoder scales into large language models, and the design turns out to be a substrate rather than a single model. We work two accounting examples by hand — ViT patch geometry and a GPT parameter budget — and finish on the scaling laws that make growing the model pay off so predictably.

The Vision Transformer

Nothing in the architecture is specific to language. A Transformer consumes a sequence of vectors; supply image patches instead of word embeddings and the same encoder classifies pictures. The Vision Transformer (ViT) does exactly this, discarding the convolution entirely.¹

The sequence $z_0$ runs through a standard encoder stack. Classification reads the final state of the class token alone, $\hat{y}=\mathrm{softmax}(z_{\text{cls}}^{(L)}W)$: the [CLS] token aggregates evidence from every patch through self-attention and serves as the pooled image representation.

Worked example: patch geometry. Take the base ViT configuration on a standard input: $H=W=224$, $C=3$ color channels, patch size $P=16$, model width $d=768$. The sequence length is

patches, so with the prepended class token the encoder sees $197$ tokens. Each patch flattens to $P^{2}C=16\times 16\times 3=768$ raw values, which the patch-embedding matrix $E\in {\mathbb{R}}^{768\times 768}$ maps to the model width — here the input and output widths coincide, but only by coincidence of this configuration. That embedding matrix alone holds $768\times 768\approx 590\text{k}$ weights, and the learned position table adds $197\times 768\approx 151\text{k}$ more. Halving the patch to $P=8$ quarters nothing about the weights but quadruples the token count to $N=784$, and since attention is $O(N^2)$, that is a $16\times$ jump in attention compute. Patch size is the single knob that trades spatial resolution against sequence length, and through the $N^2$ term it is the dominant cost lever in a ViT.

A convolution hard-wires locality and translation equivariance: a small kernel slides over the image, so nearby pixels interact first and the same filter applies everywhere. ViT bakes in none of this. Self-attention is global from layer one and the patch order is known only through learned position embeddings, so the network must learn that nearby patches are related rather than assume it. This weaker inductive bias is the central trade-off.

The cost of dropping the prior is data. On mid-size datasets a CNN of comparable size wins, because its built-in assumptions substitute for examples. Pretrained on hundreds of millions of images, ViT matches or beats CNNs: with enough data the learned bias overtakes the hand-coded one. Scale, again, is the lever.¹

The GPT family and decoder-only LLMs

The decoder-only branch is the one that scaled into large language models. Strip the cross-attention from a decoder and you have a pure autoregressive model: stacked masked-self-attention layers trained on a single objective, predicting the next token.

The striking fact about the GPT line is that the architecture barely changed across generations. GPT-1, GPT-2, GPT-3, and GPT-4 are the same decoder-only Transformer scaled up: more layers, wider $d$, more heads, and vastly more data and compute. The qualitative jumps in capability came from scale, not from a new mechanism.²

Worked example: counting a GPT's parameters. The per-layer count from the previous lesson was $12d^2$ (four $d\times d$ attention projections plus a $4d$-wide FFN). GPT-2 small uses $L=12$ layers, width $d=768$, and vocabulary $|V|=50,257$. The blocks hold

weights. The token embedding is $|V|\cdot d=50,257\times 768\approx 38.6\text{M}$, and GPT-2 ties the output projection to this same matrix, so it is not counted twice. The learned position table ($1024\times 768\approx 0.8\text{M}$) and the LayerNorm parameters are negligible. Summing $85\text{M}+38.6\text{M}\approx 124\text{M}$ recovers the published 124M-parameter count almost exactly. Two facts fall out of the arithmetic: at this modest width the embedding table is nearly a third of the model, and scaling $d$ grows the blocks quadratically but the embedding only linearly — which is why the block term dominates completely by the time you reach GPT-3's $d=12,288$, where $12d^2$ per layer is already $\approx 1.8\text{B}$ weights.

Two behaviors emerge at the top of this trajectory and postdate Goodfellow's 2016 text entirely.

• In-context learning. A sufficiently large model performs a new task from a handful of examples placed in the prompt, with no weight updates. The conditioning examples steer the forward pass; the gradient never runs. Few-shot prompting is inference, not training.
• Instruction tuning and RLHF. A pretrained LM predicts likely text, not helpful text. Supervised fine-tuning on instruction–response pairs, followed by reinforcement learning from human feedback (optimizing a reward model fit to human preference comparisons), aligns the raw next-token predictor with what a user actually wants.

The three architectural families map cleanly onto three pretraining objectives and three uses.

Other derivatives

The Transformer turned out to be a substrate, not a single model. Three directions matter here.

• T5 (text-to-text). Casts every NLP task — translation, classification, summarization, question answering — as mapping an input string to an output string, so one encoder–decoder with one objective covers them all. Task identity lives in a text prefix ("translate English to German: …"), not in the architecture.
• Multimodal (CLIP-style). A vision encoder (often a ViT) and a text encoder are trained so that matching image–caption pairs land near each other in a shared embedding space, via a contrastive objective. This grounds language in images and underpins text-conditioned generation and zero-shot classification.
• Transformer everywhere. Audio, protein sequences, time series, reinforcement learning trajectories, and code all became Transformer problems once they were expressed as token sequences. The architecture's indifference to what a token is, shown already by ViT, is why it spread across so many fields.

Scaling

Why does simply making the model bigger work so reliably? Because the test loss follows a smooth, predictable scaling law as a function of model size, dataset size, and compute. These empirical laws postdate Goodfellow's 2016 text and are the quantitative reason scale became the dominant research lever.³

A power law is a straight line on log–log axes. Plotting loss against compute, the points fall on a line of slope $-{\alpha }_C$ across many decades: the predictability is what lets large training runs be planned in advance.

Given a fixed compute budget $C$, the laws also dictate how to spend it: parameters and data should grow together. The Chinchilla result corrects an earlier bias toward huge models trained on too little data — many large models were badly under-trained.

Worked example: reading the Chinchilla ratio. GPT-3 has $N=175\text{B}$ parameters and was trained on $D=300\text{B}$ tokens, a ratio of only $D/N\approx 1.7$ tokens per parameter — far below the compute-optimal $\approx 20$. The Chinchilla model reallocated the same compute budget the other way: $N=70\text{B}$ parameters (four times smaller) trained on $D=1.4\text{T}$ tokens (four times more data), giving $D/N\approx 20$. With $C\approx 6ND$, both runs cost $6\times 175\times 300\approx 3.2\times 10^{23}$ FLOPs versus $6\times 70\times 1400\approx 5.9\times 10^{23}$ — the same order of magnitude — yet the smaller model trained on more data wins on loss. The lesson is blunt: at a fixed budget, a model can be too big, because every parameter spent past the optimum is a token of data not seen.

Smoothness at the level of loss coexists with sharp jumps at the level of behavior. Some capabilities are emergent: absent at small scale and present, sometimes abruptly, past a threshold.

Architecture has been stable while scale moved by many orders of magnitude. The lesson of the last decade is that the same decoder-only Transformer, made larger and trained on more data, keeps improving along a predictable curve, which is why scaling, not redesign, has been the dominant lever.³

The primary sources

Goodfellow, Chollet, and Stevens predate most of what this lesson covers — the Transformer itself postdates Goodfellow (2016), and ViT, the GPT scaling story, and the scaling laws postdate all three. The canonical primary sources fill the gap.

• The original Transformer. Vaswani et al., Attention Is All You Need (NeurIPS 2017), is the source for every equation in the previous lesson: scaled dot-product attention, multi-head projection, sinusoidal positional encoding, and the post-norm encoder–decoder. Everything here is a descendant of that one paper.
• Vision Transformer. Dosovitskiy et al., An Image Is Worth $16\times 16$ Words (ICLR 2021), introduced ViT and made the data-efficiency trade-off concrete: below roughly ImageNet scale a ResNet wins, but pretrained on the $300\text{M}$-image JFT set, ViT overtakes it. The $P=16$ patch and the [CLS] read-out in the worked example above are theirs.
• The GPT line. Radford et al.'s GPT-2 report (Language Models Are Unsupervised Multitask Learners, 2019) established zero-shot ability from scale; Brown et al., Language Models Are Few-Shot Learners (NeurIPS 2020), introduced GPT-3 and in-context learning — a $175\text{B}$-parameter model solving new tasks from prompt examples alone, no gradient step. Ouyang et al., Training Language Models to Follow Instructions with Human Feedback (NeurIPS 2022), is the RLHF recipe (InstructGPT) that turns a next-token predictor into an assistant.
• Scaling laws and Chinchilla. Kaplan et al., Scaling Laws for Neural Language Models (2020), fit the power laws $L(N),L(D),L(C)$. Hoffmann et al., Training Compute-Optimal Large Language Models (NeurIPS 2022) — the Chinchilla paper — corrected the parameter/data balance to $\approx 20$ tokens per parameter, the ratio traced above. Wei et al., Emergent Abilities of Large Language Models (TMLR 2022), catalogued the sharp capability jumps.
• Efficient attention. The $O(n^2)$ cost drove two lines of work: rotary position embeddings (Su et al., RoFormer, 2021), which replace the additive table with a query/key rotation and are now standard in decoder-only LLMs, and FlashAttention (Dao et al., NeurIPS 2022), an IO-aware exact-attention kernel that never materializes the full $n\times n$ matrix, cutting memory from $O(n^2)$ to $O(n)$ without changing the math.

The pattern across these sources: the architecture froze while the surrounding practice — scale, position encoding, attention kernels, and alignment — kept changing.

Takeaways

• A Transformer consumes a sequence of vectors and is indifferent to what they represent: the Vision Transformer tokenizes an image into $P\times P$ patches, embeds each linearly, prepends a [CLS] token, and classifies with a plain encoder — no convolution.
• ViT trades the CNN's locality prior for data: it must learn spatial relations from scratch, so it needs large-scale pretraining before it matches or beats a CNN. Patch size sets the sequence length $N=HW/P^2$ and, through the $O(N^2)$ attention term, dominates the compute.
• Decoder-only LLMs (the GPT line) are causal language models trained on $p(x)={\prod }_{t}p(x_t\mid x_{<t})$; GPT-1 through GPT-4 are the same architecture scaled, with in-context learning and RLHF layered on top.
• The parameter arithmetic ($L\cdot 12d^2$ blocks plus a $|V|\cdot d$ embedding) recovers GPT-2 small's 124M count exactly, and shows why the embedding table matters at small width but vanishes against the quadratic block term at scale.
• The Transformer is a substrate: T5 casts every task as text-to-text, CLIP-style models align vision and language contrastively, and the same block handles audio, proteins, and code.
• Scaling laws make test loss a power law in parameters, data, and compute; Chinchilla balances $N$ and $D$ at $\approx 20$ tokens/param (GPT-3 was badly under-trained at $1.7$), and some capabilities emerge sharply at scale — so scale, not redesign, is the dominant lever.