--- title: The Transformer Architecture module: Architectures moduleNumber: 5 lessonNumber: 6 order: 506 summary: > The Transformer is the architecture built around the attention mechanism. This first part assembles the full encoder–decoder of "Attention Is All You Need" — embeddings and positional encoding, stacked self-attention and feed-forward sublayers wrapped in residual connections and LayerNorm, masked decoding and cross-attention — works through causal masking and the three modern families (encoder-only, decoder-only, encoder–decoder), and accounts for where the parameters and the $O(n^2)$ compute actually go. topics: [Architectures] sources: - book: Goodfellow ref: "§10.11 — Explicit Memory & Attention (the Transformer postdates the 2016 text)" - book: Chollet ref: "Ch. 11 — Deep Learning for Text; The Transformer Architecture" --- The previous lesson built the [attention mechanism](/deep-learning/architectures/attention-and-transformers): scaled dot-product attention, multi-head projection, and the positional encoding that restores order to a permutation-equivariant operation. That is the _operation_. This lesson assembles the _architecture_: how those operations stack into the encoder–decoder of "Attention Is All You Need," how the stack splits into the three families that dominate modern practice, and where its parameters and its quadratic cost actually live.[^chollet-transformer] > **Definition (Transformer).** A sequence model built entirely from attention > and position-wise feed-forward layers, with no recurrence or convolution. > Token embeddings plus positional encodings feed a stack of $N$ identical > layers; each layer interleaves multi-head attention with a position-wise MLP, > and wraps every sublayer in a residual connection followed by layer > normalization. A linear projection and softmax read the top of the stack out > to a distribution over the vocabulary. ## The sublayer wrapper The atom of the architecture is not a layer but a **sublayer wrapper**: a residual skip around a function $f$, followed by [layer normalization](/deep-learning/regularization/normalization). Every attention block and every feed-forward block in the entire model is wrapped this way, so we fix the convention before stacking anything. > **Definition (Position-wise feed-forward network).** A two-layer MLP applied > independently and identically to every position, with an expansion factor of > typically $4$: > $$ > \FFN(x) = \parens{\ReLU(x W_1 + b_1)} W_2 + b_2, > \qquad W_1 \in \mathbb{R}^{d \times d_{\text{ff}}},\; W_2 \in \mathbb{R}^{d_{\text{ff}} \times d}, > $$ > with $d_{\text{ff}} = 4d$ the standard inner width. Attention mixes > _across_ positions; the FFN is the only nonlinearity acting _within_ a > position. Because its weights are shared across positions, it amounts to a > $1 \times 1$ convolution over the sequence. The expansion is not arbitrary. The FFN takes each token's $d$-vector, projects it _up_ into a wider $4d$ space, applies a pointwise nonlinearity there, and projects _back down_ to $d$. The wide hidden layer is where the per-token computation happens: it gives the nonlinearity enough room to compute many independent features of the token before compressing the result. The original paper used $\ReLU$; most modern stacks swap in $\GELU$ (a smooth gate $x \cdot \Phi(x)$, with $\Phi$ the standard-normal CDF) or a gated variant like $\SwiGLU$, which lowers loss at fixed width for reasons that remain partly empirical. Track the shapes explicitly. A batch of $B$ sequences of length $n$ enters as a tensor of shape $(B, n, d)$. Every operation below preserves the batch and sequence axes and acts only on the last (feature) axis: $$ \underbrace{(B, n, d)}_{\text{input}} \;\xrightarrow{\,W_1 \in \mathbb{R}^{d \times 4d}\,}\; \underbrace{(B, n, 4d)}_{\text{hidden}} \;\xrightarrow{\,\ReLU\,}\; (B, n, 4d) \;\xrightarrow{\,W_2 \in \mathbb{R}^{4d \times d}\,}\; \underbrace{(B, n, d)}_{\text{output}} . $$ The same $W_1, W_2$ apply at every one of the $B \times n$ positions — the FFN carries no positional index and no cross-token interaction, which is what makes it a position-wise map. **Layer normalization** is the other half of the wrapper, so its statistics matter for what follows. Given a single token vector $x \in \mathbb{R}^{d}$, LayerNorm computes the mean and variance _over the $d$ feature entries of that one token_ — not over the batch, not over the sequence — and standardizes: $$ \mu = \frac{1}{d}\sum_{k=1}^{d} x_k, \qquad \sigma^2 = \frac{1}{d}\sum_{k=1}^{d} (x_k - \mu)^2, \qquad \LN(x) = \gamma \odot \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta, $$ with learned per-feature scale $\gamma \in \mathbb{R}^{d}$ and shift $\beta \in \mathbb{R}^{d}$ (initialized to $1$ and $0$), and $\epsilon \approx 10^{-5}$ for numerical safety. Because the reduction is per-token, LayerNorm is independent of batch size and of the other positions — it behaves identically at training and inference and needs no running statistics — the reason it replaced BatchNorm for variable-length sequence models.[^gf-norm] The single addition it contributes is $2d$ parameters ($\gamma$ and $\beta$) per LayerNorm. The two wrapper styles differ only in **where** the normalization sits relative to the residual sum. | Style | Sublayer formula | Residual path | | --- | --- | --- | | Post-norm (original) | $\LN\!\parens{x + f(x)}$ | passes through LN at every layer | | Pre-norm (modern) | $x + f\!\parens{\LN(x)}$ | clean identity, never normalized | The distinction looks cosmetic but governs whether a deep stack trains at all.[^gf-norm] > **Theorem (Pre-norm residual identity).** Under pre-norm wrapping, the output > of an $N$-layer stack expands as > $$ > z_N = z_0 + \sum_{\ell=1}^{N} f_\ell\!\parens{\LN(z_{\ell-1})}, > $$ > so the input $z_0$ reaches the top through an unbroken identity path, and the > gradient $\partial z_N / \partial z_0$ contains a term equal to $1$ > independent of depth. > **Proof.** Pre-norm defines $z_\ell = z_{\ell-1} + f_\ell(\LN(z_{\ell-1}))$. > Unrolling the recurrence from $\ell = 1$ to $N$ telescopes the residual sums, > giving $z_N = z_0 + \sum_{\ell} f_\ell(\LN(z_{\ell-1}))$. > Differentiating, $\partial z_N/\partial z_0 = I + \sum_\ell \partial > f_\ell/\partial z_0$; the leading $I$ is a depth-independent identity, so no > product of Jacobians can drive the gradient to zero on the skip path. $\qed$ Post-norm breaks that identity: each residual sum is squeezed back through a LayerNorm, so the gradient path is a _product_ of $N$ normalization Jacobians and can vanish or explode with depth, which is why post-norm Transformers need a learning-rate warmup and pre-norm ones largely do not. $$ % caption: Post-norm (left) routes every skip through LayerNorm; pre-norm (right) normalizes only % the branch input, leaving a clean identity highway (blue) up the stack. \begin{tikzpicture}[>=stealth, font=\footnotesize, blk/.style={draw, minimum width=24mm, minimum height=8mm, align=center}, op/.style={draw, circle, inner sep=0pt, minimum size=6mm}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} % ---- post-norm ---- \node (pin) at (0,0) {\texttt{input x}}; \node[blk] (pf) at (0,1.5) {\texttt{sublayer f}}; \node[op] (pa) at (0,3.0) {$+$}; \node[blk, draw=green, text=green] (pn) at (0,4.5) {\texttt{LayerNorm}}; \node (pout) at (0,6.0) {\texttt{output}}; \draw[->, thick] (pin) -- (pf); \draw[->, thick] (pf) -- (pa); \draw[->, thick] (pa) -- (pn); \draw[->, thick] (pn) -- (pout); \draw[->, acc, thick] (1.3,0.2) .. controls (2.4,1.0) and (2.4,2.2) .. (1.3,3.0) node[pos=0.5, right, text=acc] {\texttt{skip}}; \node[font=\footnotesize] at (0,-0.8) {\texttt{post-norm}}; % ---- pre-norm ---- \begin{scope}[xshift=6.6cm] \node (qin) at (0,0) {\texttt{input x}}; \node[blk, draw=green, text=green] (qn) at (0,1.5) {\texttt{LayerNorm}}; \node[blk] (qf) at (0,3.0) {\texttt{sublayer f}}; \node[op] (qa) at (0,4.5) {$+$}; \node (qout) at (0,6.0) {\texttt{output}}; \draw[->, thick] (qin) -- (qn); \draw[->, thick] (qn) -- (qf); \draw[->, thick] (qf) -- (qa); \draw[->, thick] (qa) -- (qout); \draw[->, acc, thick] (1.3,0.2) .. controls (2.6,1.6) and (2.6,3.0) .. (1.3,4.5) node[pos=0.5, right, text=acc] {\texttt{identity}}; \node[font=\footnotesize] at (0,-0.8) {\texttt{pre-norm}}; \end{scope} \end{tikzpicture} $$ ## Positional encoding Attention is **permutation-equivariant**: shuffle the input tokens and the output set is the same, merely reordered. That is fatal for language, where "dog bites man" and "man bites dog" are different sentences built from identical tokens. The attention operator has no notion of position, so order has to be injected into the inputs before the first layer. The original Transformer adds a fixed **sinusoidal positional encoding** to the token embeddings. > **Definition (Sinusoidal positional encoding).** For position $\text{pos} \in > \{0, 1, \dots, n-1\}$ and feature index $i \in \{0, \dots, d/2 - 1\}$, the > encoding fills even and odd feature slots with a sine/cosine pair whose > frequency decreases geometrically across the width: > $$ > \PE(\text{pos}, 2i) = \sin\!\parens{\frac{\text{pos}}{10000^{\,2i/d}}}, > \qquad > \PE(\text{pos}, 2i+1) = \cos\!\parens{\frac{\text{pos}}{10000^{\,2i/d}}}. > $$ > The vector $\PE(\text{pos}) \in \mathbb{R}^{d}$ is added to the > token embedding at that position, giving the layer-$0$ input $z_0 = \text{Embed}(x) + > \PE$. The $10000^{2i/d}$ denominator sets the **wavelength** of each feature: the lowest feature ($i = 0$) oscillates with period $2\pi$ tokens, and the highest ($i = d/2-1$) with period $2\pi \cdot 10000$ tokens. Low-frequency channels change slowly and encode coarse, long-range position; high-frequency channels distinguish adjacent tokens. Together they give each position a unique fingerprint across a geometric band of scales, the same multi-resolution idea a place-value numeral uses to name any integer with a fixed set of digits. $$ % caption: Sinusoidal positional encoding as a frequency band. Each feature slot $i$ carries a % sine/cosine of period $2\pi\cdot 10000^{2i/d}$: low-$i$ channels (top) oscillate fast and separate % adjacent tokens; high-$i$ channels (bottom) oscillate slowly and encode coarse position. \begin{tikzpicture}[>=stealth, font=\footnotesize, x=1cm, y=1cm] \definecolor{acc}{HTML}{2348F2} \draw[->, thick] (0,-3.0) -- (7.2,-3.0) node[below right, font=\footnotesize] {\texttt{position}}; % high frequency (small i) \draw[thick] plot[domain=0:6.6, samples=120] (\x, {0.45*sin(\x*260)}); \node[anchor=west, font=\footnotesize] at (6.7,0) {\texttt{i small (fast)}}; % mid frequency \draw[thick] plot[domain=0:6.6, samples=120] (\x, {-1.2+0.45*sin(\x*90)}); \node[anchor=west, font=\footnotesize] at (6.7,-1.2) {\texttt{i mid}}; % low frequency (large i) \draw[thick] plot[domain=0:6.6, samples=120] (\x, {-2.4+0.45*sin(\x*30)}); \node[anchor=west, font=\footnotesize] at (6.7,-2.4) {\texttt{i large (slow)}}; \end{tikzpicture} $$ Two properties make the sinusoidal choice more than a lookup table: - **Relative position is linear.** For any fixed offset $k$, the encoding at $\text{pos} + k$ is a fixed linear function (a rotation, from the angle-sum identities) of the encoding at $\text{pos}$. A single learned projection inside an attention head can therefore read _relative_ distance, not just absolute index, which is what a language model actually needs. - **Length extrapolation.** Because the formula is defined for every real $\text{pos}$, a model can be evaluated on sequences longer than any it saw in training without a new parameter — the encoding simply continues. The alternative is a **learned positional embedding**: a trainable table $E_{\text{pos}} \in \mathbb{R}^{n_{\max} \times d}$ with one row per position, added just like a word embedding. Learned embeddings match or slightly beat sinusoids on in-distribution data and are the common choice for encoder-only models (BERT) and ViT, but they cap the context at the table's row count $n_{\max}$ and cannot extrapolate past it. Modern decoder-only models mostly use neither and instead rotate the queries and keys inside attention (rotary embeddings, RoPE), which keeps the linear-relative property while removing the additive table entirely. ## One encoder layer An encoder layer is two wrapped sublayers in series: multi-head self-attention, then a position-wise FFN. With pre-norm wrapping and an input $x$, $$ a = x + \MHA\!\parens{\LN(x)}, \qquad y = a + \FFN\!\parens{\LN(a)}. $$ The self-attention is _unrestricted_: every position attends to every other, so each output position is a content-weighted summary of the whole input. This is the right behavior for an _encoder_: the entire sequence is available and there is no causality constraint. Every tensor in the block has the same shape $(B, n, d)$: the width $d$ is the invariant of the residual stream, so a layer's output plugs into the next layer's input without any reshaping. Trace one block for a concrete configuration $d = 512$, $h = 8$ heads (so head width $d_k = d/h = 64$), $d_{\text{ff}} = 2048$, input $(B, n, 512)$: | Step | Operation | Output shape | | --- | --- | --- | | 0 | input $x$ | $(B, n, 512)$ | | 1 | $\LN(x)$ (per-token, over the $512$ features) | $(B, n, 512)$ | | 2 | project to $Q, K, V$, split into $8$ heads | $(B, 8, n, 64)$ each | | 3 | scores $QK^\top/\sqrt{64}$, softmax over keys | $(B, 8, n, n)$ | | 4 | weighted values, concat heads | $(B, n, 512)$ | | 5 | output projection $W_O$, add residual $x +$ | $(B, n, 512)$ | | 6 | $\LN$, up-project to $d_{\text{ff}}$ | $(B, n, 2048)$ | | 7 | ReLU, down-project, add residual | $(B, n, 512)$ | The only place the shape swells is the $(B, 8, n, n)$ score tensor in step 3 and the $(B, n, 2048)$ FFN hidden in step 6 — the two costs, quadratic-in-$n$ and wide-FFN, that the accounting section makes precise. $$ % caption: Shapes through one pre-norm encoder block ($d=512$, $h=8$, $d_k=64$, $d_{ff}=2048$). The % width $d$ is the residual-stream invariant; only the attention scores ($n\times n$) and FFN hidden % ($4d$ wide) inflate before collapsing back to $(B,n,d)$. \begin{tikzpicture}[>=stealth, font=\footnotesize, bus/.style={draw=acc, text=acc, thick, minimum width=22mm, minimum height=7mm, align=center}, wide/.style={draw, minimum width=22mm, minimum height=7mm, align=center, fill=black!8}, sh/.style={font=\scriptsize, text=black}] \definecolor{acc}{HTML}{2348F2} \node[bus] (x) at (0,0) {\texttt{input}}; \node[wide] (sc) at (0,1.7) {\texttt{attn scores}}; \node[bus] (a) at (0,3.4) {\texttt{+ residual}}; \node[wide] (ff) at (0,5.1) {\texttt{FFN hidden}}; \node[bus] (y) at (0,6.8) {\texttt{+ residual}}; \draw[->, thick] (x) -- (sc); \draw[->, thick] (sc) -- (a); \draw[->, thick] (a) -- (ff); \draw[->, thick] (ff) -- (y); \node[sh, anchor=west] at (1.5,0) {\texttt{(B, n, 512)}}; \node[sh, anchor=west] at (1.5,1.7) {\texttt{(B, 8, n, n)}}; \node[sh, anchor=west] at (1.5,3.4) {\texttt{(B, n, 512)}}; \node[sh, anchor=west] at (1.5,5.1) {\texttt{(B, n, 2048)}}; \node[sh, anchor=west] at (1.5,6.8) {\texttt{(B, n, 512)}}; \end{tikzpicture} $$ $$ % caption: One encoder layer: multi-head self-attention then a position-wise FFN, each wrapped in % add-and-norm. The blue arcs are the residual connections around each sublayer. \begin{tikzpicture}[>=stealth, font=\footnotesize, blk/.style={draw, minimum width=44mm, minimum height=9mm, align=center}, an/.style={draw, minimum width=44mm, minimum height=7mm, align=center, fill=black!8}] \definecolor{acc}{HTML}{2348F2} \node (in) at (0,0) {\texttt{input (embeddings)}}; \node[blk, draw=acc, text=acc, thick] (mha) at (0,1.5) {\texttt{multi-head self-attention}}; \node[an] (an1) at (0,2.9) {\texttt{add \& norm}}; \node[blk, draw=acc, text=acc, thick] (ffn) at (0,4.4) {\texttt{feed-forward (FFN)}}; \node[an] (an2) at (0,5.8) {\texttt{add \& norm}}; \node (out) at (0,7.1) {\texttt{output (to next layer)}}; \draw[->, thick] (in) -- (mha); \draw[->, thick] (mha) -- (an1); \draw[->, thick] (an1) -- (ffn); \draw[->, thick] (ffn) -- (an2); \draw[->, thick] (an2) -- (out); \draw[->, acc, thick] (2.5,0.25) .. controls (3.9,1.0) and (3.9,2.2) .. (2.5,2.9) node[pos=0.5, right, text=acc] {residual}; \draw[->, acc, thick] (2.5,3.15) .. controls (3.9,3.9) and (3.9,5.1) .. (2.5,5.8) node[pos=0.5, right, text=acc] {residual}; \end{tikzpicture} $$ ## One decoder layer A decoder layer has _three_ wrapped sublayers. It must generate one token at a time without access to tokens it has not yet produced, and it must read the encoder. Two additions handle these requirements. - **Masked self-attention.** Before the softmax, the decoder sets the score $S_{ij} = -\infty$ for every $j > i$, forbidding position $i$ from attending to any later position. Generation stays **causal**: token $t$ depends only on tokens $< t$. - **Cross-attention.** A middle sublayer draws its queries from the decoder state but its keys and values from the encoder's output. This is the channel through which a generated token reads the entire source sequence; in translation it is how each output word reads the sentence being translated.[^chollet-decoder] | Sublayer | Queries from | Keys / values from | Mask | | --- | --- | --- | --- | | Masked self-attention | decoder | decoder | causal (lower-triangular) | | Cross-attention | decoder | encoder output | none | | Feed-forward | — | — | — | ## The full encoder–decoder The complete Transformer stacks $N$ encoder layers and $N$ decoder layers, feeds both stacks token embeddings plus positional encodings, and routes the encoder's final output into the cross-attention sublayer of _every_ decoder layer. A linear projection and softmax convert the decoder's top state into a distribution over the vocabulary. $$ % caption: The full encoder–decoder Transformer. The encoder (left) reads the source; its output % feeds cross-attention (green) in every decoder layer, which masks self-attention and reads out % through linear-plus-softmax. \begin{tikzpicture}[>=stealth, font=\scriptsize, sub/.style={draw, minimum width=30mm, minimum height=6mm, align=center}, an/.style={draw, minimum width=30mm, minimum height=5mm, align=center, fill=black!8}, emb/.style={draw, minimum width=30mm, minimum height=6mm, align=center}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \definecolor{red}{HTML}{C0392B} % ===================== ENCODER (left) ===================== \draw[black, dashed] (-1.85,2.0) rectangle (1.85,6.1); \node[emb] (esrc) at (0,0) {\texttt{source tokens}}; \node[emb, draw=acc, text=acc] (eemb) at (0,1.0) {\texttt{embedding + pos. enc.}}; \node[sub, draw=acc, text=acc, thick] (esa) at (0,2.5) {\texttt{self-attention}}; \node[an] (ean1) at (0,3.5) {\texttt{add \& norm}}; \node[sub, draw=acc, text=acc, thick] (eff) at (0,4.6) {\texttt{feed-forward}}; \node[an] (ean2) at (0,5.6) {\texttt{add \& norm}}; \node[black] (etop) at (0,6.6) {\texttt{N layers (encoder)}}; \draw[->, thick] (esrc) -- (eemb); \draw[->, thick] (eemb) -- (esa); \draw[->, thick] (esa) -- (ean1); \draw[->, thick] (ean1) -- (eff); \draw[->, thick] (eff) -- (ean2); \draw[->, thick] (ean2) -- (etop); % ===================== DECODER (right) ===================== \begin{scope}[xshift=6.6cm] \draw[black, dashed] (-1.85,2.0) rectangle (1.85,8.2); \node[emb] (dtgt) at (0,0) {\texttt{target (shifted)}}; \node[emb, draw=acc, text=acc] (demb) at (0,1.0) {\texttt{embedding + pos. enc.}}; \node[sub, draw=red, text=red, thick] (dsa) at (0,2.5) {\texttt{masked self-attention}}; \node[an] (dan1) at (0,3.5) {\texttt{add \& norm}}; \node[sub, draw=green, text=green, thick] (dca) at (0,4.6) {\texttt{cross-attention}}; \node[an] (dan2) at (0,5.6) {\texttt{add \& norm}}; \node[sub, draw=acc, text=acc, thick] (dff) at (0,6.7) {\texttt{feed-forward}}; \node[an] (dan3) at (0,7.7) {\texttt{add \& norm}}; \node[sub, draw=acc, text=acc] (lin) at (0,8.9) {\texttt{linear + softmax}}; \node (dout) at (0,9.9) {\texttt{output probabilities}}; \draw[->, thick] (dtgt) -- (demb); \draw[->, thick] (demb) -- (dsa); \draw[->, thick] (dsa) -- (dan1); \draw[->, thick] (dan1) -- (dca); \draw[->, thick] (dca) -- (dan2); \draw[->, thick] (dan2) -- (dff); \draw[->, thick] (dff) -- (dan3); \draw[->, thick] (dan3) -- (lin); \draw[->, thick] (lin) -- (dout); \node[black, anchor=west] at (1.95,8.5) {\texttt{N layers (decoder)}}; \end{scope} % ===================== CROSS-ATTENTION ROUTING ===================== \draw[->, green, thick] (1.85,5.9) .. controls (3.0,5.9) and (3.7,4.6) .. (4.75,4.6) node[pos=0.5, above, text=green, font=\footnotesize] {\texttt{keys, values}}; \end{tikzpicture} $$ The asymmetry is the whole point. The encoder reads bidirectionally because it sees the input all at once; the decoder reads causally because it produces the output left to right. Cross-attention is the only place the two streams meet. ## Causal masking in detail Masking is implemented additively, before the softmax. Define a mask matrix $M \in \mathbb{R}^{n \times n}$ with $M_{ij} = 0$ for $j \le i$ and $M_{ij} = -\infty$ for $j > i$, and add it to the raw scores: $$ A = \softmax\!\parens{\frac{QK^\top}{\sqrt{d_k}} + M}, \qquad A_{ij} = 0 \text{ for all } j > i . $$ The $-\infty$ entries become $e^{-\infty} = 0$ after exponentiation, so the softmax renormalizes over only the allowed (lower-triangular) positions. The result is **row-causal**: row $i$ of $A$ has support only on columns $1$ through $i$. > **Theorem (Autoregressive factorization).** With causal masking, the decoder > computes $p(x_t \mid x_{ \dots, x_{t-1}$, so a single forward pass yields every factor of the chain rule > $p(x) = \prod_{t=1}^{n} p(x_t \mid x_{ **Proof.** Each decoder output at position $t$ is a function of the masked > self-attention output at $t$, whose attention row has zero weight on all > $j > t$ by construction of $M$. By induction over the stacked layers, every > intermediate representation at position $t$ depends only on positions $\le t$; > the read-out head at $t-1$ therefore depends only on $x_{ the conditional $p(x_t \mid x_{ pass because the dependencies are encoded in the mask, not in sequential > evaluation. $\qed$ $$ % caption: The causal mask. Query row $i$ attends only to key columns $j \le i$: the lower triangle % (blue) is kept, the upper triangle (shaded, $-\infty$) is blocked before the softmax. \begin{tikzpicture}[font=\scriptsize, x=0.92cm, y=0.92cm] \definecolor{acc}{HTML}{2348F2} \def\n{6} % cells \foreach \r in {1,...,6}{ \foreach \c in {1,...,6}{ \pgfmathparse{\c<=\r ? 1 : 0} \ifnum\pgfmathresult=1 \fill[acc!18] (\c-1,\n-\r) rectangle (\c,\n-\r+1); \else \fill[black] (\c-1,\n-\r) rectangle (\c,\n-\r+1); \fi } } % grid \draw[black] (0,0) grid (\n,\n); % labels \foreach \c in {1,...,6} \node[anchor=south] at (\c-0.5,\n+0.1) {key \c}; \foreach \r in {1,...,6} \node[anchor=east] at (-0.1,\n-\r+0.5) {query \r}; \node[acc, anchor=west] at (\n+0.4,4.2) {\texttt{kept (past)}}; \node[black, anchor=west] at (\n+0.4,3.2) {\texttt{blocked (future)}}; \end{tikzpicture} $$ ```algorithm caption: $\textsc{MaskedSelfAttention}(X)$ — causal scaled dot-product attention $Q \gets X W_Q$; $K \gets X W_K$; $V \gets X W_V$ $S \gets Q K^{T} / \sqrt{d_k}$ // $n \times n$ raw scores for each $i, j$ with $j > i$ do $S_{ij} \gets -\infty$ // forbid attending to the future for each row $i$ do $A_{i} \gets \softmax(S_{i})$ // mass only on $j \le i$ return $A V$ // each row blends past values only ``` ## The three families The encoder and decoder were designed to work as a pair, but each half is a complete model on its own. Modern practice splits the architecture into three families, distinguished by which stack they keep, how they mask attention, and what objective they train against. | Family | Structure | Attention masking | Training objective | Examples | | --- | --- | --- | --- | --- | | Encoder-only | encoder stack, bidirectional | none (full self-attention) | masked language modeling (predict held-out tokens) | BERT, RoBERTa | | Decoder-only | decoder stack, no cross-attention | causal (lower-triangular) | autoregressive LM (next-token prediction) | GPT, LLaMA | | Encoder–decoder | both stacks, cross-attention | encoder full, decoder causal | seq2seq (conditional generation) | original Transformer, T5 | The split follows directly from the masking. An **encoder-only** model sees the whole sequence, so it is built for _understanding_ (classification, retrieval, token tagging) and is trained by corrupting input tokens and reconstructing them. A **decoder-only** model masks causally, so each position predicts the next token, making it a pure _generator_; this is the family that scaled into large language models. An **encoder–decoder** keeps both and is built for _transduction_: mapping one sequence to another, as in translation or summarization.[^chollet-families] $$ % caption: The three families. Encoder-only (BERT) attends bidirectionally; decoder-only (GPT) % masks causally; encoder–decoder (T5) joins both halves by cross-attention (green). \begin{tikzpicture}[>=stealth, font=\scriptsize, e/.style={draw, draw=acc, text=acc, minimum width=20mm, minimum height=6mm, align=center}, d/.style={draw, draw=red, text=red, minimum width=20mm, minimum height=6mm, align=center}, io/.style={font=\scriptsize, text=black}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \definecolor{green}{HTML}{1F9D4D} % ---- encoder-only ---- \node[io] (ei) at (0,-0.7) {\texttt{tokens}}; \node[e] (e1) at (0,0.2) {\texttt{encoder}}; \node[e] (e2) at (0,1.1) {\texttt{encoder}}; \node[io] (eo) at (0,1.9) {\texttt{features}}; \draw[->, thick] (ei) -- (e1); \draw[->, thick] (e1) -- (e2); \draw[->, thick] (e2) -- (eo); \node[font=\footnotesize] at (0,-1.5) {\texttt{encoder-only}}; \node[io] at (0,-2.0) {\texttt{(BERT, bidirectional)}}; % ---- decoder-only ---- \begin{scope}[xshift=3.6cm] \node[io] (di) at (0,-0.7) {\texttt{tokens}}; \node[d] (g1) at (0,0.2) {\texttt{decoder}}; \node[d] (g2) at (0,1.1) {\texttt{decoder}}; \node[io] (do) at (0,1.9) {\texttt{next token}}; \draw[->, thick] (di) -- (g1); \draw[->, thick] (g1) -- (g2); \draw[->, thick] (g2) -- (do); \node[font=\footnotesize] at (0,-1.5) {\texttt{decoder-only}}; \node[io] at (0,-2.0) {\texttt{(GPT, causal)}}; \end{scope} % ---- encoder-decoder ---- \begin{scope}[xshift=8.4cm] \node[io] (xi) at (-1.3,-0.7) {\texttt{source}}; \node[e] (x1) at (-1.3,0.2) {\texttt{encoder}}; \node[e] (x2) at (-1.3,1.1) {\texttt{encoder}}; \node[io] (yi) at (2.1,-0.7) {\texttt{target}}; \node[d] (y1) at (2.1,0.2) {\texttt{decoder}}; \node[d] (y2) at (2.1,1.1) {\texttt{decoder}}; \node[io] (yo) at (2.1,1.9) {\texttt{output}}; \draw[->, thick] (xi) -- (x1); \draw[->, thick] (x1) -- (x2); \draw[->, thick] (yi) -- (y1); \draw[->, thick] (y1) -- (y2); \draw[->, thick] (y2) -- (yo); % cross-attention: the encoder output feeds every decoder layer through a clean % vertical bus in the gap, one horizontal feeder per decoder layer. \draw[->, green, thick] (x2.east) -- (0.4,1.1) -- (y2.west); \draw[->, green, thick] (0.4,1.1) -- (0.4,0.2) -- (y1.west); \node[font=\footnotesize] at (0.4,-1.5) {\texttt{encoder-decoder}}; \node[io] at (0.4,-2.0) {\texttt{(T5, seq2seq)}}; \end{scope} \end{tikzpicture} $$ ## The residual stream A productive way to read the stack is as a **residual stream**: a fixed-width $d$-dimensional bus running from the embeddings to the output head. Each sublayer _reads_ a normalized copy of the stream, computes an update, and _writes_ that update back by addition. No sublayer overwrites the stream; they only accumulate into it. $$ z_\ell = z_{\ell-1} + \Delta_\ell, \qquad \Delta_\ell = f_\ell\!\parens{\LN(z_{\ell-1})}, \qquad z_N = z_0 + \sum_{\ell=1}^{N} \Delta_\ell . $$ This linear-accumulation view explains why Transformers are so editable: because contributions add, an attention head and an FFN can read each other's writes, and the final representation is literally the embedding plus every increment the sublayers added. $$ % caption: The residual stream. A fixed-width bus (black) runs up the stack; each block reads a % normalized view (black) and writes an additive update back (green), never overwriting it. \begin{tikzpicture}[>=stealth, font=\footnotesize, blk/.style={draw, minimum width=24mm, minimum height=8mm, align=center}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} % the bus \draw[black, line width=2.6mm] (0,-0.4) -- (0,8.4); \node[black, anchor=south, rotate=90] at (-0.6,4.0) {\texttt{residual stream (width d)}}; \node[anchor=north] at (0,-0.55) {\texttt{embed + pos}}; \node[anchor=south] at (0,8.55) {\texttt{to output head}}; % blocks on the right, reading and writing \node[blk, draw=acc, text=acc] (b1) at (3.6,1.2) {\texttt{self-attention}}; \node[blk, draw=acc, text=acc] (b2) at (3.6,3.2) {\texttt{feed-forward}}; \node[blk, draw=acc, text=acc] (b3) at (3.6,5.2) {\texttt{self-attention}}; \node[blk, draw=acc, text=acc] (b4) at (3.6,7.2) {\texttt{feed-forward}}; \foreach \b/\y in {b1/1.2, b2/3.2, b3/5.2, b4/7.2}{ \draw[->, black, thick] (0.16,\y-0.55) -- (\b.west |- 0,\y-0.55); \draw[->, green, thick] (\b.west |- 0,\y+0.55) -- (0.16,\y+0.55); } \node[black, anchor=south, font=\footnotesize] at (1.9,0.7) {\texttt{read}}; \node[green, anchor=south, font=\footnotesize] at (1.9,7.85) {\texttt{write (add)}}; \end{tikzpicture} $$ ## Where the parameters and the compute live Two accounting questions decide how a Transformer scales: where do its weights sit, and what dominates its runtime. Both have clean closed forms.[^gf-attention-mem] **Parameters.** Per layer, attention contributes the four projection matrices $W_Q, W_K, W_V, W_O$, each $d \times d$, for $4d^2$. The FFN contributes $W_1 \in \mathbb{R}^{d \times d_{\text{ff}}}$ and $W_2 \in \mathbb{R}^{d_{\text{ff}} \times d}$; with $d_{\text{ff}} = 4d$ that is $2 \cdot 4d^2 = 8d^2$. So per layer, $$ \underbrace{4d^2}_{\text{attention}} + \underbrace{8d^2}_{\text{FFN}} = 12d^2, \qquad \frac{\text{FFN params}}{\text{layer params}} = \frac{8d^2}{12d^2} = \frac{2}{3}. $$ Two-thirds of a Transformer's weights live in the feed-forward networks, not the attention — a fact that motivates much of the work on sparsifying or scaling the FFN (mixture-of-experts) rather than attention. The $12d^2$ count ignores the small additive terms — the biases ($\sim 5d$ per layer) and the two LayerNorms' $\gamma, \beta$ ($4d$ per layer) — which are linear in $d$ and vanish against the quadratic weights for any real width. Put $d = 512$, $d_{\text{ff}} = 2048$ (the base configuration) into the table to see the actual per-layer budget: | Component | Formula | $d = 512$ | Share | | --- | --- | --- | --- | | Attention $W_Q, W_K, W_V, W_O$ | $4d^2$ | $\approx 1.05\text{M}$ | $1/3$ | | Feed-forward $W_1, W_2$ ($d_{\text{ff}} = 4d$) | $8d^2$ | $\approx 2.10\text{M}$ | $2/3$ | | LayerNorm ($\gamma, \beta$, two per layer) | $4d$ | $\approx 2\text{k}$ | $< 0.1\%$ | | **Per layer total** | $\approx 12d^2$ | $\approx 3.15\text{M}$ | $1$ | A $6$-layer encoder of this width holds $\approx 19\text{M}$ weights in its blocks; the token embedding and output projection add $2 \, |V| \, d$ on top, which for a $30\text{k}$ vocabulary is another $\approx 31\text{M}$ — a reminder that at small depth the vocabulary table, not the blocks, can dominate the count. **Compute.** The cost split is reversed. The FFN is linear in sequence length — $O(n \cdot d^2)$ per layer — but the attention matrix is quadratic. > **Theorem (Quadratic attention cost).** Computing self-attention over a > sequence of length $n$ with model width $d$ costs $O(n^2 d)$ time and $O(n^2)$ > memory for the attention matrix, independent of how the heads are split. > **Proof.** Forming the scores $QK^\top$ multiplies an $n \times d$ matrix by a > $d \times n$ matrix, costing $O(n^2 d)$ and producing an $n \times n$ matrix. > The softmax is $O(n^2)$. Multiplying the $n \times n$ weights by the $n \times > d$ values is again $O(n^2 d)$. Splitting into $h$ heads of width $d/h$ gives > $h$ products of cost $O(n^2 \cdot d/h)$, summing back to $O(n^2 d)$. The > $n \times n$ score matrix dominates memory at $O(n^2)$. $\qed$ The crossover is at $n \approx d$: for $n < d$ the per-layer cost is dominated by the $O(n d^2)$ projection and FFN work, but once $n \gtrsim d$ the $O(n^2 d)$ attention term takes over and grows quadratically. This single $n^2$ is the reason context windows are expensive and the target of nearly every efficient-attention variant. $$ % caption: Per-layer cost vs. sequence length $n$ (log–log). The projection/FFN term is linear % (black); attention is quadratic (blue). They cross near $n \approx d$, after which attention dominates. \begin{tikzpicture}[>=stealth, font=\footnotesize, x=1.0cm, y=1.0cm] \definecolor{acc}{HTML}{2348F2} \draw[->, thick] (0,0) -- (7.0,0) node[right, font=\scriptsize] {log n}; \draw[->, thick] (0,0) -- (0,4.6) node[above, font=\scriptsize] {log cost}; % linear term: slope 1 \draw[black, very thick] (0.3,0.6) -- (6.4,2.6) node[right, text=black, font=\scriptsize] {linear}; % quadratic term: slope 2 \draw[acc, very thick] (0.3,-0.1) -- (6.4,4.1) node[right, text=acc, font=\scriptsize] {quadratic}; % crossover marker (lines meet near x=2.24, y=1.24) \draw[black, dashed] (2.24,0) -- (2.24,1.24); \node[anchor=north, font=\scriptsize] at (2.24,0) {n near d}; \fill[acc] (2.24,1.24) circle (1.6pt); \end{tikzpicture} $$ That is the architecture and its cost model. The same design turns out to be indifferent to what a token _is_ — image patches, DNA bases, or program tokens all work — and the decoder-only half of it scaled into the large language models that dominate current practice. Those applications, and the empirical scaling laws that make scale pay off so reliably, continue in [Transformers in Practice](/deep-learning/architectures/transformers-in-practice). ## Takeaways - The **Transformer** is attention plus position-wise feed-forward layers, stacked with **residual + LayerNorm** wrappers, with no recurrence or convolution anywhere. - **Pre-norm** ($x + f(\LN(x))$) leaves a clean identity path up the stack, so deep Transformers train stably; **post-norm** ($\LN(x + f(x))$) squeezes every residual through a norm and needs warmup. - **Sinusoidal positional encoding** gives each position a unique fingerprint across a geometric band of frequencies, makes relative offsets a linear map, and extrapolates past the training length; learned tables and rotary (RoPE) encodings are the modern alternatives. - The **encoder** attends bidirectionally; the **decoder** adds **masked self-attention** (causal, lower-triangular) and **cross-attention** to the encoder output, and reads out through **linear + softmax**. - The **three families** specialize the design: **encoder-only** (BERT, masked LM, understanding), **decoder-only** (GPT, causal LM, generation), **encoder–decoder** (T5, seq2seq, transduction). - Causal masking realizes the autoregressive factorization $p(x) = \prod_t p(x_t \mid x_{