---
title: The Transformer Architecture
module: Transformers
moduleNumber: 5
lessonNumber: 2
order: 502
summary: >
  This part takes the scaled dot-product attention of the previous lesson and
  assembles the full transformer architecture around it: multi-head attention so several relations can be read at once, the
  transformer block of residual connections and layer norm that makes deep stacks
  trainable, positional embeddings that restore word order, the decoder-only
  language model, and the encoder, decoder, and encoder-decoder shapes — closing
  with the 2017 paper and the pre-norm, FlashAttention, and RoPE refinements that
  scaled it up.
topics: [Transformers]
sources:
  - book: Jurafsky
    ref: "§9.7.1 Transformer Blocks; §9.7.2 Multihead Attention; §9.7.3 Modeling Word Order: Positional Embeddings; §9.8 Transformers as Language Models"
  - book: Jurafsky
    ref: "Ch. 10 — Machine Translation and Encoder-Decoder Models; §10.6 Encoder-Decoder with Transformers"
---

This builds on [Transformers and Self-Attention](/natural-language-processing/transformers/transformers-and-attention),
which developed the attention operation: from each input, project a query, key, and value;
score each query against the keys; scale by $\sqrt{d_k}$, softmax into weights, and mix the
values; and mask the future to make the layer causal. The rest of the transformer exists to
run that operation in parallel across several relations at once and to stack it many layers
deep. This part builds those pieces.

## Multi-head attention

Two words can relate along several axes at once. In _the keys to the cabinet were on the
table_, the verb _were_ has a syntactic agreement link to _keys_, a semantic link to _table_,
and a different link again to _cabinet_. A single attention layer produces one set of weights
$\alpha_{ij}$ and so can emphasize only one kind of relationship at a time. **Multi-head
attention** runs several attention operations in parallel, each with its own
$\mathbf{W}^Q, \mathbf{W}^K, \mathbf{W}^V$, so each **head** can specialize in a different
relation.[^jm-heads]

> **Definition (Multi-head attention).** $h$ self-attention operations run in parallel, the
> **heads**, each with independent projection matrices $\mathbf{W}^Q_i, \mathbf{W}^K_i,
> \mathbf{W}^V_i$ and each producing an $N \times d_v$ output. The $h$ head outputs are
> concatenated and projected back to the model dimension by a final matrix $\mathbf{W}^O$.

Each head projects the input into its own lower-dimensional space, of size $d_k = d_v = d /
h$, so that running $h$ heads costs about the same as one full-width head. The heads' outputs
are concatenated and a final linear map $\mathbf{W}^O$ mixes them back to the model dimension
$d$, keeping the layer's output shape equal to its input shape so the blocks can stack:

$$
\mathrm{head}_i = \mathrm{SelfAttention}(\mathbf{Q}_i, \mathbf{K}_i, \mathbf{V}_i),
\qquad
\mathrm{MultiHead}(\mathbf{X}) =
\big(\mathrm{head}_1 \oplus \mathrm{head}_2 \oplus \cdots \oplus \mathrm{head}_h\big)\mathbf{W}^O .
$$

$$
% caption: Multi-head attention: $h$ heads each run scaled dot-product attention with their
% own query/key/value projections over the same input, and their outputs are concatenated
% and projected down to $d$ by $W^O$ so the result matches the input shape.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  head/.style={draw, fill=black!4, minimum width=20mm, minimum height=9mm, align=center, font=\scriptsize},
  tok/.style={draw, minimum width=8mm, minimum height=6mm, inner sep=1pt}]
  \definecolor{acc}{HTML}{2348F2}
  \node[tok] (x) at (0,0) {input X};
  \node[head] (h1) at (-3.3,1.7) {head 1\\Wq1, Wk1, Wv1};
  \node[head] (h2) at (-0.6,1.7) {head 2\\Wq2, Wk2, Wv2};
  \node[font=\scriptsize] (dots) at (1.7,1.7) {. . .};
  \node[head] (h4) at (3.9,1.7) {head h\\Wqh, Wkh, Wvh};
  \foreach \h in {h1,h2,h4} \draw[->, black] (x) -- (\h);
  \node[draw, fill=acc!12, minimum width=90mm, minimum height=7mm] (cat) at (0.9,3.3)
    {concatenate heads, then project down by Wo};
  \foreach \h in {h1,h2,h4} \draw[->, black] (\h) -- (cat);
  \node[tok, draw=acc, text=acc] (y) at (0.9,4.8) {output y};
  \draw[->, acc, thick] (cat) -- (y);
\end{tikzpicture}
$$

The self-attention equation is unchanged inside each head; multi-head attention is
$h$ copies of it plus a concatenation and one more linear layer.[^jm-heads]

For example, take the original transformer's numbers:
$d_{model} = 512$ and $h = 8$ heads, so each head works in $d_k = d_v = 512 / 8 = 64$. Each
head's projections $\mathbf{W}^Q_i, \mathbf{W}^K_i, \mathbf{W}^V_i$ are $512 \times 64$,
mapping a $512$-dimensional input down to a $64$-dimensional per-head space. A head produces
an $N \times 64$ output; concatenating all eight gives $N \times (8 \cdot 64) = N \times 512$,
back to the model width, and $\mathbf{W}^O$ is $512 \times 512$. The total parameter count of
the four projection matrices is the same as one full-width single head would use: eight
$64$-dimensional views cost no more than one $512$-dimensional view, because the width is
split rather than multiplied.

## The transformer block

One attention layer mixes information across positions but does little else. A **transformer
block** wraps it in a position-wise feedforward network, residual connections, and layer
normalization, which together make deep stacks trainable.[^jm-block]

- **Feedforward network.** After attention has mixed information across positions, a small
  two-layer feedforward network is applied to each position independently, giving the block
  non-linear processing capacity on top of the linear value mixing.
- **Residual connections.** Each sublayer adds its input back to its output, $\mathbf{x} +
  \mathrm{Sublayer}(\mathbf{x})$. This lets information and gradients skip the sublayer,
  giving upper layers direct access to lower ones and making very deep stacks trainable.
- **Layer normalization.** After each residual add, the summed vector is normalized to zero
  mean and unit variance across its components (then rescaled by learned gain and offset),
  keeping activations in a range that gradient descent handles well.

The block computes attention, adds and normalizes, then applies the feedforward network, and
adds and normalizes again:

$$
\mathbf{z} = \mathrm{LayerNorm}\big(\mathbf{x} + \mathrm{SelfAttn}(\mathbf{x})\big),
\qquad
\mathbf{y} = \mathrm{LayerNorm}\big(\mathbf{z} + \mathrm{FFN}(\mathbf{z})\big) .
$$

Layer norm standardizes each vector using its own mean $\mu$ and standard deviation $\sigma$
over its $d_h$ components, then applies learned gain $\gamma$ and offset $\beta$:

$$
\mu = \frac{1}{d_h}\sum_{i=1}^{d_h} x_i,
\qquad
\sigma = \sqrt{\frac{1}{d_h}\sum_{i=1}^{d_h}(x_i - \mu)^2},
\qquad
\mathrm{LayerNorm}(\mathbf{x}) = \gamma\,\frac{\mathbf{x} - \mu}{\sigma} + \beta .
$$

Because the block's input and output have the same shape, blocks stack directly, and a
transformer is $N$ of them in sequence — the original transformer used $N = 6$, and modern
language models use dozens.

$$
% caption: One transformer block: multi-head self-attention and a feedforward network, each
% wrapped in a residual add (the input routed around the sublayer) followed by layer norm.
% The output shape matches the input, so blocks stack.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  sub/.style={draw, fill=black!4, minimum width=52mm, minimum height=9mm},
  norm/.style={draw, fill=acc!12, minimum width=52mm, minimum height=7mm}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[draw, minimum width=52mm, minimum height=6mm] (in) at (0,0) {input x1 . . . xn};
  \node[sub]  (att)  at (0,1.4) {multi-head self-attention};
  \node[norm] (ln1)  at (0,2.7) {add and layer norm};
  \node[sub]  (ff)   at (0,4.1) {feedforward layer};
  \node[norm] (ln2)  at (0,5.4) {add and layer norm};
  \node[draw, draw=acc, text=acc, minimum width=52mm, minimum height=6mm] (out) at (0,6.8) {output y1 . . . yn};
  \draw[->, black] (in)  -- (att);
  \draw[->, black] (att) -- (ln1);
  \draw[->, black] (ln1) -- (ff);
  \draw[->, black] (ff)  -- (ln2);
  \draw[->, acc, thick] (ln2) -- (out);
  % residual connections routed around each sublayer
  \draw[->, red!70!black, thick] (in.east)  to[bend left=52] node[right, font=\scriptsize] {residual} (ln1.east);
  \draw[->, red!70!black, thick] (ln1.east) to[bend left=52] node[right, font=\scriptsize] {residual} (ln2.east);
\end{tikzpicture}
$$

## Positional embeddings

The block still cannot represent word order: self-attention is permutation-invariant.
The output for position $i$ is a sum over the value vectors weighted by
content-based scores; permute the inputs and the same permuted outputs come back. _the dog bit
the man_ and _the man bit the dog_ would produce identical representations, which is clearly
wrong. Recurrence encoded order for free by reading left to right; attention has to be told the
order explicitly.[^jm-pos]

The fix is a **positional embedding**: a vector that encodes _where_ a token sits, added to
its word embedding before the first block. The composite input carries both what the word is
and where it stands.

> **Definition (Positional embedding).** A vector $\mathbf{p}_i$ representing position $i$ in
> the sequence, added to the word embedding $\mathbf{e}_i$ to form the block's input
> $\mathbf{x}_i = \mathbf{e}_i + \mathbf{p}_i$. It supplies the order information that
> self-attention, being permutation-invariant, cannot otherwise recover.

There are two common recipes. **Learned** positional embeddings keep a trainable vector for
each position up to some maximum length, learned alongside the word embeddings — simple, but
positions near the length limit are seen rarely in training and generalize poorly.
**Sinusoidal** positional embeddings instead compute each position with a fixed family of
sines and cosines at geometrically spaced frequencies, so that nearby positions get similar
vectors and the function extends smoothly to any length. The original transformer used the
sinusoidal form.[^jm-pos]

$$
% caption: Positional embeddings restore word order: a learned or sinusoidal position vector
% $p_i$ is added to each word embedding $e_i$, and the composite $x_i = e_i + p_i$ is what the
% first transformer block reads.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  cell/.style={draw, minimum width=10mm, minimum height=6mm, inner sep=1pt, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \foreach \i/\w in {1/Janet, 2/will, 3/back, 4/the, 5/bill} {
    % word and position embeddings side by side, so neither arrow crosses a box
    \node[cell] (w\i) at (\i*2.4-0.55,0)   {\w};
    \node[cell] (p\i) at (\i*2.4+0.55,0)   {pos \i};
    \node[circle, draw, inner sep=1pt, font=\scriptsize] (s\i) at (\i*2.4,1.3) {+};
    \node[cell, draw=acc, text=acc] (x\i) at (\i*2.4,2.6) {x\i};
    \draw[->, black] (w\i) -- (s\i);
    \draw[->, black] (p\i) -- (s\i);
    \draw[->, acc] (s\i) -- (x\i);
  }
  \node[anchor=east, font=\scriptsize, text=black] at (1.1,0)   {word / position emb.};
  \node[anchor=east, font=\scriptsize, text=black] at (1.1,2.6) {to block 1};
\end{tikzpicture}
$$

## Transformers as language models

With the pieces assembled, a decoder-only transformer is a language model. Stack causal
transformer blocks on top of the position-augmented embeddings, put a linear layer plus
softmax over the vocabulary on top of the final block, and train it to predict the next word
by teacher forcing with a cross-entropy loss — the same objective as the recurrent language
model, but computed for **every position in parallel**, since each output depends only on
earlier inputs and no serial recurrence links them.[^jm-lm]

```algorithm
caption: $\textsc{TransformerLM}$ — next-token distributions for a whole sequence, in parallel
input: token embeddings $\mathbf{e}_1, \ldots, \mathbf{e}_N$; positional embeddings $\mathbf{p}_1, \ldots, \mathbf{p}_N$
for each position $i$ do
  $\mathbf{x}_i \gets \mathbf{e}_i + \mathbf{p}_i$
$\mathbf{H} \gets [\mathbf{x}_1; \ldots; \mathbf{x}_N]$
for each block $b = 1, 2, \ldots, L$ do
  $\mathbf{A} \gets \mathrm{MaskedMultiHead}(\mathbf{H})$ // causal mask: position $i$ sees only $j \le i$
  $\mathbf{H} \gets \mathrm{LayerNorm}(\mathbf{H} + \mathbf{A})$
  $\mathbf{H} \gets \mathrm{LayerNorm}(\mathbf{H} + \mathrm{FFN}(\mathbf{H}))$
for each position $i$ do
  $\mathbf{y}_i \gets \mathrm{softmax}(\mathbf{W}\,\mathbf{h}_i)$ // distribution over next token
return $\mathbf{y}_1, \ldots, \mathbf{y}_N$
```

## Encoder, decoder, and decoder-only

The same block appears in three architectural shapes, distinguished by which attention masks
they use and whether they read one sequence or two.[^jm-encdec]

- **Encoder-only.** A stack of bidirectional transformer blocks that turns an input sequence
  into contextual representations, with no masking and no generation. This is the shape used
  for understanding tasks — classification, tagging, retrieval.
- **Encoder-decoder.** An encoder reads the source with bidirectional attention; a decoder
  generates the target with causal attention, plus an extra **cross-attention** sublayer in
  each block. Cross-attention is ordinary attention with the queries coming from the decoder
  but the keys and values coming from the encoder's output, so each generated word attends
  over the whole source. This is the classic shape for [machine
  translation](/natural-language-processing/applications/machine-translation).
- **Decoder-only.** A single stack of causal blocks that both reads a prompt and continues
  it, with no separate encoder. This is the shape of the modern
  [large language model](/natural-language-processing/transformers/large-language-models).

$$
% caption: Encoder-decoder with transformers. The encoder (bidirectional) maps the source to
% representations; each decoder block adds a cross-attention sublayer whose queries come from
% the decoder and whose keys and values come from the encoder output.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  blk/.style={draw, fill=black!4, minimum width=26mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % encoder stack
  \node[blk] (e1) at (0,0)   {encoder block};
  \node[blk] (e2) at (0,1.2) {encoder block};
  \node[anchor=north, font=\scriptsize] at (0,-0.7) {source (bidirectional)};
  \draw[->, black] (e1) -- (e2);
  \node[anchor=south, font=\scriptsize, text=acc] (henc) at (0,2.0) {encoder output};
  \draw[->, acc] (e2) -- (henc);
  % decoder stack
  \node[blk] (d1) at (5.2,0)   {decoder block\\(causal + cross)};
  \node[blk] (d2) at (5.2,1.5) {decoder block\\(causal + cross)};
  \node[anchor=north, font=\scriptsize] at (5.2,-0.7) {target (causal)};
  \draw[->, black] (d1) -- (d2);
  \node[blk, draw=acc, text=acc] (out) at (5.2,3.0) {next word};
  \draw[->, acc] (d2) -- (out);
  % cross-attention: encoder output feeds decoder keys/values
  \draw[->, red!70!black, thick] (henc.east) to[bend left=16]
    node[above, font=\scriptsize, pos=0.5] {cross-attention (K, V)} (d2.west);
\end{tikzpicture}
$$

## The 2017 paper and what came after

Everything above is the transformer of Vaswani et al., "Attention Is All You Need"
(NeurIPS 2017), the paper that named the architecture and set its defaults.[^vaswani] Their
model was the encoder-decoder of the last section, with $N = 6$ blocks on each side,
$d_{model} = 512$, $h = 8$ heads (so $d_k = d_v = 64$), and a feedforward inner width of
$2048$. Earlier translation systems added attention on top of a recurrent backbone; the
paper's claim, reflected in the title, was that the recurrence could be deleted outright,
leaving only attention. It reached state-of-the-art English-to-
German translation while training in a fraction of the time, precisely because the layer
parallelizes where recurrence serialized. The design has barely changed since; what changed
is scale and a handful of refinements below.

**Where the layer norm goes.** The 2017 block put layer norm _after_ the residual add — the
**post-norm** arrangement of the equations above. Later work found this hard to train deep,
because the residual stream is renormalized at every block and gradients to the earliest
layers degrade. Moving the norm _inside_ the residual branch, so it acts on the sublayer
input rather than the sum — **pre-norm**, $\mathbf{x} + \mathrm{Sublayer}(\mathrm{LN}(
\mathbf{x}))$ — keeps a clean residual path from input to output and lets stacks of dozens to
hundreds of blocks train stably (Xiong et al., 2020).[^prenorm] Essentially every large model
today is pre-norm.

**The quadratic cost, addressed two ways.** Forming $\mathbf{Q}\mathbf{K}^\top$ costs $O(N^2)$
in both time and memory, which caps context length. Two lines of work address this. The first
keeps exact attention but computes it without ever writing the $N \times N$ matrix to memory:
**FlashAttention** (Dao et al., 2022) tiles the computation and fuses the softmax so memory
grows linearly in $N$, turning long-context attention from memory-bound to compute-bound and
delivering large wall-clock speedups with identical numerics.[^flash] The second changes the
operation to a cheaper approximation — sparse attention patterns (Child et al., 2019),
low-rank or kernel approximations like Linformer and Performer — trading exactness for
sub-quadratic scaling. In practice FlashAttention prevailed: at the context lengths in common
use, exact attention computed efficiently outperformed the sub-quadratic approximations.

**Positional information without an added vector.** The sinusoidal and learned embeddings of
the last section are **absolute**: they tag each token with its index. **Rotary position
embedding** (RoPE; Su et al., 2021) instead rotates the query and key vectors by an angle
proportional to their position, so that the dot product $\mathbf{q}_i \cdot \mathbf{k}_j$ ends
up depending only on the _relative_ offset $i - j$.[^rope] Relative position is what attention
usually needs — "the adjective two words back," not "the word at absolute index 47" — and RoPE
extends more gracefully to sequences longer than any seen in training. It is now the default
positional scheme in most open large language models.

None of these change the idea. Attention is still a query scored against keys, softmaxed into
weights, mixing values. The refinements make it train deeper (pre-norm), run longer (Flash),
and generalize across length (RoPE), but the operation in the numeric trace of the previous
part is the one still running inside every current model.

This is the architecture the rest of modern NLP is built on. Scaled up and pretrained on
enormous text corpora, the decoder-only transformer becomes the [large language
model](/natural-language-processing/transformers/large-language-models); the deep-learning
course gives the same machinery a second, complementary treatment. What made all of it
possible is the one move in these two parts: replace the recurrent chain with a layer where
every position attends directly to every other, in parallel.

[^jm-heads]: **Jurafsky & Martin**, §9.7.2 — Multihead Attention (Eqs. 9.43–9.45): parallel heads, each with its own projection matrices and dimension $d_k = d_v = d/h$, capture different relations; head outputs are concatenated and projected by $\mathbf{W}^O$ back to the model dimension.
[^jm-block]: **Jurafsky & Martin**, §9.7.1 — Transformer Blocks (Eqs. 9.37–9.42): self-attention and a feedforward layer, each with a residual connection and layer normalization; layer norm standardizes a vector to zero mean and unit variance and rescales by learned $\gamma$ and $\beta$.
[^jm-pos]: **Jurafsky & Martin**, §9.7.3 — Modeling Word Order: Positional Embeddings: self-attention is order-agnostic, so a learned or sinusoidal positional embedding is added to each word embedding to encode absolute position.
[^jm-lm]: **Jurafsky & Martin**, §9.8 — Transformers as Language Models (Fig. 9.21): a decoder-style transformer trained by teacher forcing with cross-entropy to predict the next word, with all positions computed in parallel unlike the serial RNN.
[^jm-encdec]: **Jurafsky & Martin**, Ch. 10, §10.6 — Encoder-Decoder with Transformers (Figs. 10.15–10.16): a bidirectional encoder and a causal decoder whose blocks add a cross-attention sublayer taking queries from the decoder and keys and values from the final encoder output.
[^vaswani]: **Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin (2017)**, "Attention Is All You Need," _NeurIPS 2017_. Introduces the transformer: an encoder-decoder built only from scaled dot-product multi-head attention and position-wise feedforward layers, with $N=6$ blocks per side, $d_{model}=512$, $h=8$ heads ($d_k=d_v=64$), feedforward width $2048$, and sinusoidal positional encodings. Reaches state-of-the-art WMT English-German/English-French translation at a large reduction in training cost relative to recurrent and convolutional baselines.
[^prenorm]: **Xiong, Yang, He, Zheng, Zheng, Xing, Zhang, Lan, Wang, Liu (2020)**, "On Layer Normalization in the Transformer Architecture," _ICML 2020_. Shows analytically and empirically that the original post-norm placement produces large gradients near the output and unstable early training requiring learning-rate warmup, while pre-norm (layer norm inside the residual branch) yields well-behaved gradients and trains deep transformers stably.
[^flash]: **Dao, Fu, Ermon, Rudra, Ré (2022)**, "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness," _NeurIPS 2022_. An exact-attention algorithm that tiles the computation and fuses the softmax so the $N \times N$ score matrix is never materialized in high-bandwidth memory; reduces memory from quadratic to linear in sequence length and gives substantial wall-clock speedups with numerically identical output.
[^rope]: **Su, Lu, Pan, Murtadha, Wen, Liu (2021)**, "RoFormer: Enhanced Transformer with Rotary Position Embedding." Rotary position embedding rotates query and key vectors by a position-dependent angle so that the attention score depends only on the relative offset $i-j$; it improves length generalization and is widely adopted in open large language models. (See also Sennrich/Child/Linformer/Performer lines cited in text for sparse and approximate attention.)