The Transformer Architecture

This builds on Transformers and Self-Attention, which developed the attention operation: from each input, project a query, key, and value; score each query against the keys; scale by $\sqrt{d_k}$, softmax into weights, and mix the values; and mask the future to make the layer causal. The rest of the transformer exists to run that operation in parallel across several relations at once and to stack it many layers deep. This part builds those pieces.

Multi-head attention

Two words can relate along several axes at once. In the keys to the cabinet were on the table, the verb were has a syntactic agreement link to keys, a semantic link to table, and a different link again to cabinet. A single attention layer produces one set of weights ${\alpha }_{ij}$ and so can emphasize only one kind of relationship at a time. Multi-head attention runs several attention operations in parallel, each with its own ${\mathbf{W}}^Q,{\mathbf{W}}^K,{\mathbf{W}}^V$, so each head can specialize in a different relation.¹

Each head projects the input into its own lower-dimensional space, of size $d_k=d_v=d/h$, so that running $h$ heads costs about the same as one full-width head. The heads' outputs are concatenated and a final linear map ${\mathbf{W}}^O$ mixes them back to the model dimension $d$, keeping the layer's output shape equal to its input shape so the blocks can stack:

The self-attention equation is unchanged inside each head; multi-head attention is $h$ copies of it plus a concatenation and one more linear layer.¹

For example, take the original transformer's numbers: $d_{model}=512$ and $h=8$ heads, so each head works in $d_k=d_v=512/8=64$. Each head's projections ${\mathbf{W}}_i^Q,{\mathbf{W}}_i^K,{\mathbf{W}}_i^V$ are $512\times 64$, mapping a $512$-dimensional input down to a $64$-dimensional per-head space. A head produces an $N\times 64$ output; concatenating all eight gives $N\times (8\cdot 64)=N\times 512$, back to the model width, and ${\mathbf{W}}^O$ is $512\times 512$. The total parameter count of the four projection matrices is the same as one full-width single head would use: eight $64$-dimensional views cost no more than one $512$-dimensional view, because the width is split rather than multiplied.

The transformer block

One attention layer mixes information across positions but does little else. A transformer block wraps it in a position-wise feedforward network, residual connections, and layer normalization, which together make deep stacks trainable.²

• Feedforward network. After attention has mixed information across positions, a small two-layer feedforward network is applied to each position independently, giving the block non-linear processing capacity on top of the linear value mixing.
• Residual connections. Each sublayer adds its input back to its output, $\mathbf{x}+\mathrm{Sublayer}(\mathbf{x})$. This lets information and gradients skip the sublayer, giving upper layers direct access to lower ones and making very deep stacks trainable.
• Layer normalization. After each residual add, the summed vector is normalized to zero mean and unit variance across its components (then rescaled by learned gain and offset), keeping activations in a range that gradient descent handles well.

The block computes attention, adds and normalizes, then applies the feedforward network, and adds and normalizes again:

Layer norm standardizes each vector using its own mean $\mu$ and standard deviation $\sigma$ over its $d_h$ components, then applies learned gain $\gamma$ and offset $\beta$:

Because the block's input and output have the same shape, blocks stack directly, and a transformer is $N$ of them in sequence — the original transformer used $N=6$, and modern language models use dozens.

Positional embeddings

The block still cannot represent word order: self-attention is permutation-invariant. The output for position $i$ is a sum over the value vectors weighted by content-based scores; permute the inputs and the same permuted outputs come back. the dog bit the man and the man bit the dog would produce identical representations, which is clearly wrong. Recurrence encoded order for free by reading left to right; attention has to be told the order explicitly.³

The fix is a positional embedding: a vector that encodes where a token sits, added to its word embedding before the first block. The composite input carries both what the word is and where it stands.

There are two common recipes. Learned positional embeddings keep a trainable vector for each position up to some maximum length, learned alongside the word embeddings — simple, but positions near the length limit are seen rarely in training and generalize poorly. Sinusoidal positional embeddings instead compute each position with a fixed family of sines and cosines at geometrically spaced frequencies, so that nearby positions get similar vectors and the function extends smoothly to any length. The original transformer used the sinusoidal form.³

Transformers as language models

With the pieces assembled, a decoder-only transformer is a language model. Stack causal transformer blocks on top of the position-augmented embeddings, put a linear layer plus softmax over the vocabulary on top of the final block, and train it to predict the next word by teacher forcing with a cross-entropy loss — the same objective as the recurrent language model, but computed for every position in parallel, since each output depends only on earlier inputs and no serial recurrence links them.⁴

Encoder, decoder, and decoder-only

The same block appears in three architectural shapes, distinguished by which attention masks they use and whether they read one sequence or two.⁵

• Encoder-only. A stack of bidirectional transformer blocks that turns an input sequence into contextual representations, with no masking and no generation. This is the shape used for understanding tasks — classification, tagging, retrieval.
• Encoder-decoder. An encoder reads the source with bidirectional attention; a decoder generates the target with causal attention, plus an extra cross-attention sublayer in each block. Cross-attention is ordinary attention with the queries coming from the decoder but the keys and values coming from the encoder's output, so each generated word attends over the whole source. This is the classic shape for machine translation.
• Decoder-only. A single stack of causal blocks that both reads a prompt and continues it, with no separate encoder. This is the shape of the modern large language model.

The 2017 paper and what came after

Everything above is the transformer of Vaswani et al., Attention Is All You Need (NeurIPS 2017), the paper that named the architecture and set its defaults.⁶ Their model was the encoder-decoder of the last section, with $N=6$ blocks on each side, $d_{model}=512$, $h=8$ heads (so $d_k=d_v=64$), and a feedforward inner width of $2048$. Earlier translation systems added attention on top of a recurrent backbone; the paper's claim, reflected in the title, was that the recurrence could be deleted outright, leaving only attention. It reached state-of-the-art English-to- German translation while training in a fraction of the time, precisely because the layer parallelizes where recurrence serialized. The design has barely changed since; what changed is scale and a handful of refinements below.

Where the layer norm goes. The 2017 block put layer norm after the residual add — the post-norm arrangement of the equations above. Later work found this hard to train deep, because the residual stream is renormalized at every block and gradients to the earliest layers degrade. Moving the norm inside the residual branch, so it acts on the sublayer input rather than the sum — pre-norm, $\mathbf{x}+\mathrm{Sublayer}(\mathrm{LN}(\mathbf{x}))$ — keeps a clean residual path from input to output and lets stacks of dozens to hundreds of blocks train stably (Xiong et al., 2020).⁷ Essentially every large model today is pre-norm.

The quadratic cost, addressed two ways. Forming $\mathbf{Q}{\mathbf{K}}^{\top }$ costs $O(N^2)$ in both time and memory, which caps context length. Two lines of work address this. The first keeps exact attention but computes it without ever writing the $N\times N$ matrix to memory: FlashAttention (Dao et al., 2022) tiles the computation and fuses the softmax so memory grows linearly in $N$, turning long-context attention from memory-bound to compute-bound and delivering large wall-clock speedups with identical numerics.⁸ The second changes the operation to a cheaper approximation — sparse attention patterns (Child et al., 2019), low-rank or kernel approximations like Linformer and Performer — trading exactness for sub-quadratic scaling. In practice FlashAttention prevailed: at the context lengths in common use, exact attention computed efficiently outperformed the sub-quadratic approximations.

Positional information without an added vector. The sinusoidal and learned embeddings of the last section are absolute: they tag each token with its index. Rotary position embedding (RoPE; Su et al., 2021) instead rotates the query and key vectors by an angle proportional to their position, so that the dot product ${\mathbf{q}}_i\cdot {\mathbf{k}}_j$ ends up depending only on the relative offset $i-j$.⁹ Relative position is what attention usually needs — the adjective two words back, not the word at absolute index 47 — and RoPE extends more gracefully to sequences longer than any seen in training. It is now the default positional scheme in most open large language models.

None of these change the idea. Attention is still a query scored against keys, softmaxed into weights, mixing values. The refinements make it train deeper (pre-norm), run longer (Flash), and generalize across length (RoPE), but the operation in the numeric trace of the previous part is the one still running inside every current model.

This is the architecture the rest of modern NLP is built on. Scaled up and pretrained on enormous text corpora, the decoder-only transformer becomes the large language model; the deep-learning course gives the same machinery a second, complementary treatment. What made all of it possible is the one move in these two parts: replace the recurrent chain with a layer where every position attends directly to every other, in parallel.