---
title: Transformers and Self-Attention
module: Transformers
moduleNumber: 5
lessonNumber: 1
order: 501
summary: >
  Recurrence forced language models to read one word at a time and to squeeze
  every dependency through a chain of hidden states. Self-attention removes the
  recurrence: at every layer each position compares itself to every other and
  reads a weighted mixture of them, in a single parallel step. This first part
  builds the attention operation from the ground up — the soft lookup, queries and
  keys and values, the scaled dot-product, the numeric trace, the matrix form, and
  the causal mask — and sets up the full transformer architecture that follows.
topics: [Transformers]
sources:
  - book: Jurafsky
    ref: "Ch. 9 — Deep Learning Architectures for Sequence Processing; §9.7 Self-Attention Networks: Transformers"
  - book: Jurafsky
    ref: "§9.7 Self-Attention Networks; scaled dot-product attention; causal masking"
---

The [recurrent network](/natural-language-processing/sequences/rnns-and-lstms) reads a
sentence one word at a time, carrying a hidden state forward. That design has two costs
that gating only mitigates. The first is **information loss**: a dependency between two
distant words must be copied through every hidden state in between, degrading at each step;
even the LSTM's cell only slows the decay. The second is **serial
computation**: the hidden state at position $t$ needs the hidden state at $t-1$, so the
sequence cannot be processed in parallel — training time grows with sentence length.[^jm-motiv]

**Self-attention** discards the recurrence. Instead of passing information down a chain, it
lets every position look **directly** at every other position and pull in a weighted mixture
of them, in one step. A dependency between word 1 and word 50 is a single comparison rather
than a chain of 50 copies. And because the output at each position is computed independently
of the others, the whole sequence is processed at once, so the computation parallelizes
across a GPU. The **transformer** is a stack of layers built almost
entirely from this one operation.

$$
% caption: A self-attention layer maps an input sequence to an output sequence of the
% same length; unlike an RNN, each output $y_i$ reads directly from every earlier input,
% and the outputs are computed independently and therefore in parallel.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  io/.style={draw, minimum width=8mm, minimum height=6mm, inner sep=1pt},
  lay/.style={draw, fill=black!4, minimum width=68mm, minimum height=8mm}]
  \definecolor{acc}{HTML}{2348F2}
  \foreach \i in {1,...,5} \node[io] (x\i) at (\i*1.3,0) {x\i};
  \node[lay] (L) at (3.9,1.6) {self-attention layer};
  \foreach \i in {1,...,5} \node[io, draw=acc, text=acc] (y\i) at (\i*1.3,3.2) {y\i};
  % inputs feed the layer
  \foreach \i in {1,...,5} \draw[->, black] (x\i) -- (L);
  % layer to each output
  \foreach \i in {1,...,5} \draw[->, acc] (L) -- (y\i);
  \node[anchor=north, font=\scriptsize, text=black] at (3.9,-0.6) {inputs (one vector per token)};
  \node[anchor=south, font=\scriptsize, text=black] at (3.9,3.7) {outputs (same length)};
\end{tikzpicture}
$$

The two costs are really one geometric fact: in a recurrent network the number of steps a
signal travels between two positions is their _distance_ in the sequence, so a dependency
between token $1$ and token $50$ passes through $49$ hidden states, and the training gradient
that would tie them passes through $49$ multiplications. In self-attention that path length is
$1$ for every pair — token $1$ and token $50$ are one dot product apart, no matter the gap.
Shorter paths mean less information decay on the forward pass and less gradient decay on the
backward pass, and they are why attention learns long-range structure that recurrence
struggles with.

$$
% caption: Path length between two positions. In an RNN a dependency between token 1 and
% token 5 must traverse every hidden state between them (length 4); in self-attention every
% pair is a single hop (length 1), regardless of distance.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  nd/.style={draw, circle, minimum size=6mm, inner sep=0pt, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  % RNN row
  \node[anchor=east, font=\scriptsize, text=black] at (-0.3,0) {RNN};
  \foreach \i in {1,...,5} \node[nd] (r\i) at (\i*1.3,0) {\i};
  \foreach \i/\j in {1/2,2/3,3/4,4/5} \draw[->, black] (r\i) -- (r\j);
  \node[anchor=west, font=\scriptsize, text=black] at (7.5,0) {path 1 to 5: length 4};
  % attention row
  \node[anchor=east, font=\scriptsize, text=acc] at (-0.3,-1.8) {attn};
  \foreach \i in {1,...,5} \node[nd, draw=acc] (a\i) at (\i*1.3,-1.8) {\i};
  \draw[->, acc, thick] (a1) to[bend right=32] (a5);
  \node[anchor=west, font=\scriptsize, text=acc] at (7.5,-1.8) {path 1 to 5: length 1};
\end{tikzpicture}
$$

## Attention as a soft lookup

Attention already appeared, in miniature, at the end of the recurrent chapter: the decoder
of an encoder-decoder model got to look back at all the encoder's hidden states, weighting
each by relevance. **Self-attention** applies that same idea inside a single sequence — a
word attends to the other words of its own sentence rather than to a separate source
text.[^jm-selfattn]

Start with the simplest version. Each input is a vector $\mathbf{x}_i$, and we want an output
$\mathbf{y}_i$ that summarizes how $\mathbf{x}_i$ relates to the words around it. Relatedness
is measured by a **dot product**: $\mathbf{x}_i \cdot \mathbf{x}_j$ is large when the two
vectors point the same way. Compute one such **score** for every earlier position, normalize
the scores with a softmax into weights $\alpha_{ij}$ that sum to one, and read off the
weighted average of the inputs:

$$
\mathrm{score}(\mathbf{x}_i, \mathbf{x}_j) = \mathbf{x}_i \cdot \mathbf{x}_j,
\qquad
\alpha_{ij} = \mathrm{softmax}\big(\mathrm{score}(\mathbf{x}_i, \mathbf{x}_j)\big)
\;\; \forall j \le i,
\qquad
\mathbf{y}_i = \sum_{j \le i} \alpha_{ij}\,\mathbf{x}_j.
$$

This is a **soft dictionary lookup**: the query $\mathbf{x}_i$ is compared against a set of
entries, the comparison scores become a probability distribution, and the answer is the
distribution-weighted blend of the entries. Nothing is recurrent, and every $\mathbf{y}_i$
is a separate sum, so all of them can be computed simultaneously.[^jm-core]

## Queries, keys, and values

Using the same vector $\mathbf{x}_i$ for all three jobs — the thing doing the comparing, the
thing being compared against, and the thing being summed — is wasteful, because those are
three genuinely different roles. Transformers give each role its own learned projection of
the input.[^jm-qkv]

> **Definition (Query, key, value).** From each input $\mathbf{x}_i$, three vectors are
> produced by three learned weight matrices: the **query** $\mathbf{q}_i = \mathbf{W}^Q
> \mathbf{x}_i$ (this word as the current focus of attention), the **key** $\mathbf{k}_i =
> \mathbf{W}^K \mathbf{x}_i$ (this word as something to be compared against), and the
> **value** $\mathbf{v}_i = \mathbf{W}^V \mathbf{x}_i$ (this word's contribution to the
> output). Attention scores a query against keys; the softmax weights then mix the values.

The score is now a dot product between the current word's query and each earlier word's key,
and the output is the weighted sum of the corresponding **values** rather than of the raw
inputs:

$$
\mathrm{score}(\mathbf{x}_i, \mathbf{x}_j) = \mathbf{q}_i \cdot \mathbf{k}_j,
\qquad
\mathbf{y}_i = \sum_{j \le i} \alpha_{ij}\,\mathbf{v}_j .
$$

The three matrices $\mathbf{W}^Q, \mathbf{W}^K, \mathbf{W}^V$ are the only learned parameters
of the attention layer; they are shared across all positions and trained end-to-end with the
rest of the network. Query and key vectors have dimension $d_k$, values have dimension $d_v$,
and the model as a whole works in dimension $d_{model}$; a single-head layer takes
$d_k = d_v = d_{model}$, though the multi-head version below will pull them apart.[^jm-qkv]

$$
% caption: One output $y_i$ from queries, keys, and values: the query $q_i$ is compared
% (dot product) against each earlier key $k_j$, the scores are softmaxed into weights, and
% the output is the weighted sum of the value vectors $v_j$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  tok/.style={draw, minimum width=8mm, minimum height=6mm, inner sep=1pt},
  qkv/.style={draw, minimum width=6mm, minimum height=5mm, inner sep=1pt, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % three earlier tokens
  \foreach \i/\x in {1/0, 2/2.4, 3/4.8} \node[tok] (x\i) at (\x,0) {x\i};
  % keys and values under each
  \foreach \i/\x in {1/0, 2/2.4, 3/4.8} {
    \node[qkv] (k\i) at (\x-0.55,1.2) {k\i};
    \node[qkv] (v\i) at (\x+0.55,1.2) {v\i};
    \draw[->, black] (x\i) -- (k\i);
    \draw[->, black] (x\i) -- (v\i);
  }
  % the current query, drawn from x3 (the focus)
  \node[qkv, draw=red, text=red] (q) at (6.6,1.2) {q3};
  \draw[->, red] (x3) to[bend right=12] (q);
  % scores: query dotted with each key
  \node[draw, minimum width=20mm, minimum height=6mm, fill=black!4] (sm) at (2.4,2.8)
    {softmax scores};
  \foreach \i in {1,2,3} \draw[->, red, dashed] (q) to[bend left=8] (k\i);
  \foreach \i in {1,2,3} \draw[->, black] (k\i) -- (sm);
  % weighted sum of values
  \node[tok, draw=acc, text=acc] (y) at (2.4,4.2) {y3};
  \draw[->, acc, thick] (sm) -- (y) node[pos=0.3, right, font=\scriptsize] {weight and sum values};
  \foreach \i in {1,2,3} \draw[->, black] (v\i) to[bend right=14] (y);
\end{tikzpicture}
$$

### Why the $\sqrt{d_k}$ scaling

A dot product of two $d_k$-dimensional vectors sums $d_k$ products. If the components are
roughly independent with unit variance, the dot product has variance proportional to $d_k$,
so as the dimension grows the scores spread out into large positive and negative values.
Feed large values into a softmax and it saturates: almost all the probability collapses onto
the single largest score, and the gradient with respect to the others goes to zero, which
stalls learning.[^jm-scale]

The fix is to divide the score by $\sqrt{d_k}$ before the softmax, rescaling the variance
back to a constant regardless of dimension. This is **scaled dot-product attention**:

$$
\mathrm{score}(\mathbf{x}_i, \mathbf{x}_j) =
\frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d_k}} .
$$

Intuitively: a softmax turns a gap between two scores into a gap between two probabilities,
but the size of the probability gap depends on the scale of the scores. Left unscaled, that
scale grows with dimension, so at large $d_k$ even an ordinary gap saturates the softmax.
Dividing by $\sqrt{d_k}$ holds the scale fixed, so the same-sized gap always produces the
same-sized probability gap.

To see why the constant matters, make the argument quantitative. Suppose two keys produce
raw dot products with the query that differ by one standard deviation of the score
distribution. Because that distribution has standard deviation $\sqrt{d_k}$, a one-sigma gap
is a raw difference of $\sqrt{d_k}$: the winning logit is $\sqrt{d_k}$, the loser is $0$. Run
those through the softmax and the winner's probability climbs sharply with dimension — at
$d_k = 4$ it wins with probability $0.88$, at $d_k = 64$ with $0.9997$, at $d_k = 1024$ it is
$1.0000$ to four decimals. The distribution has collapsed onto one word, and the gradient
that would nudge the others has vanished. Divide by $\sqrt{d_k}$ first and the same one-sigma
gap becomes a difference of exactly $1$ at every dimension, so the winner's probability holds
steady at $0.73$ regardless of $d_k$. That is the whole point of the scaling: it fixes the
softmax's operating point so the layer learns the same way whether $d_k$ is $4$ or $1024$.

$$
% caption: Without scaling, a one-standard-deviation gap between two scores sends the
% softmax winner's probability toward 1 as $d_k$ grows (blue), saturating the softmax and
% killing gradients. Dividing by $\sqrt{d_k}$ pins the effective gap at 1 and holds the
% winner near 0.73 at every dimension (dashed).
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->] (0,0) -- (6.4,0) node[right, font=\scriptsize] {dimension dk (log)};
  \draw[->] (0,0) -- (0,3.6) node[above, font=\scriptsize] {softmax winner prob};
  \foreach \y/\l in {0/0.5, 1.6/0.75, 3.2/1.0} \node[anchor=east, font=\scriptsize] at (0,\y) {\l};
  \draw[black] (0,3.2) -- (6.0,3.2);
  % unscaled: rises to 1 (dk = 4, 16, 64, 256, 1024): probs .88,.98,.9997,1,1
  \draw[acc, very thick] (0.8,2.43) -- (2.0,3.05) -- (3.2,3.19) -- (4.4,3.2) -- (5.6,3.2);
  \node[acc, anchor=south, font=\scriptsize] at (4.4,3.24) {unscaled};
  % scaled: flat at 0.73
  \draw[red, very thick, dashed] (0.8,1.47) -- (5.6,1.47);
  \node[red, anchor=south, font=\scriptsize] at (2.6,1.5) {scaled by sqrt(dk)};
  \foreach \x/\l in {0.8/4, 2.0/16, 3.2/64, 4.4/256, 5.6/1024} \node[anchor=north, font=\scriptsize] at (\x,0) {\l};
\end{tikzpicture}
$$

### A numeric trace of one attention step

For example, take three tokens and work in $d_k = d_v = 2$, computing the output $\mathbf{y}_3$ for the third position, which (being causal) attends
to positions $1$, $2$, and $3$. Suppose the projections have already produced these query,
key, and value vectors:

$$
\mathbf{q}_3 = (1, 1),
\qquad
\mathbf{k}_1 = (1, 0),\;\; \mathbf{k}_2 = (0, 1),\;\; \mathbf{k}_3 = (1, 1),
\qquad
\mathbf{v}_1 = (1, 0),\;\; \mathbf{v}_2 = (0, 2),\;\; \mathbf{v}_3 = (2, 2).
$$

**Step 1 — score.** Dot the query against each key:
$\mathbf{q}_3 \cdot \mathbf{k}_1 = 1$, $\mathbf{q}_3 \cdot \mathbf{k}_2 = 1$,
$\mathbf{q}_3 \cdot \mathbf{k}_3 = 2$.

**Step 2 — scale.** Divide by $\sqrt{d_k} = \sqrt{2} \approx 1.414$:
$0.707,\; 0.707,\; 1.414$.

**Step 3 — softmax.** Exponentiate and normalize. The weights come out
$\alpha_{3,1} = 0.248$, $\alpha_{3,2} = 0.248$, $\alpha_{3,3} = 0.503$, summing to $1$.
Position $3$ attends most to itself (its key matched the query best) and splits the rest
evenly between positions $1$ and $2$.

**Step 4 — weighted sum of values.**

$$
\mathbf{y}_3 = 0.248\,(1,0) + 0.248\,(0,2) + 0.503\,(2,2)
= (1.255,\; 1.503).
$$

The output is a blend of the three value vectors, tilted toward $\mathbf{v}_3$. Two details.
First, the scaling changed the answer: with the raw scores $1,1,2$ the
softmax would give weights $0.212, 0.212, 0.576$, a sharper peak on position $3$; the
$\sqrt{2}$ divisor softened the distribution, as intended. Second, nothing here
touched positions $4$ or beyond; had this been an encoder (bidirectional) layer, the query
would also have scored against future keys, but the causal mask forbids it.

$$
% caption: The numeric trace for $\mathbf{y}_3$. Query $(1,1)$ scores against the three
% keys, the scores are scaled by $\sqrt{2}$ and softmaxed into weights
% $(0.248, 0.248, 0.503)$, and those weights mix the value vectors into the output
% $(1.255, 1.503)$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  cell/.style={draw, minimum width=15mm, minimum height=6mm, inner sep=1pt, font=\scriptsize},
  wcell/.style={draw=acc, text=acc, minimum width=15mm, minimum height=6mm, inner sep=1pt, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[anchor=east, font=\scriptsize, text=black] at (-0.2,2.4) {key};
  \node[anchor=east, font=\scriptsize, text=black] at (-0.2,1.6) {raw score};
  \node[anchor=east, font=\scriptsize, text=black] at (-0.2,0.8) {scaled};
  \node[anchor=east, font=\scriptsize, text=black] at (-0.2,0.0) {weight};
  \foreach \c/\k/\r/\s/\a in {1/{(1,0)}/1.0/0.707/0.248, 2/{(0,1)}/1.0/0.707/0.248, 3/{(1,1)}/2.0/1.414/0.503} {
    \node[cell]  at (\c*1.9,2.4) {k\c = \k};
    \node[cell]  at (\c*1.9,1.6) {\r};
    \node[cell]  at (\c*1.9,0.8) {\s};
    \node[wcell] at (\c*1.9,0.0) {\a};
  }
  \node[cell, draw=acc, text=acc, minimum width=24mm] (y) at (3.8,-1.2) {y3 = (1.255, 1.503)};
  \foreach \c in {1,2,3} \draw[->, acc!70] (\c*1.9,-0.35) -- (y.north);
\end{tikzpicture}
$$

### The whole layer as one matrix product

Because every output is independent, the layer has an efficient batched form. Pack the $N$
input vectors as rows of a matrix $\mathbf{X} \in \mathbb{R}^{N \times d}$ and project them
all at once:

$$
\mathbf{Q} = \mathbf{X}\mathbf{W}^Q,
\qquad
\mathbf{K} = \mathbf{X}\mathbf{W}^K,
\qquad
\mathbf{V} = \mathbf{X}\mathbf{W}^V .
$$

The product $\mathbf{Q}\mathbf{K}^\top$ is an $N \times N$ matrix holding every query-key
score at once; scale it, softmax each row, and multiply by $\mathbf{V}$. The entire
self-attention computation for a sequence of $N$ tokens collapses to a single expression:

$$
\mathrm{SelfAttention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) =
\mathrm{softmax}\!\left(\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d_k}}\right)\mathbf{V} .
$$

This is the equation the entire architecture is built around. Note its cost: forming
$\mathbf{Q}\mathbf{K}^\top$ requires $N^2$ dot products, so attention is **quadratic** in the
sequence length. That quadratic cost is why transformer inputs are usually capped at a page
or a paragraph, and why more efficient attention variants are an active research
area.[^jm-matrix]

## Causal (masked) self-attention

For a **language model** the constraint is absolute: predicting the next word may use only
the words before it. If position $i$ were allowed to attend to position $i+1$, the model
would see the answer it is supposed to predict. So a language-model attention layer must be
**causal** — each query attends only to keys at its own position or earlier, exactly the
$j \le i$ condition written into the sums above.[^jm-mask]

In the matrix form the full $\mathbf{Q}\mathbf{K}^\top$ scores every query against every key,
including future ones. Those forbidden entries sit in the **upper triangle** of the matrix.
To forbid them, set the upper triangle to $-\infty$ before the softmax; $e^{-\infty} = 0$, so
every future key receives exactly zero weight and contributes nothing to the output.

$$
% caption: The causal mask on the $N \times N$ score matrix. Each row $i$ keeps the scores
% for keys $j \le i$ (lower triangle, shaded) and sets keys $j > i$ (upper triangle) to
% minus infinity, so the softmax assigns them zero weight.
\begin{tikzpicture}[>=stealth, font=\scriptsize]
  \definecolor{acc}{HTML}{2348F2}
  \def\s{0.9}
  \foreach \r in {1,...,5} {
    \foreach \c in {1,...,5} {
      \ifnum\c>\r
        \node[draw, minimum size=\s cm, inner sep=0pt, fill=black!3, text=black]
          at (\c*\s,-\r*\s) {-inf};
      \else
        \node[draw, minimum size=\s cm, inner sep=0pt, fill=acc!12]
          at (\c*\s,-\r*\s) {q\r k\c};
      \fi
    }
  }
  \node[anchor=south, font=\footnotesize] at (3*\s,-0.15) {keys (j)};
  \node[rotate=90, anchor=south, font=\footnotesize] at (0.15,-3*\s) {queries (i)};
\end{tikzpicture}
$$

The same layer, without the mask, is **bidirectional**: every position attends to the whole
sequence in both directions. Bidirectional self-attention is what an encoder uses, where the
full input is available at once; causal self-attention is what a decoder and any left-to-right
language model use. It is the identical operation with one triangular mask flipped on or
off.[^jm-mask]

With the attention operation in hand — a query scored against keys, softmaxed into weights, mixing values, optionally masked to be causal — we can assemble the full network. That is the job of the next part, which stacks attention into multiple heads, wraps it in the transformer block, restores word order, and arrives at the architectures modern NLP is built on. This continues in [The Transformer Architecture](/natural-language-processing/transformers/the-transformer-architecture).

[^jm-motiv]: **Jurafsky & Martin**, _Speech and Language Processing_ (3rd ed.), §9.7 — Self-Attention Networks: recurrent connections cause information loss over long spans and force serial computation; transformers eliminate recurrence to allow direct access to arbitrarily distant context and full parallelism.
[^jm-selfattn]: **Jurafsky & Martin**, §9.7 — Self-Attention Networks (Fig. 9.15): a self-attention layer maps a sequence to a sequence of the same length, each output attending to the inputs up to and including its own position, with per-position computations independent and hence parallelizable.
[^jm-core]: **Jurafsky & Martin**, §9.7, Eqs. 9.27–9.30 — the core attention computation: a dot-product score between elements, a softmax normalization into weights $\alpha_{ij}$, and a weighted sum of the inputs.
[^jm-qkv]: **Jurafsky & Martin**, §9.7, Eqs. 9.31–9.33 — the query, key, and value roles and their projection matrices $\mathbf{W}^Q, \mathbf{W}^K, \mathbf{W}^V$; the score as query-key dot product and the output as a weighted sum over value vectors, with dimensions $d_k$ and $d_v$.
[^jm-scale]: **Jurafsky & Martin**, §9.7, Eq. 9.34 — scaled dot-product attention: dividing the score by $\sqrt{d_k}$ keeps the dot product from growing with dimension and saturating the softmax, which would otherwise cause vanishing gradients.
[^jm-matrix]: **Jurafsky & Martin**, §9.7, Eqs. 9.35–9.36 — the packed matrix form $\mathrm{softmax}(\mathbf{Q}\mathbf{K}^\top/\sqrt{d_k})\mathbf{V}$; the $N \times N$ score matrix makes attention quadratic in sequence length.
[^jm-mask]: **Jurafsky & Martin**, §9.7 (Fig. 9.17) — causal (masked) self-attention: for language modeling the upper-triangular scores over future tokens are set to $-\infty$ so the softmax zeros them, restricting each query to keys at or before its position.