---
title: Large Language Models
module: Transformers
moduleNumber: 5
lessonNumber: 3
order: 503
summary: >
  A large language model is a decoder-only transformer trained on one objective —
  predict the next token. This first part assembles the inference side: the
  language-modeling head that turns a hidden state into a distribution over the
  vocabulary, autoregressive generation, and the decoding strategies — greedy,
  beam, and sampling with temperature, top-k, and nucleus — that read text back
  out of that distribution. Training the distribution at web scale comes next.
topics: [Transformers]
sources:
  - book: Jurafsky
    ref: "§9.3 RNNs as Language Models; §9.4 Weight Tying; §9.8 Transformers as Language Models; §9.9 Contextual Generation and Summarization"
  - book: Jurafsky
    ref: "Ch. 10 — Machine Translation and Encoder-Decoder Models; §10.5 Beam Search"
  - book: Jurafsky
    ref: "Ch. 3 — N-gram Language Models; §3.3 Sampling Sentences from a Language Model"
---

The [transformer](/natural-language-processing/transformers/transformers-and-attention)
gave us a block that reads a sequence and, at every position, produces a vector
that has attended to everything before it. A **large language model** is a
decoder-only stack of those blocks trained on the single task of predicting the
next word, over a large fraction of the written web, at large scale. The
ingredients are one architecture, one objective, and scale; nothing else is new.

The objective is the same one the [n-gram
model](/natural-language-processing/foundations/n-gram-language-models) and the
[neural language
model](/natural-language-processing/semantics/neural-language-models) were built
around: assign a probability to the next token given the tokens so far. Writing
$w_{<t} = w_1, \ldots, w_{t-1}$ for the prefix, the model estimates

$$
P(w_t \mid w_{<t}),
$$

and the probability of a whole sequence factors, by the chain rule, into a product
of these conditionals:

$$
P(w_{1:n}) \;=\; \prod_{t=1}^{n} P(w_t \mid w_{<t}).
$$

The transformer's only job is to compute the right-hand side well. Everything in
this lesson is either how the block turns its hidden state into that distribution,
how we read text out of the distribution, or how we drive the distribution to be
accurate by training on enough text.

## The language-modeling head

A stack of $L$ transformer blocks maps an input sequence to a sequence of hidden
vectors. At position $t$ the top block emits $\mathbf{h}_t \in \mathbb{R}^{d}$, a
$d$-dimensional summary of the prefix that has attended, through the causal mask,
to $w_1, \ldots, w_t$ and no further.[^slp-transformer-lm] To turn that vector into
a prediction we need a map from $\mathbb{R}^{d}$ to a probability distribution over
the vocabulary $V$. This map is the **language-modeling head**, and it is two
operations: a linear projection followed by a softmax.

The projection is called the **unembedding**. A matrix $\mathbf{U} \in
\mathbb{R}^{|V| \times d}$ scores every word in the vocabulary against the hidden
state, producing one **logit** per word:

$$
\mathbf{z}_t \;=\; \mathbf{U}\,\mathbf{h}_t \;\in\; \mathbb{R}^{|V|}.
$$

Each entry $z_t[i]$ measures how well word $i$ matches the evidence in
$\mathbf{h}_t$; the $i$-th row of $\mathbf{U}$ is effectively a second embedding of
word $i$, and the logit is its inner product with the hidden state. The softmax
then normalizes the logits into a distribution:

$$
\mathbf{y}_t \;=\; \softmax(\mathbf{z}_t),
\qquad
P(w_{t+1} = i \mid w_{1:t}) \;=\; \mathbf{y}_t[i].
$$

> **Definition (Language-modeling head).** The final layer that converts a
> transformer's top-level hidden vector $\mathbf{h}_t$ into a distribution over the
> vocabulary: an **unembedding** projection $\mathbf{z}_t = \mathbf{U}\mathbf{h}_t$
> to a logit per word, followed by a softmax. The output $\mathbf{y}_t$ is the
> model's estimate of $P(w_{t+1} \mid w_{1:t})$.

$$
% caption: The language-modeling head. The top transformer block emits a hidden
% state $\mathbf{h}_t$; the unembedding $\mathbf{U}$ scores every word in the
% vocabulary, giving a logit vector $\mathbf{z}_t$, and a softmax normalizes it
% into the next-token distribution $\mathbf{y}_t = \mathrm{softmax}(\mathbf{U}\mathbf{h}_t)$.
\begin{tikzpicture}[>=stealth, font=\small,
  box/.style={draw, minimum width=30mm, minimum height=10mm, align=center, font=\footnotesize},
  bar/.style={draw, fill=black!8, minimum width=3mm}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box, draw=acc, text=acc, thick] (h)  at (0,3.0) {hidden state h(t)\\(top block output)};
  \node[box] (u)  at (0,1.5) {unembedding U};
  \node[box] (z)  at (0,0.0) {logits z(t) = U h(t)};
  \node[box] (s)  at (0,-1.5) {softmax};
  \draw[->, acc, thick] (h) -- (u);
  \draw[->, acc, thick] (u) -- (z);
  \draw[->, acc, thick] (z) -- (s);
  % distribution bars to the right
  \begin{scope}[xshift=52mm, yshift=-18mm]
    \foreach \x/\ht in {0/0.5, 0.45/1.4, 0.9/0.3, 1.35/2.1, 1.8/0.7, 2.25/0.4, 2.7/1.1}
      \fill[acc!70] (\x,0) rectangle ++(0.3,\ht);
    \draw[->] (-0.2,0) -- (3.3,0) node[right, font=\scriptsize] {vocabulary};
    \node[font=\scriptsize, anchor=south] at (1.35,2.15) {argmax};
    \node[acc, font=\footnotesize, anchor=south west] at (0,2.6) {y(t): P(next word)};
  \end{scope}
  \draw[->, acc, thick] (s.east) .. controls (2.6,-1.5) and (3.0,-1.5) .. (3.4,-1.5);
\end{tikzpicture}
$$

The unembedding matrix is where most of a model's parameters can pile up, since
$|V|$ is large — tens of thousands of subword tokens is typical. A standard trick
is **weight tying**: reuse the input embedding matrix $\mathbf{E}$ as the
unembedding, setting $\mathbf{U} = \mathbf{E}$.[^slp-weight-tying] The rows of
$\mathbf{E}$ already learn a vector per word at the input; the rows of
$\mathbf{U}$ must learn one per word at the output; there is no reason to learn
two. Tying them improves perplexity and removes an entire $|V| \times d$ block of
parameters.

> **Definition (Weight tying).** Sharing one matrix between the input embedding
> and the output unembedding, $\mathbf{U} = \mathbf{E}$, so a single set of word
> vectors serves both to look words up on the way in and to score them on the way
> out. It reduces parameters and typically lowers perplexity.

## Autoregressive generation

The head gives us a distribution over the next token. To generate text we sample
one token from it, append that token to the context, and run the model forward
again — now conditioned on the token we just produced. Repeating this is
**autoregressive generation**: the word chosen at each step becomes part of the
input for the next.[^slp-autoregressive]

The word "autoregressive" is borrowed loosely. A true autoregressive model
predicts a value from a linear function of its own past values; a transformer
language model is deeply non-linear, but it shares the defining feature — each
output is conditioned on the outputs already produced.

```algorithm
caption: $\textsc{Generate}$ — autoregressive decoding from a language model
input: model $P_\theta$, prompt tokens $w_{1:k}$, max length $n$
$t \gets k$
repeat
  $\mathbf{y}_t \gets P_\theta(\cdot \mid w_{1:t})$ // next-token distribution from the head
  $w_{t+1} \gets \textsc{Decode}(\mathbf{y}_t)$ // pick a token: argmax, sample, ...
  append $w_{t+1}$ to the context
  $t \gets t + 1$
until $w_{t}$ is the end-of-sequence token or $t = n$
return $w_{k+1:t}$
```

The line ``$\textsc{Decode}(\mathbf{y}_t)$`` is the only free choice, and it is the
whole subject of the next section. Priming the context is what makes generation
practically useful. Instead of starting from a bare start-of-sequence marker, seed the
model with a task-appropriate prefix: a source sentence for translation, a long
article for summarization, a question for question answering.[^slp-contextual] The
model does not know it is "doing a task" — it only continues the text — but a
prefix that makes the desired output the most probable continuation turns
next-token prediction into a general-purpose interface.

## Decoding strategies

Given the distribution $\mathbf{y}_t$, how do we commit to an actual token? The
choice trades off two failure modes. Always take the most probable token and the
text is fluent but flat, and often globally suboptimal; sample too freely from the
tail and the text drifts into incoherence. Decoding strategies sit on that
spectrum.

### Greedy decoding

The simplest rule takes the single most probable token at every step:

$$
\hat{w}_{t+1} \;=\; \argmax_{w \in V} P(w \mid w_{1:t}).
$$

This is **greedy decoding**.[^slp-greedy] It is fast and deterministic, but greedy
is a local choice, and the locally best token can strand the model on a globally
worse sequence. A token that looks best now may leave no good continuation, while a
slightly-less-probable token now might open onto a much more probable completion.
Greedy search cannot see that far ahead.

### Beam search

**Beam search** addresses this by keeping the $k$ best partial
sequences at once instead of one.[^slp-beam] The number $k$ is the **beam
width**. At each step, every surviving sequence — a **hypothesis** — is extended by
every possible next token, all $k \times |V|$ candidates are scored by their total
log-probability, and only the best $k$ are kept. A hypothesis that ends in the
end-of-sequence token is complete and set aside; the search runs until the beam
empties.

$$
% caption: Beam search with width $k=2$. From each of the two surviving
% hypotheses (thick blue), all vocabulary extensions are scored by cumulative
% log-probability; only the best two survive to the next column, while greedy
% (dashed) commits to a single locally-best path and cannot recover.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  hyp/.style={draw, minimum width=13mm, minimum height=6mm, font=\scriptsize},
  keep/.style={draw=acc, text=acc, thick}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[hyp, keep] (s) at (0,0) {start};
  % step 1
  \node[hyp, keep] (a) at (3,1.3) {the};
  \node[hyp, keep] (b) at (3,0)   {a};
  \node[hyp] (c) at (3,-1.3) {an};
  \draw[->, acc] (s) -- (a);  \draw[->, acc] (s) -- (b);
  \draw[->, black] (s) -- (c);
  % step 2
  \node[hyp, keep] (d) at (6.3,1.9) {the cat};
  \node[hyp] (e) at (6.3,0.7) {the dog};
  \node[hyp, keep] (f) at (6.3,-0.5) {a cat};
  \node[hyp] (g) at (6.3,-1.7) {a dog};
  \draw[->, acc] (a) -- (d);  \draw[->, black] (a) -- (e);
  \draw[->, acc] (b) -- (f);  \draw[->, black] (b) -- (g);
  % greedy dashed
  \draw[->, red, dashed, thick] (s) to[bend right=8] (b);
  \draw[->, red, dashed, thick] (b) to[bend right=8] (g);
  \node[red, font=\scriptsize, anchor=north] at (4.6,-1.9) {greedy path};
\end{tikzpicture}
$$

Because probabilities of long strings are tiny, hypotheses are scored in log-space,
where the product becomes a sum that can be extended incrementally:

$$
\score(w_{1:t}) \;=\; \log P(w_{1:t}) \;=\; \sum_{i=1}^{t} \log P(w_i \mid w_{1:i-1}).
$$

One wrinkle: because every term is negative, longer sequences score lower purely
for being long, so a naive beam favors short outputs. The usual fix is **length
normalization** — divide the score by the number of tokens $T$ before
comparing:[^slp-length-norm]

$$
\score(w_{1:T}) \;=\; \frac{1}{T} \sum_{i=1}^{T} \log P(w_i \mid w_{1:i-1}).
$$

Beam search is standard in machine translation, where there is a single correct
answer to search for and widths of $k = 5$ to $10$ are common. For open-ended
generation it is a poor fit: maximizing probability produces bland, repetitive text.
Open-ended generation calls for sampling.

### Sampling

Instead of maximizing, we can **sample** the next token from the distribution
itself, choosing each word with probability proportional to $\mathbf{y}_t[i]$. This
is the same weighted-die idea used to visualize an n-gram model back in the
[foundations](/natural-language-processing/foundations/n-gram-language-models):
lay every word out on the unit interval with width equal to its probability, throw
a dart, and read off the word it lands in.[^slp-sampling] Pure sampling matches the
model's distribution exactly, which is both its strength and its problem — the long
tail of low-probability tokens is collectively heavy, so even a well-trained model
occasionally samples a bizarre word and the text derails. The remaining strategies
all reshape or truncate the distribution before sampling to control that tail.

### Temperature

**Temperature** rescales the logits before the softmax by a factor $\tau > 0$:

$$
\mathbf{y}_t \;=\; \softmax\!\left(\frac{\mathbf{z}_t}{\tau}\right).
$$

Dividing by a small $\tau < 1$ stretches the logits apart, sharpening the
distribution toward its peak — in the limit $\tau \to 0$ sampling becomes greedy.
A large $\tau > 1$ compresses the logits, flattening the distribution toward
uniform and making rare words more likely. Temperature does not remove any word; it
only reweights the distribution, from sharper ($\tau < 1$) to flatter ($\tau > 1$).

$$
% caption: Truncated sampling distributions. Temperature (low $\tau$) sharpens the
% full distribution; top-k keeps the k highest-probability words and renormalizes;
% top-p / nucleus keeps the smallest set whose probabilities sum to p. The kept
% mass is shaded; everything to the right of the cut is discarded.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % bar heights (a decaying distribution)
  \def\heights{{2.4,1.9,1.5,1.1,0.8,0.55,0.38,0.26,0.18,0.12}}
  % --- top-k panel ---
  \begin{scope}
    \node[font=\footnotesize, anchor=south] at (2.3,2.7) {top-k (k = 4)};
    \foreach \i in {0,...,9} {
      \pgfmathsetmacro\hh{\heights[\i]}
      \ifnum\i<4 \fill[acc!70] (\i*0.46,0) rectangle ++(0.34,\hh);
      \else \fill[black] (\i*0.46,0) rectangle ++(0.34,\hh); \fi
    }
    \draw[red, thick] (4*0.46-0.06,-0.15) -- (4*0.46-0.06,2.5);
    \node[red, font=\scriptsize, anchor=south west] at (4*0.46,2.0) {cut};
    \draw[->] (-0.1,0) -- (4.9,0) node[right, font=\scriptsize] {words};
  \end{scope}
  % --- top-p panel ---
  \begin{scope}[xshift=64mm]
    \node[font=\footnotesize, anchor=south] at (2.3,2.7) {top-p (p = 90 percent)};
    \foreach \i in {0,...,9} {
      \pgfmathsetmacro\hh{\heights[\i]}
      \ifnum\i<6 \fill[acc!70] (\i*0.46,0) rectangle ++(0.34,\hh);
      \else \fill[black] (\i*0.46,0) rectangle ++(0.34,\hh); \fi
    }
    \draw[red, thick] (6*0.46-0.06,-0.15) -- (6*0.46-0.06,2.5);
    \node[red, font=\scriptsize, anchor=south west] at (6*0.46,1.4) {mass = p};
    \draw[->] (-0.1,0) -- (4.9,0) node[right, font=\scriptsize] {words};
  \end{scope}
\end{tikzpicture}
$$

### Top-k and top-p (nucleus) sampling

The tail is heavy because it is long, not because any one word in it is likely.
**Truncated sampling** cuts the tail off and renormalizes what remains, then samples
from that.

- **Top-k sampling** keeps the $k$ highest-probability words, zeros the rest, and
  renormalizes. Sampling then never strays outside the $k$ most plausible
  continuations. The weakness is that $k$ is fixed: when the model is confident,
  $k$ words include junk; when it is unsure, $k$ words cut off good options.
- **Top-p sampling**, also called **nucleus sampling**, fixes that by cutting on
  probability mass instead of count. Sort the vocabulary by probability and keep the
  smallest set — the **nucleus** — whose cumulative probability first reaches a
  threshold $p$ (say $0.9$), discarding the rest. The cutoff adapts: a peaked
  distribution keeps few words, a flat one keeps many.

> **Definition (Nucleus sampling).** Sampling restricted to the smallest set of
> top-ranked words whose probabilities sum to at least $p$. Formally, keep the
> smallest $V^{(p)} \subseteq V$ with $\sum_{w \in V^{(p)}} P(w \mid w_{<t}) \ge p$,
> renormalize over $V^{(p)}$, and sample. Unlike top-k, the number of retained
> words varies with the model's confidence at each step.

In practice temperature, top-k, and top-p compose: a generation call might set
$\tau = 0.8$ and $p = 0.95$ together, sharpening slightly and then clipping the
tail. Greedy and beam sit at one end (deterministic, probability-maximizing);
temperature and truncated sampling occupy the other (stochastic, diversity-seeking).

### A worked decoding step

Carry one step through with numbers. Suppose the head has produced logits over six candidate
next words after the prefix _the cat_:

$$
\mathbf{z} = (\underbrace{3.0}_{\text{sat}},\; \underbrace{2.0}_{\text{ran}},\;
\underbrace{1.0}_{\text{slept}},\; \underbrace{0.5}_{\text{on}},\;
\underbrace{0.0}_{\text{the}},\; \underbrace{-1.0}_{\text{purple}}).
$$

At the default temperature $\tau = 1$ the softmax gives probabilities
$(0.605, 0.222, 0.082, 0.050, 0.030, 0.011)$. **Greedy** takes _sat_. **Pure sampling** takes
_sat_ about $60\%$ of the time but leaves real mass on the rest, including $1.1\%$ on the
implausible _purple_. Now apply each knob:

- **Temperature $\tau = 0.7$** divides the logits by $0.7$ before the softmax, sharpening the
  distribution to $(0.745, 0.179, 0.043, 0.021, 0.010, 0.002)$ — _sat_ gains, the tail shrinks.
  Temperature $\tau = 1.5$ does the reverse, flattening to $(0.461, 0.237, 0.121, 0.087,
  0.062, 0.032)$, which raises every rare word.
- **Nucleus ($p = 0.9$)** at $\tau = 1$ sorts by probability and accumulates: _sat_ ($0.605$),
  _ran_ ($0.605 + 0.222 = 0.827$), _slept_ ($0.827 + 0.082 = 0.909 \ge 0.9$). The nucleus is
  $\{\text{sat}, \text{ran}, \text{slept}\}$; the other three words are discarded and the kept
  three are renormalized to sum to one. Even a single unlucky draw of _purple_ is now
  impossible, while the plausible alternatives _ran_ and _slept_ stay available.

The two mechanisms are orthogonal: temperature reshapes the whole distribution, truncation
deletes the tail. Composing $\tau = 0.7$ then $p = 0.9$ would first sharpen, then cut — the
usual production setting for text that is varied but never derails.

$$
% caption: One decoding step over six candidates. Default softmax ($\tau = 1$) leaves a long
% tail; low temperature ($\tau = 0.7$) sharpens toward the top word; nucleus sampling
% ($p = 0.9$) keeps the top three words whose mass first reaches 0.9 (shaded) and discards
% the rest.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \begin{scope}
    \node[font=\footnotesize, anchor=south] at (1.6,3.2) {softmax (tau = 1)};
    % index / height / word ; first three (i<3) are the nucleus
    \foreach \i/\hh/\w in {0/2.42/sat, 1/0.89/ran, 2/0.33/slept} {
      \fill[acc!70] (\i*0.62,0) rectangle ++(0.46,\hh);
      \node[font=\scriptsize, anchor=north, rotate=40] at (\i*0.62+0.23,-0.05) {\w};
    }
    \foreach \i/\hh/\w in {3/0.20/on, 4/0.12/the, 5/0.04/purple} {
      \fill[black] (\i*0.62,0) rectangle ++(0.46,\hh);
      \node[font=\scriptsize, anchor=north, rotate=40] at (\i*0.62+0.23,-0.05) {\w};
    }
    \draw[red, thick] (3*0.62-0.08,-0.1) -- (3*0.62-0.08,2.7);
    \node[red, font=\scriptsize, anchor=south west] at (3*0.62,1.9) {nucleus cut (p=0.9)};
  \end{scope}
  \begin{scope}[xshift=58mm]
    \node[font=\footnotesize, anchor=south] at (1.6,3.2) {low temp (tau = 0.7)};
    \foreach \i/\hh/\w in {0/2.98/sat, 1/0.72/ran, 2/0.17/slept, 3/0.08/on, 4/0.04/the, 5/0.01/purple} {
      \fill[acc!70] (\i*0.62,0) rectangle ++(0.46,\hh);
      \node[font=\scriptsize, anchor=north, rotate=40] at (\i*0.62+0.23,-0.05) {\w};
    }
  \end{scope}
\end{tikzpicture}
$$

## From reading text out to putting it in

Everything so far assumes the distribution $P_\theta(w_t \mid w_{<t})$ is already good — the head shapes it, autoregression rolls it forward, and a decoding strategy reads text out of it. But that distribution is only as good as the parameters behind it, and those come from training on an enormous amount of text. The next part is where the parameters come from: self-supervised pretraining at web scale, the scaling laws that make its loss predictable, the KV cache that makes long-context inference affordable, and how the finished model is evaluated. This continues in [Large Language Models: Pretraining and Scaling](/natural-language-processing/transformers/llm-pretraining-and-scaling).

[^slp-transformer-lm]: **Jurafsky & Martin**, _Speech and Language Processing_ (3rd ed.), §9.8 — Transformers as Language Models: at each position the final transformer layer produces an output distribution over the vocabulary, trained by teacher forcing to predict the next word.
[^slp-weight-tying]: **Jurafsky & Martin**, §9.4 — Weight Tying: the input embedding matrix $\mathbf{E}$ and the final projection $\mathbf{U}$ both hold a set of word vectors of shape $|V| \times d_h$; tying them ($\mathbf{E} = \mathbf{U}$) improves perplexity and reduces parameter count.
[^slp-autoregressive]: **Jurafsky & Martin**, §9.3 (Autoregressive Generation): incrementally generate by sampling the next word from the softmax, feeding each sampled word back as the next input, until an end-of-sequence marker or a length limit is reached.
[^slp-contextual]: **Jurafsky & Martin**, §9.9 — Contextual Generation and Summarization: priming autoregressive generation with a task-appropriate context (a source sentence, an article) drives translation, summarization, and question answering from the same next-token machinery.
[^slp-greedy]: **Jurafsky & Martin**, §10.5 — Beam Search: choosing the single most probable token at each step, $\hat{y}_t = \argmax_{w} P(w \mid x, y_{<t})$, is greedy decoding; it is locally optimal and can miss the globally most probable sequence.
[^slp-beam]: **Jurafsky & Martin**, §10.5 — Beam Search: keep $k$ hypotheses (the beam width), extend each by every vocabulary item, score by cumulative log-probability, and prune back to the best $k$; production MT uses widths of roughly $5$–$10$.
[^slp-length-norm]: **Jurafsky & Martin**, §10.5 — Beam Search: because models assign lower probability to longer strings, hypotheses are length-normalized by dividing the summed log-probability by the number of tokens $T$ before comparison.
[^slp-sampling]: **Jurafsky & Martin**, §3.3 — Sampling Sentences from a Language Model; §10.7.3 (Monte Carlo decoding): sampling chooses each word with probability proportional to its likelihood, as with rolling a weighted die over the softmax distribution.