Large Language Models

The transformer gave us a block that reads a sequence and, at every position, produces a vector that has attended to everything before it. A large language model is a decoder-only stack of those blocks trained on the single task of predicting the next word, over a large fraction of the written web, at large scale. The ingredients are one architecture, one objective, and scale; nothing else is new.

The objective is the same one the n-gram model and the neural language model were built around: assign a probability to the next token given the tokens so far. Writing $w_{<t}=w_1,\dots ,w_{t-1}$ for the prefix, the model estimates

and the probability of a whole sequence factors, by the chain rule, into a product of these conditionals:

The transformer's only job is to compute the right-hand side well. Everything in this lesson is either how the block turns its hidden state into that distribution, how we read text out of the distribution, or how we drive the distribution to be accurate by training on enough text.

The language-modeling head

A stack of $L$ transformer blocks maps an input sequence to a sequence of hidden vectors. At position $t$ the top block emits ${\mathbf{h}}_t\in {\mathbb{R}}^d$, a $d$-dimensional summary of the prefix that has attended, through the causal mask, to $w_1,\dots ,w_t$ and no further.¹ To turn that vector into a prediction we need a map from ${\mathbb{R}}^d$ to a probability distribution over the vocabulary $V$. This map is the language-modeling head, and it is two operations: a linear projection followed by a softmax.

The projection is called the unembedding. A matrix $\mathbf{U}\in {\mathbb{R}}^{|V|\times d}$ scores every word in the vocabulary against the hidden state, producing one logit per word:

Each entry $z_t[i]$ measures how well word $i$ matches the evidence in ${\mathbf{h}}_t$; the $i$-th row of $\mathbf{U}$ is effectively a second embedding of word $i$, and the logit is its inner product with the hidden state. The softmax then normalizes the logits into a distribution:

The unembedding matrix is where most of a model's parameters can pile up, since $|V|$ is large — tens of thousands of subword tokens is typical. A standard trick is weight tying: reuse the input embedding matrix $\mathbf{E}$ as the unembedding, setting $\mathbf{U}=\mathbf{E}$.² The rows of $\mathbf{E}$ already learn a vector per word at the input; the rows of $\mathbf{U}$ must learn one per word at the output; there is no reason to learn two. Tying them improves perplexity and removes an entire $|V|\times d$ block of parameters.

Autoregressive generation

The head gives us a distribution over the next token. To generate text we sample one token from it, append that token to the context, and run the model forward again — now conditioned on the token we just produced. Repeating this is autoregressive generation: the word chosen at each step becomes part of the input for the next.³

The word autoregressive is borrowed loosely. A true autoregressive model predicts a value from a linear function of its own past values; a transformer language model is deeply non-linear, but it shares the defining feature — each output is conditioned on the outputs already produced.

The line $\textsc{Decode}(\mathbf{y}_t)$ is the only free choice, and it is the whole subject of the next section. Priming the context is what makes generation practically useful. Instead of starting from a bare start-of-sequence marker, seed the model with a task-appropriate prefix: a source sentence for translation, a long article for summarization, a question for question answering.⁴ The model does not know it is doing a task — it only continues the text — but a prefix that makes the desired output the most probable continuation turns next-token prediction into a general-purpose interface.

Decoding strategies

Given the distribution ${\mathbf{y}}_t$, how do we commit to an actual token? The choice trades off two failure modes. Always take the most probable token and the text is fluent but flat, and often globally suboptimal; sample too freely from the tail and the text drifts into incoherence. Decoding strategies sit on that spectrum.

Greedy decoding

The simplest rule takes the single most probable token at every step:

This is greedy decoding.⁵ It is fast and deterministic, but greedy is a local choice, and the locally best token can strand the model on a globally worse sequence. A token that looks best now may leave no good continuation, while a slightly-less-probable token now might open onto a much more probable completion. Greedy search cannot see that far ahead.

Beam search

Beam search addresses this by keeping the $k$ best partial sequences at once instead of one.⁶ The number $k$ is the beam width. At each step, every surviving sequence — a hypothesis — is extended by every possible next token, all $k\times |V|$ candidates are scored by their total log-probability, and only the best $k$ are kept. A hypothesis that ends in the end-of-sequence token is complete and set aside; the search runs until the beam empties.

Because probabilities of long strings are tiny, hypotheses are scored in log-space, where the product becomes a sum that can be extended incrementally:

One wrinkle: because every term is negative, longer sequences score lower purely for being long, so a naive beam favors short outputs. The usual fix is length normalization — divide the score by the number of tokens $T$ before comparing:⁷

Beam search is standard in machine translation, where there is a single correct answer to search for and widths of $k=5$ to $10$ are common. For open-ended generation it is a poor fit: maximizing probability produces bland, repetitive text. Open-ended generation calls for sampling.

Sampling

Instead of maximizing, we can sample the next token from the distribution itself, choosing each word with probability proportional to ${\mathbf{y}}_t[i]$. This is the same weighted-die idea used to visualize an n-gram model back in the foundations: lay every word out on the unit interval with width equal to its probability, throw a dart, and read off the word it lands in.⁸ Pure sampling matches the model's distribution exactly, which is both its strength and its problem — the long tail of low-probability tokens is collectively heavy, so even a well-trained model occasionally samples a bizarre word and the text derails. The remaining strategies all reshape or truncate the distribution before sampling to control that tail.

Temperature

Temperature rescales the logits before the softmax by a factor $\tau >0$:

Dividing by a small $\tau <1$ stretches the logits apart, sharpening the distribution toward its peak — in the limit $\tau \to 0$ sampling becomes greedy. A large $\tau >1$ compresses the logits, flattening the distribution toward uniform and making rare words more likely. Temperature does not remove any word; it only reweights the distribution, from sharper ($\tau <1$) to flatter ($\tau >1$).

Top-k and top-p (nucleus) sampling

The tail is heavy because it is long, not because any one word in it is likely. Truncated sampling cuts the tail off and renormalizes what remains, then samples from that.

• Top-k sampling keeps the $k$ highest-probability words, zeros the rest, and renormalizes. Sampling then never strays outside the $k$ most plausible continuations. The weakness is that $k$ is fixed: when the model is confident, $k$ words include junk; when it is unsure, $k$ words cut off good options.
• Top-p sampling, also called nucleus sampling, fixes that by cutting on probability mass instead of count. Sort the vocabulary by probability and keep the smallest set — the nucleus — whose cumulative probability first reaches a threshold $p$ (say $0.9$), discarding the rest. The cutoff adapts: a peaked distribution keeps few words, a flat one keeps many.

In practice temperature, top-k, and top-p compose: a generation call might set $\tau =0.8$ and $p=0.95$ together, sharpening slightly and then clipping the tail. Greedy and beam sit at one end (deterministic, probability-maximizing); temperature and truncated sampling occupy the other (stochastic, diversity-seeking).

A worked decoding step

Carry one step through with numbers. Suppose the head has produced logits over six candidate next words after the prefix the cat:

At the default temperature $\tau =1$ the softmax gives probabilities $(0.605,0.222,0.082,0.050,0.030,0.011)$. Greedy takes sat. Pure sampling takes sat about $60\%$ of the time but leaves real mass on the rest, including $1.1\%$ on the implausible purple. Now apply each knob:

• Temperature $\tau =0.7$ divides the logits by $0.7$ before the softmax, sharpening the distribution to $(0.745,0.179,0.043,0.021,0.010,0.002)$ — sat gains, the tail shrinks. Temperature $\tau =1.5$ does the reverse, flattening to $(0.461,0.237,0.121,0.087,0.062,0.032)$, which raises every rare word.
• Nucleus ($p=0.9$) at $\tau =1$ sorts by probability and accumulates: sat ($0.605$), ran ($0.605+0.222=0.827$), slept ($0.827+0.082=0.909\ge 0.9$). The nucleus is $\{\text{sat},\text{ran},\text{slept}\}$; the other three words are discarded and the kept three are renormalized to sum to one. Even a single unlucky draw of purple is now impossible, while the plausible alternatives ran and slept stay available.

The two mechanisms are orthogonal: temperature reshapes the whole distribution, truncation deletes the tail. Composing $\tau =0.7$ then $p=0.9$ would first sharpen, then cut — the usual production setting for text that is varied but never derails.

From reading text out to putting it in

Everything so far assumes the distribution $P_{\theta }(w_t\mid w_{<t})$ is already good — the head shapes it, autoregression rolls it forward, and a decoding strategy reads text out of it. But that distribution is only as good as the parameters behind it, and those come from training on an enormous amount of text. The next part is where the parameters come from: self-supervised pretraining at web scale, the scaling laws that make its loss predictable, the KV cache that makes long-context inference affordable, and how the finished model is evaluated. This continues in Large Language Models: Pretraining and Scaling.