Large Language Models: Pretraining and Scaling

This builds on Large Language Models, which assembled the inference side: the language-modeling head that turns a top-block hidden state into a distribution over the vocabulary, autoregressive generation, and the decoding strategies that read text out of that distribution. All of it presumed a well-trained model. This part supplies the training, and the scaling results that govern it.

Pretraining

None of the above learns anything. The distribution $P_{\theta }(w_t\mid w_{<t})$ is only as good as the parameters $\theta$, and those come from pretraining — the self-supervised phase where the model reads enormous quantities of raw text and is trained to predict the next token at every position.¹

The objective needs no human labels, which is what makes web-scale training possible. Take any document, and at every position the correct answer is simply the word that actually comes next. This is self-supervision: the data is its own answer key. The loss at position $t$ is the cross-entropy between the model's distribution ${\hat{\mathbf{y}}}_t$ and the one-hot vector for the true next word $w_{t+1}$, which reduces to the negative log-probability the model assigned to that word:²

The training loss over a sequence is the average of this over all $T$ positions:

Two properties make this tractable at scale. During training the model is fed the true prefix at every step rather than its own guesses — teacher forcing — which decouples the positions.³ And because the transformer computes every position's output independently (unlike an RNN, whose recurrence forces a serial pass), a whole training sequence is processed in parallel, one forward pass scoring the next-token prediction at every position at once.

The second requirement is data. Pretraining corpora are assembled from web crawls, books, code, and reference text, filtered and deduplicated, and tokenized into subword units so that a fixed vocabulary can represent any string, including words it has never seen. The scale is the point: hundreds of billions of tokens, so that predicting the next word forces the model to absorb syntax, factual associations, and enough of the statistics of the world to make the prediction. The representations that fall out are the ones reused when the model is fine-tuned or prompted for a downstream task.

Scaling laws

The empirical result behind large-scale pretraining is that test loss falls predictably as the model grows. Across many orders of magnitude, the loss $L$ is a power law in each of three resources — the parameter count $N$, the dataset size $D$, and the compute budget $C$ — whenever the other two are not the bottleneck:

with small positive exponents $\alpha$. A power law is a straight line on log-log axes, which is what makes it useful: measure the loss at a few affordable scales, fit the line, and extrapolate to predict the loss of a model too expensive to have trained yet. Training runs can then be budgeted in advance.

Because the loss depends on all three, a fixed compute budget poses an allocation question: spend it on more parameters or more data? Early (Kaplan) scaling work suggested pushing model size hard; the later Chinchilla result corrected it, finding that parameters and data should grow together — roughly in proportion — and that most large models had been badly undertrained, too big for the number of tokens they saw.⁴ The compute-optimal frontier is the balance point: for a given $C$, the pairing of $N$ and $D$ that minimizes loss. Off that frontier, one resource is wasted while the other bottlenecks the loss.

The reason scaling laws matter beyond bookkeeping is that some capabilities do not appear gradually. As loss crosses certain thresholds, models begin to do things — multi-step arithmetic, following an instruction from a few examples in the prompt — that smaller models could not do at all. These emergent abilities, and the related phenomenon of in-context learning, where a model performs a new task from examples given in the prompt without any weight update, are what make the scaled-up next-token predictor feel qualitatively different from the n-gram it descends from. We take them up in the next lesson.

Context length and the KV cache

The transformer attends over its entire context, and the cost of that is quadratic: attention compares every token against every other, so a context of length $n$ costs on the order of $n^2$ to process. The context length — the maximum prefix the model can attend over — is therefore both a capability (longer context means more to condition on) and a cost driver.

The cost is clearest in generation. Naively, producing token $t+1$ re-runs attention over the whole prefix $w_{1:t}$, and doing that at every step repeats enormous work. The KV cache removes the repetition: attention needs a key and a value vector for each past token, and those never change once computed, so the model stores them. Generating a new token then computes attention using the one new query against all cached keys and values, and appends its own key and value to the cache. Generation cost per token becomes linear in the context rather than quadratic, at the price of memory that grows with the context. This is why long contexts are expensive to serve.

Evaluation

How good is a language model? Two families of answer.

Perplexity measures the model against held-out text directly. It is the inverse probability the model assigns to a test set, normalized by length — equivalently, the exponentiated average negative log-probability per token:⁵

Perplexity is the model's weighted average branching factor: how many words, on average, it is choosing between at each step. Lower is better — the model is less surprised by real text. It is the training objective itself, evaluated on held-out data, which makes it the natural intrinsic metric, with one caveat: perplexities are only comparable across models that share a vocabulary and tokenization, and an improvement in perplexity does not guarantee an improvement on any downstream task.

Put numbers on it. Suppose a four-token test string the cat sat down receives per-token probabilities $0.10,0.40,0.25,0.20$ from the model. The average negative log-probability is $-\frac{1}{4}(\ln 0.10+\ln 0.40+\ln 0.25+\ln 0.20)=1.554$ nats, and perplexity is its exponential, $e^{1.554}=4.73$. The same number arrives from the other definition: the geometric mean of the four probabilities is $(0.10\cdot 0.40\cdot 0.25\cdot 0.20{)}^{1/4}=0.2115$, and perplexity is its reciprocal, $1/0.2115=4.73$. The interpretation is direct: this model is as uncertain, on average, as if it were choosing uniformly among $4.73$ words at each step. A model that assigned every token probability $1$ would have perplexity $1$ (no uncertainty); a model that assigned uniform probability $1/|V|$ to a $50,000$-word vocabulary would have perplexity $50,000$. Halving perplexity means the model is, in the branching-factor sense, twice as sure of the next word.

Benchmarks measure the model against tasks instead of text. A generative model producing free text needs task-level scoring: for translation and summarization, overlap metrics like chrF (a character-n-gram F-score) and the older word-based BLEU compare the output against human references, while embedding-based metrics like BERTScore match tokens by cosine similarity to reward paraphrase rather than exact overlap.⁶ Beyond single tasks, large models are reported on suites of question-answering, reasoning, and knowledge benchmarks — the coarse but comparable numbers that track progress across the field. Neither family is sufficient alone: perplexity is faithful but task-blind, benchmarks are task-aware but easy to overfit.

Scaling laws, GPT-3, and emergence

The scaling story above is drawn from three papers worth naming precisely, because their claims are often blurred together.

The scaling laws (Kaplan et al., 2020). Scaling Laws for Neural Language Models established the power-law form empirically: over seven orders of magnitude, test loss falls as $L(N)\approx (N_c/N{)}^{{\alpha }_N}$ in parameters and analogously in data and compute, with exponents around ${\alpha }_N\approx 0.076$ and ${\alpha }_D\approx 0.095$ for their setup.⁷ Their headline recommendation, given a fixed compute budget, was to spend most of it on model size and comparatively little on more data — train very large models on relatively few tokens and stop before convergence.

The Chinchilla correction (Hoffmann et al., 2022). Training Compute-Optimal Large Language Models re-ran the analysis more carefully and reached the opposite allocation: parameters and training tokens should scale in roughly equal proportion, about $20$ tokens per parameter at the compute-optimal point.⁸ To demonstrate it they trained Chinchilla, a $70$-billion-parameter model, on $1.4$ trillion tokens, and it outperformed the $280$-billion-parameter Gopher despite being four times smaller — because Gopher, like most models of its era, had been badly undertrained for its size. The result changed practice: compute-optimal models are smaller and trained on more tokens.

GPT-3 and in-context learning (Brown et al., 2020). Language Models Are Few-Shot Learners trained a $175$-billion-parameter decoder-only transformer and showed that, at that scale, the model performs new tasks from examples placed in the prompt with no gradient update — the in-context learning the last section named.⁹ Give it a few input-output pairs as context and it infers the pattern and continues it. This is not learning in the weight-update sense; the demonstrations live only in the forward pass. GPT-3 made few-shot prompting a headline capability and is the reason the next lesson can treat prompting as an adaptation method on par with fine-tuning.

Emergent abilities (Wei et al., 2022). Emergent Abilities of Large Language Models documented tasks — multi-step arithmetic, word unscrambling, certain benchmarks — where performance stays at chance across many model sizes and then rises sharply past a scale threshold, rather than improving smoothly.¹⁰ The claim is contested: Schaeffer et al. (2023) argued that some apparent emergence is an artifact of discontinuous metrics (exact-match scoring hides the smooth improvement a soft metric would reveal), and that under continuous metrics the jumps often vanish.¹¹ The honest summary is that scaling reliably lowers loss as a power law, that some downstream metrics improve smoothly and others appear to jump, and that how much of the jumping is real versus a measurement artifact is still an open question. State the phenomenon, not more than the evidence supports.

The whole recipe

The parts summarize in one sentence. A stack of transformer blocks with a language-modeling head estimates $P(w_t\mid w_{<t})$; trained by self-supervised next-token prediction on web-scale text, its loss falls as a power law in size, data, and compute; and at inference a decoding strategy reads text back out of the distribution. One architecture, one objective, and scale. The deep-learning course's LLM lesson works the same pipeline from the modeling side, and the alignment step that turns this fluent-but-unhelpful next-token predictor into an assistant — RLHF — is where reinforcement learning re-enters the story. What pretraining leaves us with is a model that continues text; the next lesson is about making it do what we ask.