---
title: "Large Language Models: Pretraining and Scaling"
module: Transformers
moduleNumber: 5
lessonNumber: 4
order: 504
summary: >
  A language model's next-token distribution is only as good as the parameters
  behind it. This part is where those parameters come from: self-supervised
  pretraining on web-scale text with teacher forcing and cross-entropy, the
  scaling laws that make test loss a predictable power law in parameters, data,
  and compute, the KV cache that keeps long-context inference affordable, and how
  a finished model is evaluated by perplexity and benchmarks — closing with the
  Kaplan, Chinchilla, GPT-3, and emergence papers behind the scaling story.
topics: [Transformers]
sources:
  - book: Jurafsky
    ref: "§9.9.1 Pretraining; §9.3 Teacher Forcing and Cross-Entropy"
  - book: Jurafsky
    ref: "Ch. 3 — §3.2.1 Perplexity; Ch. 10 — §10.7.3 Scaling; §10.8 MT Evaluation"
---

This builds on [Large Language Models](/natural-language-processing/transformers/large-language-models),
which assembled the inference side: the language-modeling head that turns a top-block hidden
state into a distribution over the vocabulary, autoregressive generation, and the decoding
strategies that read text out of that distribution. All of it presumed a well-trained model.
This part supplies the training, and the scaling results that govern it.

## Pretraining

None of the above learns anything. The distribution $P_\theta(w_t \mid w_{<t})$ is
only as good as the parameters $\theta$, and those come from **pretraining** — the
self-supervised phase where the model reads enormous quantities of raw text and is
trained to predict the next token at every position.[^slp-pretraining]

The objective needs no human labels, which is what makes web-scale training
possible. Take any document, and at every position the "correct answer" is simply
the word that actually comes next. This is **self-supervision**: the data is its
own answer key. The loss at position $t$ is the cross-entropy between the model's
distribution $\hat{\mathbf{y}}_t$ and the one-hot vector for the true next word
$w_{t+1}$, which reduces to the negative log-probability the model assigned to that
word:[^slp-crossent]

$$
L_{\mathrm{CE}}(\hat{\mathbf{y}}_t, w_{t+1}) \;=\; -\log \hat{\mathbf{y}}_t[w_{t+1}].
$$

The training loss over a sequence is the average of this over all $T$ positions:

$$
L \;=\; -\frac{1}{T} \sum_{t=1}^{T} \log \hat{\mathbf{y}}_t[w_{t+1}].
$$

Two properties make this tractable at scale. During training the model is fed the
**true** prefix at every step rather than its own guesses — **teacher forcing** —
which decouples the positions.[^slp-teacher] And because the transformer computes
every position's output independently (unlike an RNN, whose recurrence forces a
serial pass), a whole training sequence is processed in parallel, one forward pass
scoring the next-token prediction at every position at once.

$$
% caption: Self-supervised pretraining. A raw text sequence is shifted by one to
% form its own targets; the transformer predicts the next token at every position
% in parallel, and the loss is the average cross-entropy against the true next word.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  tok/.style={draw, minimum width=13mm, minimum height=7mm, font=\scriptsize},
  tgt/.style={draw=acc, text=acc, minimum width=13mm, minimum height=7mm, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \foreach \i/\w in {0/So, 1/long, 2/and, 3/thanks} \node[tok] (in\i) at (\i*1.7,0) {\w};
  \node[tok, font=\scriptsize, minimum width=13mm] (block) at (2.55,1.4) {transformer stack};
  \draw[black] (-0.85,0.95) rectangle (5.95,1.85);
  \foreach \i/\w in {0/long, 1/and, 2/thanks, 3/for} \node[tgt] (out\i) at (\i*1.7,2.9) {\w};
  \foreach \i in {0,...,3} {
    \draw[->, black] (in\i.north) -- (in\i.north |- 0,0.95);
    \draw[->, acc] (in\i.north |- 0,1.85) -- (out\i.south);
  }
  \node[font=\scriptsize, anchor=west] at (6.2,0) {input};
  \node[acc, font=\scriptsize, anchor=west] at (6.2,2.9) {target = shift by 1};
\end{tikzpicture}
$$

The second requirement is data. Pretraining corpora are assembled from
web crawls, books, code, and reference text, filtered and deduplicated, and
tokenized into subword units so that a fixed vocabulary can represent any string,
including words it has never seen. The scale is the point: hundreds of billions of
tokens, so that predicting the next word forces the model to absorb syntax,
factual associations, and enough of the statistics of the world to make the
prediction. The representations that fall out are the ones reused when the
model is [fine-tuned or
prompted](/natural-language-processing/transformers/fine-tuning-and-prompting) for
a downstream task.

## Scaling laws

The empirical result behind large-scale pretraining is that test loss
falls **predictably** as the model grows. Across many orders of magnitude, the loss
$L$ is a **power law** in each of three resources — the parameter count $N$, the
dataset size $D$, and the compute budget $C$ — whenever the other two are not the
bottleneck:

$$
L(N) \;\propto\; N^{-\alpha_N},
\qquad
L(D) \;\propto\; D^{-\alpha_D},
\qquad
L(C) \;\propto\; C^{-\alpha_C},
$$

with small positive exponents $\alpha$. A power law is a straight line on
log-log axes, which is what makes it useful: measure the loss at a few
affordable scales, fit the line, and extrapolate to predict the loss of a model too
expensive to have trained yet. Training runs can then be budgeted in advance.

$$
% caption: Scaling laws. On log-log axes, test loss falls as a straight line in
% parameters, data, and compute, $L \propto N^{-\alpha}$; each curve is labeled at
% its right end. Starving any one resource bends its curve flat early (dashed),
% since the neglected term dominates the loss.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->] (0,0) -- (6.6,0) node[right, font=\scriptsize] {log scale (N, D, or C)};
  \draw[->] (0,0) -- (0,4.4) node[above, font=\scriptsize] {log test loss};
  % three parallel-ish power-law lines (different exponents)
  \draw[acc, very thick] (0.5,4.0) -- (5.6,1.4);
  \draw[black, very thick] (0.5,3.4) -- (5.6,1.1);
  \draw[red, very thick] (0.5,2.7) -- (5.6,0.9);
  \node[acc, anchor=west, font=\scriptsize] at (5.65,1.4) {parameters N};
  \node[black, anchor=west, font=\scriptsize] at (5.65,1.05) {data D};
  \node[red, anchor=west, font=\scriptsize] at (5.65,0.85) {compute C};
  % a starved (bent) curve
  \draw[black, thick, dashed] (0.5,4.2) .. controls (2.2,3.5) and (3.0,3.3) .. (5.6,3.2);
  \node[black, anchor=west, font=\scriptsize] at (5.65,3.2) {bottleneck: one term dominates};
\end{tikzpicture}
$$

Because the loss depends on all three, a fixed compute budget poses an allocation
question: spend it on more parameters or more data? Early **(Kaplan)** scaling
work suggested pushing model size hard; the later **Chinchilla** result corrected
it, finding that parameters and data should grow **together** — roughly in
proportion — and that most large models had been badly undertrained, too big for
the number of tokens they saw.[^slp-scaling] The **compute-optimal** frontier is
the balance point: for a given $C$, the pairing of $N$ and $D$ that minimizes loss.
Off that frontier, one resource is wasted while the other bottlenecks the loss.

> **Definition (Compute-optimal scaling).** For a fixed compute budget $C$, the
> choice of model size $N$ and training-token count $D$ that minimizes test loss.
> The Chinchilla finding is that the optimum scales $N$ and $D$ in roughly equal
> proportion, so a compute-optimal model is smaller and trained on more tokens than
> earlier size-first practice assumed.

The reason scaling laws matter beyond bookkeeping is that some capabilities do not
appear gradually. As loss crosses certain thresholds, models begin to do things —
multi-step arithmetic, following an instruction from a few examples in the prompt —
that smaller models could not do at all. These **emergent abilities**, and the
related phenomenon of **in-context learning**, where a model performs a new task
from examples given in the prompt without any weight update, are what make the
scaled-up next-token predictor feel qualitatively different from the n-gram it
descends from. We take them up in the [next
lesson](/natural-language-processing/transformers/fine-tuning-and-prompting).

## Context length and the KV cache

The transformer attends over its entire context, and the cost of that is quadratic:
attention compares every token against every other, so a context of length $n$
costs on the order of $n^2$ to process. The **context length** — the maximum prefix
the model can attend over — is therefore both a capability (longer context means
more to condition on) and a cost driver.

The cost is clearest in generation. Naively, producing token $t+1$ re-runs
attention over the whole prefix $w_{1:t}$, and doing that at every step repeats
enormous work. The **KV cache** removes the repetition: attention needs a key and a
value vector for each past token, and those never change once computed, so the
model stores them. Generating a new token then computes attention using the one
new query against all cached keys and values, and appends its own key and value to
the cache. Generation cost per token becomes linear in the context rather than
quadratic, at the price of memory that grows with the context. This is why long
contexts are expensive to serve.

## Evaluation

How good is a language model? Two families of answer.

**Perplexity** measures the model against held-out text directly. It is the inverse
probability the model assigns to a test set, normalized by length — equivalently,
the exponentiated average negative log-probability per token:[^slp-perplexity]

$$
\PP(w_{1:n}) \;=\; P(w_{1:n})^{-\frac{1}{n}}
\;=\; \exp\!\left(-\frac{1}{n} \sum_{t=1}^{n} \log P(w_t \mid w_{<t})\right).
$$

Perplexity is the model's **weighted average branching factor**: how many words, on
average, it is choosing between at each step. Lower is better — the model is less
surprised by real text. It is the training objective itself, evaluated on held-out
data, which makes it the natural intrinsic metric, with one caveat: perplexities
are only comparable across models that share a vocabulary and tokenization, and an
improvement in perplexity does not guarantee an improvement on any downstream task.

Put numbers on it. Suppose a four-token test string _the cat sat down_ receives per-token
probabilities $0.10, 0.40, 0.25, 0.20$ from the model. The average negative log-probability
is $-\tfrac{1}{4}(\ln 0.10 + \ln 0.40 + \ln 0.25 + \ln 0.20) = 1.554$ nats, and perplexity is
its exponential, $e^{1.554} = 4.73$. The same number arrives from the other definition: the
geometric mean of the four probabilities is $(0.10 \cdot 0.40 \cdot 0.25 \cdot 0.20)^{1/4} =
0.2115$, and perplexity is its reciprocal, $1/0.2115 = 4.73$. The interpretation is direct:
this model is as uncertain, on average, as if it were choosing uniformly among $4.73$ words at
each step. A model that assigned every token probability $1$ would have perplexity $1$ (no
uncertainty); a model that assigned uniform probability $1/|V|$ to a $50{,}000$-word
vocabulary would have perplexity $50{,}000$. Halving perplexity means the model is, in the
branching-factor sense, twice as sure of the next word.

$$
% caption: Perplexity of the test string "the cat sat down". Per-token probabilities are
% turned into per-token surprisals (negative log-probabilities); their average, exponentiated,
% is the perplexity 4.73 — the effective number of equally likely choices per step.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  cell/.style={draw, minimum width=16mm, minimum height=6mm, inner sep=1pt, font=\scriptsize},
  rcell/.style={draw=acc, text=acc, minimum width=16mm, minimum height=6mm, inner sep=1pt, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[anchor=east, font=\scriptsize, text=black] at (-0.2,1.6) {token};
  \node[anchor=east, font=\scriptsize, text=black] at (-0.2,0.8) {prob};
  \node[anchor=east, font=\scriptsize, text=black] at (-0.2,0.0) {-log prob};
  \foreach \c/\w/\p/\s in {1/the/0.10/2.30, 2/cat/0.40/0.92, 3/sat/0.25/1.39, 4/down/0.20/1.61} {
    \node[cell] at (\c*1.9,1.6) {\w};
    \node[cell] at (\c*1.9,0.8) {\p};
    \node[cell] at (\c*1.9,0.0) {\s};
  }
  \node[rcell, minimum width=40mm] (pp) at (5.7,-1.2) {avg = 1.554, PP = exp(1.554) = 4.73};
  \foreach \c in {1,2,3,4} \draw[->, acc!60] (\c*1.9,-0.35) -- (pp.north);
\end{tikzpicture}
$$

**Benchmarks** measure the model against tasks instead of text. A generative model
producing free text needs task-level scoring: for translation and summarization,
overlap metrics like **chrF** (a character-n-gram F-score) and the older word-based
**BLEU** compare the output against human references, while embedding-based metrics
like **BERTScore** match tokens by cosine similarity to reward paraphrase rather
than exact overlap.[^slp-eval] Beyond single tasks, large models are reported on
suites of question-answering, reasoning, and knowledge benchmarks — the coarse but
comparable numbers that track progress across the field. Neither family is
sufficient alone: perplexity is faithful but task-blind, benchmarks are task-aware
but easy to overfit.

## Scaling laws, GPT-3, and emergence

The scaling story above is drawn from three papers worth naming precisely, because their
claims are often blurred together.

**The scaling laws (Kaplan et al., 2020).** "Scaling Laws for Neural Language Models"
established the power-law form empirically: over seven orders of magnitude, test loss falls as
$L(N) \approx (N_c/N)^{\alpha_N}$ in parameters and analogously in data and compute, with
exponents around $\alpha_N \approx 0.076$ and $\alpha_D \approx 0.095$ for their setup.[^kaplan]
Their headline recommendation, given a fixed compute budget, was to spend most of it on model
_size_ and comparatively little on more data — train very large models on relatively few
tokens and stop before convergence.

**The Chinchilla correction (Hoffmann et al., 2022).** "Training Compute-Optimal Large
Language Models" re-ran the analysis more carefully and reached the opposite allocation:
parameters and training tokens should scale in _roughly equal proportion_, about $20$ tokens
per parameter at the compute-optimal point.[^chinchilla] To demonstrate it they trained
Chinchilla, a $70$-billion-parameter model, on $1.4$ trillion tokens, and it outperformed the
$280$-billion-parameter Gopher despite being four times smaller — because Gopher, like most
models of its era, had been badly undertrained for its size. The result changed practice:
compute-optimal models are smaller and trained on more tokens.

$$
% caption: Kaplan vs. Chinchilla allocation of a fixed compute budget. Kaplan (2020) put most
% of a budget into parameters and undertrained; Chinchilla (2022) found the loss-minimizing
% point grows parameters N and tokens D together (about 20 tokens per parameter).
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->] (0,0) -- (6.2,0) node[right, font=\scriptsize] {params N (log)};
  \draw[->] (0,0) -- (0,4.0) node[above, font=\scriptsize] {tokens D (log)};
  % chinchilla frontier: D proportional to N (slope 1)
  \draw[acc, very thick] (0.4,0.5) -- (5.4,3.6);
  \node[acc, anchor=south, font=\scriptsize] at (2.9,3.7) {Chinchilla frontier: D grows with N};
  % kaplan: big N, small D (a point low-right)
  \fill[red] (4.8,1.0) circle (2.2pt);
  \node[red, anchor=north, font=\scriptsize] at (4.8,0.85) {Kaplan: big N, few tokens};
  \fill[acc] (3.1,2.3) circle (2.2pt);
  \node[acc, anchor=south east, font=\scriptsize] at (3.05,2.4) {compute-optimal};
\end{tikzpicture}
$$

**GPT-3 and in-context learning (Brown et al., 2020).** "Language Models Are Few-Shot
Learners" trained a $175$-billion-parameter decoder-only transformer and showed that, at that
scale, the model performs new tasks from _examples placed in the prompt_ with no gradient
update — the **in-context learning** the last section named.[^gpt3] Give it a few input-output
pairs as context and it infers the pattern and continues it. This is not learning in the
weight-update sense; the demonstrations live only in the forward pass. GPT-3 made few-shot
prompting a headline capability and is the reason the next lesson can treat prompting as an
adaptation method on par with fine-tuning.

**Emergent abilities (Wei et al., 2022).** "Emergent Abilities of Large Language Models"
documented tasks — multi-step arithmetic, word unscrambling, certain benchmarks — where
performance stays at chance across many model sizes and then rises sharply past a scale
threshold, rather than improving smoothly.[^emergent] The claim is contested: Schaeffer et al.
(2023) argued that some apparent emergence is an artifact of discontinuous metrics (exact-match
scoring hides the smooth improvement a soft metric would reveal), and that under continuous
metrics the jumps often vanish.[^mirage] The honest summary is that scaling reliably lowers
loss as a power law, that some downstream metrics improve smoothly and others appear to jump,
and that how much of the jumping is real versus a measurement artifact is still an open
question. State the phenomenon, not more than the evidence supports.

## The whole recipe

The parts summarize in one sentence. A stack of transformer blocks
with a language-modeling head estimates $P(w_t \mid w_{<t})$; trained by
self-supervised next-token prediction on web-scale text, its loss falls as a power
law in size, data, and compute; and at inference a decoding strategy reads text
back out of the distribution. One architecture, one objective, and scale. The
[deep-learning course's LLM
lesson](/deep-learning/large-models-and-agents/large-language-models) works the
same pipeline from the modeling side, and the alignment step that turns this
fluent-but-unhelpful next-token predictor into an assistant —
[RLHF](/reinforcement-learning/modern-deep-rl/rlhf-and-language-models) — is where
reinforcement learning re-enters the story. What pretraining leaves us with is a
model that continues text; the [next
lesson](/natural-language-processing/transformers/fine-tuning-and-prompting) is
about making it do what we ask.

[^slp-pretraining]: **Jurafsky & Martin**, §9.9.1 — Pretraining: first train a transformer language model on a large corpus in a self-supervised way, then add and fine-tune a task head; the self-supervised objective incorporates rich information about language.
[^slp-crossent]: **Jurafsky & Martin**, §9.3: the language-modeling loss is cross-entropy against the one-hot true next word, reducing to $L_{\mathrm{CE}} = -\log \hat{y}_t[w_{t+1}]$, the negative log-probability the model assigns to the correct next token, averaged over the sequence.
[^slp-teacher]: **Jurafsky & Martin**, §9.3 (Teacher Forcing): during training the model is always given the correct history $w_{1:t}$ to predict $w_{t+1}$, rather than its own previous outputs; transformers score every position in parallel.
[^slp-scaling]: **Jurafsky & Martin**, §10.7.3 and the scaling-law literature it cites: test loss falls as a power law in parameters, data, and compute; the compute-optimal balance (Chinchilla) grows model size and training tokens together rather than favoring size alone.
[^slp-perplexity]: **Jurafsky & Martin**, §3.2.1 — Perplexity: the inverse probability of the test set normalized by length, equal to the exponentiated per-token cross-entropy and interpretable as the weighted average branching factor; lower is better.
[^slp-eval]: **Jurafsky & Martin**, §10.8 — MT Evaluation: character-overlap metrics like chrF and the word-based BLEU compare output to human references, and embedding-based metrics like BERTScore match tokens by cosine similarity to credit paraphrase.
[^kaplan]: **Kaplan, McCandlish, Henighan, Brown, Chess, Child, Gray, Radford, Wu, Amodei (2020)**, "Scaling Laws for Neural Language Models." Establishes that language-model test loss follows smooth power laws in parameters, dataset size, and compute over many orders of magnitude, with small exponents; recommends, for a fixed compute budget, allocating most of it to model size and training large models on relatively little data without training to convergence.
[^chinchilla]: **Hoffmann, Borgeaud, Mensch, et al. (2022)**, "Training Compute-Optimal Large Language Models" (Chinchilla), _NeurIPS 2022_. Finds that model size and training tokens should be scaled in roughly equal proportion (about 20 tokens per parameter) at the compute-optimal frontier; the 70B Chinchilla, trained on 1.4T tokens, outperforms the 280B Gopher, showing prior large models were substantially undertrained.
[^gpt3]: **Brown, Mann, Ryder, et al. (2020)**, "Language Models Are Few-Shot Learners" (GPT-3), _NeurIPS 2020_. A 175B-parameter decoder-only transformer that performs many tasks from a few in-context demonstrations with no gradient update (in-context / few-shot learning), demonstrating that task specification can move from fine-tuning into the prompt at sufficient scale.
[^emergent]: **Wei, Tay, Bommasani, et al. (2022)**, "Emergent Abilities of Large Language Models," _TMLR 2022_. Documents tasks on which performance remains near chance across smaller scales and rises sharply beyond a threshold model size, rather than improving smoothly with scale.
[^mirage]: **Schaeffer, Miranda, Koyejo (2023)**, "Are Emergent Abilities of Large Language Models a Mirage?", _NeurIPS 2023_. Argues that many reported emergent jumps are artifacts of discontinuous, all-or-nothing metrics (e.g. exact match); under continuous or per-token metrics the same capabilities often improve smoothly and predictably with scale.