Scaling, Inference, and Alignment of Language Models

This builds on Large Language Models, which defined the object: a decoder-only Transformer trained by next-token prediction, its subword tokenizer, its pretraining objectives, and the three architecture families. That lesson fixed the architecture and named scale as the design axis. This one works out why scale pays, in what proportion, and what a trained model has to become before it is useful: training, inference, and behavior, in that order.

Scaling laws

The reason scale became the dominant lever is that the loss is predictable. Over many orders of magnitude, test cross-entropy falls as a power law in each of model size $N$ , data $D$ , and compute $C$ when the other two are not the bottleneck.¹

Given a fixed compute budget, the laws also dictate how to split it between model size and data. The Chinchilla result corrects an earlier bias toward huge models trained on too little data.²

The practical content is a frontier: for each compute budget there is one optimal model size, and runs off that line waste compute by being either too small (unable to exploit the data) or too large (under-trained).

Compute-optimal frontier (log--log). For each compute budget the optimal model size

N

grows along a line of slope

1/2

(blue); models that are too large (under-trained) or too small sit off the line (red).

Emergent abilities and in-context learning

Smoothness of loss coexists with sharp jumps in behavior. Some capabilities are emergent: near-random for small models, then rising abruptly past a scale threshold, so they are invisible by extrapolating small-model performance.³

The two curves are decoupled: the loss falls smoothly as a power law in scale, while a specific downstream metric stays flat at chance and then rises at a sharp knee. The loss curve therefore gives no warning that a capability is about to appear.

Loss versus a task metric across scale (log model size on x). The cross-entropy loss (blue) falls smoothly as a power law; a downstream capability (red) stays near chance, then rises abruptly past a scale threshold. Smooth loss hides the discontinuous behavior.

Chain-of-thought prompting elicits a further capability: asking the model to emit intermediate reasoning steps before the answer sharply improves performance on multi-step arithmetic and logic, and the gain itself is emergent, appearing only past a scale threshold.⁵ The phenomenon is that the same likelihood objective, pushed far enough, yields behaviors nobody trained for directly.

Decoding

A trained LM is a distribution $p_{θ} (x_{t} ∣ x_{< t})$ ; turning it into text requires a decoding rule. The choice trades fidelity against diversity.

Beam search keeps the $B$ highest-scoring partial sequences rather than one, maximizing total sequence log-probability; it is standard for translation but produces bland, repetitive open-ended text, because the globally most probable continuation is often degenerate.⁶ Sampling methods fix this by drawing from a truncated distribution, and nucleus sampling, truncating by cumulative mass rather than a fixed count, adapts the cutoff to how peaked $p_{t}$ is.

Strategy	Rule	Diversity	Typical use
Greedy	take the argmax	none	short factual answers
Beam ( $B$ )	keep top- $B$ sequences	low	translation, summarization
Temperature $τ$	scale logits by $1/ τ$	tunable	creative text
Top- $k$	sample from $k$ likeliest	medium	with fixed budget
Nucleus (top- $p$ )	sample from mass $\geq p$	adaptive	open-ended generation

Algorithm:

\textsc{NucleusSample}(p_t, p)

— sample one token from the top-

p

nucleus

1
sort vocabulary so $p_t(v_1) \ge p_t(v_2) \ge \dots$
2
$c \gets 0$ ; $V_p \gets \emptyset$
3
for $j \gets 1$ to $\abs{V}$ do
4
$V_p \gets V_p \cup \{v_j\}$ ; $c \gets c + p_t(v_j)$
5
if $c \ge p$ then
smallest set covering mass $p$
6
break
7
$q(v) \gets p_t(v) / c$ for $v \in V_p$ , else $0$
renormalize over the nucleus
8
return $v \sim q$

The figure shows how nucleus truncation keeps a few tokens on a peaked distribution but more on a flat one, which is the behavior a fixed top- $k$ cannot match.

Nucleus sampling keeps the smallest token set whose probability sums to

p

. On a peaked step (left) that is two tokens; on a flat step (right), four.

KV cache and inference cost

Autoregressive generation appears to cost $O (t^{2})$ at step $t$ : each new token recomputes attention over all $t$ positions. The key--value cache removes the redundancy. Once computed, the keys $K_{< t}$ and values $V_{< t}$ of past positions never change, so they are stored and reused; only the new token's query, key, and value are computed at each step.

The cache trades memory for compute, and that memory becomes the binding constraint at long context: a $30$ B model at $d = 7168$ , $48$ layers, in $16$ -bit, caches roughly $2 \cdot 48 \cdot 7168 \cdot 2$ bytes per token, about $1.4$ MB, so a $100, 000$ -token context costs $\sim 140$ GB of cache alone. This memory growth, not the arithmetic, is why long-context inference is expensive.

KV cache. Past keys and values (black) are stored once and reused; each new token computes only its own query, key, value (blue) and attends over the cache.

Parameter-efficient adaptation

Full fine-tuning updates all $N$ parameters and stores a separate copy of the model per task, infeasible at billions of parameters. Parameter-efficient methods adapt a frozen model by training a small number of new parameters.

Adapters insert small bottleneck MLPs between frozen layers and train only those; they add $< 4%$ parameters but introduce sequential layers that slow inference.⁷ Prefix / prompt tuning prepends a few trainable virtual-token vectors to the keys and values, steering a frozen model through the attention itself.⁸ The dominant method, LoRA, freezes the weights and learns a low-rank update.⁹

For example, with $d = 4096$ and $r = 8$ , the update has $2 \cdot 8 \cdot 4096 = 65, 536$ parameters instead of $409 6^{2} = 16.7$ M, a $256 \times$ reduction, with no inference-time cost because $B A$ can be folded back into $W_{0}$ after training. The assumption is that the change a task induces is intrinsically low-rank even though $W_{0}$ is full rank.

LoRA. The frozen weight

W_{0}

(

d \times d

) is left untouched; the update is the product of two thin matrices

B

(

d \times r

) and

A

(

r \times d

) with

r ≪ d

, the only trained parameters.

Method	Trainable params	Inference overhead	Mechanism
Full fine-tuning	$N$ (all)	none	update every weight
Adapters	$\sim 2$ -- $4%$	added layers (slower)	bottleneck MLP per block
Prefix / prompt	$≪ 1%$	longer keys/values	trainable virtual tokens
LoRA	$2 r d$ per matrix	none (merge $B A$ )	low-rank weight delta

Alignment

A pretrained LM predicts likely text, not helpful text. Alignment is the stack that turns the raw next-token predictor into an assistant: supervised instruction tuning, then learning from human preferences.

Instruction tuning (FLAN). Fine-tune on a large mixture of tasks phrased as natural-language instructions; the model generalizes to held-out instructions it never saw, a zero-shot ability that itself improves with the number of tuning tasks.¹⁰ This is plain supervised learning on $(instruction, response)$ pairs.

RLHF. Instruction tuning teaches format, not preference. Reinforcement learning from human feedback fits a reward model $r_{ϕ}$ to human pairwise comparisons, then optimizes the policy $π_{θ}$ against it.¹¹ The reward model is trained on the Bradley--Terry likelihood that the preferred response $y_{w}$ beats the rejected $y_{l}$ :

L_{RM} (ϕ) = - E_{(x, y_{w}, y_{l})} [log σ (r_{ϕ} (x, y_{w}) - r_{ϕ} (x, y_{l}))] .

The policy is then optimized by PPO to maximize reward while a KL penalty keeps it near the supervised reference $π_{ref}$ , preventing reward hacking:

θ max E_{x, y \sim π_{θ}} [r_{ϕ} (x, y)] - β E_{x} [KL (π_{θ} (\cdot ∣ x) ∥ π_{ref} (\cdot ∣ x))] .

Algorithm:

\textsc{RLHF}

— align a pretrained LM from human preferences

1
$\pi_{\text{ref}} \gets$ instruction-tuned (SFT) model
2
collect comparisons: for prompts $x$ , humans rank pairs $(y_w, y_l)$
3
train reward model $r_\phi$ on the Bradley--Terry loss $\mathcal{L}_{\text{RM}}$
4
$\pi_\theta \gets \pi_{\text{ref}}$
5
repeat
6
sample responses $y \sim \pi_\theta(\cdot \mid x)$
7
score with $r_\phi$ ; subtract KL penalty to $\pi_{\text{ref}}$
8
update $\theta$ by PPO on the penalized reward
policy gradient step
9
until reward plateaus
10
return $\pi_\theta$

The pipeline has three stages, each consuming the output of the last: a supervised model becomes the reference, pairwise human comparisons train a scalar reward, and the policy is optimized for high reward while a KL penalty keeps it near the reference.

The RLHF pipeline. A pretrained model is first supervised-fine-tuned (SFT) into the reference; humans rank sampled response pairs; those rankings fit a reward model r; PPO then optimizes the policy against r under a KL penalty to the reference, so the policy improves reward without drifting far from fluent text.

DPO. Direct preference optimization removes the reward model and the RL loop entirely. It shows the RLHF objective has a closed-form optimal policy, and that substituting it back yields a supervised classification loss on the preference pairs.¹²

Stage	Trains	Signal	Loop
Instruction tuning	the LM	instruction--response pairs	supervised
RLHF	reward model, then policy	human preference comparisons	RL (PPO)
DPO	the LM directly	human preference comparisons	supervised

DPO collapses the three-stage RLHF pipeline into one stable supervised step, which is why it has largely displaced PPO for preference alignment.

Takeaways

A large language model is a decoder-only Transformer trained on next-token prediction $L = - \sum_{t} log p_{θ} (x_{t} ∣ x_{< t})$ , then scaled; the architecture is fixed and scale is the design axis.
Subword tokenization (BPE merges, WordPiece by likelihood, Unigram by pruning, SentencePiece wrapping either) keeps $∣ V ∣ \approx 30$ -- $50$ k while spelling any string.
The four objectives are causal LM (GPT), masked LM (BERT), span corruption (T5), and prefix LM, distinguished entirely by the attention mask.
The three families: encoder-only (BERT/RoBERTa, understanding), decoder-only (GPT/LLaMA, generation), encoder--decoder (T5, transduction); two-thirds of the parameters live in the FFN.
Scaling laws make loss a power law in $N$ , $D$ , $C$ ; Chinchilla balances them at $N, D \propto C^{1/2}$ , about $20$ tokens per parameter.
Capabilities emerge sharply at scale, and in-context learning plus chain-of-thought let a frozen model solve new tasks from the prompt alone.
Decoding trades fidelity for diversity: greedy and beam maximize probability; temperature, top- $k$ , and nucleus sampling truncate the distribution, with top- $p$ adapting the cutoff to its shape.
The KV cache makes generation $O (n^{2} d)$ instead of $O (n^{3} d)$ but grows memory as $O (n d n_{layers})$ , the real bound on long context.
LoRA adapts a frozen model with a low-rank delta $Δ W = B A$ ( $r ≪ d$ ), cutting trainable parameters by orders of magnitude at no inference cost.
Alignment = instruction tuning (FLAN) + preference learning; RLHF fits a reward model and optimizes by PPO with a KL penalty, while DPO folds the same objective into one supervised log-sigmoid loss.

Kaplan et al., Scaling Laws for Neural Language Models, 2020 — establishes the power-law dependence of loss on model size, data, and compute over many orders of magnitude. ↩
Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla), 2022 — corrects the size/data balance: parameters and tokens should scale together, $\sim 20$ tokens per parameter. ↩
Wei et al., Emergent Abilities of Large Language Models, TMLR 2022 — abilities that are near-random below a scale threshold and rise sharply above it, invisible to small-model extrapolation. ↩
Brown et al., Language Models are Few-Shot Learners (GPT-3), NeurIPS 2020 — the $175$ B decoder-only model that demonstrated in-context (few-shot) learning without weight updates. ↩
Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022 — prompting for intermediate steps elicits multi-step reasoning, itself an emergent, scale-gated effect. ↩
Holtzman et al., The Curious Case of Neural Text Degeneration, ICLR 2020 — introduces nucleus (top- $p$ ) sampling and shows why maximizing likelihood (beam/greedy) yields degenerate open-ended text. ↩
Houlsby et al., Parameter-Efficient Transfer Learning for NLP, ICML 2019 — adapter modules: small bottleneck MLPs inserted between frozen layers, trained per task. ↩
Li & Liang, Prefix-Tuning: Optimizing Continuous Prompts for Generation, ACL 2021 — prepends trainable continuous virtual tokens to steer a frozen model through its own attention. ↩
Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022 — freezes the weights and learns a low-rank update $Δ W = B A$ , mergeable at inference with zero overhead. ↩
Wei et al., Finetuned Language Models Are Zero-Shot Learners (FLAN), ICLR 2022 — instruction tuning on a mixture of tasks yields zero-shot generalization to unseen instructions. ↩
Ouyang et al., Training Language Models to Follow Instructions with Human Feedback (InstructGPT), NeurIPS 2022 — the RLHF pipeline: SFT, a reward model on preference comparisons, then PPO with a KL penalty. ↩
Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, NeurIPS 2023 — derives the closed-form RLHF optimum and recasts alignment as one supervised log-sigmoid loss on preferences. ↩