--- title: "Scaling, Inference, and Alignment of Language Models" module: Large Models & Agents moduleNumber: 10 lessonNumber: 2 order: 1002 summary: > Once a language model is built, three questions remain: how does it improve as it grows, how is it decoded and served affordably, and how is a raw next-token predictor turned into an assistant. We derive the Kaplan power laws and the Chinchilla compute-optimal balance, trace emergent abilities and in-context learning, catalog the decoding strategies from greedy to nucleus sampling, work the KV cache that makes generation quadratic instead of cubic, cover parameter-efficient adaptation by low-rank updates (LoRA), and close on the alignment stack: instruction tuning, RLHF, and DPO. topics: [Large Models & Agents] sources: - book: Chollet ref: "Ch. 11 — Deep Learning for Text; large-scale language models" - book: Goodfellow ref: "Ch. 12 — Applications; large-scale language modeling postdates the 2016 text" --- This builds on [Large Language Models](/deep-learning/large-models-and-agents/large-language-models), which defined the object: a decoder-only Transformer trained by next-token prediction, its subword tokenizer, its pretraining objectives, and the three architecture families. That lesson fixed the architecture and named scale as the design axis. This one works out _why_ scale pays, in what proportion, and what a trained model has to become before it is useful: training, inference, and behavior, in that order. ## Scaling laws The reason scale became the dominant lever is that the loss is _predictable_. Over many orders of magnitude, test cross-entropy falls as a power law in each of model size $N$, data $D$, and compute $C$ when the other two are not the bottleneck.[^kaplan] > **Definition (Kaplan scaling laws).** With the other factors held > non-limiting, the loss obeys > $$ > L(N) \approx \parens{\frac{N_c}{N}}^{\alpha_N}, > \quad > L(D) \approx \parens{\frac{D_c}{D}}^{\alpha_D}, > \quad > L(C) \approx \parens{\frac{C_c}{C}}^{\alpha_C}, > $$ > with small positive exponents ($\alpha_N \approx 0.076$, $\alpha_D \approx 0.095$ > empirically), so each tenfold increase buys a fixed, diminishing decrement of > loss. On log--log axes each is a straight line of slope $-\alpha$. Given a fixed compute budget, the laws also dictate _how to split it_ between model size and data. The **Chinchilla** result corrects an earlier bias toward huge models trained on too little data.[^hoffmann] > **Theorem (Compute-optimal balance).** Using the FLOP count $C \approx 6 N D$ > for a Transformer and the joint loss > $L(N, D) = L_\infty + A N^{-\alpha} + B D^{-\beta}$ with $\alpha \approx \beta$, > the loss at fixed $C$ is minimized by > $$ > N_{\text{opt}} \propto C^{\,a}, \quad D_{\text{opt}} \propto C^{\,b}, > \qquad a = \frac{\beta}{\alpha+\beta},\;\; b = \frac{\alpha}{\alpha+\beta}, > $$ > so when $\alpha \approx \beta$ both exponents are near $1/2$: parameters and > tokens should grow together, at roughly $D / N \approx 20$ tokens per parameter. > **Proof.** Fix $C$ and write $D = C / 6N$. Substituting, > $L(N) = L_\infty + A N^{-\alpha} + B\,(6N/C)^{\beta}$. Differentiate and set to > zero: > $$ > \frac{dL}{dN} = -\alpha A N^{-\alpha-1} + \beta B\, 6^{\beta} C^{-\beta} N^{\beta-1} = 0. > $$ > Collecting powers of $N$ gives $N^{\alpha+\beta} = \tfrac{\alpha A}{\beta B 6^{\beta}} C^{\beta}$, > hence $N_{\text{opt}} \propto C^{\beta/(\alpha+\beta)}$. Then > $D_{\text{opt}} = C/6N_{\text{opt}} \propto C^{\,1-\beta/(\alpha+\beta)} = C^{\alpha/(\alpha+\beta)}$. > With $\alpha \approx \beta$ both exponents collapse to $1/2$, and the ratio > $D_{\text{opt}}/N_{\text{opt}} \propto C^{0}$ is constant. $\qed$ The practical content is a frontier: for each compute budget there is one optimal model size, and runs off that line waste compute by being either too small (unable to exploit the data) or too large (under-trained). $$ % caption: Compute-optimal frontier (log--log). For each compute budget the % optimal model size $N$ grows along a line of slope $1/2$ (blue); models that % are too large (under-trained) or too small sit off the line (red). \begin{tikzpicture}[>=stealth, font=\footnotesize, x=1.0cm, y=1.0cm] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \draw[->, thick] (0,0) -- (7.2,0) node[right, font=\footnotesize] {\texttt{log compute C}}; \draw[->, thick] (0,0) -- (0,4.8) node[above, font=\footnotesize] {\texttt{log optimal N}}; % the frontier: slope 1/2 \draw[acc, very thick] (0.5,0.7) -- (6.6,3.75) node[pos=0.7, above, sloped, text=acc, font=\footnotesize] {\texttt{slope 1/2}}; % optimal points on the line \fill[acc] (1.7,1.3) circle (1.9pt); \fill[acc] (3.6,2.25) circle (1.9pt); \fill[acc] (5.5,3.2) circle (1.9pt); % off-frontier runs \fill[red] (1.7,3.6) circle (1.9pt); \node[red, anchor=west, font=\footnotesize] at (1.85,3.6) {\texttt{too big (under-trained)}}; \fill[red] (5.9,1.6) circle (1.9pt); \node[red, anchor=east, font=\footnotesize] at (5.75,1.6) {\texttt{too small}}; \end{tikzpicture} $$ ## Emergent abilities and in-context learning Smoothness of loss coexists with sharp jumps in behavior. Some capabilities are **emergent**: near-random for small models, then rising abruptly past a scale threshold, so they are invisible by extrapolating small-model performance.[^wei-emergent] > **Definition (In-context learning).** A model performs a new task from examples > placed in the prompt, with no weight updates: the conditioning demonstrations > $(x_1, y_1), \dots, (x_k, y_k)$ steer the forward pass, and prediction reads off > $p_\theta(y \mid x_1, y_1, \dots, x_k, y_k, x)$. Few-shot prompting is inference, > not training; the gradient never runs.[^brown] The two curves are decoupled: the loss falls smoothly as a power law in scale, while a specific downstream metric stays flat at chance and then rises at a sharp knee. The loss curve therefore gives no warning that a capability is about to appear. $$ % caption: Loss versus a task metric across scale (log model size on x). The % cross-entropy loss (blue) falls smoothly as a power law; a downstream capability % (red) stays near chance, then rises abruptly past a scale threshold. Smooth loss % hides the discontinuous behavior. \begin{tikzpicture}[>=stealth, font=\footnotesize, x=1.0cm, y=1.0cm] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \draw[->, thick] (0,0) -- (6.6,0) node[right, font=\footnotesize] {\texttt{log model size}}; \draw[->, thick] (0,0) -- (0,3.6) node[above, font=\footnotesize] {\texttt{metric}}; % smooth decreasing loss (power law) \draw[acc, very thick] (0.4,3.2) .. controls (2.0,1.6) and (3.5,1.0) .. (6.2,0.7); \node[acc, anchor=west, font=\footnotesize] at (4.2,1.25) {\texttt{loss}}; % emergent metric: flat then sharp rise \draw[red, very thick] (0.4,0.35) -- (3.4,0.4) .. controls (4.0,0.5) and (4.3,2.6) .. (5.2,2.9) -- (6.2,3.0); \node[red, anchor=east, font=\footnotesize] at (6.1,2.55) {\texttt{accuracy}}; % threshold marker \draw[black, dashed] (4.0,0) -- (4.0,3.2); \node[black, anchor=north, font=\footnotesize] at (4.0,-0.15) {\texttt{threshold}}; \end{tikzpicture} $$ **Chain-of-thought** prompting elicits a further capability: asking the model to emit intermediate reasoning steps before the answer sharply improves performance on multi-step arithmetic and logic, and the gain itself is emergent, appearing only past a scale threshold.[^wei-cot] The phenomenon is that the same likelihood objective, pushed far enough, yields behaviors nobody trained for directly. ## Decoding A trained LM is a distribution $p_\theta(x_t \mid x_{ **Definition (Decoding strategies).** Given the next-token distribution > $p_t = p_\theta(\cdot \mid x_{ - **Greedy:** $x_t = \arg\max_{v} p_t(v)$. > - **Temperature:** rescale logits $z$ by $\tau$, $p_t(v) \propto \exp(z_v / \tau)$; > $\tau \to 0$ recovers greedy, $\tau > 1$ flattens. > - **Top-$k$:** renormalize over the $k$ highest-probability tokens. > - **Nucleus (top-$p$):** renormalize over the smallest set $V_p$ with > $\sum_{v \in V_p} p_t(v) \ge p$. **Beam search** keeps the $B$ highest-scoring partial sequences rather than one, maximizing total sequence log-probability; it is standard for translation but produces bland, repetitive open-ended text, because the globally most probable continuation is often degenerate.[^holtzman] Sampling methods fix this by drawing from a _truncated_ distribution, and nucleus sampling, truncating by cumulative mass rather than a fixed count, adapts the cutoff to how peaked $p_t$ is. | Strategy | Rule | Diversity | Typical use | | --- | --- | --- | --- | | Greedy | take the argmax | none | short factual answers | | Beam ($B$) | keep top-$B$ sequences | low | translation, summarization | | Temperature $\tau$ | scale logits by $1/\tau$ | tunable | creative text | | Top-$k$ | sample from $k$ likeliest | medium | with fixed budget | | Nucleus (top-$p$) | sample from mass $\ge p$ | adaptive | open-ended generation | ```algorithm caption: $\textsc{NucleusSample}(p_t, p)$ — sample one token from the top-$p$ nucleus sort vocabulary so $p_t(v_1) \ge p_t(v_2) \ge \dots$ $c \gets 0$; $V_p \gets \emptyset$ for $j \gets 1$ to $\abs{V}$ do $V_p \gets V_p \cup \{v_j\}$; $c \gets c + p_t(v_j)$ if $c \ge p$ then // smallest set covering mass $p$ break $q(v) \gets p_t(v) / c$ for $v \in V_p$, else $0$ // renormalize over the nucleus return $v \sim q$ ``` The figure shows how nucleus truncation keeps a few tokens on a peaked distribution but more on a flat one, which is the behavior a fixed top-$k$ cannot match. $$ % caption: Nucleus sampling keeps the smallest token set whose probability sums to % $p$. On a peaked step (left) that is two tokens; on a flat step (right), four. \begin{tikzpicture}[>=stealth, font=\scriptsize, x=1.0cm, y=1.0cm] \definecolor{acc}{HTML}{2348F2} % ---- peaked distribution ---- \begin{scope} \draw[->, thick] (0,0) -- (3.4,0) node[right, font=\footnotesize] {\texttt{tokens}}; \draw[->, thick] (0,0) -- (0,2.6) node[above, font=\footnotesize] {\texttt{prob}}; % kept bars (in nucleus): outlined accent \draw[draw=acc, thick, fill=acc!15] (0.2,0) rectangle (0.7,2.2); \draw[draw=acc, thick, fill=acc!15] (0.9,0) rectangle (1.4,1.2); % dropped bars: outlined black \draw[draw=black, thick, fill=black!8] (1.6,0) rectangle (2.1,0.4); \draw[draw=black, thick, fill=black!8] (2.3,0) rectangle (2.8,0.25); \node[acc, anchor=south, font=\footnotesize] at (1.5,2.3) {\texttt{kept}}; \node[black, anchor=north, font=\footnotesize] at (1.7,-0.2) {\texttt{peaked}}; \end{scope} % ---- flat distribution ---- \begin{scope}[xshift=5.0cm] \draw[->, thick] (0,0) -- (3.4,0) node[right, font=\footnotesize] {\texttt{tokens}}; \draw[->, thick] (0,0) -- (0,2.6) node[above, font=\footnotesize] {\texttt{prob}}; \draw[draw=acc, thick, fill=acc!15] (0.2,0) rectangle (0.7,1.1); \draw[draw=acc, thick, fill=acc!15] (0.9,0) rectangle (1.4,1.0); \draw[draw=acc, thick, fill=acc!15] (1.6,0) rectangle (2.1,0.85); \draw[draw=acc, thick, fill=acc!15] (2.3,0) rectangle (2.8,0.75); \node[acc, anchor=south, font=\footnotesize] at (1.5,1.6) {\texttt{wider nucleus}}; \node[black, anchor=north, font=\footnotesize] at (1.7,-0.2) {\texttt{flat}}; \end{scope} \end{tikzpicture} $$ ## KV cache and inference cost Autoregressive generation appears to cost $O(t^2)$ at step $t$: each new token recomputes attention over all $t$ positions. The **key--value cache** removes the redundancy. Once computed, the keys $K_{ **Theorem (Cached generation cost).** With a KV cache, generating token $t$ > costs $O(t \cdot d)$ time for the attention (one query against $t$ cached keys), > so generating a sequence of length $n$ costs $\sum_{t=1}^{n} O(t d) = O(n^2 d)$ > total, versus $O(n^3 d)$ without the cache. Cache memory grows as > $O(n \cdot d \cdot n_{\text{layers}})$. > **Proof.** Without caching, step $t$ recomputes all $t$ keys and values > ($O(t d^2)$) and the $t \times t$ attention ($O(t^2 d)$), summing to $O(n^3 d)$ > over $n$ steps. With caching, the past $K, V$ are read from memory, so step $t$ > computes only the new row: one query against $t$ keys, $O(t d)$, plus the new > $K, V$ projections $O(d^2)$. Summing the attention term over $t = 1, \dots, n$ > gives $O(n^2 d)$. Each layer stores $t$ key and value vectors of width $d$, so > after $n$ tokens the cache holds $O(n d n_{\text{layers}})$ entries. $\qed$ The cache trades memory for compute, and that memory becomes the binding constraint at long context: a $30$B model at $d = 7168$, $48$ layers, in $16$-bit, caches roughly $2 \cdot 48 \cdot 7168 \cdot 2$ bytes per token, about $1.4$ MB, so a $100{,}000$-token context costs $\sim 140$ GB of cache alone. This memory growth, not the arithmetic, is why long-context inference is expensive. $$ % caption: KV cache. Past keys and values (black) are stored once and reused; each % new token computes only its own query, key, value (blue) and attends over the % cache. \begin{tikzpicture}[>=stealth, font=\scriptsize, cell/.style={draw, black, minimum width=8mm, minimum height=6mm, inner sep=1pt, align=center}, new/.style={draw=acc, text=acc, thick, minimum width=8mm, minimum height=6mm, inner sep=1pt, align=center, fill=acc!15}] \definecolor{acc}{HTML}{2348F2} % cached K,V row \node[anchor=east, black] at (-0.2,1.0) {cache}; \node[cell] (k1) at (0.6,1.0) {$k_1$}; \node[cell] (k2) at (1.5,1.0) {$k_2$}; \node[cell] (k3) at (2.4,1.0) {$k_3$}; \node[new] (k4) at (3.3,1.0) {$k_4$}; \node[anchor=east, black] at (-0.2,0.0) {values}; \node[cell] (v1) at (0.6,0.0) {$v_1$}; \node[cell] (v2) at (1.5,0.0) {$v_2$}; \node[cell] (v3) at (2.4,0.0) {$v_3$}; \node[new] (v4) at (3.3,0.0) {$v_4$}; % new query \node[new] (q) at (3.3,2.4) {$q_4$}; % attention edges from q to all keys \draw[->, black, thick] (q) -- (k1); \draw[->, black, thick] (q) -- (k2); \draw[->, black, thick] (q) -- (k3); \draw[->, acc, thick] (q) -- (k4); \node[anchor=west, black, font=\scriptsize] at (4.0,1.7) {attend over cache}; \end{tikzpicture} $$ ## Parameter-efficient adaptation Full fine-tuning updates all $N$ parameters and stores a separate copy of the model per task, infeasible at billions of parameters. Parameter-efficient methods adapt a frozen model by training a small number of new parameters. **Adapters** insert small bottleneck MLPs between frozen layers and train only those; they add $< 4\%$ parameters but introduce sequential layers that slow inference.[^houlsby] **Prefix / prompt tuning** prepends a few trainable "virtual-token" vectors to the keys and values, steering a frozen model through the attention itself.[^li-liang] The dominant method, **LoRA**, freezes the weights and learns a low-rank update.[^hu] > **Definition (Low-rank adaptation, LoRA).** For a frozen weight matrix > $W_0 \in \mathbb{R}^{d \times d}$, parametrize the update as a product of two thin > matrices, > $$ > W = W_0 + \Delta W, \qquad \Delta W = B A, \quad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times d},\; r \ll d, > $$ > and train only $A, B$. The forward pass is > $h = W_0 x + B(A x)$; $A$ is initialized random, $B$ to zero, so training starts > at the pretrained model. The trainable count drops from $d^2$ to $2rd$. For example, with $d = 4096$ and $r = 8$, the update has $2 \cdot 8 \cdot 4096 = 65{,}536$ parameters instead of $4096^2 = 16.7$M, a $256\times$ reduction, with no inference-time cost because $BA$ can be folded back into $W_0$ after training. The assumption is that the _change_ a task induces is intrinsically low-rank even though $W_0$ is full rank. $$ % caption: LoRA. The frozen weight $W_0$ ($d \times d$) is left untouched; the % update is the product of two thin matrices $B$ ($d \times r$) and $A$ % ($r \times d$) with $r \ll d$, the only trained parameters. \begin{tikzpicture}[>=stealth, font=\footnotesize, x=1.0cm, y=1.0cm] \definecolor{acc}{HTML}{2348F2} % frozen W0: large square \draw[draw=black, thick, fill=black!8] (0,0) rectangle (2.4,2.4); \node[black] at (1.2,1.2) {$W_0$ frozen}; \node[black, anchor=north, font=\scriptsize] at (1.2,-0.1) {$d$ by $d$}; % plus \node[black] at (3.1,1.2) {$+$}; % B: tall thin \draw[draw=acc, thick, fill=acc!15] (3.8,0) rectangle (4.4,2.4); \node[acc] at (4.1,1.2) {$B$}; \node[acc, anchor=north, font=\scriptsize] at (4.1,-0.1) {$d$ by $r$}; % A: wide short \draw[draw=acc, thick, fill=acc!15] (4.8,0.9) rectangle (7.2,1.5); \node[acc] at (6.0,1.2) {$A$}; \node[acc, anchor=north, font=\scriptsize] at (6.0,0.8) {$r$ by $d$}; % brace label \node[acc, anchor=south, font=\scriptsize] at (5.6,2.4) {trained (rank $r$, small)}; \end{tikzpicture} $$ | Method | Trainable params | Inference overhead | Mechanism | | --- | --- | --- | --- | | Full fine-tuning | $N$ (all) | none | update every weight | | Adapters | $\sim 2$--$4\%$ | added layers (slower) | bottleneck MLP per block | | Prefix / prompt | $\ll 1\%$ | longer keys/values | trainable virtual tokens | | LoRA | $2rd$ per matrix | none (merge $BA$) | low-rank weight delta | ## Alignment A pretrained LM predicts likely text, not helpful text. **Alignment** is the stack that turns the raw next-token predictor into an assistant: supervised instruction tuning, then learning from human preferences. **Instruction tuning (FLAN).** Fine-tune on a large mixture of tasks phrased as natural-language instructions; the model generalizes to held-out instructions it never saw, a zero-shot ability that itself improves with the number of tuning tasks.[^wei-flan] This is plain supervised learning on $(\text{instruction}, \text{response})$ pairs. **RLHF.** Instruction tuning teaches format, not preference. **Reinforcement learning from human feedback** fits a reward model $r_\phi$ to human pairwise comparisons, then optimizes the policy $\pi_\theta$ against it.[^ouyang] The reward model is trained on the Bradley--Terry likelihood that the preferred response $y_w$ beats the rejected $y_l$: $$ \mathcal{L}_{\text{RM}}(\phi) = -\,\mathbb{E}_{(x, y_w, y_l)}\brackets{\log \sigma\!\parens{r_\phi(x, y_w) - r_\phi(x, y_l)}}. $$ The policy is then optimized by PPO to maximize reward while a KL penalty keeps it near the supervised reference $\pi_{\text{ref}}$, preventing reward hacking: $$ \max_{\theta}\; \mathbb{E}_{x,\, y \sim \pi_\theta}\brackets{ r_\phi(x, y) } \;-\; \beta\, \mathbb{E}_{x}\brackets{ \KL\!\parens{\pi_\theta(\cdot \mid x) \,\|\, \pi_{\text{ref}}(\cdot \mid x)} }. $$ ```algorithm caption: $\textsc{RLHF}$ — align a pretrained LM from human preferences $\pi_{\text{ref}} \gets$ instruction-tuned (SFT) model collect comparisons: for prompts $x$, humans rank pairs $(y_w, y_l)$ train reward model $r_\phi$ on the Bradley--Terry loss $\mathcal{L}_{\text{RM}}$ $\pi_\theta \gets \pi_{\text{ref}}$ repeat sample responses $y \sim \pi_\theta(\cdot \mid x)$ score with $r_\phi$; subtract KL penalty to $\pi_{\text{ref}}$ update $\theta$ by PPO on the penalized reward // policy gradient step until reward plateaus return $\pi_\theta$ ``` The pipeline has three stages, each consuming the output of the last: a supervised model becomes the reference, pairwise human comparisons train a scalar reward, and the policy is optimized for high reward while a KL penalty keeps it near the reference. $$ % caption: The RLHF pipeline. A pretrained model is first supervised-fine-tuned % (SFT) into the reference; humans rank sampled response pairs; those rankings fit a % reward model r; PPO then optimizes the policy against r under a KL penalty to the % reference, so the policy improves reward without drifting far from fluent text. \begin{tikzpicture}[>=stealth, font=\footnotesize, x=1.0cm, y=1.0cm, box/.style={draw, black, minimum width=17mm, minimum height=9mm, align=center, inner sep=2pt}, acc/.style={draw=acc, text=acc, thick, minimum width=17mm, minimum height=9mm, align=center, inner sep=2pt, fill=acc!12}] \definecolor{acc}{HTML}{2348F2} \node[box] (pt) at (0,0) {pretrained\\ LM}; \node[box] (sft) at (2.6,0) {SFT\\ reference}; \node[box] (rm) at (5.4,0) {reward\\ model r}; \node[acc] (pol) at (8.2,0) {policy\\ (PPO)}; \draw[->, thick] (pt) -- (sft) node[midway, above, font=\scriptsize] {instr.}; \draw[->, thick] (sft) -- (rm) node[midway, above, font=\scriptsize] {rankings}; \draw[->, acc, thick] (rm) -- (pol) node[midway, above, text=acc, font=\footnotesize] {\texttt{reward}}; % KL leash back to reference \draw[->, black, thick, dashed] (sft) to[out=-60, in=-120] node[midway, below, font=\scriptsize] {KL leash} (pol); % human comparisons feeding the reward model \node[black, anchor=south, font=\scriptsize] at (5.4,1.15) {human pairs}; \draw[->, black] (5.4,1.0) -- (rm); \end{tikzpicture} $$ **DPO.** Direct preference optimization removes the reward model and the RL loop entirely. It shows the RLHF objective has a closed-form optimal policy, and that substituting it back yields a _supervised_ classification loss on the preference pairs.[^rafailov] > **Theorem (DPO loss).** The RLHF objective is optimized directly by minimizing > $$ > \mathcal{L}_{\text{DPO}}(\theta) = -\,\mathbb{E}_{(x, y_w, y_l)}\!\brackets{ > \log \sigma\!\parens{ \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} > - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} } }, > $$ > a simple binary-classification loss over preference pairs, with no reward model > and no sampling. > **Proof sketch.** The KL-penalized objective has the analytic optimum > $\pi^{\ast}(y \mid x) \propto \pi_{\text{ref}}(y \mid x)\exp\!\parens{r(x,y)/\beta}$. > Inverting gives the implicit reward > $r(x,y) = \beta \log \tfrac{\pi^{\ast}(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)$. > Substituting this $r$ into the Bradley--Terry likelihood, the partition term > $Z(x)$ cancels in the difference $r(x,y_w) - r(x,y_l)$, leaving exactly the > log-sigmoid of the two log-ratios above. Maximizing that likelihood in $\theta$ > is DPO. $\qed$ | Stage | Trains | Signal | Loop | | --- | --- | --- | --- | | Instruction tuning | the LM | instruction--response pairs | supervised | | RLHF | reward model, then policy | human preference comparisons | RL (PPO) | | DPO | the LM directly | human preference comparisons | supervised | DPO collapses the three-stage RLHF pipeline into one stable supervised step, which is why it has largely displaced PPO for preference alignment. ## Takeaways - A **large language model** is a decoder-only Transformer trained on next-token prediction $\mathcal{L} = -\sum_t \log p_\theta(x_t \mid x_{