--- title: Reinforcement Learning from Human Feedback module: Reinforcement Learning moduleNumber: 11 lessonNumber: 5 order: 1105 summary: > Many objectives we want from a model, that it be helpful and harmless, are hard to write down but easy to judge by comparison. RLHF turns that asymmetry into a training signal: fit a reward model to pairwise human preferences under the Bradley-Terry likelihood, then fine-tune the policy to maximize that reward under a KL penalty toward a reference. We derive the reward loss, the KL-regularized RL objective and its closed-form optimum, then show how DPO inverts that optimum to collapse the whole pipeline into one supervised log-sigmoid loss, and survey IPO, KTO, RLAIF, and GRPO. topics: [Reinforcement Learning] sources: - book: Chollet ref: "Ch. 11 — aligning generative language models" --- The [large language model](/deep-learning/large-models-and-agents/large-language-models) lesson sketched RLHF in a paragraph: fit a reward model, optimize with PPO, regularize with a KL term. This lesson is the derivation. We build the reward model from the Bradley-Terry likelihood, write the KL-regularized objective the policy actually maximizes, solve it in closed form, and then show that the same closed form, inverted, turns the entire reinforcement-learning loop into a single supervised loss. The [policy gradients and PPO](/deep-learning/reinforcement-learning/policy-gradients-and-actor-critic) lesson supplies the optimizer; the [foundations of RL](/deep-learning/reinforcement-learning/foundations-of-reinforcement-learning) lesson supplies the language of policies, rewards, and returns we reuse throughout. ## The specification problem The objectives we want from an assistant, that it be helpful, harmless, and honest, resist direct specification. There is no loss function for "helpful." But a human shown two responses to the same prompt can reliably say which is better, even when they could not have written down a rule that produces the better one. RLHF is built on exactly this asymmetry. > **Definition (Preference comparison).** For a prompt $x$ and two candidate > responses $y_1, y_2$, a human annotator returns a binary preference $y_w \succ y_l$ > ("$y_w$ is preferred to $y_l$"), where $w$ marks the winner and $l$ the loser. A > preference dataset is a collection $\mathcal{D} = \set{(x^{(i)}, y_w^{(i)}, y_l^{(i)})}_{i=1}^{N}$ > of such judgments. The plan is to learn a scalar **reward** $r(x, y)$ that ranks responses the way humans do, then push a language model toward high-reward responses. The reward converts a set of discrete comparisons into a differentiable signal a policy can ascend. | Quantity | Hard to specify | Easy to elicit | | --- | --- | --- | | Absolute quality of one response | yes (no scalar in the annotator's head) | no | | Which of two responses is better | no | yes (a single click) | | A rule that generates good responses | yes | no | | A reward that _ranks_ responses | learned from comparisons | -- | ## The three-stage pipeline RLHF as deployed in modern instruction-tuned systems is three stages run in sequence. First, **supervised fine-tuning** (SFT) on curated demonstrations teaches the base model the _format_ of an answer and gives a reference policy $\pi_{\text{ref}}$. Second, a **reward model** $r_\phi$ is fit to human comparisons. Third, the policy is **RL-fine-tuned** to maximize $r_\phi$ while a KL penalty holds it near $\pi_{\text{ref}}$. $$ % caption: The three-stage RLHF pipeline. SFT yields the reference policy; the reward % model learns from human comparisons; RL fine-tunes the policy under a KL leash. \begin{tikzpicture}[>=stealth, font=\footnotesize, stage/.style={draw=acc, text=acc, thick, fill=acc!12, minimum width=27mm, minimum height=13mm, align=center}, data/.style={draw=black, thick, fill=black!8, minimum width=24mm, minimum height=11mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} % data sources (top row) \node[data] (demo) at (0,2.2) {demonstrations}; \node[data] (comp) at (4.6,2.2) {human compari-\\sons}; \node[data] (prompts) at (9.2,2.2) {prompts only}; % stages (bottom row) \node[stage] (sft) at (0,0) {1. SFT}; \node[stage] (rm) at (4.6,0) {2. reward model}; \node[stage] (rl) at (9.2,0) {3. RL f\/ine-tune}; % vertical data -> stage \draw[->, black, thick] (demo) -- (sft); \draw[->, black, thick] (comp) -- (rm); \draw[->, black, thick] (prompts) -- (rl); % horizontal flow \draw[->, acc, thick] (sft) -- (rm) node[midway, below, font=\scriptsize, text=acc] {ref}; \draw[->, acc, thick] (rm) -- (rl) node[midway, below, font=\scriptsize, text=acc] {reward}; % KL leash back from rl to sft \draw[->, black, thick, dashed] (rl.east) .. controls (12.6,0.6) and (12.6,4.9) .. (4.6,4.9) node[above, font=\scriptsize, text=black] {KL leash to reference} .. controls (-3.4,4.9) and (-3.4,0.6) .. (sft.west); \end{tikzpicture} $$ The three stages consume three different kinds of data, and that is the point: each stage extracts the signal that is cheapest to collect for what it must learn. | Stage | Data | Trains | Loss type | Output | | --- | --- | --- | --- | --- | | 1. SFT | $(x, y)$ demonstrations | base LM | supervised cross-entropy | $\pi_{\text{ref}}$ | | 2. Reward model | $(x, y_w, y_l)$ comparisons | a scalar head | Bradley-Terry log-loss | $r_\phi$ | | 3. RL fine-tune | prompts $x$ only | the policy | KL-regularized reward | $\pi_\theta$ | ## Reward modeling from pairwise preferences We need a model that, given a comparison $(x, y_w, y_l)$, assigns a high probability to the event "$y_w$ beats $y_l$." The **Bradley-Terry** model is the canonical choice: it posits a latent scalar score per item and makes the probability of one beating another a logistic function of their score difference. > **Definition (Bradley-Terry preference model).** Each response carries a latent > reward $r(x, y) \in \mathbb{R}$, and the probability that $y_w$ is preferred to $y_l$ > is the logistic of the reward gap, > $$ > P(y_w \succ y_l \mid x) \;=\; \sigma\!\parens{r(x, y_w) - r(x, y_l)} > \;=\; \frac{1}{1 + \exp\!\parens{-\parens{r(x, y_w) - r(x, y_l)}}}. > $$ > Only the _difference_ of rewards is identified: adding any function $c(x)$ to every > reward leaves all preference probabilities unchanged. The model has the right limits. When $r(x, y_w) \gg r(x, y_l)$ the gap is large and positive, so $P \to 1$; when the two rewards are equal the gap is zero and $P = \tfrac12$, a coin flip; and the curve is symmetric, so $P(y_w \succ y_l) + P(y_l \succ y_w) = 1$. $$ % caption: Bradley-Terry: preference probability is the logistic of the reward gap % $\Delta = r(x,y_w) - r(x,y_l)$, passing through $\tfrac12$ at $\Delta = 0$. \begin{tikzpicture}[>=stealth, font=\footnotesize, x=1.0cm, y=1.0cm] \definecolor{acc}{HTML}{2348F2} % axes \draw[->, thick] (-3.2,0) -- (3.4,0) node[right, font=\scriptsize] {reward gap}; \draw[->, thick] (0,-0.2) -- (0,2.6) node[above, font=\scriptsize] {P(win)}; % half line (start after the label gap so it never runs over the tick text) \draw[black, dashed] (-2.8,1.1) -- (3.2,1.1); \node[black, anchor=east, font=\scriptsize] at (-3.0,1.1) {1/2}; % one line \draw[black, dashed] (-2.8,2.2) -- (3.2,2.2); \node[black, anchor=east, font=\scriptsize] at (-3.0,2.2) {1}; % logistic curve, scaled: y = 2.2 * sigmoid(1.6 x) \draw[acc, very thick] plot[domain=-3.2:3.2, samples=80] (\x, {2.2/(1+exp(-1.6*\x))}); % marker at gap 0 \fill[acc] (0,1.1) circle (2pt); \node[acc, anchor=north west, font=\scriptsize] at (1.55,0.95) {logistic}; \end{tikzpicture} $$ ### Deriving the reward loss We fit $r_\phi$ by maximum likelihood on the comparison dataset. Treat each comparison as one Bernoulli observation whose outcome is "the labeled winner won." The likelihood of a single comparison is $P(y_w \succ y_l \mid x) = \sigma(\Delta_\phi)$ with the shorthand $\Delta_\phi = r_\phi(x, y_w) - r_\phi(x, y_l)$. The dataset log-likelihood is the sum over independent comparisons, $$ \ell(\phi) \;=\; \sum_{(x, y_w, y_l) \in \mathcal{D}} \log \sigma\!\parens{\Delta_\phi}. $$ Maximizing the log-likelihood is minimizing its negative, normalized to an expectation over the dataset. This is the reward-model objective. > **Definition (Reward-model loss).** The reward model $r_\phi$ minimizes the negative > log-likelihood of the observed preferences under Bradley-Terry, > $$ > \mathcal{L}_{\text{RM}}(\phi) > \;=\; -\,\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} > \brackets{\log \sigma\!\parens{r_\phi(x, y_w) - r_\phi(x, y_l)}}. > $$ > Architecturally, $r_\phi$ is the SFT model with the next-token head replaced by a > scalar head reading the final hidden state of the last token. The gradient explains what the loss does mechanically. Using $\sigma'(z) = \sigma(z)\parens{1 - \sigma(z)}$ and $\dfrac{d}{dz}\log\sigma(z) = 1 - \sigma(z)$, the per-comparison gradient is $$ \nabla_\phi\!\brackets{-\log\sigma(\Delta_\phi)} \;=\; -\parens{1 - \sigma(\Delta_\phi)}\,\nabla_\phi \Delta_\phi \;=\; -\,\underbrace{\sigma(-\Delta_\phi)}_{\text{prob. of being wrong}}\, \parens{\nabla_\phi r_\phi(x, y_w) - \nabla_\phi r_\phi(x, y_l)}. $$ The update raises the winner's reward and lowers the loser's, scaled by $\sigma(-\Delta_\phi)$, the model's current probability of _mis_-ranking the pair. Already-correct comparisons (large $\Delta_\phi$) get a vanishing gradient; only the contested pairs move the weights. This is the same self-weighting that makes logistic regression focus on the margin. For example, suppose the model already ranks a pair correctly with $\Delta_\phi = 4$: then $\sigma(-4) \approx 0.018$, so the pair's gradient is scaled by under two percent and it barely nudges the weights. Suppose instead the model ranks a pair backwards with $\Delta_\phi = -2$: then $\sigma(2) \approx 0.88$, an eighty-eight percent weight, and the update pushes hard to reverse the ordering. The loss concentrates its gradient on currently-misranked pairs and contributes almost nothing for pairs already ranked confidently, which is the correct behavior for a ranking objective. The identifiability caveat carries a practical consequence. Because only reward _differences_ enter the loss, $r_\phi$ is pinned only up to a per-prompt additive constant $c(x)$: the scale of the scores is fixed by the logistic, but their offset is free. When the reward feeds the RL step this is harmless, since the KL term and the advantage baseline both subtract off any prompt-level constant. It does mean reward magnitudes are not comparable across prompts, so a raw $r_\phi(x, y)$ is a within-prompt ranking signal, not an absolute quality score. ## The RL fine-tuning step With $r_\phi$ fixed, stage three optimizes the policy. Naively maximizing expected reward fails: the reward model is an imperfect proxy fit on finite data, and an unconstrained optimizer will find responses that score highly under $r_\phi$ yet are degenerate to a human. To address this, _anchor_ the policy to the SFT reference with a KL penalty. > **Definition (KL-regularized RL objective).** Given a reward $r$ and reference policy > $\pi_{\text{ref}}$, the RLHF policy maximizes expected reward minus a KL penalty, > $$ > \max_{\pi}\; > \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi(\cdot \mid x)} > \brackets{ r(x, y) } > \;-\; \beta\, \mathbb{E}_{x \sim \mathcal{D}} > \brackets{ \KL\!\parens{\pi(\cdot \mid x) \,\|\, \pi_{\text{ref}}(\cdot \mid x)} }, > $$ > where $\beta > 0$ sets the strength of the leash. As $\beta \to 0$ the policy maximizes > reward unconstrained; as $\beta \to \infty$ it is pinned to $\pi_{\text{ref}}$. The KL term serves two purposes. It keeps the policy in the region where the reward model was trained and is therefore trustworthy, and it preserves the fluency and breadth the base model already has, which the sparse reward would otherwise erode. Without the anchor three failures follow directly. The optimizer exploits the reward model's blind spots and finds high-scoring gibberish (reward hacking); it collapses onto the single highest-reward response and loses diversity (mode collapse); and it drifts so far from the data $\pi_{\text{ref}}$ was fit on that $r_\phi$ is extrapolating rather than judging (distribution drift). The two terms pull in opposite directions: reward pulls the policy away from the reference toward high-scoring responses, and the KL pulls it back toward the reference, with $\beta$ setting the balance point. $$ % caption: The KL-anchored objective as a tug-of-war. Reward pulls the policy away from % the reference toward high-scoring responses; the KL penalty pulls it back. The % coefficient $\beta$ sets the balance point. \begin{tikzpicture}[>=stealth, font=\footnotesize, anchor2/.style={draw=black, thick, fill=black!8, minimum width=22mm, minimum height=12mm, align=center, font=\scriptsize}, pol/.style={draw=acc, text=acc, thick, fill=acc!12, minimum width=22mm, minimum height=12mm, align=center, font=\scriptsize}, hi/.style={draw=red, text=red, thick, fill=red!8, minimum width=22mm, minimum height=12mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[anchor2] (ref) at (0,0) {reference (SFT)}; \node[pol] (pi) at (5.0,0) {policy}; \node[hi] (hi) at (10.0,0) {high reward}; % KL pulls policy back to reference \draw[->, black, very thick] (pi.west) -- (ref.east) node[midway, above, font=\scriptsize, text=black] {KL pulls back}; % reward pulls policy toward high reward \draw[->, red, very thick] (pi.east) -- (hi.west) node[midway, above, font=\scriptsize, text=red] {reward pulls away}; % balance annotation \node[acc, anchor=north, font=\scriptsize] at (5.0,-0.95) {balance set by beta}; \end{tikzpicture} $$ ### Per-token reward shaping Generation is a token-level Markov decision process: the state is the prompt plus the tokens emitted so far, the action is the next token, and the trajectory reward $r_\phi(x, y)$ arrives only at the terminal token. To turn the sequence-level objective into a dense per-step signal PPO can consume, the KL penalty is distributed across tokens, giving a per-token reward $$ \tilde r_t \;=\; \underbrace{r_\phi(x, y)\,\mathbb{1}[t = T]}_{\text{terminal reward}} \;-\; \beta\,\underbrace{\log \frac{\pi_\theta(y_t \mid x, y_{=stealth, font=\footnotesize, tok/.style={draw=black, thick, fill=black!8, minimum width=10mm, minimum height=8mm, inner sep=1pt, align=center}, endtok/.style={draw=acc, text=acc, thick, fill=acc!12, minimum width=10mm, minimum height=8mm, inner sep=1pt, align=center}] \definecolor{acc}{HTML}{2348F2} % token chain \node[tok] (t1) at (0,0) {$y_1$}; \node[tok] (t2) at (1.5,0) {$y_2$}; \node[tok] (t3) at (3.0,0) {$y_3$}; \node[font=\footnotesize, black] (dots) at (4.3,0) {. . .}; \node[endtok] (tT) at (5.6,0) {$y_T$}; \draw[->, black, thick] (t1) -- (t2); \draw[->, black, thick] (t2) -- (t3); \draw[->, black, thick] (t3) -- (dots); \draw[->, black, thick] (dots) -- (tT); % per-token KL penalty: each arrow points down AT a label that sits clear below it \foreach \n in {t1,t2,t3,tT} \draw[->, black, thick] (\n.south) -- ++(0,-0.5); \node[black, anchor=north, font=\scriptsize] at (0,-1.02) {KL}; \node[black, anchor=north, font=\scriptsize] at (1.5,-1.02) {KL}; \node[black, anchor=north, font=\scriptsize] at (3.0,-1.02) {KL}; \node[black, anchor=north, font=\scriptsize] at (5.6,-1.02) {KL}; % terminal reward: arrow points up AT the label above it \draw[->, acc, thick] (tT.north) -- ++(0,0.55); \node[acc, anchor=south, font=\scriptsize] at (5.6,1.12) {+ reward}; \end{tikzpicture} $$ ### PPO as the optimizer The shaped reward feeds a standard actor-critic loop. A **value head** $V_\psi$ is attached to the policy to estimate the expected return from each token, and the advantage $\hat A_t$ (typically by generalized advantage estimation) measures how much better the chosen token did than the value baseline. PPO then takes a clipped policy-gradient step on $\hat A_t$, which the [policy gradients lesson](/deep-learning/reinforcement-learning/policy-gradients-and-actor-critic) develops in full. The clip is what keeps each update small enough that the on-policy reward estimates stay valid. ```algorithm caption: $\textsc{RLHF}(\pi_{\text{ref}}, \mathcal{D}_{\text{cmp}}, \mathcal{D}_{\text{prompt}}, \beta)$ — align a policy from preferences train reward model $r_\phi$ by minimizing $\mathcal{L}_{\text{RM}}$ on $\mathcal{D}_{\text{cmp}}$ $\pi_\theta \gets \pi_{\text{ref}}$; initialize value head $V_\psi$ repeat sample prompts $x \sim \mathcal{D}_{\text{prompt}}$ and rollouts $y \sim \pi_\theta(\cdot \mid x)$ score each rollout: $r \gets r_\phi(x, y)$ // terminal reward for each token $t$ do $\tilde r_t \gets r\cdot\mathbb{1}[t{=}T] - \beta\log\dfrac{\pi_\theta(y_t \mid x, y_{ **Definition (Reward over-optimization).** As the policy is pushed to higher proxy > reward $r_\phi$, the _true_ human-judged quality first rises with it, then peaks and > falls, while $r_\phi$ keeps climbing. The policy has learned to exploit errors in the > reward model rather than to improve. The gap typically widens with the KL distance > $\sqrt{\KL(\pi_\theta \| \pi_{\text{ref}})}$ traveled from the reference. The KL coefficient $\beta$ trades these two failure directions against each other. Too large, and the policy never moves; too small, and it over-optimizes the proxy into degeneracy. $$ % caption: Reward over-optimization. True quality (blue) rises then falls as the policy % travels in KL from the reference, while the proxy reward (red) keeps climbing. \begin{tikzpicture}[>=stealth, font=\footnotesize, x=1.0cm, y=1.0cm] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \draw[->, thick] (0,0) -- (6.6,0) node[right, font=\scriptsize] {KL from reference}; \draw[->, thick] (0,0) -- (0,3.4) node[above, font=\scriptsize] {quality}; % proxy reward: monotone rising, concave \draw[red, very thick] plot[domain=0.1:6.2, samples=80] (\x, {3.0*(1 - exp(-0.55*\x))}); \node[red, anchor=south, font=\scriptsize] at (4.9,1.45) {proxy reward}; % true quality: rise then fall (a hump) \draw[acc, very thick] plot[domain=0.1:6.2, samples=80] (\x, {2.7*exp(-0.5*(\x-2.3)*(\x-2.3)/1.9)}); \node[acc, anchor=south, font=\scriptsize] at (2.3,2.85) {true quality}; % peak marker \draw[black, dashed] (2.3,0) -- (2.3,2.7); \node[black, anchor=north, font=\scriptsize] at (2.3,-0.1) {best KL}; \end{tikzpicture} $$ Two further levers control the same trade-off. The **value head** must be well-fit or the advantage estimates are biased, which destabilizes the policy step; in practice the value and policy share the backbone but use separate heads. And **advantage estimation** by GAE introduces a bias-variance parameter ($\lambda$) that, set well, smooths the sparse terminal reward into a usable per-token learning signal without injecting too much variance from long rollouts. ## Direct preference optimization PPO-based RLHF is operationally heavy: a separate reward model, a value head, on-policy sampling, and a famously finicky RL loop. **Direct preference optimization** (DPO) removes all of it by exploiting a fact about the KL-regularized objective: its optimum has a closed form, and that closed form can be inverted to express the reward in terms of the policy. Substituting the inverted reward into the Bradley-Terry loss yields a _supervised_ objective on the policy directly, with no reward model and no sampling. > **Theorem (DPO objective).** The policy that maximizes the KL-regularized RL objective > for a reward $r$ is the same policy that minimizes the supervised loss > $$ > \mathcal{L}_{\text{DPO}}(\theta) > = -\,\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\! > \brackets{ > \log \sigma\!\parens{ > \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} > - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} > } > }, > $$ > a binary-classification loss over preference pairs that needs no explicit reward > model, no value head, and no on-policy rollouts. > **Proof.** Fix a prompt $x$ and maximize the per-prompt objective > $\mathbb{E}_{y \sim \pi}[r(x,y)] - \beta\KL\parens{\pi \,\|\, \pi_{\text{ref}}}$ > over distributions $\pi(\cdot \mid x)$. Expanding the KL and dropping the constant > $\log Z$ that we introduce next, the objective equals > $$ > -\beta \sum_y \pi(y \mid x)\, > \log \frac{\pi(y \mid x)}{\pi_{\text{ref}}(y \mid x)\exp\!\parens{r(x,y)/\beta}}. > $$ > Define $Z(x) = \sum_y \pi_{\text{ref}}(y \mid x)\exp\!\parens{r(x,y)/\beta}$ and the > normalized distribution > $\pi^{\ast}(y \mid x) = \tfrac{1}{Z(x)}\pi_{\text{ref}}(y \mid x)\exp\!\parens{r(x,y)/\beta}$. > Multiplying and dividing the argument of the log by $Z(x)$ rewrites the objective as > $-\beta\KL\parens{\pi \,\|\, \pi^{\ast}} + \beta\log Z(x)$, and since > $\log Z(x)$ is independent of $\pi$, the objective is maximized exactly when the KL is > zero, i.e. $\pi = \pi^{\ast}$: > $$ > \pi^{\ast}(y \mid x) > \;=\; \frac{1}{Z(x)}\,\pi_{\text{ref}}(y \mid x)\, > \exp\!\parens{\tfrac{1}{\beta}\,r(x, y)}. > $$ > This is the closed-form optimum. Now _invert_ it: solve for the reward by taking the > log of both sides, > $$ > r(x, y) > \;=\; \beta \log \frac{\pi^{\ast}(y \mid x)}{\pi_{\text{ref}}(y \mid x)} \;+\; \beta \log Z(x). > $$ > Substitute this reward into the Bradley-Terry probability > $P(y_w \succ y_l \mid x) = \sigma\parens{r(x, y_w) - r(x, y_l)}$. The intractable > partition term $\beta\log Z(x)$ appears in _both_ rewards and cancels in the > difference, leaving > $$ > r(x, y_w) - r(x, y_l) > = \beta \log \frac{\pi^{\ast}(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} > - \beta \log \frac{\pi^{\ast}(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}. > $$ > Maximizing the Bradley-Terry likelihood of the observed preferences in $\pi^{\ast}$, > parametrized as $\pi_\theta$, is the same as minimizing the negative log-sigmoid of > this difference, which is $\mathcal{L}_{\text{DPO}}(\theta)$. $\qed$ The derivation is a change of variable: the reward and the optimal policy are two encodings of the same object, and DPO works in policy space so it never has to fit the reward at all. The implicit reward DPO assigns to a response is the quantity it pushes up or down. $$ % caption: The DPO change of variable. The closed-form optimum maps a reward to a policy % ($\pi^{*}\propto\pi_{\text{ref}}\exp(r/\beta)$); taking logs inverts it, mapping the % policy back to a reward ($r=\beta\log(\pi/\pi_{\text{ref}})+\beta\log Z$). DPO fits the % policy side, so the reward is never instantiated. \begin{tikzpicture}[>=stealth, font=\footnotesize, spc/.style={draw=black, thick, fill=black!8, minimum width=32mm, minimum height=15mm, align=center, font=\scriptsize}, psp/.style={draw=acc, text=acc, thick, fill=acc!12, minimum width=32mm, minimum height=15mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[spc] (rew) at (0,0) {reward space\\reward model}; \node[psp] (pol) at (7.0,0) {policy space\\policy vs reference}; % forward: reward -> policy (closed-form optimum) \draw[->, black, thick] (rew.north east) .. controls (3.5,1.5) .. (pol.north west) node[midway, above, font=\scriptsize, text=black] {optimum: reweight by exp(reward / beta)}; % inverse: policy -> reward (take logs) \draw[->, acc, thick] (pol.south west) .. controls (3.5,-1.5) .. (rew.south east) node[midway, below, font=\scriptsize, text=acc] {invert: reward = beta log(policy / ref)}; % DPO annotation on the policy side \node[acc, anchor=north, font=\scriptsize, align=center] at (7.0,-1.15) {DPO trains here}; \end{tikzpicture} $$ > **Definition (Implicit reward).** DPO never instantiates $r_\phi$, but the optimum it > targets corresponds to the implicit reward > $\hat r(x, y) = \beta \log \dfrac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$, > the $\beta$-scaled log-likelihood ratio of the trained policy to the reference. The > DPO loss is precisely the Bradley-Terry loss on this implicit reward. In words: the optimal policy is the reference distribution _reweighted_ by $\exp(r/\beta)$, which tilts probability mass toward high-reward responses while the KL leash keeps the shape close to the reference. $$ % caption: The closed-form optimum reweights the reference by $\exp(r/\beta)$, tilting % mass from low-reward (left) toward high-reward (right) responses; smaller $\beta$ tilts harder. \begin{tikzpicture}[>=stealth, font=\footnotesize, x=1.0cm, y=1.0cm] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} % axes \draw[->, thick] (-0.2,0) -- (6.6,0) node[right, font=\scriptsize] {responses (low to high reward)}; \draw[->, thick] (0,-0.2) -- (0,2.9) node[above, font=\scriptsize] {prob}; % reference: symmetric bump centered at 3.0 (black) \draw[black, very thick] plot[domain=0.2:6.2, samples=80] (\x, {2.0*exp(-0.5*(\x-3.0)*(\x-3.0)/1.1)}); \node[black, anchor=south east, font=\scriptsize] at (2.6,1.95) {reference}; % optimal: same bump shifted right + sharpened (green) = reweighted by exp(r/beta) \draw[green, very thick] plot[domain=0.2:6.2, samples=80] (\x, {2.45*exp(-0.5*(\x-4.0)*(\x-4.0)/0.7)}); \node[green, anchor=south west, font=\scriptsize] at (4.75,2.05) {optimal policy}; % tilt arrow \draw[->, acc, thick] (3.0,2.45) .. controls (3.5,2.7) .. (4.0,2.55); \node[acc, anchor=south, font=\scriptsize] at (3.5,2.7) {tilt by exp(reward / beta)}; \end{tikzpicture} $$ $$ % caption: RLHF versus DPO. RLHF fits a reward then optimizes through RL; DPO inverts % the same optimum to train the policy directly on preferences in one supervised step. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw=black, thick, fill=black!8, minimum width=24mm, minimum height=11mm, align=center, font=\scriptsize}, acb/.style={draw=acc, text=acc, thick, fill=acc!12, minimum width=24mm, minimum height=11mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} % shared start \node[box] (pref) at (0,0) {preferences\\$(x, y_w, y_l)$}; % ---- RLHF path (top) ---- \node[box] (rm) at (4.2,1.6) {reward model}; \node[box] (rl) at (8.4,1.6) {RL (PPO)\\+ KL leash}; \node[acb] (pi1) at (12.4,1.6) {policy}; \draw[->, black, thick] (pref.north) .. controls (1.8,1.6) .. (rm); \draw[->, black, thick] (rm) -- (rl); \draw[->, black, thick] (rl) -- (pi1); \node[black, font=\scriptsize] at (6.3,2.6) {RLHF: two stages, sampling}; % ---- DPO path (bottom) ---- \node[acb] (pi2) at (12.4,-1.6) {policy}; \draw[->, acc, thick] (pref.south) .. controls (1.8,-1.6) and (8.4,-1.6) .. (pi2) node[midway, below, font=\scriptsize, text=acc] {DPO: one supervised log-sigmoid loss}; % link: implicit reward (Latin words only; no Greek in nodes) \node[black, font=\scriptsize, align=center] at (8.4,-0.45) {implicit reward = beta log(policy / ref)}; \end{tikzpicture} $$ | Property | RLHF (PPO) | DPO | | --- | --- | --- | | Reward model | explicit $r_\phi$, trained separately | implicit, $\beta\log(\pi_\theta/\pi_{\text{ref}})$ | | Sampling | on-policy rollouts each step | none; offline on fixed pairs | | Extra networks | reward model + value head | none beyond reference copy | | Loss | clipped policy gradient | supervised log-sigmoid | | Stability | sensitive to many knobs | stable, supervised-style | ## Variants DPO opened a family of preference-optimization losses that change the link function, the data format, or the optimizer while keeping the offline, RL-free spirit. **IPO** (identity preference optimization) observes that DPO's log-sigmoid can drive the implicit reward gap to infinity and overfit when a pair is always labeled the same way; it replaces the Bradley-Terry log-sigmoid with a squared loss that regresses the implicit reward gap toward a finite target $\tfrac{1}{2\beta}$, which bounds the solution and resists over-fitting deterministic preferences. **KTO** (Kahneman-Tversky optimization) drops the _paired_ requirement entirely. Drawing on prospect theory, it treats each response as an independent "good" or "bad" example and maximizes a human-utility function with reference-dependent gains and losses, so it trains on unpaired thumbs-up / thumbs-down data that is far cheaper to collect than matched comparisons. **RLAIF and Constitutional AI** replace the human labeler with a model. An AI critic, prompted with a written "constitution" of principles, generates the preference labels (and self-revisions), so the comparison dataset is produced without human annotation of each pair; humans write the principles once instead of judging thousands of pairs. The reward-modeling and RL machinery is otherwise unchanged. **GRPO** (group-relative policy optimization) keeps the RL loop but discards the value head: for each prompt it samples a _group_ of responses, scores them, and uses the group's mean reward as the baseline, so the advantage of a response is its reward minus the group average. This removes the separate critic network while keeping on-policy optimization, trading the learned baseline for a Monte-Carlo one. | Method | Data | Optimizer | Extra nets | Stability | Compute | | --- | --- | --- | --- | --- | --- | | RLHF-PPO | paired comparisons | RL (clipped PG) | reward + value | sensitive | high (sampling) | | DPO | paired comparisons | supervised | reference only | stable | low (offline) | | IPO | paired comparisons | supervised (squared) | reference only | stable, anti-overfit | low (offline) | | KTO | unpaired labels | supervised | reference only | stable | low (offline) | | GRPO | sampled groups | RL (group baseline) | reward only | moderate | medium | ## The alignment frontier Chollet's chapter predates the alignment stack, so the citations are the primary papers: the RLHF/InstructGPT pipeline,[^ouyang-rlhf] the Bradley-Terry model of paired comparisons,[^bradley-terry] reward over-optimization,[^gao-overopt] DPO,[^rafailov-dpo] Constitutional AI / RLAIF,[^bai-cai] and GRPO.[^shao-grpo] Three threads connect this lesson to current practice and to the [large-language-model alignment](/deep-learning/large-models-and-agents/scaling-inference-and-alignment) lesson. **The reward model is the binding constraint.** Every failure in this lesson traces to the reward model being a finite-data proxy, and the biggest practical gains have come from making it better: larger preference datasets, ensembles that flag their own disagreement (high variance is a signal to stop optimizing), and process rewards that score _each reasoning step_ rather than only the final answer. A better reward model pushes the peak of the over-optimization hump further out, which is worth more than any refinement of the RL loop. **Verifiable rewards sidestep the proxy entirely.** When the task has a checkable answer — a math problem, a passing test suite — the reward need not be _learned_ at all: it is $+1$ if the answer is correct and $0$ otherwise. GRPO with such a verifiable reward is the training method behind the reasoning models of the [AI-agents lesson](/deep-learning/large-models-and-agents/ai-agents), and it removes the reward-hacking failure mode by construction, because a correct answer cannot be gamed. The Goodhart problem is a property of _proxy_ rewards; a verifiable reward is not a proxy. **DPO and PPO are converging in practice.** DPO's simplicity made it the default for offline preference data, but PPO-style online methods (and GRPO) remain stronger when fresh rollouts and a verifiable or strong reward model are available, because they optimize against the current policy's own mistakes rather than a fixed offline set. The field now mixes them: a DPO or SFT warm start, then online RL with a reward model or verifier for the final polish. The reading is that RLHF turned an unspecifiable objective into a learnable one through the preference asymmetry, DPO showed the RL loop was optional for offline data, and the frontier has moved to making the reward trustworthy — better models, ensembles, process rewards, and, where possible, verifiable rewards that are not proxies at all. ## Takeaways - RLHF exploits an **asymmetry**: helpfulness is hard to specify but easy to judge by comparison, so we learn a reward from pairwise preferences instead of writing a loss. - The **pipeline** is three stages: SFT gives a reference policy, a reward model fits human comparisons, and RL fine-tuning maximizes that reward under a KL leash. - Reward modeling uses **Bradley-Terry**, $P(y_w \succ y_l) = \sigma(r(x,y_w) - r(x,y_l))$, and the reward loss is the negative log-sigmoid of the reward gap; its gradient self-weights toward currently-misranked pairs. - The RL step maximizes $\mathbb{E}[r(x,y)] - \beta\KL(\pi \| \pi_{\text{ref}})$; the KL penalty is distributed per token and the policy is optimized by **PPO** with a value head and advantage estimation. - The KL leash guards against **reward over-optimization** (Goodhart): proxy reward rises monotonically while true quality peaks and falls, and $\beta$ tunes the trade-off. - **DPO** solves the KL-regularized objective in closed form, $\pi^{\ast} \propto \pi_{\text{ref}}\exp(r/\beta)$, inverts it to an implicit reward $\beta\log(\pi_\theta/\pi_{\text{ref}})$, and substitutes into Bradley-Terry, collapsing the pipeline into one supervised log-sigmoid loss. - **Variants** trade off data and machinery: IPO bounds the gap with a squared loss, KTO uses unpaired prospect-theory utilities, RLAIF / Constitutional AI replace the human labeler with a principled model critic, and GRPO drops the value head for a group-relative baseline. - **The frontier:** the reward model is the binding constraint, so ensembles and process rewards matter more than the RL loop; **verifiable rewards** (correct-answer checks) sidestep reward hacking entirely and drive GRPO-trained reasoning models. [^ouyang-rlhf]: **Ouyang et al.**, _Training Language Models to Follow Instructions with Human Feedback_ (InstructGPT), NeurIPS 2022 — the three-stage RLHF pipeline: SFT, a Bradley-Terry reward model on comparisons, and PPO with a KL penalty. [^bradley-terry]: **Bradley & Terry**, _Rank Analysis of Incomplete Block Designs_, Biometrika 1952 — the paired-comparison model in which the probability one item beats another is a logistic of their latent-score difference. [^gao-overopt]: **Gao, Schulman & Hilton**, _Scaling Laws for Reward Model Overoptimization_, ICML 2023 — shows true quality peaks then falls as proxy reward is optimized, with the gap growing with KL distance from the reference. [^rafailov-dpo]: **Rafailov et al.**, _Direct Preference Optimization: Your Language Model is Secretly a Reward Model_, NeurIPS 2023 — derives the closed-form KL-regularized optimum and inverts it into a supervised log-sigmoid loss on preference pairs. [^bai-cai]: **Bai et al.**, _Constitutional AI: Harmlessness from AI Feedback_, 2022 — replaces human preference labels with an AI critic guided by a written set of principles (RLAIF), reducing human annotation to writing the constitution. [^shao-grpo]: **Shao et al.**, _DeepSeekMath: Pushing the Limits of Mathematical Reasoning_ (GRPO), 2024 — group-relative policy optimization uses a group of sampled responses' mean reward as the baseline, dropping the value head from the RL loop.