Reinforcement Learning from Human Feedback

The large language model lesson sketched RLHF in a paragraph: fit a reward model, optimize with PPO, regularize with a KL term. This lesson is the derivation. We build the reward model from the Bradley-Terry likelihood, write the KL-regularized objective the policy actually maximizes, solve it in closed form, and then show that the same closed form, inverted, turns the entire reinforcement-learning loop into a single supervised loss. The policy gradients and PPO lesson supplies the optimizer; the foundations of RL lesson supplies the language of policies, rewards, and returns we reuse throughout.

The specification problem

The objectives we want from an assistant, that it be helpful, harmless, and honest, resist direct specification. There is no loss function for helpful. But a human shown two responses to the same prompt can reliably say which is better, even when they could not have written down a rule that produces the better one. RLHF is built on exactly this asymmetry.

The plan is to learn a scalar reward $r(x,y)$ that ranks responses the way humans do, then push a language model toward high-reward responses. The reward converts a set of discrete comparisons into a differentiable signal a policy can ascend.

The three-stage pipeline

RLHF as deployed in modern instruction-tuned systems is three stages run in sequence. First, supervised fine-tuning (SFT) on curated demonstrations teaches the base model the format of an answer and gives a reference policy ${\pi }_{\text{ref}}$. Second, a reward model $r_{\varphi }$ is fit to human comparisons. Third, the policy is RL-fine-tuned to maximize $r_{\varphi }$ while a KL penalty holds it near ${\pi }_{\text{ref}}$.

The three stages consume three different kinds of data, and that is the point: each stage extracts the signal that is cheapest to collect for what it must learn.

Reward modeling from pairwise preferences

We need a model that, given a comparison $(x,y_w,y_l)$, assigns a high probability to the event $y_w$ beats $y_l$. The Bradley-Terry model is the canonical choice: it posits a latent scalar score per item and makes the probability of one beating another a logistic function of their score difference.

The model has the right limits. When $r(x,y_w)\gg r(x,y_l)$ the gap is large and positive, so $P\to 1$; when the two rewards are equal the gap is zero and $P=\frac{1}{2}$, a coin flip; and the curve is symmetric, so $P(y_w\succ y_l)+P(y_l\succ y_w)=1$.

Deriving the reward loss

We fit $r_{\varphi }$ by maximum likelihood on the comparison dataset. Treat each comparison as one Bernoulli observation whose outcome is the labeled winner won. The likelihood of a single comparison is $P(y_w\succ y_l\mid x)=\sigma ({\Delta }_{\varphi })$ with the shorthand ${\Delta }_{\varphi }=r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l)$. The dataset log-likelihood is the sum over independent comparisons,

Maximizing the log-likelihood is minimizing its negative, normalized to an expectation over the dataset. This is the reward-model objective.

The gradient explains what the loss does mechanically. Using ${\sigma }^{\prime }(z)=\sigma (z)\left(1-\sigma (z)\right)$ and $\frac{d}{dz}\log \sigma (z)=1-\sigma (z)$, the per-comparison gradient is

The update raises the winner's reward and lowers the loser's, scaled by $\sigma (-{\Delta }_{\varphi })$, the model's current probability of mis-ranking the pair. Already-correct comparisons (large ${\Delta }_{\varphi }$) get a vanishing gradient; only the contested pairs move the weights. This is the same self-weighting that makes logistic regression focus on the margin.

For example, suppose the model already ranks a pair correctly with ${\Delta }_{\varphi }=4$: then $\sigma (-4)\approx 0.018$, so the pair's gradient is scaled by under two percent and it barely nudges the weights. Suppose instead the model ranks a pair backwards with ${\Delta }_{\varphi }=-2$: then $\sigma (2)\approx 0.88$, an eighty-eight percent weight, and the update pushes hard to reverse the ordering. The loss concentrates its gradient on currently-misranked pairs and contributes almost nothing for pairs already ranked confidently, which is the correct behavior for a ranking objective.

The identifiability caveat carries a practical consequence. Because only reward differences enter the loss, $r_{\varphi }$ is pinned only up to a per-prompt additive constant $c(x)$: the scale of the scores is fixed by the logistic, but their offset is free. When the reward feeds the RL step this is harmless, since the KL term and the advantage baseline both subtract off any prompt-level constant. It does mean reward magnitudes are not comparable across prompts, so a raw $r_{\varphi }(x,y)$ is a within-prompt ranking signal, not an absolute quality score.

The RL fine-tuning step

With $r_{\varphi }$ fixed, stage three optimizes the policy. Naively maximizing expected reward fails: the reward model is an imperfect proxy fit on finite data, and an unconstrained optimizer will find responses that score highly under $r_{\varphi }$ yet are degenerate to a human. To address this, anchor the policy to the SFT reference with a KL penalty.

The KL term serves two purposes. It keeps the policy in the region where the reward model was trained and is therefore trustworthy, and it preserves the fluency and breadth the base model already has, which the sparse reward would otherwise erode. Without the anchor three failures follow directly. The optimizer exploits the reward model's blind spots and finds high-scoring gibberish (reward hacking); it collapses onto the single highest-reward response and loses diversity (mode collapse); and it drifts so far from the data ${\pi }_{\text{ref}}$ was fit on that $r_{\varphi }$ is extrapolating rather than judging (distribution drift). The two terms pull in opposite directions: reward pulls the policy away from the reference toward high-scoring responses, and the KL pulls it back toward the reference, with $\beta$ setting the balance point.

Per-token reward shaping

Generation is a token-level Markov decision process: the state is the prompt plus the tokens emitted so far, the action is the next token, and the trajectory reward $r_{\varphi }(x,y)$ arrives only at the terminal token. To turn the sequence-level objective into a dense per-step signal PPO can consume, the KL penalty is distributed across tokens, giving a per-token reward

Summed over the trajectory, the second term telescopes into the sequence-level $\mathrm{KL}$, so this shaping restates the objective above one step at a time. Every token pays a small price for drifting from the reference, and the whole response collects the terminal reward.

PPO as the optimizer

The shaped reward feeds a standard actor-critic loop. A value head $V_{\psi }$ is attached to the policy to estimate the expected return from each token, and the advantage ${\hat{A}}_t$ (typically by generalized advantage estimation) measures how much better the chosen token did than the value baseline. PPO then takes a clipped policy-gradient step on ${\hat{A}}_t$, which the policy gradients lesson develops in full. The clip is what keeps each update small enough that the on-policy reward estimates stay valid.

Failure modes: reward hacking and the KL leash

The reward model is a proxy, not the true objective, and optimizing a proxy hard enough breaks it. This is Goodhart's law: a measure that is optimized ceases to measure what it once did.

The KL coefficient $\beta$ trades these two failure directions against each other. Too large, and the policy never moves; too small, and it over-optimizes the proxy into degeneracy.

Two further levers control the same trade-off. The value head must be well-fit or the advantage estimates are biased, which destabilizes the policy step; in practice the value and policy share the backbone but use separate heads. And advantage estimation by GAE introduces a bias-variance parameter ($\lambda$) that, set well, smooths the sparse terminal reward into a usable per-token learning signal without injecting too much variance from long rollouts.

Direct preference optimization

PPO-based RLHF is operationally heavy: a separate reward model, a value head, on-policy sampling, and a famously finicky RL loop. Direct preference optimization (DPO) removes all of it by exploiting a fact about the KL-regularized objective: its optimum has a closed form, and that closed form can be inverted to express the reward in terms of the policy. Substituting the inverted reward into the Bradley-Terry loss yields a supervised objective on the policy directly, with no reward model and no sampling.

The derivation is a change of variable: the reward and the optimal policy are two encodings of the same object, and DPO works in policy space so it never has to fit the reward at all. The implicit reward DPO assigns to a response is the quantity it pushes up or down.

In words: the optimal policy is the reference distribution reweighted by $\exp (r/\beta )$, which tilts probability mass toward high-reward responses while the KL leash keeps the shape close to the reference.

Variants

DPO opened a family of preference-optimization losses that change the link function, the data format, or the optimizer while keeping the offline, RL-free spirit.

IPO (identity preference optimization) observes that DPO's log-sigmoid can drive the implicit reward gap to infinity and overfit when a pair is always labeled the same way; it replaces the Bradley-Terry log-sigmoid with a squared loss that regresses the implicit reward gap toward a finite target $\frac{1}{2\beta }$, which bounds the solution and resists over-fitting deterministic preferences.

KTO (Kahneman-Tversky optimization) drops the paired requirement entirely. Drawing on prospect theory, it treats each response as an independent good or bad example and maximizes a human-utility function with reference-dependent gains and losses, so it trains on unpaired thumbs-up / thumbs-down data that is far cheaper to collect than matched comparisons.

RLAIF and Constitutional AI replace the human labeler with a model. An AI critic, prompted with a written constitution of principles, generates the preference labels (and self-revisions), so the comparison dataset is produced without human annotation of each pair; humans write the principles once instead of judging thousands of pairs. The reward-modeling and RL machinery is otherwise unchanged.

GRPO (group-relative policy optimization) keeps the RL loop but discards the value head: for each prompt it samples a group of responses, scores them, and uses the group's mean reward as the baseline, so the advantage of a response is its reward minus the group average. This removes the separate critic network while keeping on-policy optimization, trading the learned baseline for a Monte-Carlo one.

The alignment frontier

Chollet's chapter predates the alignment stack, so the citations are the primary papers: the RLHF/InstructGPT pipeline,¹ the Bradley-Terry model of paired comparisons,² reward over-optimization,³ DPO,⁴ Constitutional AI / RLAIF,⁵ and GRPO.⁶ Three threads connect this lesson to current practice and to the large-language-model alignment lesson.

The reward model is the binding constraint. Every failure in this lesson traces to the reward model being a finite-data proxy, and the biggest practical gains have come from making it better: larger preference datasets, ensembles that flag their own disagreement (high variance is a signal to stop optimizing), and process rewards that score each reasoning step rather than only the final answer. A better reward model pushes the peak of the over-optimization hump further out, which is worth more than any refinement of the RL loop.

Verifiable rewards sidestep the proxy entirely. When the task has a checkable answer — a math problem, a passing test suite — the reward need not be learned at all: it is $+1$ if the answer is correct and $0$ otherwise. GRPO with such a verifiable reward is the training method behind the reasoning models of the AI-agents lesson, and it removes the reward-hacking failure mode by construction, because a correct answer cannot be gamed. The Goodhart problem is a property of proxy rewards; a verifiable reward is not a proxy.

DPO and PPO are converging in practice. DPO's simplicity made it the default for offline preference data, but PPO-style online methods (and GRPO) remain stronger when fresh rollouts and a verifiable or strong reward model are available, because they optimize against the current policy's own mistakes rather than a fixed offline set. The field now mixes them: a DPO or SFT warm start, then online RL with a reward model or verifier for the final polish.

The reading is that RLHF turned an unspecifiable objective into a learnable one through the preference asymmetry, DPO showed the RL loop was optional for offline data, and the frontier has moved to making the reward trustworthy — better models, ensembles, process rewards, and, where possible, verifiable rewards that are not proxies at all.

Takeaways

• RLHF exploits an asymmetry: helpfulness is hard to specify but easy to judge by comparison, so we learn a reward from pairwise preferences instead of writing a loss.
• The pipeline is three stages: SFT gives a reference policy, a reward model fits human comparisons, and RL fine-tuning maximizes that reward under a KL leash.
• Reward modeling uses Bradley-Terry, $P(y_w\succ y_l)=\sigma (r(x,y_w)-r(x,y_l))$, and the reward loss is the negative log-sigmoid of the reward gap; its gradient self-weights toward currently-misranked pairs.
• The RL step maximizes $\mathbb{E}[r(x,y)]-\beta \mathrm{KL}(\pi \Vert {\pi }_{\text{ref}})$; the KL penalty is distributed per token and the policy is optimized by PPO with a value head and advantage estimation.
• The KL leash guards against reward over-optimization (Goodhart): proxy reward rises monotonically while true quality peaks and falls, and $\beta$ tunes the trade-off.
• DPO solves the KL-regularized objective in closed form, ${\pi }^{\ast }\propto {\pi }_{\text{ref}}\exp (r/\beta )$, inverts it to an implicit reward $\beta \log ({\pi }_{\theta }/{\pi }_{\text{ref}})$, and substitutes into Bradley-Terry, collapsing the pipeline into one supervised log-sigmoid loss.
• Variants trade off data and machinery: IPO bounds the gap with a squared loss, KTO uses unpaired prospect-theory utilities, RLAIF / Constitutional AI replace the human labeler with a principled model critic, and GRPO drops the value head for a group-relative baseline.
• The frontier: the reward model is the binding constraint, so ensembles and process rewards matter more than the RL loop; verifiable rewards (correct-answer checks) sidestep reward hacking entirely and drive GRPO-trained reasoning models.