RLHF and Language Models

A large language model is trained on one objective: given the text so far, put high probability on the token that actually came next in the corpus. Trained at scale, the model absorbs the syntax, facts, and styles of the internet, but it is not an assistant. Ask it a question and it is as likely to continue with a plausible list of other questions, because on the web a question is often followed by more questions. The model is doing exactly what it was trained to do — imitate the distribution of text — and that is not the same as being helpful, honest, and harmless.¹

This is a mismatch between the objective that was optimized (next-token likelihood) and the objective we actually want (a useful assistant). No amount of extra pretraining fixes it, because the loss cannot see the difference: a helpful answer and an unhelpful-but-fluent continuation can be equally probable under the data. Fixing it needs a new signal, one that scores behavior rather than likelihood. The trouble is that the behavior we want is almost impossible to write down as a formula. Reinforcement learning from human feedback (RLHF) constructs that signal from human judgment, and it is the reason the trial-and-error idea at the root of this course now aligns the largest models ever built.

The alignment problem: hard to specify, easy to compare

Suppose you wanted to write a reward function for a good answer. You would have to encode helpfulness, factual accuracy, appropriate refusals, tone, format, and the absence of a thousand failure modes, all as a scalar. Nobody can do this. Any explicit rule you write is either too coarse to capture what you mean or so specific that a capable optimizer finds and exploits the gap between your rule and your intent. Specifying the reward is the whole difficulty of the problem.

Yet there is one thing humans do easily and reliably: shown two answers to the same prompt, a person can usually say which is better. They cannot tell you the number a good answer deserves, but they can rank a pair. That asymmetry — absolute reward hard to state, relative preference easy to state — is what RLHF exploits: it never asks a human for a score, only for comparisons, and it builds the reward from them.

The insight that a reward can be learned from preferences rather than hand-specified predates language models. Christiano et al. showed it on control tasks in 2017: a simulated robot learned a backflip — a behavior no one had written a reward for — from a few hundred human comparisons between short video clips of its behavior.² RLHF for language models is that idea carried to a domain where the space of behaviors is the space of all text.

The RLHF pipeline

InstructGPT¹ assembled the components into the three-stage pipeline that is now standard. Each stage produces the input the next one needs.

Stage 1 — supervised fine-tuning

Start from a pretrained base model and fine-tune it on a modest set of demonstrations: prompts paired with high-quality answers written by humans. This is ordinary supervised learning — maximize the likelihood of the demonstrated answer given the prompt — and its job is narrow. It moves the model out of the continue the document regime and into the answer the instruction regime, so that its outputs are at least in the right format and roughly on task. The result is the SFT model, written ${\pi }_{\text{SFT}}$. It is a competent instruction-follower but not yet tuned to human preference, and it is the starting point (and, shortly, the anchor) for everything that follows.

Stage 2 — the reward model

Now build the reward from comparisons. Sample a prompt $x$, draw two (or more) candidate answers from the SFT model, and ask a human which they prefer. This produces a dataset of triples $(x,y_w,y_l)$ where $y_w$ (the winner) was preferred over $y_l$ (the loser). From this we fit a reward model$r_{\varphi }(x,y)$ — a network, usually the SFT model with its output head replaced by a scalar — that scores how good an answer $y$ is for prompt $x$.

The link between a scalar reward and a pairwise preference is the Bradley-Terry model. It posits that the probability a human prefers $y_w$ to $y_l$ is a logistic function of the reward difference:

where $\sigma$ is the logistic sigmoid. The larger the reward gap in the winner's favor, the more confidently the model predicts the observed preference. Fitting $r_{\varphi }$ is then maximum likelihood under this model: minimize the negative log-likelihood of the human choices,

This is a binary classification loss over pairs. Note what it determines and what it leaves free: only reward differences are constrained, so $r_{\varphi }$ is identified only up to an additive constant — a fact DPO exploits later. The reward model distills thousands of individual human judgments into a single differentiable function that can score any answer the policy might produce, including ones no human ever saw.

Worked example: what the reward gap means in probability

The sigmoid link makes the reward difference interpretable directly as a confidence. Suppose the reward model, after training, scores a winning answer at $r_{\varphi }(x,y_w)=2.1$ and a losing one at $r_{\varphi }(x,y_l)=1.4$. The gap is $\Delta =0.7$, so the model predicts a human prefers $y_w$ with probability $\sigma (0.7)=1/(1+e^{-0.7})\approx 0.668$. A gap of $1.0$ gives $\sigma (1.0)\approx 0.731$; a gap of $3.0$ gives $\sigma (3.0)\approx 0.953$; a gap of $0$ gives exactly $0.5$, indifference. The per-example loss on a pair the model gets right with confidence $0.668$ is $-\log 0.668\approx 0.403$ nats; on a pair it gets wrong (predicts $y_l$ with the same reward gap reversed) the loss is $-\log 0.332\approx 1.103$ nats, nearly three times larger. The loss penalizes confident mistakes far more than unconfident ones, the behavior a calibrated preference model needs.

Stage 3 — policy optimization against the reward

With a reward in hand, the alignment problem finally becomes a reinforcement learning problem. The environment is a bandit-like one: the state is the prompt $x$, the action is the full generated answer $y\sim {\pi }_{\theta }(\cdot \mid x)$, and the reward is $r_{\varphi }(x,y)$ delivered once the answer is complete. We want the policy ${\pi }_{\theta }$ that produces high-reward answers, which is a policy-optimization objective — precisely the setting the policy-gradient methods of the approximation module were built for.

But maximizing $r_{\varphi }$ alone fails. The reward model is an imperfect approximation of human preference, accurate near the answers it was trained on and unreliable far from them. A policy free to move anywhere will find the answers where $r_{\varphi }$ is mistakenly high and collapse onto them. The fix is a leash: keep ${\pi }_{\theta }$ from drifting too far from the reference model ${\pi }_{\text{ref}}$ (the SFT model) that generated the training answers, by penalizing the KL divergence between them. The RLHF objective is

where $\beta >0$ controls how tight the leash is. In practice the KL term is folded into the per-token reward and the whole thing is optimized with PPO — the clipped surrogate objective from the deep-RL module is the optimizer that does the actual work here, so RLHF is PPO pointed at a learned reward with a KL regularizer rather than a new algorithm. The reward each token sees is the terminal reward minus a running KL penalty,

so every token that makes the answer more probable under ${\pi }_{\theta }$ than under ${\pi }_{\text{ref}}$ is charged for the deviation.

Worked example: the KL charge on a single answer

Take a short answer of five tokens and suppose the policy has, over training, pushed the log-probability of each token above the reference by the amounts $\log ({\pi }_{\theta }/{\pi }_{\text{ref}})=(0.4,0.1,0.9,0.2,0.5)$ nats — the third token is where the policy has moved most aggressively. The summed log-ratio is $0.4+0.1+0.9+0.2+0.5=2.1$ nats. With a leash of $\beta =0.2$, the KL charge subtracted from the terminal reward is $0.2\times 2.1=0.42$. If the reward model scored the answer at $r_{\varphi }=1.8$, the effective reward the PPO update actually optimizes is $R=1.8-0.42=1.38$. Loosen the leash to $\beta =0.05$ and the charge falls to $0.105$, leaving $R=1.695$ — the policy keeps almost all of its reward and is barely penalized for drifting. Tighten it to $\beta =0.5$ and the charge is $1.05$, leaving $R=0.75$; now the gain from the reward model is nearly cancelled by the cost of having moved. The single scalar $\beta$ determines how heavily distance from the reference counts against the reward, token by token.

The output is the aligned policy — the model a user actually talks to. Every consumer chat assistant in wide use went through some version of this loop.

Reward hacking and why the KL leash matters

Drop the KL penalty ($\beta =0$) and the policy maximizes $r_{\varphi }$ to the exclusion of everything else — and because $r_{\varphi }$ is only a proxy for human preference, the maximum of the proxy is not the maximum of the truth. The policy discovers the reward model's blind spots: it learns that padding answers with hedged, agreeable filler scores well, or that a certain phrasing the reward model over-rewards can be repeated, and it drifts into degenerate text that the reward model scores highly and a human finds useless or bizarre. This is reward hacking (equivalently, over-optimization): optimizing the measure until it stops tracking the thing it was meant to measure.

The KL penalty is the defense. It keeps ${\pi }_{\theta }$ inside the neighborhood of ${\pi }_{\text{ref}}$ where the reward model saw training data and is therefore trustworthy, trading a little reward for staying on reliable ground. The tradeoff is real and has a characteristic shape: as you loosen the leash (raise the KL budget), true quality first rises — the policy is genuinely improving — then peaks and falls as over-optimization sets in, even though the measured reward keeps climbing.³

Choosing $\beta$ (or an explicit KL target) is therefore central: it stops the optimizer from exploiting the reward model outside the region where it is accurate.

Direct Preference Optimization

The three-stage pipeline works, but it is heavy: a separate reward model to train and store, an on-policy RL loop with its own instabilities, sampling from the policy at every step, a value head, reward and KL bookkeeping. Direct Preference Optimization (DPO)⁴ asks whether the reward model is necessary at all, and shows by a change of variables that it is not.

The idea in one line: instead of learning a reward and then optimizing a policy against it, notice that the optimal policy is a stand-in for the reward, so you can train the policy on the preferences directly and skip the middle step. The two equations below make that precise — the first writes the best policy in terms of the reward, the second inverts it to write the reward in terms of the policy.

The derivation starts from the RLHF objective itself. For a fixed reward $r$, the policy that maximizes reward minus $\beta \mathrm{KL}$ has a known closed form — it is the reference policy reweighted by the exponentiated reward:

where $Z(x)$ is a normalizing constant. This is not something you can sample from directly — $Z(x)$ sums over all possible answers — which is why the standard pipeline needs RL. But the relation can be inverted to write the reward in terms of the optimal policy:

Now substitute this expression for $r$ into the Bradley-Terry preference loss. Because the loss depends only on the reward difference $r(x,y_w)-r(x,y_l)$, the intractable $\beta \log Z(x)$ term is identical for both answers to the same prompt and cancels exactly. What remains is a loss written entirely in terms of the policy we want to train and the frozen reference, with no reward model anywhere:

This is a single supervised classification loss on the preference pairs. It raises the policy's log-probability of the winning answer relative to the reference, lowers it for the loser, with $\beta$ playing the same leash role it did before — the reference appears in the loss, so DPO is implicitly KL-regularized by construction. There is no reward model to fit, no sampling from the policy during training, no RL loop. The reward model has not disappeared; it is implicit in the ratio $\beta \log ({\pi }_{\theta }/{\pi }_{\text{ref}})$ — the reward the change of variables identified.

What the DPO gradient actually does

Differentiating the loss shows how DPO behaves. Write ${\hat{r}}_{\theta }(x,y)=\beta \log {\pi }_{\theta }(y\mid x)/{\pi }_{\text{ref}}(y\mid x)$ for the implicit reward. The gradient of the DPO loss is

Read the two pieces. The vector part raises the log-probability of the winner and lowers the loser's — ordinary contrastive learning. The scalar weight in front is the point: it is $\sigma$ of the reward gap in the wrong direction, so it is large exactly when the implicit reward has the pair backwards (the model currently scores the loser above the winner) and small when the model already ranks them correctly. DPO concentrates its gradient on the pairs it currently ranks wrong and applies little to the ones it ranks right — a built-in hard-example weighting that a naive maximum-likelihood objective on the winners alone would not have. This is why DPO does not simply collapse onto the winning answers: once a pair is ranked correctly with margin, its gradient weight vanishes.

Worked example: one DPO update

Suppose for a pair the current policy assigns log-ratios $\log {\pi }_{\theta }/{\pi }_{\text{ref}}$ of $0.5$ to the winner and $0.8$ to the loser — the model has it backwards, preferring the loser. With $\beta =0.1$ the implicit rewards are ${\hat{r}}_w=0.05$ and ${\hat{r}}_l=0.08$, so the weight is $\sigma (0.08-0.05)=\sigma (0.03)\approx 0.507$, near its maximum of $0.5$: almost the full gradient is applied, pushing the winner's probability up and the loser's down. After several such updates the log-ratios might flip to $0.9$ (winner) and $0.3$ (loser); now the implicit rewards are ${\hat{r}}_w=0.09$ and ${\hat{r}}_l=0.03$, so the weight is $\sigma ({\hat{r}}_l-{\hat{r}}_w)=\sigma (-0.06)\approx 0.485$ — already below the $0.5$ maximum, and it keeps shrinking as the margin grows. The update throttles itself as it succeeds.

DPO is simpler, more stable, and cheaper, and it matches or exceeds PPO-based RLHF on many preference benchmarks, which is why it became a default for open-model alignment. The tradeoffs are worth naming: DPO learns only from the fixed preference dataset (it never generates fresh on-policy samples to be scored, so it cannot discover and correct its own new failure modes the way an online PPO loop can), and it inherits whatever coverage gaps the preference data has. The explicit reward model remains useful when you want to score many candidates, run best-of-$n$ sampling, or keep improving with fresh comparisons.

Beyond human labels: RLAIF and verifiable rewards

Two directions push past the bottleneck of human comparisons.

The first replaces the human labeler with a model. In RL from AI feedback (RLAIF) and Constitutional AI⁵, the preference labels are produced by an LLM prompted with a written set of principles (a constitution) rather than by people. A model critiques and revises its own answers against the principles, and generates the preference pairs used to train the reward model or run DPO. This scales preference collection far beyond what human labeling can reach and makes the values being optimized for explicit and auditable in the constitution's text, at the cost of trusting the labeler model's judgment.

The second sidesteps preference modeling entirely when the task has a checkable answer. For math, code, and formal reasoning, correctness is not a matter of taste — a solution is right or wrong, and a verifier (a unit-test suite, a symbolic checker, a numeric comparison) supplies a verifiable reward with no reward model and no human in the loop. Running policy optimization against such a reward is RL with verifiable rewards (RLVR), and it is what underlies recent reasoning-focused models: the policy is rewarded for producing chains of thought that lead to verifiably correct answers, and the same PPO-style optimization that RLHF uses drives the improvement. The signal is cleaner and harder to hack, but it applies only where correctness is machine-checkable, which is a small slice of what we ask assistants to do.

Beyond the standard pipeline: critic-free RL, process rewards, and reasoning models

The InstructGPT recipe fixed the shape of RLHF, but the years after it reworked almost every stage. Four developments matter for where the field now sits.

Group-relative policy optimization (GRPO). PPO on a language model carries a value network — a second model as large as the policy — whose only job is to estimate the baseline for advantage computation. GRPO, introduced with the DeepSeekMath work, removes it.⁶ Instead of learning a value function, GRPO samples a group of $G$ answers to the same prompt, scores them all with the reward model, and uses the group's own mean and standard deviation as the baseline: the advantage of answer $i$ is ${\hat{A}}_i=(r_i-\mathrm{mean}(r_{1:G}))/\mathrm{std}(r_{1:G})$. An answer better than its siblings gets a positive advantage, worse than them a negative one, with no separate critic to train or store. The saving is real — half the memory and one fewer network to stabilize — and GRPO is the optimizer behind the DeepSeek-R1 reasoning models.⁷

Process rewards versus outcome rewards. A verifiable reward on a math problem scores only the final answer — right or wrong — which gives the policy no signal about where a long chain of reasoning went wrong. A process reward model (PRM) scores each intermediate step instead, and Lightman et al. showed that supervising the process rather than only the outcome trained more reliable math solvers, because dense per-step feedback assigns credit far more precisely than a single terminal bit.⁸ The tradeoff is labeling cost: step-level labels are far more expensive to collect than final-answer checks.

Iterative and online DPO. DPO's main weakness — it never generates fresh samples, so it cannot correct failures outside its fixed dataset — is patched by running it in rounds: train with DPO, sample fresh answers from the improved policy, label the new pairs (by humans, an AI judge, or a verifier), and repeat. Each round's on-policy data closes coverage gaps the previous dataset missed, recovering some of the online-exploration advantage that plain PPO-RLHF has over one-shot DPO while keeping DPO's simplicity.⁹

Best-of-$n$ and rejection sampling. The explicit reward model keeps a use even where DPO wins on training: at inference, sample $n$ answers, score them all with $r_{\varphi }$, and return the best. This best-of-$n$ sampling spends compute to buy quality with no further training, and its offline cousin, rejection-sampling fine-tuning, generates many answers, keeps only the reward model's top picks, and fine-tunes on them — a lightweight alternative to a full RL loop that several production systems use as a first alignment pass before or instead of PPO.

Why this closes the loop

The first lesson of this course defined reinforcement learning as learning to act from a scalar reward signal found by trial and error, with no supervisor handing over the right answer. RLHF is that definition applied to the largest models we build. There is no labeled correct response to imitate for an open-ended request; there is only a comparison signal, distilled into a reward, and a policy improved against it by the same policy-gradient and PPO machinery the rest of the course developed. The deep-learning course covers the same method from the language-model side in its RL from human feedback lesson.

Pretraining gives the model its knowledge; RLHF makes that knowledge usable. The trial-and-error principle that started as a way to teach a simulated agent to walk is now the step that turns a fluent next-token predictor into something a person can actually use, which makes it, for the moment, reinforcement learning's largest real-world application.