--- title: RLHF and Language Models module: Modern Deep Reinforcement Learning moduleNumber: 5 lessonNumber: 17 order: 517 summary: > A language model trained to predict the next token is fluent but not helpful, honest, or harmless — the objective it was optimized for is not the objective we want. RLHF closes that gap by turning the one thing humans do reliably, comparing two outputs, into a reward. We build the three-stage pipeline: supervised fine-tuning, a Bradley-Terry reward model fit to preference pairs, then PPO against that reward with a KL penalty keeping it near the reference policy. We then cover reward hacking and why the KL penalty matters, Direct Preference Optimization, which folds the reward model into a single classification loss, and the RLAIF and verifiable-reward variants. This pipeline is what makes the largest models usable as assistants. topics: [Deep RL] sources: - book: Sutton & Barto ref: "Ch. 13 — Policy Gradient Methods (background for the policy-optimization stage)" - book: Grokking Deep RL ref: "Ch. 11–12 — policy-based and actor-critic methods (PPO background)" --- A large language model is trained on one objective: given the text so far, put high probability on the token that actually came next in the corpus. Trained at scale, the model absorbs the syntax, facts, and styles of the internet, but it is not an assistant. Ask it a question and it is as likely to continue with a plausible list of _other_ questions, because on the web a question is often followed by more questions. The model is doing exactly what it was trained to do — imitate the distribution of text — and that is not the same as being helpful, honest, and harmless.[^instructgpt] This is a mismatch between the **objective that was optimized** (next-token likelihood) and the **objective we actually want** (a useful assistant). No amount of extra pretraining fixes it, because the loss cannot see the difference: a helpful answer and an unhelpful-but-fluent continuation can be equally probable under the data. Fixing it needs a new signal, one that scores _behavior_ rather than _likelihood_. The trouble is that the behavior we want is almost impossible to write down as a formula. **Reinforcement learning from human feedback (RLHF)** constructs that signal from human judgment, and it is the reason the [trial-and-error idea](/reinforcement-learning/foundations/what-is-reinforcement-learning) at the root of this course now aligns the largest models ever built. ## The alignment problem: hard to specify, easy to compare Suppose you wanted to write a reward function for "a good answer." You would have to encode helpfulness, factual accuracy, appropriate refusals, tone, format, and the absence of a thousand failure modes, all as a scalar. Nobody can do this. Any explicit rule you write is either too coarse to capture what you mean or so specific that a capable optimizer finds and exploits the gap between your rule and your intent. Specifying the reward is the whole difficulty of the problem. Yet there is one thing humans do easily and reliably: shown two answers to the same prompt, a person can usually say which is better. They cannot tell you the _number_ a good answer deserves, but they can rank a _pair_. That asymmetry — absolute reward hard to state, relative preference easy to state — is what RLHF exploits: it never asks a human for a score, only for comparisons, and it builds the reward from them. $$ % caption: The alignment gap. Pretraining and supervised fine-tuning optimize % likelihood of text; what we want is helpful, honest, harmless behavior. The two % objectives do not coincide, and the reward that would close the gap is easy to % judge by comparison but nearly impossible to write down. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=34mm, minimum height=13mm, align=center}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[box] (like) at (0,0) {maximize\\next-token likelihood}; \node[box, draw=acc, text=acc] (want) at (6.6,0) {helpful, honest,\\harmless assistant}; \draw[<->, red, thick] (like) -- (want) node[midway, above, black, font=\scriptsize] {objective gap}; \node[align=center, font=\scriptsize, black] at (3.3,-1.15) {reward hard to write down,\\preference easy to compare}; \end{tikzpicture} $$ > **Definition (Alignment problem).** The gap between the objective a model was > trained on (here, likelihood of observed text) and the objective its user > actually intends (helpful, honest, harmless behavior). Alignment methods supply > a training signal for the intended objective when it cannot be written as an > explicit loss. The insight that a reward can be learned from _preferences_ rather than hand-specified predates language models. Christiano et al. showed it on control tasks in 2017: a simulated robot learned a backflip — a behavior no one had written a reward for — from a few hundred human comparisons between short video clips of its behavior.[^christiano] RLHF for language models is that idea carried to a domain where the space of behaviors is the space of all text. ## The RLHF pipeline InstructGPT[^instructgpt] assembled the components into the three-stage pipeline that is now standard. Each stage produces the input the next one needs. $$ % caption: The three-stage RLHF pipeline. Supervised fine-tuning turns a base % model into an instruction-follower; a reward model is fit to human preference % comparisons; then policy optimization improves the SFT model against that reward % with a KL leash back to it. \begin{tikzpicture}[>=stealth, font=\footnotesize, stg/.style={draw, minimum width=32mm, minimum height=17mm, align=center}, data/.style={draw, dashed, black, minimum width=30mm, minimum height=11mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[stg] (sft) at (0,0) {1. supervised\\f\/ine-tuning\\(SFT)}; \node[stg] (rm) at (4.7,0) {2. reward model\\training}; \node[stg, draw=acc, text=acc, thick] (rl) at (9.4,0) {3. RL policy\\optimization\\(PPO + KL)}; \draw[->, acc, thick] (sft) -- (rm); \draw[->, acc, thick] (rm) -- (rl); % data sources \node[data] (d1) at (0,2.1) {demonstrations\\(prompt, ideal answer)}; \node[data] (d2) at (4.7,2.1) {comparisons\\(answer A vs B)}; \draw[->, black] (d1) -- (sft); \draw[->, black] (d2) -- (rm); % SFT feeds the reference for the KL leash \draw[->, black] (sft.south) .. controls (2.5,-1.8) and (7.0,-1.8) .. (rl.south) node[midway, below, black!70, font=\scriptsize] {SFT model = reference for KL penalty}; \end{tikzpicture} $$ ### Stage 1 — supervised fine-tuning Start from a pretrained base model and fine-tune it on a modest set of demonstrations: prompts paired with high-quality answers written by humans. This is ordinary supervised learning — maximize the likelihood of the demonstrated answer given the prompt — and its job is narrow. It moves the model out of the "continue the document" regime and into the "answer the instruction" regime, so that its outputs are at least in the right format and roughly on task. The result is the **SFT model**, written $\pi_{\text{SFT}}$. It is a competent instruction-follower but not yet tuned to human preference, and it is the starting point (and, shortly, the anchor) for everything that follows. ### Stage 2 — the reward model Now build the reward from comparisons. Sample a prompt $x$, draw two (or more) candidate answers from the SFT model, and ask a human which they prefer. This produces a dataset of triples $(x, y_w, y_l)$ where $y_w$ (the "winner") was preferred over $y_l$ (the "loser"). From this we fit a **reward model** $r_\phi(x, y)$ — a network, usually the SFT model with its output head replaced by a scalar — that scores how good an answer $y$ is for prompt $x$. The link between a scalar reward and a pairwise preference is the **Bradley-Terry model**. It posits that the probability a human prefers $y_w$ to $y_l$ is a logistic function of the reward _difference_: $$ P\bigl(y_w \succ y_l \mid x\bigr) \;=\; \sigma\!\bigl(r_\phi(x, y_w) - r_\phi(x, y_l)\bigr) \;=\; \frac{\exp r_\phi(x, y_w)}{\exp r_\phi(x, y_w) + \exp r_\phi(x, y_l)}, $$ where $\sigma$ is the logistic sigmoid. The larger the reward gap in the winner's favor, the more confidently the model predicts the observed preference. Fitting $r_\phi$ is then maximum likelihood under this model: minimize the negative log-likelihood of the human choices, $$ \mathcal{L}(\phi) \;=\; -\,\mathbb{E}_{(x,\, y_w,\, y_l)\,\sim\,\mathcal{D}} \Bigl[\log \sigma\!\bigl(r_\phi(x, y_w) - r_\phi(x, y_l)\bigr)\Bigr]. $$ This is a binary classification loss over pairs. Note what it determines and what it leaves free: only reward _differences_ are constrained, so $r_\phi$ is identified only up to an additive constant — a fact DPO exploits later. The reward model distills thousands of individual human judgments into a single differentiable function that can score any answer the policy might produce, including ones no human ever saw. > **Definition (Bradley-Terry preference model).** A model of pairwise choice in > which the probability of preferring item $i$ to item $j$ is > $\sigma(s_i - s_j)$, a logistic function of their latent-score difference. In > RLHF the scores are the reward-model outputs $r_\phi(x, y)$, and the reward > model is fit by maximum likelihood on human comparisons. #### Worked example: what the reward gap means in probability The sigmoid link makes the reward _difference_ interpretable directly as a confidence. Suppose the reward model, after training, scores a winning answer at $r_\phi(x, y_w) = 2.1$ and a losing one at $r_\phi(x, y_l) = 1.4$. The gap is $\Delta = 0.7$, so the model predicts a human prefers $y_w$ with probability $\sigma(0.7) = 1 / (1 + e^{-0.7}) \approx 0.668$. A gap of $1.0$ gives $\sigma(1.0) \approx 0.731$; a gap of $3.0$ gives $\sigma(3.0) \approx 0.953$; a gap of $0$ gives exactly $0.5$, indifference. The per-example loss on a pair the model gets right with confidence $0.668$ is $-\log 0.668 \approx 0.403$ nats; on a pair it gets _wrong_ (predicts $y_l$ with the same reward gap reversed) the loss is $-\log 0.332 \approx 1.103$ nats, nearly three times larger. The loss penalizes confident mistakes far more than unconfident ones, the behavior a calibrated preference model needs. $$ % caption: The Bradley-Terry link from reward gap to preference probability. The % probability a human prefers the winner is the logistic sigmoid of the reward % difference: 0.5 at a zero gap, rising through 0.67 at a gap of 0.7 and 0.95 at a % gap of 3. Only differences are identified, so the curve is unchanged by adding a % constant to every reward. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \draw[->, black] (-4.2,0) -- (4.4,0) node[anchor=north, black] {reward gap r(win) - r(lose)}; \draw[->, black] (0,-0.15) -- (0,3.7) node[anchor=east, black] {P(win preferred)}; % gridlines at 0.5 and 1.0 \draw[black, dashed] (-4.0,1.6) -- (4.0,1.6); \node[anchor=east, font=\scriptsize, black] at (-0.1,1.6) {0.5}; \draw[black, dashed] (-4.0,3.2) -- (4.0,3.2); \node[anchor=east, font=\scriptsize, black] at (-0.1,3.2) {1.0}; % sigmoid curve, scaled: y = 3.2 * sigma(x) \draw[acc, very thick, smooth] plot coordinates { (-4.0,0.058)(-3.0,0.152)(-2.0,0.379)(-1.0,0.860)(-0.5,1.219) (0,1.6)(0.5,1.981)(0.7,2.137)(1.0,2.340)(2.0,2.821)(3.0,3.048)(4.0,3.142)}; % markers at 0.7 and 3.0 \fill[red] (0.7,2.137) circle (2.2pt); \node[red, anchor=north west, font=\scriptsize] at (0.75,2.1) {gap 0.7, P = 0.67}; \fill[red] (3.0,3.048) circle (2.2pt); \node[red, anchor=north west, font=\scriptsize] at (2.5,3.0) {gap 3, P = 0.95}; \end{tikzpicture} $$ ### Stage 3 — policy optimization against the reward With a reward in hand, the alignment problem finally becomes a reinforcement learning problem. The environment is a bandit-like one: the "state" is the prompt $x$, the "action" is the full generated answer $y \sim \pi_\theta(\cdot \mid x)$, and the reward is $r_\phi(x, y)$ delivered once the answer is complete. We want the policy $\pi_\theta$ that produces high-reward answers, which is a policy-optimization objective — precisely the setting the [policy-gradient methods](/reinforcement-learning/approximation/policy-gradient-methods) of the approximation module were built for. But maximizing $r_\phi$ alone fails. The reward model is an _imperfect_ approximation of human preference, accurate near the answers it was trained on and unreliable far from them. A policy free to move anywhere will find the answers where $r_\phi$ is mistakenly high and collapse onto them. The fix is a leash: keep $\pi_\theta$ from drifting too far from the reference model $\pi_{\text{ref}}$ (the SFT model) that generated the training answers, by penalizing the KL divergence between them. The RLHF objective is $$ \max_{\theta}\; \mathbb{E}_{x \sim \mathcal{D},\; y \sim \pi_\theta(\cdot \mid x)} \Bigl[\, r_\phi(x, y) \,\Bigr] \;-\; \beta\, \mathbb{E}_{x \sim \mathcal{D}} \Bigl[\, \mathrm{KL}\!\bigl(\pi_\theta(\cdot \mid x)\,\big\|\,\pi_{\text{ref}}(\cdot \mid x)\bigr) \Bigr], $$ where $\beta > 0$ controls how tight the leash is. In practice the KL term is folded into the per-token reward and the whole thing is optimized with **[PPO](/reinforcement-learning/deep-rl/actor-critic-and-ppo)** — the clipped surrogate objective from the deep-RL module is the optimizer that does the actual work here, so RLHF is PPO pointed at a learned reward with a KL regularizer rather than a new algorithm. The reward each token sees is the terminal reward minus a running KL penalty, $$ R(x, y) \;=\; r_\phi(x, y) \;-\; \beta \sum_{t} \log \frac{\pi_\theta(y_t \mid x, y_{ **Definition (Reward hacking / over-optimization).** The failure in which a > policy optimized against a learned or proxy reward exploits the gap between the > proxy and the true objective, driving the proxy up while true quality > plateaus or falls. In RLHF it appears as the KL between policy and reference > grows: the policy leaves the region where the reward model is accurate. The KL penalty is the defense. It keeps $\pi_\theta$ inside the neighborhood of $\pi_{\text{ref}}$ where the reward model saw training data and is therefore trustworthy, trading a little reward for staying on reliable ground. The tradeoff is real and has a characteristic shape: as you loosen the leash (raise the KL budget), true quality first rises — the policy is genuinely improving — then peaks and falls as over-optimization sets in, even though the _measured_ reward keeps climbing.[^overopt] $$ % caption: The reward-vs-KL tradeoff. As the policy moves away from the reference % (rising KL), the reward-model score keeps climbing, but true quality rises, % peaks, then falls as over-optimization sets in. The KL leash sets an operating % point near the peak of true quality rather than the peak of measured reward. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \draw[->, black] (-0.2,0) -- (9.4,0) node[anchor=north, black, font=\footnotesize] {KL from reference}; \draw[->, black] (0,-0.2) -- (0,4.6) node[anchor=east, black, font=\footnotesize] {score}; % measured reward: keeps rising, saturating \draw[red, thick, smooth] plot coordinates {(0.2,0.4)(1.2,1.5)(2.4,2.5)(3.6,3.2)(4.8,3.7)(6.0,4.0)(7.2,4.2)(8.6,4.3)}; \node[red, anchor=west, font=\footnotesize] at (8.65,4.3) {measured reward}; % true quality: rises then falls (inverted-U) \draw[acc, very thick, smooth] plot coordinates {(0.2,0.4)(1.2,1.6)(2.4,2.7)(3.4,3.25)(4.0,3.4)(4.8,3.25)(6.0,2.6)(7.2,1.8)(8.6,1.1)}; \node[acc, anchor=west, font=\footnotesize] at (8.65,1.1) {true quality}; % operating point near the peak of true quality \draw[black, dashed] (4.0,0) -- (4.0,3.4); \fill[acc] (4.0,3.4) circle (2.4pt); \node[black, anchor=south, font=\scriptsize] at (4.0,3.5) {KL leash operating point}; \end{tikzpicture} $$ Choosing $\beta$ (or an explicit KL target) is therefore central: it stops the optimizer from exploiting the reward model outside the region where it is accurate. ## Direct Preference Optimization The three-stage pipeline works, but it is heavy: a separate reward model to train and store, an on-policy RL loop with its own instabilities, sampling from the policy at every step, a value head, reward and KL bookkeeping. **Direct Preference Optimization (DPO)**[^dpo] asks whether the reward model is necessary at all, and shows by a change of variables that it is not. The idea in one line: instead of learning a reward and then optimizing a policy against it, notice that the optimal policy _is_ a stand-in for the reward, so you can train the policy on the preferences directly and skip the middle step. The two equations below make that precise — the first writes the best policy in terms of the reward, the second inverts it to write the reward in terms of the policy. The derivation starts from the RLHF objective itself. For a fixed reward $r$, the policy that maximizes reward minus $\beta\,\mathrm{KL}$ has a known closed form — it is the reference policy reweighted by the exponentiated reward: $$ \pi^\star(y \mid x) \;=\; \frac{1}{Z(x)}\, \pi_{\text{ref}}(y \mid x)\, \exp\!\Bigl(\tfrac{1}{\beta}\, r(x, y)\Bigr), $$ where $Z(x)$ is a normalizing constant. This is not something you can sample from directly — $Z(x)$ sums over all possible answers — which is why the standard pipeline needs RL. But the relation can be _inverted_ to write the reward in terms of the optimal policy: $$ r(x, y) \;=\; \beta \, \log \frac{\pi^\star(y \mid x)}{\pi_{\text{ref}}(y \mid x)} \;+\; \beta \, \log Z(x). $$ Now substitute this expression for $r$ into the Bradley-Terry preference loss. Because the loss depends only on the reward _difference_ $r(x, y_w) - r(x, y_l)$, the intractable $\beta \log Z(x)$ term is identical for both answers to the same prompt and **cancels exactly**. What remains is a loss written entirely in terms of the policy we want to train and the frozen reference, with no reward model anywhere: $$ \mathcal{L}_{\text{DPO}}(\theta) \;=\; -\,\mathbb{E}_{(x,\, y_w,\, y_l)} \left[ \log \sigma\!\left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} \;-\; \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]. $$ This is a single supervised classification loss on the preference pairs. It raises the policy's log-probability of the winning answer relative to the reference, lowers it for the loser, with $\beta$ playing the same leash role it did before — the reference appears in the loss, so DPO is implicitly KL-regularized by construction. There is no reward model to fit, no sampling from the policy during training, no RL loop. The reward model has not disappeared; it is _implicit_ in the ratio $\beta \log(\pi_\theta / \pi_{\text{ref}})$ — the reward the change of variables identified. #### What the DPO gradient actually does Differentiating the loss shows how DPO behaves. Write $\hat r_\theta(x, y) = \beta \log \pi_\theta(y \mid x) / \pi_{\text{ref}}(y \mid x)$ for the implicit reward. The gradient of the DPO loss is $$ \nabla_\theta \mathcal{L}_{\text{DPO}} = -\,\beta\,\mathbb{E}_{(x, y_w, y_l)}\Bigl[ \underbrace{\sigma\!\bigl(\hat r_\theta(x, y_l) - \hat r_\theta(x, y_w)\bigr)}_{\text{weight}} \bigl( \nabla_\theta \log \pi_\theta(y_w \mid x) - \nabla_\theta \log \pi_\theta(y_l \mid x) \bigr) \Bigr]. $$ Read the two pieces. The vector part raises the log-probability of the winner and lowers the loser's — ordinary contrastive learning. The scalar weight in front is the point: it is $\sigma$ of the reward gap _in the wrong direction_, so it is _large_ exactly when the implicit reward has the pair backwards (the model currently scores the loser above the winner) and _small_ when the model already ranks them correctly. DPO concentrates its gradient on the pairs it currently ranks wrong and applies little to the ones it ranks right — a built-in hard-example weighting that a naive maximum-likelihood objective on the winners alone would not have. This is why DPO does not simply collapse onto the winning answers: once a pair is ranked correctly with margin, its gradient weight vanishes. #### Worked example: one DPO update Suppose for a pair the current policy assigns log-ratios $\log \pi_\theta / \pi_{\text{ref}}$ of $0.5$ to the winner and $0.8$ to the loser — the model has it backwards, preferring the loser. With $\beta = 0.1$ the implicit rewards are $\hat r_w = 0.05$ and $\hat r_l = 0.08$, so the weight is $\sigma(0.08 - 0.05) = \sigma(0.03) \approx 0.507$, near its maximum of $0.5$: almost the full gradient is applied, pushing the winner's probability up and the loser's down. After several such updates the log-ratios might flip to $0.9$ (winner) and $0.3$ (loser); now the implicit rewards are $\hat r_w = 0.09$ and $\hat r_l = 0.03$, so the weight is $\sigma(\hat r_l - \hat r_w) = \sigma(-0.06) \approx 0.485$ — already below the $0.5$ maximum, and it keeps shrinking as the margin grows. The update throttles itself as it succeeds. $$ % caption: RLHF versus DPO. Classic RLHF fits an explicit reward model, then runs % a PPO loop that samples from the policy, scores with the reward model, and % updates under a KL penalty. DPO folds preference learning into one supervised % loss on the pairs, with the reward implicit in the policy-to-reference log-ratio. % The RLHF reward model is $r_\phi(x, y)$. \begin{tikzpicture}[>=stealth, font=\scriptsize, b/.style={draw, minimum width=26mm, minimum height=11mm, align=center}] \definecolor{acc}{HTML}{2348F2} % top row: RLHF \node[black!70, font=\footnotesize, anchor=west] at (-0.4,1.55) {RLHF}; \node[b] (pref1) at (0.9,0.6) {preference\\pairs}; \node[b] (rmb) at (4.3,0.6) {reward model\\r(x, y)}; \node[b, draw=acc, text=acc] (ppo) at (7.7,0.6) {PPO loop\\(+ KL leash)}; \draw[->, acc, thick] (pref1) -- (rmb); \draw[->, acc, thick] (rmb) -- (ppo); % bottom row: DPO \node[black!70, font=\footnotesize, anchor=west] at (-0.4,-1.75) {DPO}; \node[b] (pref2) at (0.9,-2.4) {preference\\pairs}; \node[b, draw=acc, text=acc] (dpo) at (7.7,-2.4) {one supervised\\classif\/ication loss}; \draw[->, acc, thick] (pref2) -- (dpo) node[midway, above, black!70] {no reward model, no RL loop}; \end{tikzpicture} $$ DPO is simpler, more stable, and cheaper, and it matches or exceeds PPO-based RLHF on many preference benchmarks, which is why it became a default for open-model alignment. The tradeoffs are worth naming: DPO learns only from the fixed preference dataset (it never generates fresh on-policy samples to be scored, so it cannot discover and correct its own new failure modes the way an online PPO loop can), and it inherits whatever coverage gaps the preference data has. The explicit reward model remains useful when you want to score many candidates, run best-of-$n$ sampling, or keep improving with fresh comparisons. | | RLHF (PPO) | DPO | | --- | --- | --- | | Reward model | explicit, trained separately | implicit in $\beta \log(\pi_\theta/\pi_{\text{ref}})$ | | Training signal | on-policy samples scored by $r_\phi$ | fixed offline preference pairs | | Optimization | RL loop (PPO + KL penalty) | one supervised classification loss | | KL regularization | explicit penalty term | built into the loss via the reference | | Failure discovery | can find its own new failures online | limited to the dataset's coverage | | Cost / stability | heavier, more moving parts | lighter, more stable | ## Beyond human labels: RLAIF and verifiable rewards Two directions push past the bottleneck of human comparisons. The first replaces the human labeler with a model. In **RL from AI feedback (RLAIF)** and **Constitutional AI**[^constitutional], the preference labels are produced by an LLM prompted with a written set of principles (a "constitution") rather than by people. A model critiques and revises its own answers against the principles, and generates the preference pairs used to train the reward model or run DPO. This scales preference collection far beyond what human labeling can reach and makes the values being optimized for explicit and auditable in the constitution's text, at the cost of trusting the labeler model's judgment. The second sidesteps preference modeling entirely when the task has a _checkable_ answer. For math, code, and formal reasoning, correctness is not a matter of taste — a solution is right or wrong, and a verifier (a unit-test suite, a symbolic checker, a numeric comparison) supplies a **verifiable reward** with no reward model and no human in the loop. Running policy optimization against such a reward is **RL with verifiable rewards (RLVR)**, and it is what underlies recent reasoning-focused models: the policy is rewarded for producing chains of thought that lead to verifiably correct answers, and the same PPO-style optimization that RLHF uses drives the improvement. The signal is cleaner and harder to hack, but it applies only where correctness is machine-checkable, which is a small slice of what we ask assistants to do. | Variant | Where the reward comes from | Best suited to | | --- | --- | --- | | RLHF | human pairwise comparisons | open-ended helpfulness, tone, safety | | RLAIF / Constitutional | model judgments against written principles | scaling preference labels; explicit values | | RLVR | a verifier checks correctness | math, code, formal reasoning | ## Beyond the standard pipeline: critic-free RL, process rewards, and reasoning models The InstructGPT recipe fixed the shape of RLHF, but the years after it reworked almost every stage. Four developments matter for where the field now sits. **Group-relative policy optimization (GRPO).** PPO on a language model carries a value network — a second model as large as the policy — whose only job is to estimate the baseline for advantage computation. **GRPO**, introduced with the DeepSeekMath work, removes it.[^grpo] Instead of learning a value function, GRPO samples a _group_ of $G$ answers to the same prompt, scores them all with the reward model, and uses the group's own mean and standard deviation as the baseline: the advantage of answer $i$ is $\hat A_i = (r_i - \mean(r_{1:G})) / \std(r_{1:G})$. An answer better than its siblings gets a positive advantage, worse than them a negative one, with no separate critic to train or store. The saving is real — half the memory and one fewer network to stabilize — and GRPO is the optimizer behind the DeepSeek-R1 reasoning models.[^deepseekr1] $$ % caption: PPO versus GRPO advantage estimation. PPO trains a separate value % network to predict the per-token baseline, then forms the advantage as reward % minus that baseline. GRPO samples a group of answers to one prompt, scores each, % and uses the group mean as the baseline (advantage = reward minus group mean, % divided by the group spread), removing the value network entirely. \begin{tikzpicture}[>=stealth, font=\scriptsize, b/.style={draw, minimum width=26mm, minimum height=11mm, align=center}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % PPO row \node[black!70, font=\footnotesize, anchor=west] at (-0.4,1.35) {PPO}; \node[b] (ppol) at (1.0,0.5) {one answer\\+ value net}; \node[b] (ppob) at (4.6,0.5) {baseline from\\value net}; \node[b, draw=acc, text=acc] (ppoa) at (8.2,0.5) {advantage =\\r - baseline}; \draw[->, black, thick] (ppol) -- (ppob); \draw[->, acc, thick] (ppob) -- (ppoa); % GRPO row \node[black!70, font=\footnotesize, anchor=west] at (-0.4,-1.75) {GRPO}; \node[b] (grpl) at (1.0,-2.6) {group of G\\answers, no critic}; \node[b] (grpb) at (4.6,-2.6) {baseline =\\group mean}; \node[b, draw=acc, text=acc] (grpa) at (8.2,-2.6) {advantage =\\(r - mean) / std}; \draw[->, black, thick] (grpl) -- (grpb); \draw[->, acc, thick] (grpb) -- (grpa); \end{tikzpicture} $$ **Process rewards versus outcome rewards.** A verifiable reward on a math problem scores only the final answer — right or wrong — which gives the policy no signal about _where_ a long chain of reasoning went wrong. A **process reward model (PRM)** scores each intermediate step instead, and Lightman et al. showed that supervising the process rather than only the outcome trained more reliable math solvers, because dense per-step feedback assigns credit far more precisely than a single terminal bit.[^prm] The tradeoff is labeling cost: step-level labels are far more expensive to collect than final-answer checks. $$ % caption: Outcome reward versus process reward on a chain of reasoning. An outcome % reward model scores only the final answer, so a wrong step early gets no signal % until the end; a process reward model scores each step, localizing the error and % giving dense credit. Correct steps are marked +, the faulty step -, the outcome a % single terminal check. \begin{tikzpicture}[>=stealth, font=\scriptsize, st/.style={draw, minimum width=13mm, minimum height=9mm, align=center}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % steps \foreach \i/\x in {1/0, 2/1.9, 3/3.8, 4/5.7} { \node[st] (s\i) at (\x,0) {step \i}; } \draw[->, black] (s1) -- (s2); \draw[->, black] (s2) -- (s3); \draw[->, black] (s3) -- (s4); \node[st, draw=acc, text=acc] (ans) at (7.9,0) {answer}; \draw[->, acc] (s4) -- (ans); % process reward marks above each step \node[acc, anchor=south, font=\scriptsize] at (0,0.55) {+}; \node[acc, anchor=south, font=\scriptsize] at (1.9,0.55) {+}; \node[red, anchor=south, font=\scriptsize] at (3.8,0.55) {- (error here)}; \node[acc, anchor=south, font=\scriptsize] at (5.7,0.55) {+}; \node[black, anchor=north, font=\scriptsize] at (2.85,-0.7) {process reward: score every step}; % outcome reward below the answer \node[red, anchor=north, font=\scriptsize] at (7.9,-0.55) {outcome reward:}; \node[red, anchor=north, font=\scriptsize] at (7.9,-1.05) {one terminal bit}; \end{tikzpicture} $$ **Iterative and online DPO.** DPO's main weakness — it never generates fresh samples, so it cannot correct failures outside its fixed dataset — is patched by running it in _rounds_: train with DPO, sample fresh answers from the improved policy, label the new pairs (by humans, an AI judge, or a verifier), and repeat. Each round's on-policy data closes coverage gaps the previous dataset missed, recovering some of the online-exploration advantage that plain PPO-RLHF has over one-shot DPO while keeping DPO's simplicity.[^onlinedpo] **Best-of-$n$ and rejection sampling.** The explicit reward model keeps a use even where DPO wins on training: at _inference_, sample $n$ answers, score them all with $r_\phi$, and return the best. This **best-of-$n$** sampling spends compute to buy quality with no further training, and its offline cousin, **rejection-sampling fine-tuning**, generates many answers, keeps only the reward model's top picks, and fine-tunes on them — a lightweight alternative to a full RL loop that several production systems use as a first alignment pass before or instead of PPO. ## Why this closes the loop The [first lesson of this course](/reinforcement-learning/foundations/what-is-reinforcement-learning) defined reinforcement learning as learning to act from a scalar reward signal found by trial and error, with no supervisor handing over the right answer. RLHF is that definition applied to the largest models we build. There is no labeled "correct" response to imitate for an open-ended request; there is only a comparison signal, distilled into a reward, and a policy improved against it by the same [policy-gradient](/reinforcement-learning/approximation/policy-gradient-methods) and [PPO](/reinforcement-learning/deep-rl/actor-critic-and-ppo) machinery the rest of the course developed. The deep-learning course covers the same method from the language-model side in its [RL from human feedback](/deep-learning/reinforcement-learning/rl-from-human-feedback) lesson. Pretraining gives the model its knowledge; RLHF makes that knowledge usable. The trial-and-error principle that started as a way to teach a simulated agent to walk is now the step that turns a fluent next-token predictor into something a person can actually use, which makes it, for the moment, reinforcement learning's largest real-world application. [^instructgpt]: **Ouyang, Wu, Jiang, et al. (2022)**, "Training language models to follow instructions with human feedback", NeurIPS (the InstructGPT paper) — establishes the three-stage SFT / reward-model / PPO pipeline and shows a 1.3B aligned model preferred over a 175B unaligned one. [^christiano]: **Christiano, Leike, Brown, Martic, Legg, Amodei (2017)**, "Deep Reinforcement Learning from Human Preferences", NeurIPS — learns a reward from a few hundred human comparisons between trajectory segments, including behaviors (e.g. a backflip) with no hand-written reward. [^overopt]: **Gao, Schulman, Hilton (2023)**, "Scaling Laws for Reward Model Overoptimization", ICML — measures how true quality diverges from measured reward as KL from the reference grows, quantifying the over-optimization tradeoff the KL penalty controls. [^dpo]: **Rafailov, Sharma, Mitchell, Manning, Ermon, Finn (2023)**, "Direct Preference Optimization: Your Language Model is Secretly a Reward Model", NeurIPS — derives the change of variables that turns the RLHF objective into a single classification loss on preference pairs, eliminating the explicit reward model and RL loop. [^constitutional]: **Bai, Kadavath, Kundu, et al. (2022)**, "Constitutional AI: Harmlessness from AI Feedback", arXiv:2212.08073 — replaces human harmlessness labels with model self-critique and revision against a written set of principles, generating AI preference labels for training. [^grpo]: **Shao, Wang, Zhu, et al. (2024)**, "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models", arXiv:2402.03300 — introduces Group Relative Policy Optimization (GRPO), which drops PPO's value network and estimates the advantage baseline from the mean reward of a sampled group of answers to the same prompt. [^deepseekr1]: **DeepSeek-AI (2025)**, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning", arXiv:2501.12948 — trains reasoning models with GRPO against verifiable (rule-based) rewards, showing chain-of-thought reasoning can be elicited by RL with a checkable outcome signal. [^prm]: **Lightman, Kosaraju, Burda, et al. (2023)**, "Let's Verify Step by Step", arXiv:2305.20050 — trains a process reward model that scores each reasoning step and shows step-level (process) supervision outperforms outcome-only supervision on mathematical reasoning, releasing the PRM800K step-label dataset. [^onlinedpo]: **Guo, Zhang, Liu, et al. (2024)**, "Direct Language Model Alignment from Online AI Feedback", arXiv:2402.04792 — runs DPO iteratively on freshly sampled, on-policy responses labeled by an AI judge, recovering online-exploration benefits that one-shot offline DPO lacks. See also **Xu et al. (2023)**, "Some things are more CRINGE than others" and the iterative-DPO line for the same idea.