--- title: "Offline RL: Implicit Methods, Sequence Models, and Beyond" module: Modern Deep Reinforcement Learning moduleNumber: 5 lessonNumber: 10 order: 510 summary: > A companion to the offline-RL problem lesson. Policy constraint and conservative value estimation both still query a learned value function; implicit methods (IQL) avoid querying it off the data at all, using an in-sample expectile backup. We then build pessimism into a learned model (MOPO, COMBO) and drop bootstrapping entirely with Decision Transformer's return-conditioned sequence modeling, closing with offline-to-online fine-tuning, diffusion planners, and the offline view of RLHF. The one rule throughout: without online correction, be pessimistic about what you cannot verify. topics: [Deep RL] sources: - book: Sutton & Barto ref: "Ch. 11 — Off-policy Methods with Approximation; §11.3 The Deadly Triad" - book: Grokking Deep RL ref: "Ch. 12 — sample-efficient and off-policy actor-critic; batch/offline learning" --- This builds on [Offline RL: The Problem and Value-Based Fixes](/reinforcement-learning/modern-deep-rl/offline-rl), which diagnosed the core failure — bootstrapping queries the value function at out-of-distribution actions, those queries come back optimistic, and with no online audit the errors compound — and built two families of fixes. **Policy constraint** (BCQ) keeps the policy near the behavior policy; **conservative value estimation** (CQL) pushes down the $Q$-values of unseen actions to lower-bound the truth. Both still learn and query a value function off the data, if only to suppress it. This lesson takes a third family that never queries it at all, then a model-based branch and a sequence-modeling view that reroute around bootstrapping entirely — and the modern lines that grew from them. ## Fix family 3 — implicit methods (IQL) Both families so far still query $Q_\theta$ at policy actions during training — BCQ restricts those actions, CQL penalizes their values — which means both still touch the out-of-distribution region and both need a knob to control how much. **Implicit Q-Learning (IQL)**[^iql] removes the query entirely: it never evaluates $Q_\theta$ at any action outside the dataset, not even to compute the target. The trick is to replace the Bellman $\max_{a'} Q(s', a')$ — the operation that seeks unseen high-value actions — with an **expectile** of the value distribution over the actions that _are_ in the data. IQL fits a state-value function $V_\psi$ by expectile regression against $Q$ on in-sample actions, then uses $V_\psi(s')$ in place of the max in the Q-target: $$ L_V(\psi) = \mathbb{E}_{(s,a)\sim\mathcal{D}}\Bigl[L^\tau_2\bigl(Q_{\bar\theta}(s,a) - V_\psi(s)\bigr)\Bigr], \qquad L_Q(\theta) = \mathbb{E}_{(s,a,s')\sim\mathcal{D}}\Bigl[\bigl(r + \gamma V_\psi(s') - Q_\theta(s,a)\bigr)^2\Bigr], $$ where $L^\tau_2(u) = \lvert \tau - \mathbb{1}(u<0)\rvert\, u^2$ is the **expectile loss** with $\tau \in (0.5, 1)$. Because $L^\tau_2$ penalizes over- and under-estimates asymmetrically, an expectile with $\tau$ near $1$ approximates the _maximum_ over the in-support actions — it estimates the value of the best action the data actually took at that state, without ever proposing an action of its own. Every argument to $Q_\theta$ comes straight from $\mathcal{D}$, so extrapolation error never enters. > **Definition (Expectile regression).** Regression under the asymmetric loss > $L^\tau_2(u) = \lvert \tau - \mathbb{1}(u<0)\rvert\,u^2$. The minimizer is the > $\tau$-expectile of the target distribution; $\tau = 0.5$ gives the mean, and > $\tau \to 1$ approaches the maximum. IQL uses it to estimate the best _in-support_ > action value without querying out-of-distribution actions. The policy is then extracted separately by **advantage-weighted regression**: behavior-clone the data, but weight each logged action by $\exp\bigl(\beta\,(Q(s,a) - V(s))\bigr)$, up-weighting actions that beat the state value and down-weighting the rest. This yields a policy that stays inside the data by construction (it only ever imitates logged actions) while preferring the high-advantage ones. IQL is simple, has no separate generative model, and is among the strongest and most stable offline methods in practice. **A worked expectile.** To see why the asymmetric loss reaches toward the max, take a state $s$ where the data logged three actions with $Q$-values $\{2, 5, 8\}$ (equal frequency). The ordinary mean-squared fit ($\tau = 0.5$) returns $V(s) = 5$, the mean. With $\tau = 0.9$ the loss weights positive residuals ($Q > V$) nine times as heavily as negative ones, so $V$ is pulled upward until the residual pull balances: the $\tau$-expectile solves $\tau\sum_{Q>V}(Q-V) = (1-\tau)\sum_{Q=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \draw[->, black] (-0.2,0) -- (0,0) -- (0,3.6) node[anchor=east, black, font=\footnotesize] {value}; \draw[->, black] (0,0) -- (8.6,0) node[anchor=north, black, font=\footnotesize] {tau}; \foreach \x/\l in {0.6/0.5,4.2/0.9,7.8/1.0} \node[black, font=\scriptsize, anchor=north] at (\x,0) {\l}; % logged action values as horizontal guides \draw[black, dashed] (0,0.7) -- (8.2,0.7); \node[black, font=\scriptsize, anchor=south west] at (8.25,0.62) {Q=2}; \draw[black, dashed] (0,1.75) -- (8.2,1.75); \node[black, font=\scriptsize, anchor=south west] at (8.25,1.67) {Q=5}; \draw[black, dashed] (0,2.8) -- (8.2,2.8); \node[black, font=\scriptsize, anchor=south west] at (8.25,2.72) {Q=8}; % fitted expectile rising \draw[acc, very thick] plot coordinates {(0.6,1.75)(4.2,2.5)(7.8,2.8)}; \fill[acc] (0.6,1.75) circle (2pt); \fill[acc] (4.2,2.5) circle (2pt); \fill[acc] (7.8,2.8) circle (2pt); \node[acc, anchor=south, font=\scriptsize] at (0.6,1.8) {mean}; \node[acc, anchor=south, font=\scriptsize] at (4.2,2.55) {about 7.1}; \node[acc, anchor=south east, font=\scriptsize] at (7.7,2.9) {in-sample max}; \end{tikzpicture} $$ $$ % caption: The three model-free families as escalating strictness about the % out-of-distribution region. BCQ still queries OOD actions but restricts them to a % small ball near the data; CQL queries them only to push their values down; IQL % never queries any action outside the dataset at all. Strictness increases left to % right. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=30mm, minimum height=15mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[box, draw=acc, text=acc] (bcq) at (0,0) {BCQ\\query, restricted\\to small ball}; \node[box, draw=acc, text=acc] (cql) at (4.2,0) {CQL\\query, but push\\values down}; \node[box, draw=acc, text=acc, thick] (iql) at (8.4,0) {IQL\\never query\\o\/f\/f-data at all}; \draw[->, black, thick] (bcq) -- (cql); \draw[->, black, thick] (cql) -- (iql); \node[black, anchor=north, font=\scriptsize] at (4.2,-1.4) {stricter about out-of-distribution actions}; \end{tikzpicture} $$ The three families line up as escalating strictness about the out-of-distribution region. | Family | Method | Mechanism | Touches OOD actions? | | --- | --- | --- | --- | | Policy constraint | BCQ | generate on-support candidates, perturb, argmax | yes, but restricted to a small ball | | Conservative value | CQL | penalize $Q_\theta$ on policy actions, pull up data actions | yes, to push their values down | | Implicit | IQL | expectile of in-sample values; AWR policy extraction | no — only in-dataset actions are ever queried | ## Model-based offline RL The same pessimism principle transfers to model-based RL, where you fit a dynamics model $\hat{p}(s' \mid s, a)$ from $\mathcal{D}$ and plan or train inside it. A learned model lets you generate synthetic rollouts, but it inherits the identical problem one level up: the model is accurate on transitions like those in $\mathcal{D}$ and unreliable off-support, and a planner that maximizes return will drive the rollout straight into the regions the model is most wrong about and most optimistic. The fix is again pessimism, now **penalized by model uncertainty**. **MOPO**[^mopo] subtracts a penalty proportional to the model's predicted uncertainty (e.g. the disagreement across an ensemble of dynamics models) from the reward used in synthetic rollouts, $$ \tilde r(s, a) \;=\; \hat r(s, a) \;-\; \lambda\, u(s, a), $$ so trajectories that wander into unmodeled territory are discouraged; the agent is optimized on a **pessimistic MDP** whose reward is lowest exactly where the model is least trustworthy. **COMBO**[^combo] reaches the same end without an explicit uncertainty estimate by folding a CQL-style value penalty onto state-action pairs generated by model rollouts, pushing down values on synthetic (potentially out-of-distribution) samples while pulling up values on real data. Both realize the same instruction as the model-free methods, applied to the model rather than the value: **be pessimistic under the model — do not trust rollouts you cannot verify.** ## Decision Transformer: offline RL as sequence modeling A different tradition abandons value functions and Bellman backups altogether. **Decision Transformer**[^dt] reframes offline RL as _conditional sequence modeling_: treat a trajectory as a sequence of tokens and train a causal Transformer to predict the next action, exactly as a language model predicts the next word. The only twist is what the sequence conditions on. Each trajectory is tokenized as an interleaving of **return-to-go** $\hat R_t = \sum_{t'\ge t} r_{t'}$, state, and action: $$ \bigl(\hat R_1,\, s_1,\, a_1,\; \hat R_2,\, s_2,\, a_2,\; \ldots,\; \hat R_T,\, s_T,\, a_T\bigr), $$ and the model is trained by ordinary supervised next-action prediction to output $a_t$ given all tokens up to $s_t$. Because the action is conditioned on the _desired future return_, at test time you prompt the model with a high target return, feed the current state, and let it autoregressively generate the action that achieves that return — in effect reading off the action taken, at this state, by logged trajectories that went on to earn that much return. $$ % caption: Decision Transformer tokenizes a trajectory as interleaved % return-to-go, state, and action, and a causal Transformer predicts each next % action from the tokens before it. Conditioning on a desired return turns offline % RL into supervised next-token prediction, with no value function or bootstrapping. \begin{tikzpicture}[>=stealth, font=\footnotesize, tok/.style={draw, minimum width=9mm, minimum height=8mm, align=center, font=\scriptsize}, ret/.style={tok, draw=red, text=red}, act/.style={tok, draw=acc, text=acc}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % input token row \node[ret] (R1) at (0,0) {R-to-go}; \node[tok] (s1) at (1.1,0) {state}; \node[act] (a1) at (2.2,0) {action}; \node[ret] (R2) at (3.5,0) {R-to-go}; \node[tok] (s2) at (4.6,0) {state}; \node[act] (a2) at (5.7,0) {action}; \node[ret] (R3) at (7.0,0) {R-to-go}; \node[tok] (s3) at (8.1,0) {state}; % transformer band \node[draw, fill=acc!8, minimum width=90mm, minimum height=8mm, anchor=west, font=\small] (tf) at (-0.5,1.7) {causal Transformer (masked self-attention)}; % inputs into transformer \foreach \n in {R1,s1,a1,R2,s2,a2,R3,s3} \draw[->, black] (\n.north) -- (\n.north |- tf.south); % predicted actions out \node[act] (pa1) at (2.2,3.4) {action}; \node[act] (pa2) at (5.7,3.4) {action}; \node[act] (pa3) at (8.1,3.4) {action}; \draw[->, acc] (2.2,2.1) -- (pa1); \draw[->, acc] (5.7,2.1) -- (pa2); \draw[->, acc] (8.1,2.1) -- (pa3); \node[acc, anchor=west, font=\scriptsize] at (8.6,3.4) {predicted}; \end{tikzpicture} $$ There is no Bellman backup, no bootstrapping, and therefore _no out-of-distribution overestimation to begin with_ — the model only ever imitates actions that appear in the data, so the entire failure mode this lesson is about cannot arise. That is both its appeal and its limit: sequence modeling cannot **stitch** together sub-optimal trajectories the way dynamic programming can (recombining the good half of one trajectory with the good half of another), so on datasets that reward such recombination the value-based methods above can outperform it. Decision Transformer established that a large chunk of offline RL is reachable by supervised learning alone, and seeded return-conditioned and Transformer-based control as a research direction in its own right. $$ % caption: The stitching limitation. Two logged trajectories each solve half the % task: trajectory A reaches the midpoint, trajectory B goes from the midpoint to % the goal. Dynamic programming can recombine their good halves into an optimal % path through the midpoint; a sequence model that only imitates whole logged % trajectories cannot, since neither trajectory alone reaches the goal. \begin{tikzpicture}[>=stealth, font=\footnotesize, st/.style={circle, draw, minimum size=6mm, inner sep=0pt, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \definecolor{grn}{HTML}{1F9D4D} \node[st] (s) at (0,0) {start}; \node[st] (m) at (4.4,0) {mid}; \node[st, fill=grn!15] (g) at (8.8,0) {goal}; % trajectory A: start -> mid \draw[->, acc, thick] (s) to[bend left=28] node[midway, above, font=\scriptsize] {logged A} (m); % trajectory B: mid -> goal \draw[->, red, thick] (m) to[bend left=28] node[midway, above, font=\scriptsize] {logged B} (g); % stitched optimal (DP) \draw[->, grn, very thick, dashed] (s) to[bend right=22] node[midway, below, font=\scriptsize] {DP stitches A then B} (g); \node[black, anchor=north, font=\scriptsize] at (4.4,-1.7) {sequence model imitates whole trajectories, cannot stitch}; \end{tikzpicture} $$ ## Fine-tuning, generative planners, and RLHF Offline RL postdates Sutton & Barto's text, and the years since Decision Transformer have pushed it in three directions the classic treatment does not reach. **Offline-to-online fine-tuning.** A policy trained offline is only as good as the log; the natural next move is to deploy it, collect a little fresh data, and continue. The obstacle is that a pessimistic offline value function is _badly calibrated_ for online exploration — it deliberately underrates unseen actions, so naively switching to online RL causes a sharp unlearning dip as the conservative values get corrected all at once. **AWAC** (Nair et al. 2020) uses advantage-weighted updates precisely so the transition from offline to online is smooth, and IQL's advantage-weighted extraction was designed with the same fine-tuning use in mind. **Cal-QL** (Nakamoto et al. 2023) fixes the dip directly by _calibrating_ the conservative lower bound so it sits above the behavior policy's value rather than far below every value, which lets online fine-tuning improve monotonically instead of crashing first. The lesson is that pessimism, ideal for pure offline learning, must be relaxed the moment online correction returns. **Return-conditioned and generative planners.** Decision Transformer opened a line that treats control as generation. **RvS** (Emmons et al. 2021) showed that much of Decision Transformer's performance is reachable by a plain feed-forward network conditioned on the return or goal — the Transformer is not essential; the _conditioning_ is. A second branch replaces autoregressive next-action prediction with **diffusion**: **Diffuser** (Janner et al. 2022) trains a diffusion model over whole trajectories and plans by sampling high-return trajectories and executing their first action, and **Decision Diffuser** (Ajay et al. 2023) conditions that generation on returns, constraints, or skills. These generative planners inherit Decision Transformer's freedom from bootstrapping (no out-of-distribution overestimation) while, unlike the causal Transformer, recombining trajectory segments through the denoising process, partially recovering the stitching that sequence models lack. $$ % caption: Diffusion planning. A diffusion model is trained to denoise whole % trajectories drawn from the offline dataset. At plan time it starts from noise % and denoises toward a high-return trajectory (optionally return-conditioned), % then executes only the first action and replans. The denoising can recombine % segments, unlike a purely autoregressive sequence model. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=24mm, minimum height=11mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[box] (noise) at (0,0) {pure noise\\trajectory}; \node[box, draw=acc, text=acc, thick] (denoise) at (4.2,0) {denoise\\(di\/f\/fusion model)}; \node[box, draw=red, text=red] (traj) at (8.6,0) {high-return\\trajectory}; \draw[->, acc, thick] (noise) -- (denoise); \draw[->, acc, thick] (denoise) -- (traj) node[midway, above, font=\scriptsize] {return-conditioned}; \draw[->, red, thick] (traj.south) to[out=270, in=270] node[pos=0.5, below, font=\scriptsize, text=red] {execute f\/irst action, then replan} (denoise.south); \end{tikzpicture} $$ **Offline RL inside RLHF.** The single most visible deployment of offline-style RL is in aligning large language models. **Direct Preference Optimization** (Rafailov et al. 2023) reframes reinforcement learning from human feedback as a supervised objective on a fixed preference dataset — no reward model rolled out online, no policy-gradient sampling — which is offline learning from logged human comparisons. The connection is not incidental: DPO's derivation runs through the same maximum-entropy / KL-constrained-policy machinery, and the pessimism-toward-unseen principle reappears as a KL penalty keeping the tuned model near the reference policy. Offline RL's core discipline — improve a policy from a fixed dataset without drifting off its support — turns out to be exactly the discipline alignment needs. ## The through-line: pessimism without a safety net Every method here is a different encoding of one idea. Online RL can afford optimism because the environment audits it — an overvalued action gets tried and corrected. Offline, that audit is gone, and unchecked optimism about unseen actions compounds through bootstrapping into a value function detached from reality. So each family refuses to trust what the data cannot confirm. | Approach | How it enforces pessimism | | --- | --- | | BCQ / policy constraint | keep the policy near $\pi_\beta$; never query far-off actions | | CQL / conservative value | push down $Q_\theta$ off the data; provably lower-bound the value | | IQL / implicit | never query $Q_\theta$ off the data at all (in-sample expectile) | | MOPO / COMBO / model-based | penalize reward by model uncertainty; pessimistic MDP | | Decision Transformer | imitate return-conditioned; no bootstrapping, no OOD query | The lesson's one rule: **without online correction, you must be pessimistic about what you cannot verify.** It is the same off-policy instability from the [deadly triad](/reinforcement-learning/approximation/off-policy-and-the-deadly-triad), met not by importance sampling or gradient corrections but by a discipline about trust — because the one stabilizer offline RL removes is the ability to check. [^iql]: **Kostrikov, Nair, Levine (2021)**, "Offline Reinforcement Learning with Implicit Q-Learning", arXiv:2110.06169 (ICLR 2022) — expectile regression to approximate the in-support max, with advantage-weighted policy extraction, avoiding out-of-distribution action queries entirely. [^mopo]: **Yu, Thomas, Yu, Ermon, Zou, Levine, Finn, Ma (2020)**, "MOPO: Model-based Offline Policy Optimization", NeurIPS — subtracts a model-uncertainty penalty from the reward, optimizing on a pessimistic MDP. [^combo]: **Yu, Kumar, Rafailov, Rajeswaran, Levine, Finn (2021)**, "COMBO: Conservative Offline Model-Based Policy Optimization", NeurIPS — combines model-based rollouts with a CQL-style value penalty, without an explicit uncertainty estimate. [^dt]: **Chen, Lu, Rajeswaran, Lee, Grover, Laskin, Abbeel, Srinivas, Mordatch (2021)**, "Decision Transformer: Reinforcement Learning via Sequence Modeling", NeurIPS — casts offline RL as return-conditioned autoregressive sequence modeling with a causal Transformer. [^awac]: **Nair, Gupta, Dalal, Levine (2020)**, "AWAC: Accelerating Online Reinforcement Learning with Offline Datasets", arXiv:2006.09359 — advantage-weighted actor-critic that pretrains offline and fine-tunes online without the destabilizing value-correction dip. [^calql]: **Nakamoto, Zhai, Singh, Sobol Mark, Ma, Finn, Kumar, Levine (2023)**, "Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning", NeurIPS — calibrates CQL's conservative lower bound so online fine-tuning improves monotonically instead of unlearning first. [^rvs]: **Emmons, Eysenbach, Kostrikov, Levine (2021)**, "RvS: What is Essential for Offline RL via Supervised Learning?", arXiv:2112.10751 (ICLR 2022) — shows return- and goal-conditioned feed-forward policies match Decision Transformer, isolating conditioning (not the Transformer) as the essential ingredient. [^diffuser]: **Janner, Du, Tenenbaum, Levine (2022)**, "Planning with Diffusion for Flexible Behavior Synthesis", ICML — Diffuser: a diffusion model over full trajectories that plans by sampling high-return trajectories and executing their first action. [^dd]: **Ajay, Du, Gupta, Tenenbaum, Jaakkola, Agrawal (2023)**, "Is Conditional Generative Modeling all you need for Decision-Making?", ICLR — Decision Diffuser: return-, constraint-, and skill-conditioned trajectory diffusion for offline decision-making. [^dpo]: **Rafailov, Sharma, Mitchell, Ermon, Manning, Finn (2023)**, "Direct Preference Optimization: Your Language Model is Secretly a Reward Model", NeurIPS — recasts RLHF as a supervised objective on a fixed preference dataset, an offline-learning view of language-model alignment with a KL penalty toward the reference policy.