Offline RL: Implicit Methods, Sequence Models, and Beyond

This builds on Offline RL: The Problem and Value-Based Fixes, which diagnosed the core failure — bootstrapping queries the value function at out-of-distribution actions, those queries come back optimistic, and with no online audit the errors compound — and built two families of fixes. Policy constraint (BCQ) keeps the policy near the behavior policy; conservative value estimation (CQL) pushes down the $Q$-values of unseen actions to lower-bound the truth.

Both still learn and query a value function off the data, if only to suppress it. This lesson takes a third family that never queries it at all, then a model-based branch and a sequence-modeling view that reroute around bootstrapping entirely — and the modern lines that grew from them.

Fix family 3 — implicit methods (IQL)

Both families so far still query $Q_{\theta }$ at policy actions during training — BCQ restricts those actions, CQL penalizes their values — which means both still touch the out-of-distribution region and both need a knob to control how much. Implicit Q-Learning (IQL)¹ removes the query entirely: it never evaluates $Q_{\theta }$ at any action outside the dataset, not even to compute the target.

The trick is to replace the Bellman ${\max }_{a^{\prime }}Q(s^{\prime },a^{\prime })$ — the operation that seeks unseen high-value actions — with an expectile of the value distribution over the actions that are in the data. IQL fits a state-value function $V_{\psi }$ by expectile regression against $Q$ on in-sample actions, then uses $V_{\psi }(s^{\prime })$ in place of the max in the Q-target:

where $L_2^{\tau }(u)=|\tau -1(u<0)|u^2$ is the expectile loss with $\tau \in (0.5,1)$. Because $L_2^{\tau }$ penalizes over- and under-estimates asymmetrically, an expectile with $\tau$ near $1$ approximates the maximum over the in-support actions — it estimates the value of the best action the data actually took at that state, without ever proposing an action of its own. Every argument to $Q_{\theta }$ comes straight from $\mathcal{D}$, so extrapolation error never enters.

The policy is then extracted separately by advantage-weighted regression: behavior-clone the data, but weight each logged action by $\exp (\beta (Q(s,a)-V(s)))$, up-weighting actions that beat the state value and down-weighting the rest. This yields a policy that stays inside the data by construction (it only ever imitates logged actions) while preferring the high-advantage ones. IQL is simple, has no separate generative model, and is among the strongest and most stable offline methods in practice.

A worked expectile. To see why the asymmetric loss reaches toward the max, take a state $s$ where the data logged three actions with $Q$-values $\{2,5,8\}$ (equal frequency). The ordinary mean-squared fit ($\tau =0.5$) returns $V(s)=5$, the mean. With $\tau =0.9$ the loss weights positive residuals ($Q>V$) nine times as heavily as negative ones, so $V$ is pulled upward until the residual pull balances: the $\tau$-expectile solves $\tau {\sum }_{Q>V}(Q-V)=(1-\tau ){\sum }_{Q<V}(V-Q)$. Plugging $V=7$ gives $0.9\cdot (8-7)=0.9$ on the left and $0.1\cdot [(7-2)+(7-5)]=0.1\cdot 7=0.7$ on the right — close, and nudging $V$ to about $7.1$ balances them. So the $\tau =0.9$ expectile is roughly $7.1$, far above the mean of $5$ and approaching the best logged value $8$, without the fit ever proposing an action of its own. Push $\tau \to 1$ and $V\to 8$, the in-sample max. That climb toward the best observed action, never an imagined one, is the whole trick.

A worked AWR weight. For the policy, suppose at that same state the logged action $a$ has $Q(s,a)=8$ so its advantage is $A=Q-V=8-7.1=0.9$, while a different logged action $a^{\prime }$ has $Q(s,a^{\prime })=2$, advantage $A^{\prime }=2-7.1=-5.1$. With inverse-temperature $\beta =1$ the AWR weights are $e^{0.9}\approx 2.46$ for $a$ and $e^{-5.1}\approx 0.006$ for $a^{\prime }$: the behavior-cloning loss all but ignores the poor action and imitates the good one strongly. The policy never leaves the set of logged actions, so no out-of-distribution query is possible, yet it concentrates on the advantageous ones.

The three families line up as escalating strictness about the out-of-distribution region.

Model-based offline RL

The same pessimism principle transfers to model-based RL, where you fit a dynamics model $\hat{p}(s^{\prime }\mid s,a)$ from $\mathcal{D}$ and plan or train inside it. A learned model lets you generate synthetic rollouts, but it inherits the identical problem one level up: the model is accurate on transitions like those in $\mathcal{D}$ and unreliable off-support, and a planner that maximizes return will drive the rollout straight into the regions the model is most wrong about and most optimistic.

The fix is again pessimism, now penalized by model uncertainty. MOPO² subtracts a penalty proportional to the model's predicted uncertainty (e.g. the disagreement across an ensemble of dynamics models) from the reward used in synthetic rollouts,

so trajectories that wander into unmodeled territory are discouraged; the agent is optimized on a pessimistic MDP whose reward is lowest exactly where the model is least trustworthy. COMBO³ reaches the same end without an explicit uncertainty estimate by folding a CQL-style value penalty onto state-action pairs generated by model rollouts, pushing down values on synthetic (potentially out-of-distribution) samples while pulling up values on real data. Both realize the same instruction as the model-free methods, applied to the model rather than the value: be pessimistic under the model — do not trust rollouts you cannot verify.

Decision Transformer: offline RL as sequence modeling

A different tradition abandons value functions and Bellman backups altogether. Decision Transformer⁴ reframes offline RL as conditional sequence modeling: treat a trajectory as a sequence of tokens and train a causal Transformer to predict the next action, exactly as a language model predicts the next word. The only twist is what the sequence conditions on. Each trajectory is tokenized as an interleaving of return-to-go ${\hat{R}}_t={\sum }_{t^{\prime }\ge t}{r}_{t^{\prime }}$, state, and action:

and the model is trained by ordinary supervised next-action prediction to output $a_t$ given all tokens up to $s_t$. Because the action is conditioned on the desired future return, at test time you prompt the model with a high target return, feed the current state, and let it autoregressively generate the action that achieves that return — in effect reading off the action taken, at this state, by logged trajectories that went on to earn that much return.

There is no Bellman backup, no bootstrapping, and therefore no out-of-distribution overestimation to begin with — the model only ever imitates actions that appear in the data, so the entire failure mode this lesson is about cannot arise. That is both its appeal and its limit: sequence modeling cannot stitch together sub-optimal trajectories the way dynamic programming can (recombining the good half of one trajectory with the good half of another), so on datasets that reward such recombination the value-based methods above can outperform it. Decision Transformer established that a large chunk of offline RL is reachable by supervised learning alone, and seeded return-conditioned and Transformer-based control as a research direction in its own right.

Fine-tuning, generative planners, and RLHF

Offline RL postdates Sutton & Barto's text, and the years since Decision Transformer have pushed it in three directions the classic treatment does not reach.

Offline-to-online fine-tuning. A policy trained offline is only as good as the log; the natural next move is to deploy it, collect a little fresh data, and continue. The obstacle is that a pessimistic offline value function is badly calibrated for online exploration — it deliberately underrates unseen actions, so naively switching to online RL causes a sharp unlearning dip as the conservative values get corrected all at once. AWAC (Nair et al. 2020) uses advantage-weighted updates precisely so the transition from offline to online is smooth, and IQL's advantage-weighted extraction was designed with the same fine-tuning use in mind. Cal-QL (Nakamoto et al. 2023) fixes the dip directly by calibrating the conservative lower bound so it sits above the behavior policy's value rather than far below every value, which lets online fine-tuning improve monotonically instead of crashing first. The lesson is that pessimism, ideal for pure offline learning, must be relaxed the moment online correction returns.

Return-conditioned and generative planners. Decision Transformer opened a line that treats control as generation. RvS (Emmons et al. 2021) showed that much of Decision Transformer's performance is reachable by a plain feed-forward network conditioned on the return or goal — the Transformer is not essential; the conditioning is. A second branch replaces autoregressive next-action prediction with diffusion: Diffuser (Janner et al. 2022) trains a diffusion model over whole trajectories and plans by sampling high-return trajectories and executing their first action, and Decision Diffuser (Ajay et al. 2023) conditions that generation on returns, constraints, or skills. These generative planners inherit Decision Transformer's freedom from bootstrapping (no out-of-distribution overestimation) while, unlike the causal Transformer, recombining trajectory segments through the denoising process, partially recovering the stitching that sequence models lack.

Offline RL inside RLHF. The single most visible deployment of offline-style RL is in aligning large language models. Direct Preference Optimization (Rafailov et al. 2023) reframes reinforcement learning from human feedback as a supervised objective on a fixed preference dataset — no reward model rolled out online, no policy-gradient sampling — which is offline learning from logged human comparisons. The connection is not incidental: DPO's derivation runs through the same maximum-entropy / KL-constrained-policy machinery, and the pessimism-toward-unseen principle reappears as a KL penalty keeping the tuned model near the reference policy. Offline RL's core discipline — improve a policy from a fixed dataset without drifting off its support — turns out to be exactly the discipline alignment needs.

The through-line: pessimism without a safety net

Every method here is a different encoding of one idea. Online RL can afford optimism because the environment audits it — an overvalued action gets tried and corrected. Offline, that audit is gone, and unchecked optimism about unseen actions compounds through bootstrapping into a value function detached from reality. So each family refuses to trust what the data cannot confirm.

The lesson's one rule: without online correction, you must be pessimistic about what you cannot verify. It is the same off-policy instability from the deadly triad, met not by importance sampling or gradient corrections but by a discipline about trust — because the one stabilizer offline RL removes is the ability to check.