Offline RL: The Problem and Value-Based Fixes

Every method so far assumed a running environment. The agent acts, the environment returns a reward and a next state, and that fresh feedback keeps the estimates accurate — a value that drifts too high produces actions that get tried, fail, and get corrected. Offline reinforcement learning removes exactly that loop. You are handed a fixed dataset $\mathcal{D}$ of transitions logged by some earlier policy, and you must return the best policy you can without collecting a single new transition. No exploration, no rollouts, no chance to test an optimistic guess against the environment.¹

This is off-policy learning taken to its limit. Off-policy methods already learn about a target policy $\pi$ from data generated by a different behavior policy; offline RL is the special case where the behavior data is all there is and $\pi$ can never be run to gather more. The deadly triad — function approximation, bootstrapping, off-policy training — is present in full, and one of its stabilizers, the ability to visit the states your policy actually induces, is gone. The result is a failure mode sharp enough to have its own name, and a family of fixes that all reduce to a single instruction: be pessimistic about what the data cannot verify.

This lesson sets up the problem — the failure mode, and how you even score a policy from a log — then builds the two value-based families of fix: policy constraint (BCQ) and conservative value estimation (CQL). The implicit methods, model-based pessimism, and the sequence-modeling view of Decision Transformer continue in a companion lesson.

The setting and why it matters

Formally, offline RL fixes a static dataset

collected by a behavior policy ${\pi }_{\beta }$ (possibly a mixture of several policies, or logged human operators). The learner sees $\mathcal{D}$ once, fits a policy $\pi$, and is evaluated by deploying $\pi$ in the true environment. Between fitting and deployment there is no interaction. The empirical state-action distribution of the data,

is the only region of the world the learner has ground truth for. Anything outside its support is a guess.

The motivation is that in the domains where RL would help most, online interaction is expensive, slow, or dangerous, while logged data already exists in bulk.

In each case the promise is the same one supervised learning delivered on perception: turn a large static dataset into a decision-making system, with all the learning done offline and only the final policy deployed. The obstacle is that a policy is not a label. To evaluate a candidate action the value function must estimate outcomes for actions the dataset may never have taken, and estimation without feedback is where offline RL fails.

The core failure: distributional shift and extrapolation error

Run an ordinary off-policy algorithm — Q-learning, or an actor-critic like DDPG — on a fixed dataset and it typically diverges or returns a policy far worse than ${\pi }_{\beta }$, even when $\mathcal{D}$ is large and the network has ample capacity. The culprit is not optimization but the bootstrapping target.

Q-learning fits $Q_{\theta }$ to the Bellman target

The maximization — or the actor $\pi$ chasing high value — deliberately seeks the action $a^{\prime }$ that maximizes the current $Q$. On a full environment that is fine: if the max lands on an overvalued action, the agent tries it, observes the true low reward, and the estimate is corrected. Offline, the transition $(s^{\prime },a^{\prime })$ for that chosen $a^{\prime }$ may simply not be in $\mathcal{D}$. The network's value there is pure extrapolation, and the argmax preferentially selects wherever the extrapolation happens to overshoot.

Two properties make this fatal offline rather than merely inconvenient.

The first is selection bias toward overestimation. Errors in $Q_{\theta }$ are roughly symmetric — some out-of-distribution actions read too high, some too low — but the ${\max }_{a^{\prime }}$ (or a greedy actor) is a systematic search for the highest value, so it lands on the positive errors by construction. This is the same overestimation that motivated Double Q-learning, but without any online experience to eventually reveal the truth.

The second is compounding with no corrective feedback. Bootstrapping feeds the inflated target back in: an overestimate at $(s^{\prime },a^{\prime })$ raises the target $y$ for $(s,a)$, which raises $Q_{\theta }(s,a)$, which raises the target for whatever transitions bootstrap off $s$, and so on around the Bellman backup. Online, a rollout would eventually visit the offending action and puncture the bubble. Offline, nothing punctures it. The overestimation propagates and amplifies until the value function is dominated by fictitious high-value actions and the induced policy is garbage.

The diagnosis determines the fix. The value function is trustworthy inside the data support and unreliable outside it, and the standard Bellman backup does the one thing guaranteed to exploit the unreliable region. Every offline method below enforces the same rule a different way: do not trust, or do not query, values off the data.

A worked example: how the bubble inflates

Take a two-state chain: states $s_0$ and $s_1$, with two actions $\{L,R\}$, discount $\gamma =0.9$, and a tiny logged dataset that only ever took $R$:

Action $L$ is never logged from either state. Suppose the true environment is dull: $L$ from anywhere gives reward $0$ and self-loops, so $Q^{\ast }(s,L)=0$, while $R$ earns the returns $Q^{\ast }(s_1,R)=1/(1-\gamma )=10$ and $Q^{\ast }(s_0,R)=\gamma \cdot 10=9$. A well-behaved learner should recover exactly these.

Now initialize a network whose generalization happens to read the unseen action optimistically: say it predicts $Q_{\theta }(s_1,L)=3$ before any training (a pure extrapolation, off-support, positive by bad luck). Run one Bellman backup on the logged $(s_1,R,1,s_1)$ transition with a greedy max target:

The max reached off the data and pulled the fictitious $3$ into the target for a logged transition. $Q_{\theta }(s_1,R)$ is now dragged toward $3.7$ — a target the logged data never justifies — and the inflation does not stop. The next backup on $(s_0,R,0,s_1)$ reads ${\max }_{a^{\prime }}{Q}_{\theta }(s_1,a^{\prime })=3.7$ (or higher, if $L$ still reads higher), so $Q_{\theta }(s_0,R)\to 0+0.9\cdot 3.7=3.33$, and if the network's extrapolation of $Q_{\theta }(s_0,L)$ is also optimistic the same overshoot re-enters at $s_0$. Each sweep the phantom value at $L$ feeds a higher target, the higher target raises the logged $Q$, and with no rollout ever executing $L$ to observe its true $0$ reward, nothing pushes the estimate back down. Online, one visit to $(s_1,L)$ would return reward $0$ and collapse the bubble in a single update; offline, the $L$-value is never corrected and the whole chain drifts up unboundedly. The induced greedy policy then prefers $L$ — an action the data shows nothing about and the environment rewards not at all.

The failure is averted by one change: a value at $L$ that the learner does not trust. Each family below installs precisely that.

Off-policy evaluation: scoring a policy from the log

Before improving a policy offline you often want to score one: given a candidate $\pi$ and only the log $\mathcal{D}$ from ${\pi }_{\beta }$, estimate $J(\pi )={\mathbb{E}}_{\pi }[{\sum }_t{\gamma }^t{r}_t]$ without ever running $\pi$. This is off-policy evaluation (OPE), and it is both a subproblem inside offline RL and the way a practitioner decides whether a learned policy is safe to deploy. It inherits the same distributional-shift problem in a purer form.

The unbiased classical estimator is importance sampling. A trajectory $\tau$ logged under ${\pi }_{\beta }$ is reweighted by the ratio of the probability $\pi$ would have produced it to the probability ${\pi }_{\beta }$ did:

It is unbiased, but the product of per-step ratios is its undoing: over a horizon of $T$ steps the weight $w_k$ is a product of $T+1$ factors, and its variance grows exponentially in $T$. A single trajectory whose actions $\pi$ strongly prefers but ${\pi }_{\beta }$ rarely took gets an astronomically large weight and dominates the average, so the estimate is unbiased but so noisy as to be useless past short horizons — the curse of horizon.

Two standard repairs reduce the variance. Weighted importance sampling (WIS) self-normalizes by dividing by ${\sum }_k{w}_k$ instead of $m$; this introduces a small bias but is consistent and dramatically lower-variance in practice. Doubly-robust estimators combine importance sampling with a learned value-function model: the model supplies a low-variance baseline, and importance sampling corrects only its residual, so the estimator is unbiased if either the model or the ratios are right, and low-variance when the model is even roughly accurate. A third route abandons per-step ratios entirely and estimates the ratio of state-action occupancies ${\rho }_{\pi }(s,a)/{\rho }_{{\pi }_{\beta }}(s,a)$ directly, sidestepping the horizon product (marginalized OPE). None escapes the basic limit: a policy far from ${\pi }_{\beta }$ cannot be evaluated reliably from ${\pi }_{\beta }$'s data — the same support limit that constrains offline learning constrains offline evaluation.

Fix family 1 — policy constraint (BCQ)

The most direct reading of the diagnosis is: never let the target action stray far from actions the behavior policy would take. If $\pi$ only ever proposes actions with support in $\mathcal{D}$, then $Q_{\theta }$ is only ever queried where it is constrained by data, and extrapolation error never enters the backup.

Batch-Constrained deep Q-learning (BCQ)² implements this with a learned generative model of the behavior policy. A conditional variational autoencoder $G_{\omega }(s)$ is trained on $\mathcal{D}$ to reproduce the actions ${\pi }_{\beta }$ took at each state — it is a density model of actions that plausibly appear in the data. At decision time BCQ samples a handful of candidate actions from $G_{\omega }(s)$, allows each a small learned perturbation ${\xi }_{\varphi }$ bounded by a radius $\Phi$, and picks the candidate with the highest $Q$:

The generative model keeps the argmax on-support; the bounded perturbation lets $\pi$ improve on ${\pi }_{\beta }$ locally without leaping to unseen actions. Setting $\Phi =0$ and one sample recovers behavior cloning; larger $\Phi$ and more samples recover something closer to Q-learning. BCQ was the paper that named extrapolation error and showed standard off-policy deep RL fails on data its own earlier version generated.

The tension in policy-constraint methods is the choice of $\Phi$ (or the divergence weight): too tight and $\pi$ can never improve on the logged behavior; too loose and out-of-distribution actions leak back in. That tuning knob is the recurring cost of constraining the policy rather than the value, which is what the next family attacks instead.

Fix family 2 — conservative value estimation (CQL)

Instead of constraining which actions the policy may take, constrain the value function itself: deliberately push down the Q-values of out-of-distribution actions so the argmax has no incentive to pick them. If unseen actions look bad, the greedy policy stays on the data on its own, no generative model required.

Conservative Q-Learning (CQL)³ adds one term to the standard Bellman error. The term maximizes $Q_{\theta }$ on state-action pairs from the data and minimizes it on actions sampled from the current policy (the ones at risk of being out-of-distribution overestimates). In its practical form, at each state the penalty pushes down a log-sum-exp over all actions and pulls up the value of the logged action:

where $\hat{\mathcal{B}}$ is the empirical Bellman operator and $\alpha$ trades off conservatism against fitting the data. The log-sum-exp is high wherever any action's value is high, so minimizing it flattens the optimistic peaks; the second expectation protects the values of actions actually in the data. Kumar et al. prove this learns a $Q_{\theta }$ that lower-bounds the true $Q^{\pi }$ (for large enough $\alpha$) — the value function is provably pessimistic, so its greedy policy cannot be fooled by a fictitious high-value action.

CQL's advantage over policy constraint is that it needs no generative model of ${\pi }_{\beta }$ and degrades gracefully: even a loosely tuned $\alpha$ still yields a value function biased toward pessimism, which is the safe direction offline.

Where this leaves us

The setup is now in place. Offline RL fails because bootstrapping queries the value function at out-of-distribution actions the log never covers; those queries come back optimistic, and with no online audit the errors compound through the Bellman backup into a value function detached from reality. Off-policy evaluation lets you score a candidate policy from the log without deploying it, but it inherits the same shift.

Two families of fixes are now in hand, both enforcing the one rule — trust only what the data confirms:

• Policy constraint (BCQ) keeps the learned policy close to the behavior policy, so the value function is never asked about far-off actions in the first place.
• Conservative value estimation (CQL) lets the policy roam but pushes down the $Q$-values of out-of-distribution actions, provably lower-bounding the true value.

A third, subtler family never queries the value function off the data at all; a model-based branch builds pessimism into a learned MDP; and a sequence-modeling view sidesteps bootstrapping entirely. Those — IQL, MOPO/COMBO, and Decision Transformer, plus the modern fine-tuning and generative-planning lines — continue in Offline RL: Implicit Methods, Sequence Models, and Beyond.