---
title: "Offline RL: The Problem and Value-Based Fixes"
module: Modern Deep Reinforcement Learning
moduleNumber: 5
lessonNumber: 9
order: 509
summary: >
  Offline reinforcement learning learns a policy from a fixed logged dataset with
  no further environment interaction — off-policy learning pushed to the extreme,
  and it breaks for the extreme version of the same reason. Bootstrapping queries
  the value function at out-of-distribution actions the data never covers, those
  errors are optimistic, and with no online feedback to correct them they compound
  through the Bellman backup. This lesson sets up the failure and off-policy
  evaluation, then builds the first two families of pessimistic fixes: policy
  constraint (BCQ) and conservative value estimation (CQL). A companion lesson
  takes up implicit methods, model-based offline RL, and Decision Transformer.
topics: [Deep RL]
sources:
  - book: Sutton & Barto
    ref: "Ch. 11 — Off-policy Methods with Approximation; §11.3 The Deadly Triad"
  - book: Grokking Deep RL
    ref: "Ch. 12 — sample-efficient and off-policy actor-critic; batch/offline learning"
---

Every method so far assumed a running environment. The agent acts, the environment
returns a reward and a next state, and that fresh feedback keeps the
estimates accurate — a value that drifts too high produces actions that get tried,
fail, and get corrected. **Offline reinforcement learning** removes exactly that
loop. You are handed a fixed dataset $\mathcal{D}$ of transitions logged by some
earlier policy, and you must return the best policy you can _without collecting a
single new transition_. No exploration, no rollouts, no chance to test an
optimistic guess against the environment.[^offline-survey]

This is [off-policy learning](/reinforcement-learning/approximation/off-policy-and-the-deadly-triad)
taken to its limit. Off-policy methods already learn about a target policy $\pi$
from data generated by a different behavior policy; offline RL is the special case
where the behavior data is _all there is_ and $\pi$ can never be run to gather
more. The [deadly triad](/reinforcement-learning/approximation/off-policy-and-the-deadly-triad)
— function approximation, bootstrapping, off-policy training — is present in full,
and one of its stabilizers, the ability to visit the states your policy actually
induces, is gone. The result is a failure mode sharp enough to have its own name,
and a family of fixes that all reduce to a single instruction: **be pessimistic
about what the data cannot verify.**

This lesson sets up the problem — the failure mode, and how you even _score_ a policy
from a log — then builds the two value-based families of fix: policy constraint (BCQ)
and conservative value estimation (CQL). The implicit methods, model-based pessimism,
and the sequence-modeling view of Decision Transformer continue in a
[companion lesson](/reinforcement-learning/modern-deep-rl/offline-rl-part-2).

## The setting and why it matters

Formally, offline RL fixes a static dataset

$$
\mathcal{D} = \{(s_i, a_i, r_i, s_i')\}_{i=1}^N,
$$

collected by a **behavior policy** $\pi_\beta$ (possibly a mixture of several
policies, or logged human operators). The learner sees $\mathcal{D}$ once, fits a
policy $\pi$, and is evaluated by deploying $\pi$ in the true environment. Between
fitting and deployment there is no interaction. The empirical **state-action
distribution** of the data,

$$
d^{\pi_\beta}(s,a) \;=\; d^{\pi_\beta}(s)\,\pi_\beta(a \mid s),
$$

is the only region of the world the learner has ground truth for. Anything outside
its support is a guess.

The motivation is that in the domains where RL would help most, online interaction
is expensive, slow, or dangerous, while _logged data already exists in bulk_.

| Domain | Why online interaction is blocked | What logged data exists |
| --- | --- | --- |
| Medicine | a bad treatment policy harms patients | years of electronic health records, dosing logs |
| Autonomous driving / robotics | a bad action crashes hardware | fleet-scale driving logs, teleoperation traces |
| Recommendation | live experiments cost revenue and annoy users | billions of impression-click histories |
| Industrial control | exploration can damage the plant | sensor logs from existing controllers |

In each case the promise is the same one supervised learning delivered on
perception: turn a large static dataset into a decision-making system, with all the
learning done offline and only the final policy deployed. The obstacle is that a
policy is not a label. To evaluate a candidate action the value function must
estimate outcomes for actions _the dataset may never have taken_, and estimation
without feedback is where offline RL fails.

> **Definition (Offline / batch RL).** The problem of learning a policy $\pi$ to
> maximize expected return using only a fixed dataset $\mathcal{D}$ of transitions
> collected by a behavior policy $\pi_\beta$, with no additional environment
> interaction during or after training. Also called _batch RL_.

## The core failure: distributional shift and extrapolation error

Run an ordinary off-policy algorithm — Q-learning, or an actor-critic like
[DDPG](/reinforcement-learning/modern-deep-rl/continuous-control) — on a fixed
dataset and it typically diverges or returns a policy far worse than $\pi_\beta$,
even when $\mathcal{D}$ is large and the network has ample capacity. The culprit is
not optimization but the **bootstrapping target**.

Q-learning fits $Q_\theta$ to the Bellman target

$$
y \;=\; r + \gamma\, \max_{a'} Q_{\bar\theta}(s', a'),
\qquad\text{or for actor-critic}\qquad
y \;=\; r + \gamma\, Q_{\bar\theta}\bigl(s', \pi(s')\bigr).
$$

The maximization — or the actor $\pi$ chasing high value — deliberately seeks the
action $a'$ that _maximizes_ the current $Q$. On a full environment that is fine:
if the max lands on an overvalued action, the agent tries it, observes the true low
reward, and the estimate is corrected. Offline, the transition $(s', a')$ for that
chosen $a'$ may simply not be in $\mathcal{D}$. The network's value there is pure
**extrapolation**, and the argmax preferentially selects wherever the
extrapolation happens to overshoot.

> **Definition (Out-of-distribution action).** At a state $s$, an action $a$ with
> little or no support under the data distribution — $d^{\pi_\beta}(s, a)\approx 0$.
> The value $Q_\theta(s, a)$ at such a point is unconstrained by any observed
> return and is fit only by the network's generalization.

> **Definition (Extrapolation error).** The error in $Q_\theta(s, a)$ at
> out-of-distribution $(s, a)$, arising because no transition in $\mathcal{D}$
> constrains the estimate there. Unlike ordinary approximation error, it is not
> reduced by more data of the same distribution.

Two properties make this fatal offline rather than merely inconvenient.

The first is **selection bias toward overestimation**. Errors in $Q_\theta$ are
roughly symmetric — some out-of-distribution actions read too high, some too low —
but the $\max_{a'}$ (or a greedy actor) is a systematic search for the _highest_
value, so it lands on the positive errors by construction. This is the same
overestimation that motivated Double Q-learning, but without any online experience
to eventually reveal the truth.

The second is **compounding with no corrective feedback**. Bootstrapping feeds the
inflated target back in: an overestimate at $(s', a')$ raises the target $y$ for
$(s, a)$, which raises $Q_\theta(s,a)$, which raises the target for whatever
transitions bootstrap off $s$, and so on around the Bellman backup. Online, a
rollout would eventually visit the offending action and puncture the bubble.
Offline, nothing punctures it. The overestimation propagates and amplifies until
the value function is dominated by fictitious high-value actions and the induced
policy is garbage.

$$
% caption: The out-of-distribution overestimation blowup. Within the data support
% (shaded) the fitted $Q_\theta$ tracks the true $Q$; outside it, the Bellman
% $\max$ selects wherever the extrapolation overshoots, and bootstrapping feeds
% that inflated value back as a target, so the error compounds with no online
% return to correct it.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, black] (-0.2,0) -- (9.2,0) node[anchor=north, black, font=\footnotesize] {action $a$};
  \draw[->, black] (0,-0.2) -- (0,4.4) node[anchor=east, black, font=\footnotesize] {value};
  % data support region
  \fill[acc!8] (2.4,0) rectangle (5.6,4.2);
  \draw[acc!45, dashed] (2.4,0) -- (2.4,4.2);
  \draw[acc!45, dashed] (5.6,0) -- (5.6,4.2);
  \node[acc, font=\footnotesize, anchor=south] at (4.0,4.2) {data support};
  % true Q: a gentle bump peaking inside support
  \draw[black, thick, smooth]
    plot coordinates {(0.4,1.0)(1.4,1.25)(2.4,1.6)(3.2,2.05)(4.0,2.2)(4.8,2.0)(5.6,1.6)(6.6,1.3)(7.6,1.15)(8.7,1.05)};
  \node[black, anchor=west, font=\footnotesize] at (8.75,1.05) {true value};
  % fitted Q: matches inside support, blows up outside on the right
  \draw[red, thick, smooth]
    plot coordinates {(0.4,1.0)(1.4,1.2)(2.4,1.62)(3.2,2.05)(4.0,2.2)(4.8,2.02)(5.6,1.7)(6.4,2.5)(7.2,3.4)(8.0,4.0)(8.7,4.15)};
  \node[red, anchor=west, font=\footnotesize] at (5.9,3.05) {f\/itted value};
  % the argmax picks the overshoot
  \fill[red] (8.55,4.12) circle (2.2pt);
  \draw[red!70, ->] (7.5,3.9) to[bend left=15] (8.5,4.08);
  \node[red, font=\footnotesize, anchor=east] at (7.5,3.85) {argmax picks OOD overshoot};
\end{tikzpicture}
$$

The diagnosis determines the fix. The value function is trustworthy _inside_ the data
support and unreliable outside it, and the standard Bellman backup does the one
thing guaranteed to exploit the unreliable region. Every offline method below
enforces the same rule a different way: **do not trust, or do not query, values off
the data.**

### A worked example: how the bubble inflates

Take a two-state chain: states
$s_0$ and $s_1$, with two actions $\{L, R\}$, discount $\gamma = 0.9$, and a
tiny logged dataset that only ever took $R$:

$$
\mathcal{D} = \bigl\{\,(s_0, R, 0, s_1),\ (s_1, R, 1, s_1)\,\bigr\}.
$$

Action $L$ is never logged from either state. Suppose the true environment is dull:
$L$ from anywhere gives reward $0$ and self-loops, so $Q^\ast(s, L) = 0$, while $R$
earns the returns $Q^\ast(s_1, R) = 1/(1-\gamma) = 10$ and $Q^*(s_0, R) = \gamma\cdot
10 = 9$. A well-behaved learner should recover exactly these.

Now initialize a network whose generalization happens to read the unseen action
optimistically: say it predicts $Q_\theta(s_1, L) = 3$ before any training (a pure
extrapolation, off-support, positive by bad luck). Run one Bellman backup on the
logged $(s_1, R, 1, s_1)$ transition with a greedy max target:

$$
y = r + \gamma \max_{a'} Q_\theta(s_1, a')
  = 1 + 0.9 \cdot \max(\underbrace{Q_\theta(s_1,R)}_{\approx 1},\ \underbrace{Q_\theta(s_1,L)}_{=3})
  = 1 + 0.9 \cdot 3 = 3.7 .
$$

The max reached _off the data_ and pulled the fictitious $3$ into the target for a
_logged_ transition. $Q_\theta(s_1, R)$ is now dragged toward $3.7$ — a target the logged data never justifies — and the inflation does not stop.
The next backup on $(s_0, R, 0, s_1)$ reads $\max_{a'} Q_\theta(s_1, a') = 3.7$ (or
higher, if $L$ still reads higher), so $Q_\theta(s_0, R) \to 0 + 0.9 \cdot 3.7 =
3.33$, and if the network's extrapolation of $Q_\theta(s_0, L)$ is also optimistic
the same overshoot re-enters at $s_0$. Each sweep the phantom value at $L$ feeds a
higher target, the higher target raises the logged $Q$, and with no rollout ever
executing $L$ to observe its true $0$ reward, nothing pushes the estimate back
down. Online, one visit to $(s_1, L)$ would return reward $0$ and collapse the
bubble in a single update; offline, the $L$-value is never corrected and the whole
chain drifts up unboundedly. The induced greedy policy then _prefers $L$_ — an
action the data shows nothing about and the environment rewards not at all.

$$
% caption: The overestimation bubble on the two-state chain. A single optimistic
% off-support value at the unseen action $L$ (dashed) is pulled into the Bellman
% max, inflating the target for a logged transition; the inflated logged value
% feeds the next backup, and with no rollout to execute $L$ the estimate climbs
% each sweep while the true value (flat, low) is never queried.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (-0.2,0) -- (7.4,0) node[anchor=north, black, font=\footnotesize] {backup sweep};
  \draw[->, black] (0,-0.2) -- (0,4.4) node[anchor=east, black, font=\footnotesize] {Q at unseen L};
  % sweep ticks
  \foreach \i/\x in {0/0.6,1/2.0,2/3.4,3/4.8,4/6.2} \node[black, font=\scriptsize, anchor=north] at (\x,0) {\i};
  % true value flat low
  \draw[black, thick] (0.4,0.7) -- (7.0,0.7);
  \node[black, anchor=west, font=\scriptsize] at (5.0,0.95) {true value = 0};
  % inflating estimate
  \draw[red, thick] plot coordinates {(0.6,1.6)(2.0,2.3)(3.4,3.05)(4.8,3.65)(6.2,4.1)};
  \foreach \x/\y in {0.6/1.6,2.0/2.3,3.4/3.05,4.8/3.65,6.2/4.1} \fill[red] (\x,\y) circle (1.8pt);
  \node[red, anchor=east, font=\scriptsize] at (5.9,3.75) {estimate climbs};
  \draw[red!70, ->] (2.3,2.3) to[bend left=12] (3.2,3.0);
  \node[red, anchor=north, font=\scriptsize] at (1.7,1.5) {no rollout corrects it};
\end{tikzpicture}
$$

The failure is averted by one change: a value at $L$ that the learner does not
trust. Each family below installs precisely that.

## Off-policy evaluation: scoring a policy from the log

Before improving a policy offline you often want to _score_ one: given a candidate
$\pi$ and only the log $\mathcal{D}$ from $\pi_\beta$, estimate $J(\pi) =
\mathbb{E}_\pi[\sum_t \gamma^t r_t]$ without ever running $\pi$. This is **off-policy
evaluation** (OPE), and it is both a subproblem inside offline RL and the way a
practitioner decides whether a learned policy is safe to deploy. It inherits the
same distributional-shift problem in a purer form.

The unbiased classical estimator is **importance sampling**. A trajectory $\tau$
logged under $\pi_\beta$ is reweighted by the ratio of the probability $\pi$ would
have produced it to the probability $\pi_\beta$ did:

$$
\hat J_{\text{IS}}
= \frac{1}{m}\sum_{k=1}^{m}
\Bigl(\underbrace{\textstyle\prod_{t=0}^{T} \frac{\pi(a_t \mid s_t)}{\pi_\beta(a_t \mid s_t)}}_{\text{cumulative ratio }w_k}\Bigr)
\Bigl(\textstyle\sum_{t=0}^{T}\gamma^t r_t\Bigr).
$$

It is unbiased, but the product of per-step ratios is its undoing: over a horizon
of $T$ steps the weight $w_k$ is a product of $T{+}1$ factors, and its variance
grows _exponentially_ in $T$. A single trajectory whose actions $\pi$ strongly
prefers but $\pi_\beta$ rarely took gets an astronomically large weight and
dominates the average, so the estimate is unbiased but so noisy as to be useless
past short horizons — the **curse of horizon**.

$$
% caption: Importance-sampling weights for off-policy evaluation. Each logged
% trajectory is reweighted by the product of per-step ratios pi over pi-beta. Most
% weights are near zero (the target policy would rarely take those actions) while a
% few are enormous, so a handful of trajectories dominate the estimate and its
% variance explodes with the horizon.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (-0.2,0) -- (8.6,0) node[anchor=north, black, font=\footnotesize] {logged trajectory};
  \draw[->, black] (0,-0.2) -- (0,3.9) node[anchor=east, black, font=\footnotesize] {IS weight};
  \foreach \x/\h in {0.5/0.15,1.1/0.05,1.7/0.25,2.3/0.1,2.9/3.4,3.5/0.08,4.1/0.2,4.7/0.05,5.3/0.12,5.9/0.06,6.5/2.1,7.1/0.09,7.7/0.15,8.3/0.04} {
    \fill[acc!70] (\x,0) rectangle (\x+0.34,\h);
  }
  \node[red, anchor=south, font=\scriptsize] at (2.9,3.4) {one weight dominates};
  \node[black, anchor=west, font=\scriptsize] at (3.9,0.55) {most near zero};
\end{tikzpicture}
$$

Two standard repairs reduce the variance. **Weighted importance sampling** (WIS)
self-normalizes by dividing by $\sum_k w_k$ instead of $m$; this introduces a small
bias but is consistent and dramatically lower-variance in practice. **Doubly-robust**
estimators combine importance sampling with a learned value-function model: the
model supplies a low-variance baseline, and importance sampling corrects only its
residual, so the estimator is unbiased if _either_ the model or the ratios are
right, and low-variance when the model is even roughly accurate. A third route
abandons per-step ratios entirely and estimates the ratio of _state-action
occupancies_ $\rho_\pi(s,a)/\rho_{\pi_\beta}(s,a)$ directly, sidestepping the
horizon product (marginalized OPE). None escapes the basic limit: a policy far
from $\pi_\beta$ cannot be evaluated reliably from $\pi_\beta$'s data — the same
support limit that constrains offline _learning_ constrains offline _evaluation_.

> **Definition (Off-policy evaluation).** Estimating the value $J(\pi)$ of a target
> policy $\pi$ from a fixed dataset generated by a behavior policy $\pi_\beta$,
> without executing $\pi$. Importance sampling is unbiased but has variance
> exponential in the horizon; weighted and doubly-robust variants trade a little
> bias for much lower variance.

## Fix family 1 — policy constraint (BCQ)

The most direct reading of the diagnosis is: never let the target action stray far
from actions the behavior policy would take. If $\pi$ only ever proposes actions
with support in $\mathcal{D}$, then $Q_\theta$ is only ever queried where it is
constrained by data, and extrapolation error never enters the backup.

**Batch-Constrained deep Q-learning (BCQ)**[^bcq] implements this with a learned
generative model of the behavior policy. A conditional variational autoencoder
$G_\omega(s)$ is trained on $\mathcal{D}$ to reproduce the actions $\pi_\beta$ took
at each state — it is a density model of "actions that plausibly appear in the
data." At decision time BCQ samples a handful of candidate actions from
$G_\omega(s)$, allows each a small learned perturbation $\xi_\phi$ bounded by a
radius $\Phi$, and picks the candidate with the highest $Q$:

$$
\pi(s) \;=\; \argmax_{a_i + \xi_\phi(s, a_i)}\;
Q_\theta\bigl(s,\; a_i + \xi_\phi(s, a_i)\bigr),
\qquad a_i \sim G_\omega(s),\;\; \lvert \xi_\phi \rvert \le \Phi.
$$

The generative model keeps the argmax _on-support_; the bounded perturbation lets
$\pi$ improve on $\pi_\beta$ locally without leaping to unseen actions. Setting
$\Phi = 0$ and one sample recovers behavior cloning; larger $\Phi$ and more samples
recover something closer to Q-learning. BCQ was the paper that named
**extrapolation error** and showed standard off-policy deep RL fails on data its
own earlier version generated.

$$
% caption: BCQ's constrained argmax. A generative model of the behavior policy
% proposes candidate actions clustered on the data support (blue); each is allowed
% a small bounded perturbation (arrows); the argmax picks the best perturbed
% candidate. Because every candidate stays near the data, the value function is
% never queried at the far-off-support actions where it overshoots (red region).
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (-0.2,0) -- (9.2,0) node[anchor=north, black, font=\footnotesize] {action $a$};
  % data support band
  \fill[acc!8] (2.2,0) rectangle (5.4,3.0);
  \node[acc, anchor=south, font=\scriptsize] at (3.8,3.0) {data support};
  % off-support danger
  \fill[red!8] (6.4,0) rectangle (9.0,3.0);
  \node[red, anchor=south, font=\scriptsize] at (7.7,3.0) {o\/f\/f-support: overshoot};
  % candidate samples from generative model
  \foreach \x in {2.6,3.3,3.9,4.6,5.1} \fill[acc] (\x,0.5) circle (2pt);
  % bounded perturbations
  \foreach \x in {2.6,3.3,3.9,4.6,5.1} \draw[acc!70, ->] (\x,0.5) -- (\x+0.28,0.5);
  \node[acc, anchor=north, font=\scriptsize] at (3.9,0.35) {candidates + small perturbation};
  % chosen
  \fill[acc] (4.6,0.5) circle (3pt);
  \draw[acc, ->] (4.6,1.6) -- (4.6,0.65);
  \node[acc, anchor=south, font=\scriptsize] at (4.6,1.6) {argmax (on support)};
\end{tikzpicture}
$$

> **Definition (Policy constraint).** A class of offline methods that restrict the
> learned policy $\pi$ to stay close to the behavior policy $\pi_\beta$ — by
> generative sampling (BCQ), an explicit divergence penalty
> $D\bigl(\pi(\cdot\mid s)\,\Vert\,\pi_\beta(\cdot\mid s)\bigr)$ (e.g. BEAR, BRAC),
> or a KL/MMD term — so the value function is only queried on in-distribution
> actions.

The tension in policy-constraint methods is the choice of $\Phi$ (or the
divergence weight): too tight and $\pi$ can never improve on the logged behavior;
too loose and out-of-distribution actions leak back in. That tuning knob is the
recurring cost of constraining the _policy_ rather than the _value_, which is what
the next family attacks instead.

## Fix family 2 — conservative value estimation (CQL)

Instead of constraining which actions the policy may take, constrain the value
function itself: deliberately **push down** the Q-values of out-of-distribution
actions so the argmax has no incentive to pick them. If unseen actions look bad,
the greedy policy stays on the data on its own, no generative model required.

**Conservative Q-Learning (CQL)**[^cql] adds one term to the standard Bellman
error. The term maximizes $Q_\theta$ on state-action pairs _from the data_ and
minimizes it on actions sampled from the current policy (the ones at risk of being
out-of-distribution overestimates). In its practical form, at each state the
penalty pushes down a log-sum-exp over all actions and pulls up the value of the
logged action:

$$
\min_\theta\;
\alpha\,
\underbrace{\mathbb{E}_{s\sim\mathcal{D}}\!\left[\log\!\sum_{a}\exp Q_\theta(s,a)
\;-\; \mathbb{E}_{a\sim\pi_\beta}\!\bigl[Q_\theta(s,a)\bigr]\right]}_{\text{conservative penalty}}
\;+\;
\tfrac12\,\mathbb{E}_{\mathcal{D}}\!\bigl[(Q_\theta - \hat{\mathcal{B}}Q_{\bar\theta})^2\bigr],
$$

where $\hat{\mathcal{B}}$ is the empirical Bellman operator and $\alpha$ trades off
conservatism against fitting the data. The log-sum-exp is high wherever _any_
action's value is high, so minimizing it flattens the optimistic peaks; the second
expectation protects the values of actions actually in the data. Kumar et al. prove
this learns a $Q_\theta$ that **lower-bounds** the true $Q^\pi$ (for large enough
$\alpha$) — the value function is provably pessimistic, so its greedy policy cannot
be fooled by a fictitious high-value action.

$$
% caption: Conservative versus naive value landscapes at a fixed state. The naive
% $Q_\theta$ (dashed) fits the data in-support but extrapolates upward off it,
% handing the argmax a fictitious peak. CQL adds a penalty that pushes down
% out-of-distribution values, so the conservative $Q_\theta$ (solid) lower-bounds
% the true value and its argmax stays on the data.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (-0.2,0) -- (9.2,0) node[anchor=north, black, font=\footnotesize] {action $a$};
  \draw[->, black] (0,-0.2) -- (0,4.4) node[anchor=east, black, font=\footnotesize] {value};
  \fill[acc!8] (2.6,0) rectangle (5.4,4.2);
  \node[acc, font=\footnotesize, anchor=south] at (4.0,4.2) {data support};
  % true Q
  \draw[black, thick, smooth]
    plot coordinates {(0.4,1.1)(1.6,1.4)(2.6,1.9)(3.4,2.35)(4.0,2.45)(4.6,2.3)(5.4,1.85)(6.6,1.45)(7.8,1.2)(8.7,1.1)};
  \node[black, anchor=west, font=\footnotesize] at (8.75,1.1) {true value};
  % naive: overshoots off-support (dashed)
  \draw[red, thick, dashed, smooth]
    plot coordinates {(0.4,2.6)(1.6,3.0)(2.6,2.05)(3.4,2.4)(4.0,2.5)(4.6,2.35)(5.4,2.1)(6.6,3.1)(7.8,3.7)(8.7,3.95)};
  \node[red, anchor=east, font=\footnotesize] at (8.35,4.25) {naive value};
  % conservative: lower bound (solid acc)
  \draw[acc, very thick, smooth]
    plot coordinates {(0.4,0.5)(1.6,0.75)(2.6,1.75)(3.4,2.2)(4.0,2.3)(4.6,2.15)(5.4,1.7)(6.6,0.85)(7.8,0.55)(8.7,0.5)};
  \node[acc, anchor=west, font=\footnotesize] at (5.6,0.72) {conservative value};
  % argmax markers
  \fill[acc] (3.9,2.3) circle (2.2pt);
  \fill[red] (8.6,3.92) circle (2.2pt);
  \draw[acc!70, ->] (2.7,3.05) to[bend right=10] (3.85,2.42);
  \node[acc, font=\footnotesize, anchor=south] at (2.4,3.0) {argmax on data};
\end{tikzpicture}
$$

> **Definition (Conservative value estimation).** A class of offline methods that
> regularize the value function to be pessimistic — pushing down $Q_\theta$ at
> out-of-distribution actions so that the learned $Q_\theta$ lower-bounds the true
> value. CQL is the canonical instance; the penalty is added directly to the
> Bellman-error objective.

CQL's advantage over policy constraint is that it needs no generative model of
$\pi_\beta$ and degrades gracefully: even a loosely tuned $\alpha$ still yields a
value function biased toward pessimism, which is the safe direction offline.

## Where this leaves us

The setup is now in place. Offline RL fails because bootstrapping queries the value
function at out-of-distribution actions the log never covers; those queries come back
optimistic, and with no online audit the errors compound through the Bellman backup
into a value function detached from reality. Off-policy evaluation lets you _score_ a
candidate policy from the log without deploying it, but it inherits the same shift.

Two families of fixes are now in hand, both enforcing the one rule — trust only what the
data confirms:

- **Policy constraint (BCQ)** keeps the learned policy close to the behavior policy, so
  the value function is never asked about far-off actions in the first place.
- **Conservative value estimation (CQL)** lets the policy roam but pushes down the
  $Q$-values of out-of-distribution actions, provably lower-bounding the true value.

A third, subtler family never queries the value function off the data at all; a
model-based branch builds pessimism into a learned MDP; and a sequence-modeling view
sidesteps bootstrapping entirely. Those — IQL, MOPO/COMBO, and Decision Transformer,
plus the modern fine-tuning and generative-planning lines — continue in
[Offline RL: Implicit Methods, Sequence Models, and Beyond](/reinforcement-learning/modern-deep-rl/offline-rl-part-2).

[^offline-survey]: **Levine, Kumar, Tucker, Fu (2020)**, "Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems", arXiv:2005.01643 — the survey that frames distributional shift as the central offline challenge and organizes the method families.
[^bcq]: **Fujimoto, Meger, Precup (2019)**, "Off-Policy Deep Reinforcement Learning without Exploration", ICML — introduces batch-constrained Q-learning (BCQ) and names extrapolation error; shows standard off-policy deep RL fails on fixed datasets.
[^cql]: **Kumar, Zhou, Tucker, Levine (2020)**, "Conservative Q-Learning for Offline Reinforcement Learning", NeurIPS — the conservative penalty that pushes down out-of-distribution Q-values and provably lower-bounds the true value.