---
title: "Value-Function Geometry and Gradient-TD Methods"
module: Approximate Solution Methods
moduleNumber: 3
lessonNumber: 6
order: 306
summary: >
  Why does the deadly triad diverge, and how do you stop it? This lesson develops
  the geometry that explains the failure: value functions as vectors, the
  projection operator onto the representable subspace, and the split between the
  Bellman error, the value error, and the projected Bellman error: the three
  objectives have different minimizers. The projected Bellman error is the
  learnable one, and Gradient-TD methods (GTD2, TDC) do true stochastic
  gradient descent on it, staying stable even off-policy at $O(d)$ cost. Emphatic
  TD reweights states instead, and a survey of variance-reduction techniques
  closes the gap between stability and usable learning.
topics: [Approximation]
sources:
  - book: Sutton & Barto
    ref: "Ch. 11 — Off-policy Methods with Approximation; §11.4 Linear Value-function Geometry; §11.5 Gradient Descent in the Bellman Error"
  - book: Sutton & Barto
    ref: "§11.7 Gradient-TD Methods; §11.8 Emphatic-TD Methods; §11.9 Reducing Variance"
---

This builds on
[off-policy methods and the deadly triad](/reinforcement-learning/approximation/off-policy-and-the-deadly-triad),
which showed that function approximation, bootstrapping, and off-policy training
together can send the weights to infinity — on a seven-state problem with linearly
independent features, no less. That lesson named the failure but did not explain
it. This lesson does, by picturing value functions as points in a vector space,
and from that picture builds the two families of methods that make off-policy
learning stable.

## Linear value-function geometry

To understand _why_ the triad diverges and how to fix it, it helps to picture value
functions as vectors.[^sb-geom] Enumerate the states $\mathcal{S} = \{s_1, \dots,
s_{|\mathcal{S}|}\}$; then any value function $v$ is the vector $[v(s_1), \dots,
v(s_{|\mathcal{S}|})]^\top$ in an $|\mathcal{S}|$-dimensional space. The whole space
holds every conceivable value function; most correspond to no policy at all.

A linear approximator with $d < |\mathcal{S}|$ parameters can only reach a
$d$-dimensional _subspace_ of that space — the plane of all $v_{\mathbf{w}} =
\mathbf{X}\mathbf{w}$ representable by some weight vector. The true value $v_\pi$ of a
policy generally lies _outside_ this subspace, too complex to represent exactly.

$$
% caption: Linear value-function geometry. The true value $v_\pi$ lies outside the
% subspace of representable functions; the projection operator $\Pi$ drops it to the
% closest representable point $\Pi v_\pi$, and the Bellman operator $B_\pi$ (gray) takes
% points out of the subspace before projection maps them back.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % the representable subspace (a slanted plane)
  \draw[black, fill=black!4] (-0.4,0) -- (6.6,0) -- (5.4,1.5) -- (-1.6,1.5) -- cycle;
  \node[anchor=north west, font=\scriptsize, black] at (-1.5,0.02) {subspace: values representable as v(w)};
  % point v_w in the plane
  \fill[acc] (1.4,0.7) circle (2pt);
  \node[acc, anchor=north, font=\scriptsize] at (1.4,0.58) {v(w)};
  % projection of v_pi into plane
  \fill[black] (4.1,0.75) circle (2pt);
  \node[black, anchor=north west, font=\scriptsize] at (4.15,0.72) {proj of v-pi};
  % true value above
  \fill[black] (4.6,3.35) circle (2.4pt);
  \node[black, anchor=west, font=\footnotesize] at (4.72,3.4) {v-pi};
  % value error line (vertical drop)
  \draw[black, thick] (4.6,3.35) -- (4.1,0.78);
  \node[black, anchor=south west, font=\scriptsize] at (4.72,2.0) {value error VE};
  % Bellman image of v_w outside the plane
  \fill[red] (2.55,2.55) circle (2pt);
  \node[red, anchor=south west, font=\scriptsize] at (2.6,2.58) {B-pi v(w)};
  \draw[red, thick] (1.4,0.7) -- (2.55,2.55);
  \node[red, anchor=south east, font=\scriptsize] at (1.55,1.85) {Bellman error BE};
  % projection of Bellman image back into plane
  \fill[red] (2.55,0.68) circle (2pt);
  \node[red, anchor=north, font=\scriptsize] at (2.55,0.6) {proj B-pi v(w)};
  \draw[red, dashed] (2.55,2.55) -- (2.55,0.68);
  % PBE arrow inside plane
  \draw[->, red, thick] (1.4,0.7) -- (2.5,0.68);
  \node[red, anchor=south, font=\scriptsize] at (1.95,0.72) {PBE};
\end{tikzpicture}
$$

To measure how far apart two value functions are, Euclidean distance is wrong: some
states matter more because they are visited more often. Weight each state by the
distribution $\mu$ (typically the on-policy distribution) and use the **$\mu$-norm**

$$
\lVert v \rVert_\mu^2 \;\doteq\; \sum_{s \in \mathcal{S}} \mu(s)\, v(s)^2.
$$

The **projection operator** $\Pi$ takes any value function to the representable one
closest in this norm:

$$
\Pi v \;\doteq\; v_{\mathbf{w}}
\quad\text{where}\quad
\mathbf{w} \;=\; \argmin_{\mathbf{w} \in \mathbb{R}^d} \lVert v - v_{\mathbf{w}} \rVert_\mu^2,
\qquad
\Pi \;\doteq\; \mathbf{X}\bigl(\mathbf{X}^\top \mathbf{D}\, \mathbf{X}\bigr)^{-1}\mathbf{X}^\top \mathbf{D},
$$

where $\mathbf{X}$ stacks the feature vectors as rows and $\mathbf{D}$ is the diagonal
matrix of the weights $\mu(s)$. The projection of the true value, $\Pi v_\pi$, is the
best representable approximation in the value-error sense, and the point Monte Carlo
methods converge to.

### Bellman error versus projected Bellman error

TD methods aim somewhere else. The **Bellman operator** $B_\pi$ applies one sweep of
the Bellman equation:

$$
(B_\pi v)(s) \;\doteq\; \sum_a \pi(a \mid s) \sum_{s', r} p(s', r \mid s, a)\bigl[r + \gamma\, v(s')\bigr].
$$

Its unique fixed point is the true value: $v_\pi = B_\pi v_\pi$, which is just the
Bellman equation rewritten. For any representable $v_{\mathbf{w}}$, the difference
$\bar\delta_{\mathbf{w}} = B_\pi v_{\mathbf{w}} - v_{\mathbf{w}}$ is the **Bellman error
vector** — the expectation of the TD error at each state — and its squared $\mu$-norm is
the **mean square Bellman error**:

$$
\overline{BE}(\mathbf{w}) \;=\; \lVert \bar\delta_{\mathbf{w}} \rVert_\mu^2.
$$

Here is the geometric crux. Applying $B_\pi$ to a representable value function generally
produces one _outside_ the subspace (the gray arrow leaving the plane in the figure). In
pure dynamic programming, without approximation, you follow $B_\pi$ repeatedly and
converge to $v_\pi$. With approximation you cannot: after one Bellman step you must
_project_ back into the subspace to have a representable function, then step out again,
then project back. The quantity TD methods drive to zero is the residual _inside_ the
subspace — the **projected Bellman error**:

$$
\overline{PBE}(\mathbf{w}) \;=\; \lVert \Pi\, \bar\delta_{\mathbf{w}} \rVert_\mu^2.
$$

> **Definition (Projected Bellman error, $\overline{PBE}$).** The squared $\mu$-norm of
> the Bellman error vector _after projecting it back into the representable subspace_,
> $\lVert \Pi\, \bar\delta_{\mathbf{w}} \rVert_\mu^2$. With linear approximation there is
> always a weight vector $\mathbf{w}_{\text{TD}}$ making it zero — the TD fixed point.
> Minimizing $\overline{PBE}$ (rather than $\overline{BE}$ or $\overline{VE}$) is the
> target of stable off-policy methods.

The three objectives generally have _different_ minimizers: $\overline{VE}$ is minimized
at $\Pi v_\pi$, $\overline{BE}$ at some other point, and $\overline{PBE}$ at the TD fixed
point $\mathbf{w}_{\text{TD}}$. Semi-gradient TD, when it converges, converges to
$\mathbf{w}_{\text{TD}}$ — but under off-policy training it need not converge at all.

A tempting fix is to do true gradient descent on the Bellman error itself (the
**residual-gradient** algorithm), which is a genuine SGD method and converges. But two
results rule it out as a general approach. First, on the A-split example it converges to the
_wrong_ values, minimizing a temporal-smoothing surrogate rather than accuracy. Second,
and more fundamentally, the $\overline{BE}$ is **not learnable**: two Markov reward
processes can produce identical data yet have different $\overline{BE}$-minimizing weight
vectors, so no algorithm working from feature vectors, actions, and rewards alone can
minimize it.[^sb-be] That leaves the $\overline{PBE}$, which _is_ learnable, as the
objective worth pursuing.

## Gradient-TD methods

**Gradient-TD methods** perform true stochastic gradient descent on the
$\overline{PBE}$.[^sb-gtd] Because they follow an actual gradient, they inherit SGD's
robust convergence — stable even under off-policy training and nonlinear approximation —
at $O(d)$ cost per step, only about double that of semi-gradient TD.

The derivation expands the $\overline{PBE}$ in matrix form and differentiates. The
key point in the algebra: the gradient factors into three pieces, and
two of them share a random quantity ($\mathbf{x}_{t+1}$), so sampling each
independently and multiplying would double-count the noise. Gradient-TD works
around that one obstacle.

The gradient turns out to be a product of three expectations,

$$
\nabla \overline{PBE}(\mathbf{w}) \;=\; 2\, \mathbb{E}\bigl[\rho_t (\gamma\, \mathbf{x}_{t+1} - \mathbf{x}_t)\, \mathbf{x}_t^\top\bigr]\; \mathbb{E}\bigl[\mathbf{x}_t \mathbf{x}_t^\top\bigr]^{-1}\; \mathbb{E}\bigl[\rho_t\, \delta_t\, \mathbf{x}_t\bigr],
$$

and the trouble is that the first and last factors both depend on the next feature
$\mathbf{x}_{t+1}$, so sampling them independently and multiplying gives a _biased_
estimate — the same double-sampling problem that afflicts the residual-gradient method.
The Gradient-TD solution is to estimate and store the product of the last two factors as a
second parameter vector $\mathbf{v}$,

$$
\mathbf{v} \;\approx\; \mathbb{E}\bigl[\mathbf{x}_t \mathbf{x}_t^\top\bigr]^{-1}\, \mathbb{E}\bigl[\rho_t\, \delta_t\, \mathbf{x}_t\bigr],
$$

which is the solution to a least-squares problem and is learned by an
importance-weighted LMS rule on a second, faster step size $\beta$:

$$
\mathbf{v}_{t+1} \;\doteq\; \mathbf{v}_t + \beta\, \rho_t\, \bigl(\delta_t - \mathbf{v}_t^\top \mathbf{x}_t\bigr)\, \mathbf{x}_t.
$$

Given $\mathbf{v}_t$, the main weight vector is updated by a rule that is $O(d)$ if the
inner product $\mathbf{x}_t^\top \mathbf{v}_t$ is formed first. Two members of the family:

$$
\text{GTD2:}\quad \mathbf{w}_{t+1} \;=\; \mathbf{w}_t + \alpha\, \rho_t\, (\mathbf{x}_t - \gamma\, \mathbf{x}_{t+1})\, (\mathbf{x}_t^\top \mathbf{v}_t),
$$

$$
\text{TDC / GTD(0):}\quad \mathbf{w}_{t+1} \;=\; \mathbf{w}_t + \alpha\, \rho_t\, \bigl(\delta_t\, \mathbf{x}_t - \gamma\, \mathbf{x}_{t+1}\, (\mathbf{x}_t^\top \mathbf{v}_t)\bigr).
$$

**TDC** (TD with gradient correction) is the semi-gradient TD update
$\alpha \rho_t \delta_t \mathbf{x}_t$ plus a correction term $-\alpha \rho_t \gamma
\mathbf{x}_{t+1}(\mathbf{x}_t^\top \mathbf{v}_t)$ that supplies the missing half of the
gradient. That single correction is the difference between divergence and convergence.

$$
% caption: TDC on Baird's counterexample. The $\overline{PBE}$ falls to zero and the
% weight block $w_1$ through $w_6$ settles; the individual components need not reach
% zero, so the value error $\overline{VE}$ stays near 2, but the system is stable.
% Step sizes $\alpha = 0.005$, $\beta = 0.05$.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, black] (0,0) -- (6.4,0) node[anchor=north east, font=\scriptsize] {sweeps};
  \draw[->, black] (0,-0.2) -- (0,4.0) node[anchor=south east, font=\scriptsize] {value};
  \draw[black, dashed] (0,0.3) -- (6.0,0.3);
  \node[anchor=east, font=\scriptsize] at (-0.08,0.3) {0};
  % PBE decays to zero
  \draw[red, very thick] (0,2.7) .. controls (0.9,1.1) and (1.6,0.4) .. (6.0,0.3);
  \node[red, anchor=west, font=\scriptsize] at (6.05,0.3) {PBE};
  % VE settles near 2
  \draw[acc, very thick] (0,3.4) .. controls (0.8,2.2) and (1.6,1.75) .. (6.0,1.7);
  \node[acc, anchor=west, font=\scriptsize] at (6.05,1.7) {VE};
  % weight block settles near a positive constant
  \draw[black, thick] (0,2.35) .. controls (0.9,1.5) and (1.8,1.35) .. (6.0,1.35);
  \node[black, anchor=west, font=\scriptsize] at (6.05,1.35) {w1-w6};
\end{tikzpicture}
$$

On Baird's counterexample TDC does exactly what the theory promises: the
$\overline{PBE}$ decays to zero and the weights stop growing. Notice the components do
_not_ all reach zero, so the value error stays near $2$ — the $\overline{PBE}$ fixed point
is not the best approximation in the $\overline{VE}$ sense — but the divergence is gone.
GTD2 and TDC are **two-time-scale** methods: the correction estimate $\mathbf{v}$ (fast,
step size $\beta$) must track ahead of the primary weights $\mathbf{w}$ (slow, step size
$\alpha$), which the convergence proofs formalize as a _cascade_ requiring $\beta \to 0$
and $\alpha / \beta \to 0$. Gradient-TD is currently the best-understood and most widely
used family of stable off-policy methods, with extensions to action values, control, and
[eligibility traces](/reinforcement-learning/approximation/eligibility-traces).

## A note on emphatic-TD

Gradient-TD attacks the second part of the problem — the wrong update distribution — by
building a true gradient method. **Emphatic-TD** attacks it differently: keep the simple,
computationally cheap semi-gradient update, but _reweight the states_ so the update
distribution recovers the on-policy properties that make linear semi-gradient TD
stable.[^sb-etd] Linear semi-gradient TD is stable on-policy precisely because the state
distribution matches the transitions under $\pi$; emphatic-TD manufactures that match by
scaling each update with an **emphasis** $M_t$ that accumulates responsibility for the
current state through a recursive trace,

$$
\mathbf{w}_{t+1} \;=\; \mathbf{w}_t + \alpha\, M_t\, \rho_t\, \delta_t\, \nabla \hat{v}(S_t, \mathbf{w}_t),
\qquad
M_t \;=\; \gamma\, \rho_{t-1}\, M_{t-1} + I_t,
$$

with the **interest** $I_t$ specifying how much we care about accurately valuing each
state. In expectation, emphatic-TD converges to zero value error on Baird's
counterexample; in practice its variance is so high that consistent results are hard to
obtain, which is why the next section turns to variance-reduction techniques for all
of these methods.

## Reducing variance

Stability is only half the problem. Gradient-TD and emphatic-TD keep the weights
from diverging, but off-policy learning is _inherently_ higher-variance than
on-policy learning, and that variance is what makes these methods slow and their
results erratic in practice.[^sb-variance] The intuition is plain: if the data you
receive is only loosely related to the policy you are trying to evaluate, you
should expect to learn less about it. In the extreme you learn nothing — you
cannot learn to drive by cooking dinner. Only when the target and behavior
policies visit similar states and take similar actions is real progress possible.

Yet that overlap is what off-policy learning is for: any policy has many
neighbors, similar-but-not-identical policies with considerable overlap, and the
whole reason to learn off-policy is to generalize across that vast family of
related policies from one stream of experience. So the variance cannot simply be
avoided; it has to be _managed_.

### Where the variance comes from

The specific source is the importance-sampling ratio. As we saw, off-policy
targets are corrected by products of per-step ratios $\rho_t = \pi(A_t\mid
S_t)/b(A_t\mid S_t)$. Each ratio is $1$ in expectation, and successive ratios are
uncorrelated, so their products are $1$ in expectation too — but their _values_
swing from zero up to very large numbers, and a product of such terms has very
high variance.[^sb-variance]

A worked case shows how fast the product blows up. Let the behavior policy be
uniform over two actions, $b(a \mid s) = 0.5$, while the target policy is nearly
deterministic, $\pi(a_1 \mid s) = 0.9$ and $\pi(a_2 \mid s) = 0.1$. The per-step
ratio is then $\rho = 0.9/0.5 = 1.8$ when the behavior policy happens to sample
$a_1$ and $\rho = 0.1/0.5 = 0.2$ when it samples $a_2$; each occurs with
probability $0.5$, and the mean is $0.5 \cdot 1.8 + 0.5 \cdot 0.2 = 1$, as it must
be. Over a horizon of $n = 10$ steps the cumulative ratio $\rho_{t:t+9}$ is a
product of ten such draws. Its mean stays exactly $1$, but its _spread_ is enormous:
an all-$a_1$ trajectory (probability $0.5^{10} \approx 0.001$) carries
$1.8^{10} \approx 357$, while a trajectory with even one $a_2$ is scaled down by
the $0.2$ factor, and a mostly-$a_2$ trajectory carries a ratio near $0.2^{10}
\approx 10^{-7}$. The variance of the product grows geometrically in $n$: a rare
trajectory contributes a step roughly $357$ times the nominal size, and that single
step dominates the average of thousands of ordinary ones.

$$
% caption: The distribution of the 10-step importance ratio $\rho_{t:t+9}$ for
% $b = (0.5, 0.5)$, $\pi = (0.9, 0.1)$. Almost all mass sits below 1 (products of
% mostly-$0.2$ factors), so the mean of exactly 1 is held up by a thin tail reaching
% $1.8^{10} \approx 357$. That heavy tail is the variance the SGD update inherits.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (0,0) -- (6.6,0) node[anchor=north east, font=\scriptsize] {ratio value (log scale)};
  \draw[->, black] (0,0) -- (0,3.6) node[anchor=south east, font=\scriptsize] {probability};
  % tall bars near small values
  \foreach \x/\h in {0.5/3.1, 1.1/2.1, 1.7/1.25, 2.3/0.7, 2.9/0.38} {
    \draw[acc, very thick] (\x,0) -- (\x,\h);
  }
  % thin tail bars toward the large ratio
  \foreach \x/\h in {3.6/0.22, 4.3/0.12, 5.0/0.06, 5.7/0.03} {
    \draw[red, very thick] (\x,0) -- (\x,\h);
  }
  \node[anchor=north, font=\scriptsize] at (0.5,-0.08) {tiny};
  \draw[black, dashed] (2.05,0) -- (2.05,3.3);
  \node[black, anchor=south, font=\scriptsize] at (2.05,3.3) {mean = 1};
  \node[red, anchor=south, font=\scriptsize] at (5.7,0.1) {357};
  \node[red, anchor=west, font=\scriptsize] at (4.4,1.6) {heavy tail};
\end{tikzpicture}
$$

The damage is done because these ratios multiply the step size in the SGD update:
a large $\rho$ means a large step, which stochastic gradient
descent cannot tolerate. SGD relies on averaging many small, noisy steps to estimate
the true gradient; an occasional enormous step can move the parameter vector into
a region of the space with a completely different gradient, and the averaging
argument breaks. The obvious defense — shrinking the step size until no single step
is dangerous — makes the _expected_ step tiny and learning slow.

$$
% caption: Why importance-sampling variance hurts SGD. Small ratios give safe
% steps (blue); an occasional large ratio produces a giant step (red) that
% overshoots into a region with a different gradient, and no amount of averaging
% recovers a good direction. Shrinking the step size to contain the outlier makes
% the ordinary steps uselessly small.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % loss contours (a bowl)
  \foreach \r in {0.7,1.4,2.1} \draw[black] (0,0) ellipse ({\r*1.5} and \r);
  \fill[black] (0,0) circle (2pt);
  \node[anchor=north, font=\scriptsize] at (0,-2.3) {minimum};
  % start point
  \coordinate (p0) at (-3.3,1.7);
  \fill[acc] (p0) circle (2pt);
  \node[acc, anchor=south, font=\scriptsize] at (-3.3,1.85) {start};
  % small safe steps
  \coordinate (p1) at (-2.5,1.25);
  \coordinate (p2) at (-1.8,0.9);
  \draw[acc, thick, ->] (p0) -- (p1);
  \draw[acc, thick, ->] (p1) -- (p2);
  \node[acc, anchor=east, font=\scriptsize] at (-2.6,0.95) {small ratios: safe};
  % giant step overshoots
  \draw[red, very thick, ->] (p2) -- (3.6,-1.9);
  \node[red, anchor=west, font=\scriptsize] at (1.2,-1.0) {large ratio: overshoot};
\end{tikzpicture}
$$

### Five ideas for taming it

No single technique suffices, but several complementary strategies each reduce
the variance.[^sb-variance]

- **Weighted importance sampling.** Chapter 5 already showed that
  [weighted importance sampling](/reinforcement-learning/tabular-methods/monte-carlo-methods)
  is far better behaved than the ordinary kind — its updates have much lower
  variance because the largest weight any sample can carry is bounded. The catch is
  that adapting weighted importance sampling to function approximation is genuinely
  hard, and can probably only be done approximately, at $O(d)$ cost.

- **Avoiding ratios entirely.** The
  [tree-backup](/reinforcement-learning/tabular-methods/n-step-bootstrapping)
  algorithm performs off-policy learning _without_ any importance-sampling ratios
  at all, by backing up the expected value under the target policy rather than
  correcting sampled actions. Extending that idea to the function-approximation
  case yields stable and more efficient methods, since where there are no ratios
  there is no ratio-variance.

- **Momentum and averaging.** Because the trouble is occasional large, uncorrelated
  steps, optimizer machinery that smooths the update sequence helps directly:
  momentum, Polyak–Ruppert averaging (averaging the iterates rather than trusting
  the latest one), and their extensions all damp the effect of a single outlier
  step.

- **Adaptive, per-component step sizes.** Rather than one global step size
  limited by the worst-case ratio, methods that set a _separate_,
  adaptively-tuned step size for each component of the parameter vector — and
  "importance-weight-aware" updates designed with these products in mind — let the
  learning rate respond to the local variance instead of being throttled globally.

- **Constraining the target toward the behavior.** A complementary strategy defines
  the target policy partly _by reference to_ the behavior policy, so that it can
  never differ enough to create huge ratios in the first place. "Recognizer"-style
  constructions cap how far $\pi$ may stray from $b$, trading a little generality
  for a lot of stability.

$$
% caption: Five levers on off-policy variance. Some cut the variance at its source
% (weighted IS, tree-backup, constrained targets); others absorb it after the fact
% (momentum/averaging, adaptive step sizes). They are complementary and can be
% combined.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  src/.style={draw, minimum width=34mm, minimum height=9mm, align=center, font=\scriptsize},
  abs/.style={draw, minimum width=34mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[font=\footnotesize, text=acc] at (-3.0,3.0) {cut at the source};
  \node[src, draw=acc] (a) at (-3.0,2.15) {weighted IS};
  \node[src, draw=acc] (b) at (-3.0,1.05) {tree-backup: no ratios};
  \node[src, draw=acc] (c) at (-3.0,-0.05) {constrain target to b};
  \node[font=\footnotesize, text=red] at (3.0,3.0) {absorb after the fact};
  \node[abs, draw=red] (d) at (3.0,2.15) {momentum / averaging};
  \node[abs, draw=red] (e) at (3.0,1.05) {adaptive step sizes};
  \node[align=center, font=\footnotesize] (v) at (0,-1.5) {lower-variance\\of\/f-policy update};
  \draw[->, acc] (a) -- (v);
  \draw[->, acc] (b) -- (v);
  \draw[->, acc] (c) -- (v);
  \draw[->, red] (d) -- (v);
  \draw[->, red] (e) -- (v);
\end{tikzpicture}
$$

None of these is decisive on its own, and the chapter is candid that off-policy
variance "will probably always remain a challenge." Which of these methods combine
effectively with the stable Gradient-TD and emphatic-TD updates is still an open
question — the whole area is young. But the framing is what matters: stability and
variance are _separate_ problems. Gradient-TD and emphatic-TD solve the first;
these techniques address the second; and a usable off-policy method under function
approximation generally needs both.

## The triad in deep reinforcement learning

Sutton and Barto present the triad as a theoretical hazard demonstrated on small
linear problems. The obvious question — does it actually break the deep networks
that now dominate practice? — was answered empirically after the second edition, and
the answer is more nuanced than the theory alone suggests.

Van Hasselt, Doron, Strub, Hessel, Sonnerat, and Modayil (2018, "Deep Reinforcement
Learning and the Deadly Triad") ran a controlled study on the 57-game Atari suite,
varying the ingredients of the triad in a deep Q-network and measuring how often the
action-value estimates blew up.[^vh-triad] With rewards clipped to $[-1, 1]$ and
$\gamma = 0.99$, the largest possible true action value is about $1/(1-\gamma) = 100$;
they flagged any estimate that exceeded roughly that bound as **soft divergence**.
Their findings sharpen Sutton and Barto's picture on four points.

- **Unbounded divergence is rare in practice.** Soft divergence — estimates growing
  well past the reachable range — did occur, but the values essentially never ran off
  to infinity (floating-point NaN). The deep networks in current use apparently sit in
  a regime where the triad distorts the estimates without making them diverge, which is
  why DQN works at all despite carrying all three ingredients.
- **Longer multi-step returns diverge _less_.** Increasing the bootstrapping horizon
  (using $n$-step returns instead of one-step targets) reduced both the frequency of
  soft divergence and the number of unstable runs, and improved control performance.
  More of the target comes from real returns and less from the bootstrapped estimate,
  weakening the leg of the triad that drives instability.
- **Correcting overestimation helps.** Target networks and
  [double Q-learning](/reinforcement-learning/deep-rl/deep-q-networks) — which was
  designed to cut the maximization bias in the Q-learning target — were the most stable
  variants. Reducing overestimation and decoupling the bootstrap target from the
  weights being updated both damp the runaway feedback.
- **Bigger networks are not automatically safer.** The intuition that a
  higher-capacity approximator, closer to tabular, should be more stable did _not_ hold:
  larger networks showed _more_ soft divergence, even though they also reached the best
  final performance. Capacity and stability trade off rather than align.

These results reconcile the theory with practice. The triad's danger is real — van
Hasselt's own linear controls still diverge to infinity — but the specific
architecture of deep Q-learning (target networks, experience replay, clipped rewards,
multi-step returns) keeps the deep case in the soft-divergence regime where learning,
though sometimes unstable, does not explode.

On the theory side, two lines extend Sutton and Barto's stable methods. Sutton, Mahmood, and
White (2016, "An Emphatic Approach to the Problem of Off-policy Temporal-Difference
Learning," _JMLR_) gave the emphatic-TD method of §11.8 its formal footing: they proved
that emphasizing linear TD($\lambda$)'s updates by the emphasis weighting makes the
expected update **stable** under off-policy training, with a single learned weight
vector and a single step size — simpler than the two-time-scale Gradient-TD family,
at the cost of the higher variance the lesson already flagged.[^smw-emphatic] More
recently, Zhang, Yao, and Whiteson (2021, "Breaking the Deadly Triad with a Target
Network," _ICML_) proved that a **target network** — the same trick DQN uses
heuristically — combined with a projection or ratio clipping yields a genuinely
convergent off-policy prediction and control algorithm under linear approximation,
turning DQN's empirical stabilizer into a method with a convergence guarantee.[^zyw-target]
Together these results close much of the gap between Sutton and Barto's cautionary
counterexamples and the systems that run in production: the triad is still dangerous,
but there are now several principled ways to avoid its divergence.

## Where this leaves us

Off-policy learning with approximation is the point where the clean guarantees of
tabular RL give out. The first part of the challenge — correcting update targets with
importance sampling — is routine, if variance-prone. The second part — the instability of
bootstrapping under an off-policy update distribution — is the deadly triad, and it is
real enough to send a linearly-independent seven-state problem to infinity. Two responses
restore stability without abandoning any leg of the triad: Gradient-TD methods, which do
true SGD on the projected Bellman error at $O(d)$ cost, and emphatic-TD methods, which
reweight updates back toward the on-policy distribution. Both remain relatively young and
unsettled. The most consequential practical answer arrives later:
[deep Q-networks](/reinforcement-learning/deep-rl/deep-q-networks) keep function
approximation, bootstrapping, and off-policy replay all at once, and stabilize the triad
empirically with a target network and a replay buffer rather than by giving anything up.

[^sb-geom]: **Sutton & Barto**, §11.4 — Linear Value-function Geometry, and Figure 11.3: value functions as vectors, the $d$-dimensional representable subspace, the $\mu$-norm (11.11), the projection operator $\Pi$ (11.12) and its matrix form (11.14)–(11.16), the Bellman operator $B_\pi$ (11.20) with fixed point $v_\pi$ (11.21), the Bellman error vector (11.17)–(11.19), and the projected Bellman error $\overline{PBE}$ (11.22) with its TD fixed point $\mathbf{w}_{\text{TD}}$.
[^sb-be]: **Sutton & Barto**, §11.5 — Gradient Descent in the Bellman Error (the naive residual-gradient algorithm (11.23), the A-split and A-presplit examples showing $\overline{BE}$ minimization finds the wrong values), and §11.6 — The Bellman Error is Not Learnable: two MRPs generating identical data with different $\overline{BE}$-minimizers, so $\overline{BE}$ cannot be estimated from feature vectors, actions, and rewards alone, while $\overline{PBE}$ and $\overline{TDE}$ are learnable (Figure 11.4).
[^sb-gtd]: **Sutton & Barto**, §11.7 — Gradient-TD Methods: SGD on the $\overline{PBE}$, the matrix expansion (11.25)–(11.27), the product-of-expectations gradient and its double-sampling obstacle, the stored auxiliary vector $\mathbf{v}$ (11.28) learned by an importance-weighted LMS rule, the GTD2 and TDC / GTD(0) updates (11.29 and following), Figure 11.5 showing TDC drive the $\overline{PBE}$ to zero on Baird's counterexample, and the two-time-scale / cascade convergence argument.
[^sb-variance]: **Sutton & Barto**, §11.9 — Reducing Variance: off-policy learning as inherently higher-variance than on-policy ("you can't learn to drive by cooking dinner"); the raison d'être of generalizing to related-but-not-identical policies; the source of variance in products of policy ratios (mean 1 by (5.13), uncorrelated, but of high variance) multiplying the SGD step size and producing occasional very large steps; and the surveyed remedies — weighted importance sampling with its $O(d)$-approximation difficulty (Mahmood and Sutton, 2015), the tree-backup route to ratio-free off-policy learning (Munos et al., 2016; Mahmood, Yu and Sutton, 2017), momentum and Polyak–Ruppert averaging, adaptive per-component and importance-weight-aware step sizes (Jacobs, 1988; Karampatziakis and Langford, 2010), and constraining the target to the behavior via "recognizers" (Precup et al., 2006).
[^sb-etd]: **Sutton & Barto**, §11.8 — Emphatic-TD Methods: reweighting states to recover the on-policy distribution's stability under semi-gradient TD, the emphasis $M_t = \gamma \rho_{t-1} M_{t-1} + I_t$ and interest $I_t$, the one-step emphatic-TD update, Figure 11.6 showing convergence in expectation on Baird's counterexample, and the high-variance caveat that motivates the variance-reduction methods of §11.9.
[^vh-triad]: **H. van Hasselt, Y. Doron, F. Strub, M. Hessel, N. Sonnerat, and J. Modayil**, "Deep Reinforcement Learning and the Deadly Triad," arXiv:1812.02648 (2018). A controlled study on the 57-game Atari suite measuring how the triad's ingredients affect divergence of a deep Q-network: unbounded divergence is rare (soft divergence, above the reachable value bound $\approx 1/(1-\gamma)$, is common but values stay finite); longer multi-step returns reduce instability; target networks and double Q-learning are the most stable; and larger networks show more soft divergence though they perform best. Grounds the claim that DQN's architecture keeps the deep triad in a soft-divergence regime.
[^smw-emphatic]: **R. S. Sutton, A. R. Mahmood, and M. White**, "An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning," _Journal of Machine Learning Research_ 17(73):1–29 (2016). Proves that emphasizing linear TD($\lambda$)'s updates makes its expected update stable under off-policy training, with one learned weight vector and one step size — simpler than Gradient-TD; the formal basis for the emphatic-TD method of §11.8.
[^zyw-target]: **S. Zhang, H. Yao, and S. Whiteson**, "Breaking the Deadly Triad with a Target Network," _Proceedings of the 38th International Conference on Machine Learning_ (ICML 2021). Shows that a target network combined with a projection / ratio truncation gives a provably convergent off-policy prediction and control algorithm under linear function approximation, supplying a convergence guarantee for the target-network trick that DQN uses only heuristically.