---
title: "Sharpening DQN: Improvements and the Distributional Idea"
module: Modern Deep Reinforcement Learning
moduleNumber: 5
lessonNumber: 1
order: 501
summary: >
  In the years after the 2015 DQN paper, a stream of focused improvements each fixed
  one weakness of the baseline without disturbing its frame. This lesson recaps five
  that keep the scalar $Q$-value — Double DQN, multi-step returns, dueling networks,
  prioritized replay, and NoisyNets, each changing a different slot of the same
  Q-learning loop — then develops the sixth, distributional RL, which changes the
  objective itself: learn the whole return distribution $Z(s,a)$. We build the
  distributional Bellman equation and the C51 categorical algorithm, projection step
  and all, worked end to end on real numbers. A companion lesson takes up QR-DQN,
  Rainbow, and the modern distributional line.
topics: [Deep RL]
sources:
  - book: Grokking Deep RL
    ref: "Ch. 10 — Sample-efficient value-based methods (Double DQN, dueling, PER); Ch. 12 — Advanced actor-critic (context for distributional targets)"
  - book: Sutton & Barto
    ref: "§16.5 — Human-level Video Game Play (the DQN baseline these methods extend)"
---

[Deep Q-networks](/reinforcement-learning/deep-rl/deep-q-networks) made value-based
deep RL work: approximate $q_\ast$ with a neural network, train it by bootstrapped
semi-gradient regression, and stabilize the deadly triad with a replay buffer and a
frozen target network. In the five years after the 2015 Nature paper, a stream of
focused papers each fixed one weakness of that baseline without disturbing the
frame. Six of them proved to matter enough to keep, and one — **distributional
reinforcement learning** — turned out to be less a patch than a change of
objective: instead of predicting the _average_ return, predict the _whole
distribution_ of returns and act on its mean.

This lesson takes five of the six improvements — the ones that keep the scalar
$Q$-value and adjust the target, the network, the data, or the exploration — and then
the sixth and deepest, **distributional RL**, which changes what the value head
predicts. We stop after the C51 algorithm and its projection; the quantile methods,
the **Rainbow** agent that folds all six together, and the modern distributional line
continue in a
[companion lesson](/reinforcement-learning/modern-deep-rl/distributional-and-rainbow-part-2).[^rainbow]
The through-line is that every fix changes _one slot_ of the same Q-learning loop,
so they compose.

## Six ways to sharpen DQN

Think of the DQN agent as a machine with a few interchangeable slots: which data it
learns from, what its network looks like, what target it regresses toward, and how it
explores. Each of the six improvements swaps out one slot and leaves the rest alone.
Three of the six were introduced alongside the baseline
[in the DQN lesson](/reinforcement-learning/deep-rl/deep-q-networks) and are recapped
here only as a checklist; the other three are new. Each changes one slot of the
agent, and — this is the point Rainbow rests on — they change _different_ slots, so
they compose.

| Improvement | Slot it changes | One-line fix |
| --- | --- | --- |
| Double DQN | the target | select the greedy action with the online net, evaluate it with the target net, removing the max-overestimation bias |
| Dueling networks | the network | split into a state-value stream $V(s)$ and an advantage stream $A(s,a)$ |
| Prioritized replay | which data | sample transitions in proportion to their absolute TD error |
| Multi-step returns | the target | bootstrap after $n$ real rewards instead of one |
| Distributional (C51) | what is predicted | learn the return distribution $Z(s,a)$, not just its mean $Q(s,a)$ |
| NoisyNets | exploration | learnable noise on the weights replaces $\varepsilon$-greedy |

**Double DQN** (van Hasselt et al., 2016) decouples action _selection_ from action
_evaluation_ so that noise which inflates an action's estimate is not both chosen and
trusted; the target becomes $r + \gamma\, Q(s', \arg\max_{a'} Q(s',a';\mathbf{w}); \mathbf{w}^-)$.[^ddqn]
**Dueling networks** (Wang et al., 2016) restructure the head into a scalar
$V(s;\beta)$ and a per-action advantage $A(s,a;\alpha)$ recombined with the mean
advantage subtracted, so the state value is trained by every action's update.[^dueling]
**Prioritized experience replay** (Schaul et al., 2016) draws transition $i$ with
probability $P(i) \propto p_i^{\,\alpha}$, $p_i = |\delta_i| + \epsilon$, and corrects
the induced bias with importance-sampling weights $w_i = (N \cdot P(i))^{-\beta}$.[^per]
Those three were covered in full earlier. The remaining three deserve a paragraph
each.

### Multi-step returns

One-step Q-learning bootstraps immediately: the target uses one real reward $r$ and
then trusts the network for everything after. Waiting for $n$ real rewards before
bootstrapping trades a little variance for much less bias early in training, exactly
as in [$n$-step TD](/reinforcement-learning/tabular-methods/n-step-bootstrapping).
The **$n$-step target** replaces the single reward with a truncated $n$-step return:

$$
R_t^{(n)} \;=\; \sum_{k=0}^{n-1} \gamma^{k}\, r_{t+k},
\qquad
y_t \;=\; R_t^{(n)} + \gamma^{n} \max_{a'} Q(s_{t+n}, a'; \mathbf{w}^-).
$$

Reward information propagates $n$ times faster along a trajectory, so credit reaches
early states in far fewer updates. Rainbow uses $n = 3$. In principle a multi-step
target off-policy needs importance-sampling corrections; in practice, with short $n$
and a replay buffer of recent-ish data, the uncorrected version works well and is
what Rainbow ships.[^multistep]

For example, suppose a transition earns
rewards $r_t = 0$, $r_{t+1} = 0$, $r_{t+2} = 1$ before the agent bootstraps, with
$\gamma = 0.99$, and the current network badly misestimates the state at horizon $h$
as $Q(s_{t+h}, \cdot) = 5$ when the true continuation value is $2$. The one-step
target is $0 + 0.99 \cdot 5 = 4.95$, dominated by the wrong bootstrap: almost the
entire target is the network's own error. The three-step target is
$0 + 0.99 \cdot 0 + 0.99^2 \cdot 1 + 0.99^3 \cdot 5 = 0.980 + 4.851 = 5.831$, and the
true three-step value is $0.980 + 0.99^3 \cdot 2 = 0.980 + 1.940 = 2.920$. The
absolute bootstrap error has shrunk from $|0.99 \cdot (5-2)| = 2.97$ in the one-step
target to $|0.99^3 (5-2)| = 2.91$ — modest here, but the deeper point is that the
_two real rewards_ now anchor the target, so the network's mistake is diluted by
real reward signal. The cost is variance: a
three-step return sums three stochastic rewards, so its spread across episodes is
larger. Short $n$ (Rainbow's $3$) sits near the sweet spot for Atari — enough real
reward to cut early bias, not so many steps that variance swamps it.

$$
% caption: The n-step bias-variance trade. A one-step target (top) is almost all
% bootstrap, so a wrong network value passes straight through; an n-step target
% (bottom) anchors on n real rewards first, diluting the bootstrap error, at the
% price of summing more stochastic rewards (higher variance). Rainbow uses n=3.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  rw/.style={draw=acc, fill=acc!12, minimum width=10mm, minimum height=7mm, font=\scriptsize},
  bs/.style={draw=red, fill=red!10, minimum width=15mm, minimum height=7mm, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % one-step
  \node[anchor=east, font=\scriptsize] at (-0.2,1.3) {1-step:};
  \node[rw] (a0) at (0.9,1.3) {r};
  \node[bs] (a1) at (3.0,1.3) {g Q(s-next)};
  \node[anchor=west, font=\scriptsize, text=red] at (4.0,1.3) {mostly bootstrap};
  % three-step
  \node[anchor=east, font=\scriptsize] at (-0.2,-0.3) {3-step:};
  \node[rw] (b0) at (0.9,-0.3) {r};
  \node[rw] (b1) at (2.0,-0.3) {g r};
  \node[rw] (b2) at (3.3,-0.3) {gg r};
  \node[bs] (b3) at (5.2,-0.3) {ggg Q};
  \node[anchor=west, font=\scriptsize, text=acc] at (6.2,-0.3) {3 real rewards f\/irst};
\end{tikzpicture}
$$

### NoisyNets for exploration

$\varepsilon$-greedy explores by flipping a coin at every state and, with probability
$\varepsilon$, ignoring everything the network has learned. It is state-independent
and clumsy: on games needing a long, specific sequence of actions to reach the first
reward — Montezuma's Revenge is the standing example — uniform random jitter almost
never stumbles onto that sequence. **NoisyNets** (Fortunato et al., 2018) replace the
$\varepsilon$ schedule with _learnable_ noise injected into the network's linear
layers.[^noisy] A noisy linear layer replaces the ordinary weight $\mathbf{w}$ and
bias $\mathbf{b}$ with

$$
\mathbf{w} = \boldsymbol{\mu}_w + \boldsymbol{\sigma}_w \odot \boldsymbol{\varepsilon}_w,
\qquad
\mathbf{b} = \boldsymbol{\mu}_b + \boldsymbol{\sigma}_b \odot \boldsymbol{\varepsilon}_b,
$$

where $\boldsymbol{\mu}$ and $\boldsymbol{\sigma}$ are learned parameters and
$\boldsymbol{\varepsilon}$ is sampled noise resampled each forward pass. The
network _learns how much_ noise to add, and _where_: in states where exploration
still pays it can keep $\boldsymbol{\sigma}$ large, and in states it has mastered it
can drive $\boldsymbol{\sigma}$ toward zero, annealing its own exploration per state
rather than on a global clock. The extra parameters are trained by the ordinary
gradient — no new loss term — and at evaluation the noise is switched off.

For a layer with $p$ inputs and $q$ outputs, independent noise would need $pq$
samples of $\varepsilon_w$ per forward pass — expensive for the large linear layers
of a value network. Fortunato et al. use **factorized Gaussian noise** to cut that to
$p + q$: sample one vector $\varepsilon^{\text{in}} \in \mathbb{R}^p$ and one
$\varepsilon^{\text{out}} \in \mathbb{R}^q$, pass each through
$f(x) = \sgn(x)\sqrt{|x|}$, and set the weight noise to the outer
product $\varepsilon_{w,jk} = f(\varepsilon^{\text{out}}_j)\, f(\varepsilon^{\text{in}}_k)$
with bias noise $\varepsilon_{b,j} = f(\varepsilon^{\text{out}}_j)$. The rank-one
structure trades a little noise diversity for a large saving in sampling cost, and it
is what Rainbow ships. The initialization matters: $\boldsymbol{\sigma}$ starts near
$\sigma_0 / \sqrt{p}$ (with $\sigma_0 = 0.5$), so early training is noticeably
stochastic and $\boldsymbol{\sigma}$ shrinks only where the return gradient
indicates exploration no longer pays.

$$
% caption: A NoisyNet linear layer. Each weight is a learned mean plus a learned
% scale times fresh noise; the layer learns per-weight sigma, so exploration becomes
% state-dependent and self-annealing. Factorized noise samples one input vector and
% one output vector and forms their outer product, needing p+q samples not p*q.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  cell/.style={draw, minimum width=11mm, minimum height=7mm, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[cell] (mu) at (0,0) {mean mu};
  \node[font=\scriptsize] at (1.5,0) {+};
  \node[cell, draw=acc, text=acc] (sig) at (3.0,0) {scale sig};
  \node[font=\scriptsize] at (4.4,0) {x};
  \node[cell, draw=red, text=red] (eps) at (5.9,0) {noise eps};
  \node[font=\scriptsize] at (7.3,0) {=};
  \node[cell, thick] (w) at (8.7,0) {weight w};
  \node[anchor=north, font=\scriptsize, text=black] at (0,-0.55) {learned};
  \node[anchor=north, font=\scriptsize, text=acc] at (3.0,-0.55) {learned};
  \node[anchor=north, font=\scriptsize, text=red] at (5.9,-0.55) {resampled each pass};
\end{tikzpicture}
$$


## The distributional idea

The other five improvements adjust the target, the network, the data, or the
exploration; distributional RL changes what the value head _predicts_. It is the
central idea of this lesson, so we state the shift precisely before any algorithm.

The return from a state-action pair,

$$
Z(s,a) \;=\; \sum_{t=0}^{\infty} \gamma^{t}\, r_t
\quad\text{starting from } (s,a) \text{ and following } \pi,
$$

is a **random variable**. Reward is stochastic, transitions are stochastic, and the
policy may be stochastic, so the same $(s,a)$ produces a spread of returns across
episodes. The action-value function throws that spread away and keeps only its mean:

$$
Q(s,a) \;=\; \mathbb{E}\bigl[Z(s,a)\bigr].
$$

Distributional RL keeps the whole random variable $Z(s,a)$ and learns its
distribution. A state where an action is a coin flip between a big win and a big loss
and a state where it reliably yields the average of the two can share the same
$Q$-value while having very different return distributions; the distributional
agent distinguishes them.[^c51]

$$
% caption: Two actions with the same expected return $Q(s,a)=\mathbb{E}[Z(s,a)]$ but
% very different return distributions $Z(s,a)$. The value function (dashed line) sees
% only the shared mean; the distributional view sees the bimodal gamble on the left
% and the tight single mode on the right.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % --- left: bimodal ---
  \draw[black, ->] (-0.2,0) -- (4.4,0) node[anchor=north east, text=black] {return};
  \draw[black, ->] (0,0) -- (0,3.0) node[anchor=south east, text=black] {prob.};
  \foreach \x/\h in {0.4/0.4, 0.8/1.1, 1.2/2.1, 1.6/1.2, 2.0/0.5, 2.4/0.5, 2.8/1.2, 3.2/2.0, 3.6/1.0}
    \draw[acc, line width=3pt] (\x,0) -- (\x,\h);
  \draw[red, dashed, thick] (2.0,0) -- (2.0,2.6);
  \node[red, anchor=south, font=\scriptsize] at (2.0,2.6) {mean Q};
  \node[anchor=south, font=\scriptsize] at (2.0,-0.9) {bimodal gamble};
  % --- right: unimodal, same mean ---
  \begin{scope}[xshift=6.0cm]
    \draw[black, ->] (-0.2,0) -- (4.4,0) node[anchor=north east, text=black] {return};
    \draw[black, ->] (0,0) -- (0,3.0) node[anchor=south east, text=black] {prob.};
    \foreach \x/\h in {1.2/0.5, 1.6/1.4, 2.0/2.6, 2.4/1.4, 2.8/0.5}
      \draw[acc, line width=3pt] (\x,0) -- (\x,\h);
    \draw[red, dashed, thick] (2.0,0) -- (2.0,2.85);
    \node[red, anchor=south, font=\scriptsize] at (2.0,2.85) {mean Q};
    \node[anchor=south, font=\scriptsize] at (2.0,-0.9) {tight single mode};
  \end{scope}
\end{tikzpicture}
$$

### The distributional Bellman equation

The ordinary Bellman equation relates expected values: $Q(s,a) = \mathbb{E}[r + \gamma
Q(s',a')]$. The distributional version relates the random variables themselves,
before any expectation is taken:

$$
Z(s,a) \;\overset{D}{=}\; r + \gamma\, Z(s', a'),
$$

where $\overset{D}{=}$ means "equal in distribution" and $s', a'$ are drawn from the
transition and the policy. Read the right-hand side as an operation _on
distributions_: sampling a next state-action pair $(s',a')$ selects one of the
distributions $Z(s',a')$; multiplying by $\gamma$ **scales** it toward zero; adding
the reward $r$ **shifts** it along the return axis; and mixing over the stochastic
$(r, s', a')$ **superposes** the results into a new distribution. That composite map
is the **distributional Bellman operator** $\mathcal{T}^\pi$, and its fixed point is
$Z^\pi$.[^c51] Taking expectations of both sides collapses it back to the familiar
scalar Bellman equation, which is why any solution to the distributional problem also
solves the ordinary one — distributional RL loses nothing and carries strictly more
information.

$$
% caption: The distributional Bellman operator acting on a return distribution:
% discounting by $\gamma$ scales the distribution toward the origin, adding the reward
% $r$ shifts it right, and mixing over next-states superposes the pieces. The result
% is the new distribution $Z(s,a)$ for the current pair.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % --- source distribution Z(s', a') ---
  \draw[black, ->] (0,0) -- (3.4,0);
  \draw[black, ->] (0,0) -- (0,2.4);
  \foreach \x/\h in {1.4/0.7, 1.8/1.4, 2.2/2.0, 2.6/1.3, 3.0/0.6}
    \draw[acc, line width=2.6pt] (\x,0) -- (\x,\h);
  \node[anchor=north, font=\scriptsize] at (1.7,-0.15) {Z(s-next, a-next)};
  % scale arrow
  \draw[->, black, thick] (3.7,1.0) -- (4.6,1.0);
  \node[font=\scriptsize, anchor=south] at (4.15,1.05) {scale by g};
  % --- scaled (compressed toward 0) ---
  \begin{scope}[xshift=4.9cm]
    \draw[black, ->] (0,0) -- (3.0,0);
    \draw[black, ->] (0,0) -- (0,2.4);
    \foreach \x/\h in {0.7/0.7, 0.9/1.4, 1.1/2.0, 1.3/1.3, 1.5/0.6}
      \draw[acc, line width=2.6pt] (\x,0) -- (\x,\h);
    \node[anchor=north, font=\scriptsize] at (1.1,-0.15) {g Z};
  \end{scope}
  % shift arrow
  \draw[->, black, thick] (8.3,1.0) -- (9.2,1.0);
  \node[font=\scriptsize, anchor=south] at (8.75,1.05) {shift by r};
  % --- shifted (moved right) ---
  \begin{scope}[xshift=9.5cm]
    \draw[black, ->] (0,0) -- (3.4,0) node[anchor=north east, text=black] {return};
    \draw[black, ->] (0,0) -- (0,2.4);
    \foreach \x/\h in {1.7/0.7, 1.9/1.4, 2.1/2.0, 2.3/1.3, 2.5/0.6}
      \draw[red, line width=2.6pt] (\x,0) -- (\x,\h);
    \node[red, anchor=north, font=\scriptsize] at (1.55,-0.15) {Z(s, a)};
  \end{scope}
\end{tikzpicture}
$$

### C51: a categorical distribution on fixed atoms

To learn $Z$ with a network, you must choose how to represent a distribution.
**C51** (Bellemare, Dabney & Munos, 2017) takes the most direct route: fix a finite
grid of $N$ possible return values — the **atoms** — and let the network output a
probability for each.[^c51] The atoms are spaced evenly on $[V_{\min}, V_{\max}]$,

$$
z_i \;=\; V_{\min} + (i-1)\,\Delta z,
\qquad
\Delta z = \frac{V_{\max} - V_{\min}}{N - 1},
\qquad i = 1, \dots, N,
$$

and for each state-action pair the network emits a softmax over the atoms, giving
probabilities $p_i(s,a)$ that sum to one. The approximate return distribution is the
categorical

$$
Z_\theta(s,a) \;=\; \sum_{i=1}^{N} p_i(s,a)\, \delta_{z_i},
\qquad
Q(s,a) = \sum_{i=1}^{N} z_i\, p_i(s,a),
$$

with $\delta_{z_i}$ a point mass at atom $z_i$. The mean $Q$ used to act greedily is
just the atom values weighted by their probabilities. The "51" is the atom count the
paper found best; $[V_{\min}, V_{\max}] = [-10, 10]$ was standard for Atari with
reward clipping.

> **Definition (Categorical value distribution).** A return distribution represented
> by $N$ fixed atoms $z_1 < \dots < z_N$ spanning $[V_{\min}, V_{\max}]$ and a
> learned probability vector $\mathbf{p}(s,a) = (p_1, \dots, p_N)$ over them. The
> support is fixed; only the probabilities are learned. The action value is the mean
> $Q(s,a) = \sum_i z_i\, p_i(s,a)$.

### The projection step

Fixing the atoms creates one difficulty, and resolving it is the whole technical
content of C51. Apply the distributional Bellman operator to a categorical
distribution and the atoms move: each atom $z_j$ of the next-state distribution maps
to $r + \gamma z_j$, and in general **that value is not one of the atoms**
$\{z_1, \dots, z_N\}$. The scaled-and-shifted distribution lives on a displaced grid
that no longer lines up with the fixed support the network can represent.

The fix is a **projection** $\Phi$ that redistributes each shifted atom's probability
mass onto the two nearest fixed atoms, splitting it in inverse proportion to distance
(and clipping to $[V_{\min}, V_{\max}]$ at the ends). Formally, the mass that the
Bellman update places at $\hat{T}z_j = r + \gamma z_j$ is spread onto atom $z_i$ by
the amount

$$
\bigl(\Phi \hat{\mathcal{T}} Z\bigr)_i \;=\; \sum_{j=1}^{N}
\left[ 1 - \frac{\bigl|[\hat{T}z_j]_{V_{\min}}^{V_{\max}} - z_i\bigr|}{\Delta z} \right]_0^1
p_j(s', a^\ast),
$$

where $[\,\cdot\,]_a^b$ clips to $[a,b]$ and $a^\ast$ is the greedy next action. The
inner bracket is the linear "how close is the shifted atom to $z_i$" weight — one when
they coincide, zero a full $\Delta z$ away — so each shifted atom's mass lands
entirely within the grid. The result is a valid categorical distribution on the fixed
atoms, and it becomes the regression target.

$$
% caption: The C51 projection. A next-state atom's mass at $z_j$ is scaled and shifted
% to $\hat{T}z_j = r + \gamma z_j$, which falls between two fixed atoms; the projection
% $\Phi$ splits that mass onto the two neighbors in inverse proportion to distance, so
% the target lives on the same grid the network predicts.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % fixed atom grid (baseline)
  \draw[black] (0,0) -- (8.4,0);
  \foreach \i/\x in {1/0.6, 2/1.8, 3/3.0, 4/4.2, 5/5.4, 6/6.6, 7/7.8}
    { \draw[black] (\x,-0.08) -- (\x,0.08);
      \node[black, anchor=north, font=\scriptsize] at (\x,-0.12) {z\i}; }
  % the shifted atom (lands between z4 and z5)
  \coordinate (shift) at (4.75,0);
  \draw[red, line width=2.6pt] (4.75,0) -- (4.75,2.1);
  \node[red, anchor=south, font=\scriptsize] at (4.75,2.1) {shifted mass at r + g zj};
  % split arrows to the two neighbors
  \draw[->, acc, thick] (4.6,1.4) to[bend right=18] (4.28,0.12);
  \draw[->, acc, thick] (4.9,1.4) to[bend left=18] (5.32,0.12);
  \draw[acc, line width=2.6pt] (4.2,0) -- (4.2,1.25);
  \draw[acc, line width=2.6pt] (5.4,0) -- (5.4,0.65);
  \node[acc, anchor=east, font=\scriptsize] at (4.12,0.9) {closer: more mass};
  \node[acc, anchor=west, font=\scriptsize] at (5.48,0.5) {farther: less};
\end{tikzpicture}
$$

C51 then minimizes the cross-entropy between the network's predicted categorical
$Z_\theta(s,a)$ and the projected target $\Phi \hat{\mathcal{T}} Z_{\theta^-}$ — a
classification-style loss over atoms, not a squared TD error. Everything else is
DQN: replay buffer, target network $\theta^-$, $\varepsilon$-greedy (or NoisyNet)
behavior, and greedy action selection by the mean $Q$. On the Atari suite C51 beat
every prior single-improvement variant, and it did so while still acting on the mean —
the richer target simply produces a better-shaped mean.[^c51]

```algorithm
caption: $\textsc{Categorical-Update}$ (C51) — one gradient step on a sampled transition
input: transition $(s, a, r, s')$, atoms $z_1, \ldots, z_N$, target weights $\theta^-$
$a^{\ast} \gets \arg\max_{a'} \sum_i z_i\, p_i(s', a'; \theta^{-})$ // greedy next action by mean
$m_i \gets 0$ for $i = 1, \ldots, N$ // projected target, per atom
for $j = 1, \ldots, N$ do
  $\hat{T} z_j \gets [\,r + \gamma z_j\,]_{V_{\min}}^{V_{\max}}$ // scale, shift, clip
  $b \gets (\hat{T} z_j - V_{\min}) / \Delta z$ // fractional atom index
  $l \gets \lfloor b \rfloor$
  $u \gets \lceil b \rceil$
  $m_l \gets m_l + p_j(s', a^{\ast}; \theta^{-})\,(u - b)$ // mass to lower neighbor
  $m_u \gets m_u + p_j(s', a^{\ast}; \theta^{-})\,(b - l)$ // mass to upper neighbor
take a gradient step on the cross-entropy $-\sum_i m_i \log p_i(s, a; \theta)$
```

### A projection worked end to end

For example, take a small grid to keep the arithmetic legible: $N = 5$ atoms on $[V_{\min}, V_{\max}] = [0, 4]$,
so $\Delta z = (4 - 0)/(5 - 1) = 1$ and the atoms are $z = (0, 1, 2, 3, 4)$. Suppose
the greedy next-state distribution puts its mass on just two atoms,
$p(s', a^\ast) = (0,\, 0,\, 0.6,\, 0.4,\, 0)$ — probability $0.6$ at $z_3 = 2$ and $0.4$
at $z_4 = 3$. Let the reward be $r = 0.5$ and the discount $\gamma = 0.9$.

Each atom is scaled and shifted by $\hat T z_j = r + \gamma z_j$, then clipped to
$[0, 4]$:

$$
\hat T z_3 = 0.5 + 0.9 \cdot 2 = 2.3,
\qquad
\hat T z_4 = 0.5 + 0.9 \cdot 3 = 3.2.
$$

Neither $2.3$ nor $3.2$ is a grid atom, so each must be split. The fractional index of
a shifted value is $b = (\hat T z_j - V_{\min})/\Delta z$; here $b_3 = 2.3$ and
$b_4 = 3.2$. For the first, $l = \lfloor 2.3 \rfloor = 2$ and
$u = \lceil 2.3 \rceil = 3$ (atoms $z_2 = 2$ and $z_3 = 3$ in _index_ terms — indices
$2$ and $3$ in the zero-based atom array). The mass $0.6$ splits by distance:
$(u - b) = 0.7$ to the lower neighbor and $(b - l) = 0.3$ to the upper. So atom index
$2$ (value $2$) receives $0.6 \cdot 0.7 = 0.42$ and atom index $3$ (value $3$) receives
$0.6 \cdot 0.3 = 0.18$. For the second, $b_4 = 3.2$ gives $l = 3$, $u = 4$, splitting
$0.4$ into $0.4 \cdot 0.8 = 0.32$ at atom index $3$ and $0.4 \cdot 0.2 = 0.08$ at atom
index $4$. Collecting per atom:

$$
m = (\,0,\ 0,\ 0.42,\ 0.18 + 0.32,\ 0.08\,) = (0,\ 0,\ 0.42,\ 0.50,\ 0.08).
$$

The masses sum to $1.00$, as they must, and this vector $m$ — a valid categorical on
the fixed grid — is the cross-entropy target for the current pair. Its mean is
$0 \cdot 0 + 1 \cdot 0 + 2 \cdot 0.42 + 3 \cdot 0.50 + 4 \cdot 0.08 = 0.84 + 1.50 +
0.32 = 2.66$, which equals $r + \gamma \mathbb{E}[Z(s',a^*)] = 0.5 + 0.9 \cdot (2
\cdot 0.6 + 3 \cdot 0.4) = 0.5 + 0.9 \cdot 2.4 = 2.66$ exactly. The projection moves
mass around but preserves the mean — the scalar Bellman backup is a by-product of the
distributional one.

$$
% caption: The worked projection. Two source atoms at z=2 (mass 0.6) and z=3 (mass
% 0.4) are scaled-shifted to 2.3 and 3.2, each landing between grid atoms; the mass
% splits by inverse distance onto neighbors, giving the target (0, 0, 0.42, 0.50,
% 0.08) whose mean 2.66 matches the scalar Bellman backup.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % grid baseline
  \draw[black] (0,0) -- (8.6,0);
  \foreach \i/\x/\v in {0/0/0, 1/2.0/1, 2/4.0/2, 3/6.0/3, 4/8.0/4}
    { \draw[black] (\x,-0.08) -- (\x,0.08);
      \node[black, anchor=north, font=\scriptsize] at (\x,-0.14) {\v}; }
  % shifted values 2.3 and 3.2 (x = value*2)
  \draw[red, dashed] (4.6,0) -- (4.6,2.0);
  \node[red, anchor=south, font=\scriptsize] at (4.6,2.0) {2.3};
  \draw[red, dashed] (6.4,0) -- (6.4,0.9);
  \node[red, anchor=west, font=\scriptsize] at (6.5,0.7) {3.2};
  % projected bars
  \draw[acc, line width=4pt] (4.0,0) -- (4.0,1.26);  % 0.42
  \draw[acc, line width=4pt] (6.0,0) -- (6.0,1.50);  % 0.50
  \draw[acc, line width=4pt] (8.0,0) -- (8.0,0.24);  % 0.08
  \node[acc, anchor=south, font=\scriptsize] at (4.0,1.26) {0.42};
  \node[acc, anchor=south, font=\scriptsize] at (6.0,1.50) {0.50};
  \node[acc, anchor=south, font=\scriptsize] at (8.0,0.24) {0.08};
  \node[anchor=north, font=\scriptsize, text=black] at (4.3,-0.7) {return atom value};
\end{tikzpicture}
$$


## Where this leaves us

Two threads are now in hand. Five improvements — Double DQN, multi-step returns,
dueling networks, prioritized replay, and NoisyNets — each sharpen one slot of the
Q-learning loop without touching the others, which is what lets them stack. And
distributional RL reframes the objective: predict the whole return distribution
$Z(s,a)$, act on its mean $Q = \mathbb{E}[Z]$, and get a richer, better-shaped
estimate for free. C51 realizes that idea with a fixed grid of atoms and a projection
that snaps each Bellman-updated distribution back onto the grid.

C51 fixes the return values and learns their probabilities. The natural complement —
fix the probabilities and learn the values — gives **QR-DQN**, which drops the
projection entirely. That method, the reasons the distribution helps even when you act
on the mean, and the **Rainbow** agent that assembles all six improvements at once
(with its component ablation) continue in
[Distributional RL and Rainbow](/reinforcement-learning/modern-deep-rl/distributional-and-rainbow-part-2).

[^rainbow]: **Hessel et al. (2018)**, "Rainbow: Combining Improvements in Deep Reinforcement Learning", _AAAI_ — the integrated agent combining Double DQN, dueling networks, prioritized replay, multi-step returns, distributional (C51) learning, and NoisyNets; new state of the art on the 57-game Atari benchmark with markedly better sample efficiency; and the component ablation showing prioritized replay and multi-step returns most important, the distributional head growing in importance over training, and Double DQN largely redundant given the distributional target.
[^ddqn]: **van Hasselt, Guez & Silver (2016)**, "Deep Reinforcement Learning with Double Q-learning", _AAAI_ — decoupling action selection (online weights) from evaluation (target weights) to correct the maximization overestimation bias inherited from tabular Q-learning; target $r + \gamma\, Q(s', \arg\max_{a'} Q(s',a';\mathbf{w}); \mathbf{w}^-)$.
[^dueling]: **Wang et al. (2016)**, "Dueling Network Architectures for Deep Reinforcement Learning", _ICML_ — the two-stream network with a scalar state-value $V(s)$ and a per-action advantage $A(s,a)$ recombined with the mean advantage subtracted for identifiability, giving lower-variance, more sample-efficient learning of $V$.
[^per]: **Schaul et al. (2016)**, "Prioritized Experience Replay", _ICLR_ — sampling transitions with probability $P(i) \propto p_i^\alpha$, priority $p_i = |\delta_i| + \epsilon$ the absolute TD error, and importance-sampling weights $w_i = (N\,P(i))^{-\beta}$ (with $\beta$ annealed to 1) correcting the bias from non-uniform sampling.
[^multistep]: **Sutton & Barto**, _Reinforcement Learning: An Introduction_ (2nd ed.), Ch. 7 — $n$-step bootstrapping and the $n$-step return $R_t^{(n)} = \sum_{k=0}^{n-1}\gamma^k r_{t+k} + \gamma^n \max_{a'} Q(s_{t+n},a')$; Rainbow (Hessel et al., 2018) adopts the uncorrected multi-step target with $n=3$.
[^noisy]: **Fortunato et al. (2018)**, "Noisy Networks for Exploration", _ICLR_ — replacing $\varepsilon$-greedy with learnable parametric noise $\mathbf{w} = \boldsymbol{\mu}_w + \boldsymbol{\sigma}_w \odot \boldsymbol{\varepsilon}_w$ on the network's linear layers, so the agent learns a state-dependent amount of exploration trained by the ordinary gradient.
[^c51]: **Bellemare, Dabney & Munos (2017)**, "A Distributional Perspective on Reinforcement Learning", _ICML_ — the return distribution $Z(s,a)$ with $Q = \mathbb{E}[Z]$, the distributional Bellman equation $Z(s,a) \overset{D}{=} r + \gamma Z(s',a')$, and the C51 algorithm: $N=51$ fixed atoms on $[V_{\min},V_{\max}]=[-10,10]$, a softmax over atoms, the categorical projection $\Phi$ of the scaled-shifted target onto the fixed grid, and a cross-entropy loss; state of the art on Atari while acting on the mean.