---
title: "Distributional RL and Rainbow"
module: Modern Deep Reinforcement Learning
moduleNumber: 5
lessonNumber: 2
order: 502
summary: >
  A companion to the DQN improvements lesson. C51 fixed the return atoms and learned
  their probabilities; QR-DQN does the reverse — fix the probabilities, learn the
  values — which removes the projection and trains with a quantile loss. We cover why
  the distribution helps even when you act on the mean, then assemble Rainbow: all six
  improvements in one Q-learning loop, with the component ablation that shows each
  one's real weight. The distributional line then runs on through IQN, FQF, and
  Agent57, the first agent to beat the human baseline on all 57 Atari games.
topics: [Deep RL]
sources:
  - book: Grokking Deep RL
    ref: "Ch. 10 — Sample-efficient value-based methods; Ch. 12 — Advanced actor-critic (context for distributional targets)"
  - book: Sutton & Barto
    ref: "§16.5 — Human-level Video Game Play (the DQN baseline these methods extend)"
---

This builds on
[Sharpening DQN: Improvements and the Distributional Idea](/reinforcement-learning/modern-deep-rl/distributional-and-rainbow),
which recapped the five improvements that keep the scalar $Q$-value and then developed
the sixth — **distributional RL** — through the distributional Bellman equation and
the **C51** algorithm. Recall the shift: the return $Z(s,a)$ is a random variable,
$Q(s,a) = \mathbb{E}[Z(s,a)]$ is only its mean, and C51 learns the whole distribution
by fixing a grid of return atoms $z_1 < \dots < z_N$ and learning a probability for
each, snapping every Bellman-updated distribution back onto the grid with a projection
$\Phi$.

This lesson picks up the complement to C51, folds all six improvements into one
agent — **Rainbow** — and follows the distributional line to its modern endpoint.

## Quantile methods: QR-DQN

C51 fixes a set of return values and learns how much probability sits on each. The
equally natural dual is to fix the probabilities and learn _where_ the return values
sit. That swap is QR-DQN, and it removes the projection step.

### QR-DQN: learn the quantiles instead

C51 fixes the return values and learns their probabilities. **QR-DQN** (Dabney et
al., 2018) does the reverse: fix the probabilities and learn the return
values.[^qr] It represents $Z(s,a)$ by $N$ quantiles — the network outputs $N$
values $\theta_1(s,a), \dots, \theta_N(s,a)$, one for each of the fixed cumulative
probabilities $\tau_i = \tfrac{2i-1}{2N}$ (the midpoints of $N$ equal-probability
bins). The distribution is a mixture of point masses at the learned locations,

$$
Z_\theta(s,a) \;=\; \frac{1}{N} \sum_{i=1}^{N} \delta_{\theta_i(s,a)},
\qquad
Q(s,a) = \frac{1}{N} \sum_{i=1}^{N} \theta_i(s,a).
$$

Swapping which axis is fixed removes the projection entirely: because the support
is _learned_, the Bellman-updated atoms need not be snapped back onto a grid. In its
place QR-DQN minimizes the **quantile (Huber) regression loss**, which trains each
$\theta_i$ toward the $\tau_i$-quantile of the target return distribution. The trade
is clean: C51 carries a fixed support and a projection but a simple cross-entropy;
QR-DQN drops the support constraint and the projection but pays with an asymmetric
quantile loss. Both learn a distribution; they differ only in which coordinate of the
histogram they let the network move.

The asymmetry is what makes a regression target a _quantile_ estimate. Ordinary
squared error, minimized, returns the mean; the **pinball loss** at level $\tau$
weights over- and under-shoots differently,

$$
\rho_\tau(u) = u\,\bigl(\tau - \mathbb{1}[u < 0]\bigr),
\qquad u = \hat T\theta_j - \theta_i,
$$

and its minimizer is the $\tau$-quantile of the target. Here $u = \hat T\theta_j -
\theta_i$ is the target minus the estimate, so $u > 0$ means the estimate sat _below_
the target (an under-estimate). If $u > 0$ the loss is $\tau\, u$; if $u < 0$ it is
$(\tau - 1)\,u = (1-\tau)\,|u|$. For $\tau = 0.9$ an under-estimate ($u>0$) is
penalized $9\times$ as hard as an over-estimate, so the estimate is dragged upward
until only $10\%$ of the target mass remains above it — the definition of the
$0.9$-quantile. QR-DQN replaces the kink at $u = 0$ with a Huber softening (the
_quantile Huber loss_) for a smoother gradient near zero. For example, to estimate
the median ($\tau = 0.5$) of target samples $\{1, 2, 6\}$, the pinball loss weights
overshoot and undershoot equally, so it is minimized at the middle value $2$ — the
sample median — whereas squared error would return the mean $3$, pulled up by the
outlier. Fixing $\tau = 0.5$ recovers the median; sweeping $\tau_i = (2i-1)/2N$ across
the $N$ heads recovers the whole distribution.

$$
% caption: The pinball (quantile) loss at level tau=0.9, with u = target minus
% estimate. Under-estimates (u>0) are weighted by tau=0.9 and cost steeply;
% over-estimates (u<0) by 1-tau=0.1 and cost gently, so the minimizer rises to the
% 0.9-quantile. Squared error (dashed) is symmetric and returns the mean instead.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[black, ->] (-2.9,0) -- (2.8,0) node[anchor=north east, text=black] {error u};
  \draw[black, ->] (0,-0.2) -- (0,2.7) node[anchor=south west, text=black] {loss};
  % left branch (u<0) gentle slope 0.1; right branch (u>0) steep slope 0.9
  \draw[acc, very thick] (-2.4,0.24) -- (0,0) -- (2.4,2.16);
  \node[acc, anchor=south, font=\scriptsize] at (1.6,2.2) {pinball tau=0.9};
  \node[acc, anchor=east, font=\scriptsize] at (-1.6,0.62) {gentle side: over-estimate};
  \node[acc, anchor=north, font=\scriptsize] at (1.6,-0.35) {steep side: under-estimate};
  % squared error dashed
  \draw[red, dashed, thick] plot[domain=-1.6:1.6, samples=40] (\x, {0.55*\x*\x});
  \node[red, anchor=south, font=\scriptsize] at (-1.65,1.3) {squared (mean)};
\end{tikzpicture}
$$

$$
% caption: Two ways to represent the same return distribution. C51 (left) fixes the
% return atoms on the horizontal axis and learns the bar heights (probabilities);
% QR-DQN (right) fixes the probabilities (equal-height bars) and learns the atom
% positions along the return axis.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % --- C51: fixed x, learned heights ---
  \draw[black, ->] (0,0) -- (4.0,0) node[anchor=north east, text=black] {return};
  \draw[black, ->] (0,0) -- (0,2.7) node[anchor=south east, text=black] {prob.};
  \foreach \x/\h in {0.6/0.5, 1.2/1.2, 1.8/2.2, 2.4/1.5, 3.0/0.7, 3.6/0.3}
    \draw[acc, line width=3pt] (\x,0) -- (\x,\h);
  \node[anchor=south, font=\scriptsize] at (1.9,-0.95) {C51: f\/ixed atoms, learned heights};
  % --- QR-DQN: fixed heights, learned x ---
  \begin{scope}[xshift=6.2cm]
    \draw[black, ->] (0,0) -- (4.0,0) node[anchor=north east, text=black] {return};
    \draw[black, ->] (0,0) -- (0,2.7) node[anchor=south east, text=black] {prob.};
    \foreach \x in {0.5, 1.1, 1.5, 1.8, 2.5, 3.4}
      \draw[red, line width=3pt] (\x,0) -- (\x,1.5);
    \node[anchor=south, font=\scriptsize] at (1.9,-0.95) {QR-DQN: f\/ixed heights, learned atoms};
  \end{scope}
\end{tikzpicture}
$$

### Why the distribution helps even when you act on the mean

C51 and QR-DQN still select actions by $\arg\max_a Q(s,a) = \arg\max_a
\mathbb{E}[Z(s,a)]$ — the extra information is thrown away at decision time. So why
does modeling the full distribution improve the resulting policy? Three reasons the
papers give.[^c51] First, the distributional target is a **richer, denser
learning signal**: predicting fifty-one probabilities (or fifty-one quantiles) per
action shapes the representation with far more constraints than a single scalar, and
a better representation yields a better-estimated mean. Second, it **interacts well
with function approximation**: a categorical target reframes the regression as a
classification over atoms, whose cross-entropy loss is better-behaved for a neural
network than chasing a moving scalar, and it keeps predictions bounded to
$[V_{\min}, V_{\max}]$. Third, learning the distribution appears to **reduce the
chattering and instability** that plague approximate value iteration, giving more
stable updates. The mean you act on is simply estimated better when it is estimated
as one summary of a distribution the network was forced to get right.

## Rainbow: all six at once

Each improvement was published on its own, measured against the DQN baseline. The
natural question — do they add up, or do they overlap and cancel? — is what
**Rainbow** (Hessel et al., 2018) answered by integrating all six into one agent
and measuring the whole.[^rainbow] The integration is mostly a matter of choosing
compatible pieces:

- The value head is **C51's distributional output**, and the network uses the
  **dueling** decomposition — but applied to the atom logits: a shared state-value
  stream and a per-action advantage stream, combined before the softmax.
- The target is the **multi-step** ($n=3$) **distributional** target, run through
  **Double**-DQN action selection: the online network picks $a^\ast$, the target
  network supplies its distribution.
- Transitions are drawn by **prioritized replay**, with the priority set to the
  **distributional loss** (the KL / cross-entropy) rather than the absolute TD error,
  since there is no longer a scalar TD error to take the absolute value of.
- Exploration is **NoisyNets** throughout, so the $\varepsilon$ schedule is dropped
  entirely.

$$
% caption: Rainbow as one Q-learning loop with each of the six improvements dropped
% into a different slot. Prioritized replay picks the data; the multi-step Double-DQN
% distributional target sets the regression target; the dueling distributional head
% is the network; NoisyNets replaces the exploration. Because the slots are distinct,
% the six compose without conflict.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  slot/.style={draw, minimum width=30mm, minimum height=11mm, align=center, font=\scriptsize},
  imp/.style={draw=acc, text=acc, minimum width=30mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[slot] (data) at (0,0) {slot: which data};
  \node[slot] (net) at (0,-1.9) {slot: the network};
  \node[slot] (tgt) at (0,-3.8) {slot: the target};
  \node[slot] (exp) at (0,-5.7) {slot: exploration};
  \node[imp] (idata) at (5.4,0) {prioritized replay};
  \node[imp] (inet) at (5.4,-1.9) {dueling + distributional};
  \node[imp] (itgt) at (5.4,-3.8) {multi-step Double-DQN};
  \node[imp] (iexp) at (5.4,-5.7) {NoisyNets};
  \draw[->, acc, thick] (data) -- (idata);
  \draw[->, acc, thick] (net) -- (inet);
  \draw[->, acc, thick] (tgt) -- (itgt);
  \draw[->, acc, thick] (exp) -- (iexp);
\end{tikzpicture}
$$

Nothing here is a new algorithm; it is the same Q-learning skeleton — act, store,
sample, form a bootstrapped target, regress toward it, periodically sync the target
network — with a distributional head, a multi-step Double-DQN target, prioritized
sampling, and learned exploration noise all slotted in at once. On the 57-game Atari
benchmark Rainbow set a new state of the art in both final score and
_sample efficiency_: it reached the previous best DQN's performance in a fraction of
the frames.

### The ablation

The paper's most cited figure is the **ablation study**: retrain Rainbow six times,
each run with one component removed, and compare against the full agent. A component
matters in proportion to how much removing it hurts. Some of the results were
unexpected.[^rainbow]

$$
% caption: Rainbow ablation (schematic, following Hessel et al. 2018). Each curve is
% the full agent with one component removed; a curve that falls well below the full
% Rainbow marks a component that contributes strongly. Prioritized replay and the
% multi-step target hurt most when removed; removing Double DQN barely moved the
% distributional agent.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[black, ->] (0,0) -- (8.6,0) node[anchor=north east, text=black] {training frames};
  \draw[black, ->] (0,0) -- (0,4.4) node[anchor=south east, text=black] {median human-normalized score};
  % full Rainbow (top)
  \draw[acc, very thick] (0,0.2) .. controls (2.2,2.9) and (4.8,3.9) .. (7.2,4.15);
  \node[acc, anchor=west, font=\scriptsize] at (7.25,4.15) {full Rainbow};
  % no double (essentially on top of full)
  \draw[black, thick] (0,0.2) .. controls (2.2,2.8) and (4.8,3.8) .. (7.2,3.95);
  \node[black, anchor=west, font=\scriptsize] at (7.25,3.75) {no Double};
  % no dueling
  \draw[black, thick] (0,0.2) .. controls (2.2,2.5) and (4.8,3.3) .. (7.2,3.45);
  \node[black, anchor=west, font=\scriptsize] at (7.25,3.4) {no dueling};
  % no noisy
  \draw[black, thick] (0,0.2) .. controls (2.2,2.2) and (4.8,3.0) .. (7.2,3.1);
  \node[black, anchor=west, font=\scriptsize] at (7.25,3.05) {no NoisyNet};
  % no distributional
  \draw[red, thick] (0,0.2) .. controls (2.2,1.7) and (4.8,2.3) .. (7.2,2.5);
  \node[red, anchor=west, font=\scriptsize] at (7.25,2.5) {no distributional};
  % no multi-step
  \draw[red, thick] (0,0.2) .. controls (2.2,1.4) and (4.8,1.9) .. (7.2,2.05);
  \node[red, anchor=west, font=\scriptsize] at (7.25,2.0) {no multi-step};
  % no prioritized (worst)
  \draw[red, very thick] (0,0.2) .. controls (2.2,1.1) and (4.8,1.5) .. (7.2,1.6);
  \node[red, anchor=west, font=\scriptsize] at (7.25,1.6) {no PER};
\end{tikzpicture}
$$

**Prioritized replay and multi-step returns were the two most important
components**: removing either produced the largest drop, and multi-step in particular
sped up early learning across almost every game. **The distributional head was next**,
its contribution growing over training and dominating in the later stages — evidence
that the richer target matters most late in training. **Dueling and
NoisyNets helped moderately**, each carrying a subset of games. **Double DQN, on its
own, barely mattered inside Rainbow** — the distributional target already bounds the
values and curbs the overestimation that Double DQN was invented to fix, so its
separate correction becomes largely redundant. That last finding is the clearest
illustration of _why an integrated study was needed_: a component's value is not a
fixed property but depends on what else is in the agent.

| Component removed | Effect when ablated | Reading |
| --- | --- | --- |
| Prioritized replay | large drop | most important single component |
| Multi-step returns | large drop, worst early | fastest early learning |
| Distributional (C51) | drop, growing late | richest target, pays off over time |
| Dueling | moderate drop | helps on a subset of games |
| NoisyNets | moderate drop | better exploration on hard-exploration games |
| Double DQN | negligible | redundant given the distributional head |

## Beyond Rainbow: implicit quantiles, full parametrization, and Agent57

Rainbow was not the end of the distributional line. Work after 2018 made the return
distribution _more expressive_ while keeping the same Q-learning skeleton, then
folded the result into agents that finally cleared the whole Atari suite.

**IQN** (Dabney et al., 2018) turns QR-DQN's fixed quantile levels into a _continuous_
sampling of them.[^iqn] QR-DQN learns $N$ heads at the fixed levels
$\tau_i = (2i-1)/2N$. IQN instead learns a single network that, given a quantile level
$\tau \sim U[0,1]$ sampled fresh, outputs the return at _that_ level — an **implicit
quantile function** $Z_\tau(s,a)$. Because $\tau$ is an input, the network represents
the entire inverse CDF rather than a fixed histogram, and the action value is a Monte
Carlo average $Q(s,a) \approx \tfrac{1}{K}\sum_k Z_{\tau_k}(s,a)$ over sampled levels.
Two payoffs follow. First, expressiveness is no longer capped by a head count: more
samples buy a finer distribution at test time without retraining. Second, the
sampling of $\tau$ can be reshaped to encode **risk sensitivity** — the reason the
distributional view matters even though the agents still act on the mean. Sample
$\tau$ from the low end and the induced policy is risk-averse (it optimizes a
conservative quantile, a CVaR-like objective); sample from the high end and it is
risk-seeking. On the 57-game Atari benchmark IQN surpassed Rainbow's distributional
component and closed much of the gap to a full Rainbow with far less machinery.

**FQF** (Yang et al., 2019) takes the last step: it also learns _where_ to place the
quantile levels.[^fqf] IQN samples $\tau$ uniformly; FQF adds a small "fraction
proposal" network that outputs an adjusted set of $\tau$ values, trained to minimize
the $1$-Wasserstein distance between the quantile approximation and the true
distribution — so the atoms cluster where the distribution has structure and thin out
where it is flat. It is the fully-parametrized quantile function: both the levels and
the values are learned, generalizing C51 (fixed values, learned probabilities) and
QR-DQN (fixed probabilities, learned values) at once.

$$
% caption: The distributional family by what each fixes and learns. C51 fixes the
% return atoms and learns probabilities; QR-DQN fixes quantile levels and learns
% return values; IQN samples levels continuously; FQF learns the levels too. Each
% step frees one more coordinate of the return distribution.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=26mm, minimum height=15mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[box] (c51) at (0,0)   {C51\\f\/ixed atoms\\learn probs};
  \node[box] (qr)  at (3.4,0) {QR-DQN\\f\/ixed levels\\learn values};
  \node[box, draw=acc, text=acc] (iqn) at (6.8,0) {IQN\\sample levels\\learn values};
  \node[box, draw=red, text=red] (fqf) at (10.2,0) {FQF\\learn levels\\+ values};
  \draw[->, black, thick] (c51) -- (qr);
  \draw[->, black, thick] (qr) -- (iqn);
  \draw[->, black, thick] (iqn) -- (fqf);
  \node[anchor=south, font=\scriptsize, text=black] at (5.4,0.9) {each step frees one more coordinate};
\end{tikzpicture}
$$

The endpoint of the value-based line is **Agent57** (Badia et al., 2020), the first
agent to exceed the human baseline on _all_ 57 Atari games — including the
hard-exploration holdouts Montezuma's Revenge and Pitfall that had resisted every
DQN-style method.[^agent57] It layers three things onto the distributional
Q-learning core: a family of policies indexed by a discount and an exploration weight,
so a single network holds both near-sighted exploiting policies and far-sighted
exploring ones; a separate intrinsic-reward stream (the never-give-up novelty bonus)
to drive the hard-exploration games; and a bandit that picks, per episode, which
policy in the family to follow. The distributional head is still there — the ancestry
runs straight from C51 through Rainbow and IQN — but the win on the last few games came
from bolting on the directed exploration this course treats in the
[exploration lesson](/reinforcement-learning/modern-deep-rl/exploration). Value-based
deep RL, begun by DQN and consolidated by Rainbow, was completed when the
distributional target was combined with a serious exploration mechanism.

## Where this leaves value-based deep RL

Every improvement from DQN to Rainbow kept the Q-learning frame — act, store to a replay buffer, sample a
mini-batch, build a bootstrapped target with a frozen target network, regress toward
it — and changed exactly one slot: the target (Double DQN, multi-step), the network
(dueling), the sampling (prioritized replay), the exploration (NoisyNets), or the
thing being predicted (distributional). Because they touched different slots they
composed, and Rainbow's ablation confirmed that the sum genuinely exceeds any part.

The deepest of the six, distributional RL, is the one that reframed the objective:
learn the return _distribution_ $Z(s,a)$ and recover $Q = \mathbb{E}[Z]$ as a
by-product. That perspective — a value is a distribution, not a number — carries
beyond discrete-action Atari. It reappears in the distributional critics of
continuous-control agents and in [risk-sensitive control](/reinforcement-learning/modern-deep-rl/continuous-control),
where an agent that knows the _spread_ of returns, not just their mean, can prefer the
safe action over the equal-mean gamble. Rainbow is the endpoint of the value-based
line begun by DQN; the policy-based and model-based lines take up from
[actor-critic and PPO](/reinforcement-learning/deep-rl/actor-critic-and-ppo) and
[model-based RL](/reinforcement-learning/modern-deep-rl/model-based-rl).

[^rainbow]: **Hessel et al. (2018)**, "Rainbow: Combining Improvements in Deep Reinforcement Learning", _AAAI_ — the integrated agent combining Double DQN, dueling networks, prioritized replay, multi-step returns, distributional (C51) learning, and NoisyNets; new state of the art on the 57-game Atari benchmark with markedly better sample efficiency; and the component ablation showing prioritized replay and multi-step returns most important, the distributional head growing in importance over training, and Double DQN largely redundant given the distributional target.
[^c51]: **Bellemare, Dabney & Munos (2017)**, "A Distributional Perspective on Reinforcement Learning", _ICML_ — the return distribution $Z(s,a)$ with $Q = \mathbb{E}[Z]$, the distributional Bellman equation $Z(s,a) \overset{D}{=} r + \gamma Z(s',a')$, and the C51 algorithm: $N=51$ fixed atoms on $[V_{\min},V_{\max}]=[-10,10]$, a softmax over atoms, the categorical projection $\Phi$ of the scaled-shifted target onto the fixed grid, and a cross-entropy loss; state of the art on Atari while acting on the mean.
[^qr]: **Dabney et al. (2018)**, "Distributional Reinforcement Learning with Quantile Regression", _AAAI_ — the QR-DQN complement to C51: fix $N$ equal-probability quantile levels $\tau_i = (2i-1)/2N$ and learn the atom _locations_ $\theta_i(s,a)$, eliminating the projection and training with the quantile (Huber) regression loss.
[^iqn]: **Dabney, Ostrovski, Silver, Munos (2018)**, "Implicit Quantile Networks for Distributional Reinforcement Learning", _ICML_ — represents the return's inverse CDF by a network taking a sampled quantile level $\tau \sim U[0,1]$ as input, $Z_\tau(s,a)$, so the full distribution is learned rather than a fixed histogram; enables risk-sensitive policies by reshaping the sampling of $\tau$; surpasses QR-DQN and Rainbow's distributional component on Atari.
[^fqf]: **Yang, Zhao, Lin, Qin, Bian, Liu (2019)**, "Fully Parameterized Quantile Function for Distributional Reinforcement Learning", _NeurIPS_ — FQF adds a fraction-proposal network that learns the quantile _levels_ (minimizing the $1$-Wasserstein error) on top of IQN's learned quantile _values_, so both coordinates of the quantile function are parameterized.
[^agent57]: **Badia, Piot, Kapturowski, Sprechmann, Vitvitskyi, Guo, Blundell (2020)**, "Agent57: Outperforming the Atari Human Benchmark", _ICML_ — the first agent above the human baseline on all 57 Atari games; a parameterized family of policies over discounts and exploration weights, a never-give-up intrinsic-reward stream for hard-exploration games, and a meta-controller (bandit) selecting the policy per episode, on a distributional Q-learning core.