---
title: "DQN Improvements: Double, Dueling, and Prioritized Replay"
module: Deep Reinforcement Learning
moduleNumber: 4
lessonNumber: 2
order: 402
summary: >
  Three refinements that turn plain DQN into the standard modern value-based agent,
  each touching a different part of the system. Double DQN fixes the maximization
  bias in the target by splitting action selection from evaluation; dueling networks
  restructure the network around a state value and per-action advantages; prioritized
  replay changes which transitions are learned from. We close with Rainbow, which
  combines them, and the distributional view that predicts the whole return
  distribution rather than its mean.
topics: [Deep RL]
sources:
  - book: Grokking Deep RL
    ref: "Ch. 9 — Double DQN; Ch. 10 — Dueling DDQN and PER"
---

This builds on [Deep Q-Networks](/reinforcement-learning/deep-rl/deep-q-networks),
which set up the neural action-value function, the deadly triad, and the two
stabilizers — experience replay and a target network — that make deep Q-learning
train. Those give a working agent; the three refinements here each improve a
different component without changing the overall structure.

## Improvement one: Double DQN

DQN inherits a flaw from tabular Q-learning: the $\max$ operator **overestimates**.
The target $r + \gamma \max_{a'} Q(s', a'; \mathbf{w}^-)$ takes the maximum over
_estimated_ values, and estimates are noisy — some above the truth, some below.
Taking the max systematically prefers the ones that happen to be inflated, so the
target carries a persistent positive bias, and the bias compounds through
bootstrapping.[^gd-double]

The problem is that $\max_{a'} Q(s', a'; \mathbf{w}^-)$ answers two
questions with one network: _which_ action is best (an $\arg\max$), and _how good_ is
it (evaluating that action). Writing the max as
$Q\bigl(s', \arg\max_{a'} Q(s', a'; \mathbf{w}^-); \mathbf{w}^-\bigr)$ makes the
double role explicit. Both questions go to the same weights,
so a value inflated by noise is both _selected_ and _trusted_, biasing the answer in
the same direction twice.

**Double DQN** decouples the two. Use the **online** network to _select_ the greedy
action, and the **target** network to _evaluate_ it:[^gd-double]

$$
y^{\text{DDQN}} \;=\; r + \gamma\, Q\!\left(s',\; \arg\max_{a'} Q(s', a'; \mathbf{w});\; \mathbf{w}^-\right).
$$

Because selection and evaluation now come from networks with different weights, an
error that inflates an action's value in one network is unlikely to inflate it in
the other, and the two "cross-validate" each other's estimates. The change is one
line in the target — the online weights $\mathbf{w}$ choose $a'$, the frozen weights
$\mathbf{w}^-$ score it — and it reliably yields better policies at no extra network
cost, reusing the two networks DQN already maintains.

**How much bias, quantitatively.** The overestimation has a
clean bound. Suppose in some state the true action values are all equal to a common
$V_\ast$, so no action is genuinely better, and the estimates are unbiased but noisy:
each $Q(s', a; \mathbf{w}^-) = V_\ast + \eta_a$ with the errors $\eta_a$ independent,
zero-mean, and of comparable spread. The single-network target uses
$\max_a (V_\ast + \eta_a) = V_\ast + \max_a \eta_a$, and the maximum of several zero-mean
noises is _positive_ in expectation — for $m$ actions with variance $\sigma^2$ it
grows roughly like $\sigma\sqrt{2 \ln m}$. So the target overshoots the truth by an
amount that increases with both the noise and the number of actions; van Hasselt,
Guez, and Silver (2016) show this bias compounds through bootstrapping and measurably
inflates DQN's value estimates on Atari, sometimes by an order of magnitude, without
a matching gain in score.[^ddqn-paper] Double DQN's decoupled estimate cancels the correlation
between which action is picked and which error inflates it, so its target is far
closer to unbiased.

For example, say three next-actions have true value $10$ each, and the two
networks estimate:

| action | online $Q(s', \cdot; \mathbf{w})$ | target $Q(s', \cdot; \mathbf{w}^-)$ |
| --- | --- | --- |
| $a_1$ | $10.3$ | $9.7$ |
| $a_2$ | $11.1$ | $9.9$ |
| $a_3$ | $9.6$ | $10.4$ |

Plain DQN takes $\max$ of the target column: $10.4$, an overestimate of $+0.4$ over
the truth of $10$. Double DQN picks the greedy action from the _online_ column
($a_2$, at $11.1$) and evaluates _that_ action in the target column: $Q(s', a_2;
\mathbf{w}^-) = 9.9$, an estimate of $-0.1$ from the truth. The online network's
overestimate of $a_2$ is not confirmed by the target network, and the target
network's overestimate of $a_3$ never enters the target, because $a_3$ was not
selected. The decoupling turns $+0.4$ into $-0.1$.

$$
% caption: Double DQN. The online network names the greedy next action ("action a3
% looks best"), and the target network reports that action's value — separating
% selection from evaluation so noise does not inflate the target twice.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  net/.style={draw, minimum width=30mm, minimum height=20mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[net, draw=acc, text=acc] (on) at (0,0)
    {online network (w)\\ \, \\Q(s-next, a1) = 3.5\\Q(s-next, a2) = 1.2\\Q(s-next, a3) = 3.9};
  \node[net] (tg) at (7.4,0)
    {target network (w-minus)\\ \, \\Q(s-next, a1) = 3.8\\Q(s-next, a2) = 1.0\\Q(s-next, a3) = 3.6};
  \draw[->, red, thick] (on.east) -- node[above, font=\scriptsize, align=center] {select: "a3 is best"} (tg.west);
  \node[red, font=\scriptsize, anchor=north] at (3.7,-1.6) {evaluate a3: value 3.6 is the target};
\end{tikzpicture}
$$

## Improvement two: dueling networks

Double DQN changed the target; the **dueling architecture** changes the _network_,
and it does so without touching the control algorithm at all.[^gd-dueling] The
insight is about the [action-advantage function](/reinforcement-learning/foundations/value-functions-and-optimality)
$A(s,a) = Q(s,a) - V(s)$: in many states the choice of action barely matters (the
cart-pole balanced and upright is fine either way), and there the useful quantity is
$V(s)$ — how good the state is — not the near-identical Q-values across actions.
Forcing a single stream to learn $Q$ directly wastes capacity relearning $V(s)$
inside every action's estimate.

A dueling network shares the early layers (the convolutions, on Atari) and then
**splits into two streams**: one outputs a single scalar $V(s; \mathbf{w}, \beta)$,
the other a vector of advantages $A(s, a; \mathbf{w}, \alpha)$. They recombine into
Q-values. The naive recombination $Q = V + A$ is unidentifiable — add a constant to
$V$ and subtract it from $A$ and $Q$ is unchanged — so DQN subtracts the mean
advantage to make the decomposition unique:

$$
Q(s, a; \mathbf{w}, \alpha, \beta) \;=\; V(s; \mathbf{w}, \beta) + \left(A(s, a; \mathbf{w}, \alpha) - \frac{1}{|\mathcal{A}|} \sum_{a'} A(s, a'; \mathbf{w}, \alpha)\right).
$$

$$
% caption: The dueling architecture. Shared layers feed two heads — a scalar state
% value $V(s)$ and a per-action advantage $A(s,a)$ — recombined (advantages centered
% on their mean) into the Q-values. The control algorithm is unchanged; only the
% network differs.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  b/.style={draw, minimum width=17mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[b] (in) at (-3.0,0) {state s};
  \node[b] (sh) at (-0.6,0) {shared\\layers};
  \node[b, draw=acc, text=acc] (v) at (2.4,1.1) {value\\V(s)};
  \node[b, draw=red, text=red] (a) at (2.4,-1.1) {advantage\\A(s, a)};
  \node[b] (q) at (5.6,0) {combine\\Q(s, a)};
  \draw[->, black, thick] (in) -- (sh);
  \draw[->, acc, thick] (sh) -- (v);
  \draw[->, red, thick] (sh) -- (a);
  \draw[->, acc, thick] (v) -- (q);
  \draw[->, red, thick] (a) -- (q);
  \node[font=\scriptsize, black, align=center] at (2.4,-2.4) {V is one node; A has one node per action};
\end{tikzpicture}
$$

Subtracting the mean shifts $V$ and $A$ off their true values by a constant, but it
leaves the _relative_ ranking of actions intact, so the greedy policy is unaffected.
The benefit is that the state-value stream is trained by _every_ action's update
rather than only the action taken, so $V(s)$ — the component shared across all
actions in a state — is learned more efficiently and with lower variance. Dueling
plugs straight into Double DQN; the combination is often called dueling DDQN.

## Improvement three: prioritized experience replay

Plain replay samples the buffer uniformly, spending equal effort on transitions the
agent already predicts perfectly and on the rare surprising ones that carry the most
to learn. **Prioritized experience replay** samples the surprising ones more
often.[^gd-per] The measure of surprise is the **absolute TD error**: a transition
whose target is far from the current prediction is one the agent misjudged, and
replaying it yields the most learning. The priority of transition $i$ is

$$
p_i \;=\; |\delta_i| + \epsilon,
$$

with $\delta_i$ the TD error and a small $\epsilon$ so that even zero-error
transitions keep a chance of being replayed. Sampling strictly by priority is
brittle — noisy errors would trap the agent on a handful of transitions — so PER
samples **stochastically**, drawing transition $i$ with probability

$$
P(i) \;=\; \frac{p_i^{\,\alpha}}{\sum_k p_k^{\,\alpha}},
$$

where $\alpha \in [0, 1]$ interpolates between uniform replay ($\alpha = 0$) and
pure greedy prioritization ($\alpha = 1$). Every transition retains a nonzero
sampling probability, monotone in its error.

One correction remains. Sampling non-uniformly changes the _distribution_ of the
updates, which biases the expectation the gradient is estimating — precisely the
kind of distribution mismatch that endangers off-policy learning. PER offsets it
with **importance-sampling weights** that scale each transition's update down in
proportion to how often it is over-sampled,

$$
w_i \;=\; \bigl(N \cdot P(i)\bigr)^{-\beta}, \qquad \tilde w_i = \frac{w_i}{\max_j w_j},
$$

normalized by their maximum so weights only ever scale updates down and keep training
stable. The exponent $\beta$ is annealed toward 1 over training, fully correcting the
bias by the end, when the agent is near convergence and the bias would matter most.
The weights multiply into the loss, so PER changes _which_ transitions are seen and
_how much_ each one counts, without changing the target.

For example, take five transitions whose
absolute TD errors are $|\delta| = (2.0,\ 0.5,\ 0.1,\ 1.4,\ 0.02)$ and set $\epsilon
= 0$, $\alpha = 0.6$. The priorities raised to $\alpha$ are $p_i^{\alpha} =
(2.0^{0.6},\ 0.5^{0.6},\ 0.1^{0.6},\ 1.4^{0.6},\ 0.02^{0.6}) = (1.52,\ 0.66,\ 0.25,\
1.23,\ 0.09)$, summing to $3.75$. Dividing gives sampling probabilities $P =
(0.41,\ 0.18,\ 0.07,\ 0.33,\ 0.02)$. The high-error transition is drawn about
$20\times$ more often than the near-zero one — but the $\alpha = 0.6$ exponent keeps
even the $|\delta| = 0.02$ transition at a $2\%$ chance, so nothing is starved.
Compare $\alpha = 0$, which flattens every $P_i$ to the uniform $0.20$. The
importance weight for the top transition, with $N = 5$ and $\beta = 0.5$, is
$w_1 = (5 \times 0.41)^{-0.5} = 2.05^{-0.5} = 0.70$; the rarest transition gets
$w_5 = (5 \times 0.02)^{-0.5} = 0.10^{-0.5} = 3.16$. After normalizing by the max,
$\tilde w_1 = 0.70/3.16 = 0.22$ and $\tilde w_5 = 1.0$: the over-sampled transition's
update is scaled down to a fifth, exactly undoing its over-representation.

## Rainbow and distributional value learning

Double DQN, dueling, and prioritized replay were each published as a separate
improvement, and each helped on its own. The natural question is whether they
compose. **Rainbow** (Hessel et al., 2018, AAAI) answers it by combining six
extensions into one agent and ablating each: Double DQN, dueling networks,
prioritized replay, multi-step returns, distributional value learning, and noisy
networks for exploration. On the 57-game Atari benchmark the combination
substantially outperforms any single component, and the ablations show that removing
prioritized replay or multi-step returns hurts most — evidence that the gains are
largely complementary rather than redundant.[^rainbow] Rainbow is the practical answer to
"which DQN variant should I run": most of them, together.

Two of its components go beyond anything above. **Multi-step returns** replace the
one-step target with an $n$-step one, $y = \sum_{k=0}^{n-1} \gamma^k r_{t+k} +
\gamma^n \max_{a'} Q(s_{t+n}, a'; \mathbf{w}^-)$, trading a little off-policy bias
(the intermediate actions came from an older policy) for faster reward propagation —
the same bias-variance dial that reappears in [GAE](/reinforcement-learning/deep-rl/actor-critic-and-ppo).
**Noisy networks** (Fortunato et al., 2018) replace $\varepsilon$-greedy with learned
parametric noise on the weights, so exploration is state-dependent and annealed by
gradient descent rather than a hand-set schedule.

The deepest departure is **distributional RL** (Bellemare, Dabney, and Munos, 2017,
ICML). DQN predicts the _expected_ return $Q(s,a)$; the distributional view predicts
the full _distribution_ $Z(s,a)$ of returns, of which $Q$ is only the mean. Their
C51 agent represents $Z$ as a categorical distribution over 51 fixed atoms and trains
it with a distributional Bellman update, minimizing a cross-entropy to the projected
target distribution. Learning the whole distribution turns out to be a better
_auxiliary_ signal for the shared representation, not merely a richer output, and C51
beat the prior state of the art on Atari. A later refinement, **quantile regression
DQN** (Dabney et al., 2018, AAAI), predicts the distribution's quantiles instead of
fixed-position probabilities, removing C51's need to guess the return range in
advance.[^distributional] These are developed in the [distributional and Rainbow](/reinforcement-learning/modern-deep-rl/distributional-and-rainbow)
lesson; here the point is that the value _function_ DQN learns is itself a modeling
choice, and predicting more than its mean improves performance.

$$
% caption: Expected versus distributional value. DQN predicts a single number
% $Q(s,a)$, the mean return (left). Distributional agents such as C51 predict the
% whole return distribution $Z(s,a)$ over a set of atoms (right), of which $Q$ is
% only the average; the extra structure sharpens the learned representation.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % left: single scalar
  \draw[black, ->] (0,0) -- (3.4,0) node[right, black, font=\scriptsize] {return};
  \draw[black, ->] (0,0) -- (0,2.6) node[above, black, font=\scriptsize] {};
  \draw[acc, very thick] (2.0,0) -- (2.0,2.1);
  \fill[acc] (2.0,2.1) circle (2pt);
  \node[acc, anchor=south, font=\scriptsize] at (2.0,2.15) {Q = mean};
  \node[font=\scriptsize, black, anchor=north] at (1.7,-0.2) {DQN: one number};
  % right: distribution
  \begin{scope}[xshift=5.0cm]
    \draw[black, ->] (0,0) -- (4.2,0) node[right, black, font=\scriptsize] {return};
    \draw[black, ->] (0,0) -- (0,2.6) node[above, black, font=\scriptsize] {prob};
    \foreach \x/\h in {0.5/0.4, 1.0/0.9, 1.5/1.6, 2.0/2.1, 2.5/1.7, 3.0/1.0, 3.5/0.5}
      \draw[red, very thick] (\x,0) -- (\x,\h);
    \draw[acc, dashed, thick] (2.05,0) -- (2.05,2.3);
    \node[acc, anchor=south, font=\scriptsize] at (2.4,2.15) {mean = Q};
    \node[font=\scriptsize, black, anchor=north] at (2.0,-0.2) {C51: full distribution Z};
  \end{scope}
\end{tikzpicture}
$$

## Where this leaves us

Deep Q-networks are the value-based branch of deep reinforcement learning: approximate
$q_\ast$ with a neural network, train it by bootstrapped semi-gradient regression, and
counter the deadly triad with two stabilizers — a replay buffer that makes correlated
online data look IID, and a frozen target network that turns a moving target into a
sequence of stationary regressions. The three refinements each improve a different
part: Double DQN fixes the maximization bias in the _target_, dueling
networks restructure the _network_ around $V$ and $A$, and prioritized replay improves
_which data_ is learned from. Together they are the standard modern value-based agent.

DQN learns a value function and reads a policy off it by $\arg\max$. The
complementary branch parameterizes and optimizes the policy directly, which handles
continuous actions and stochastic policies that a value-based $\arg\max$ cannot —
[actor-critic and PPO](/reinforcement-learning/deep-rl/actor-critic-and-ppo). Both
branches, and the search-and-model methods that extend them, come together in the
[case studies](/reinforcement-learning/deep-rl/case-studies) that carried deep RL from
Atari to Go and beyond.


[^gd-double]: **Morales**, Ch. 9, "Double DQN" — Q-learning's overestimation from taking the max of noisy estimates; unwrapping the max into an argmax (selection) and an evaluation; and the DDQN target $r + \gamma Q(s', \arg\max_{a'} Q(s',a';\theta); \theta^-)$ using online weights to select and target weights to evaluate (van Hasselt, 2015).
[^gd-dueling]: **Morales**, Ch. 10, "Dueling DDQN" — the dueling architecture's shared layers splitting into a state-value stream $V(s)$ and an advantage stream $A(s,a)$, the aggregation $Q = V + (A - \frac{1}{|\mathcal{A}|}\sum_{a'} A(s,a'))$ subtracting the mean advantage for identifiability, and the more efficient, lower-variance learning of $V(s)$ (Wang et al., 2015).
[^gd-per]: **Morales**, Ch. 10, "PER: Prioritizing the replay of meaningful experiences" — the absolute TD error as priority $p_i = |\delta_i| + \epsilon$, stochastic prioritization $P(i) = p_i^\alpha / \sum_k p_k^\alpha$ interpolating uniform and greedy replay, and the weighted importance-sampling correction $w_i = (N \cdot P(i))^{-\beta}$ normalized by its max with $\beta$ annealed to 1 (Schaul et al., 2015). The canonical paper is **Schaul, Quan, Antonoglou, and Silver** (2016), "Prioritized Experience Replay", _ICLR_.
[^ddqn-paper]: **van Hasselt, Guez, and Silver** (2016), "Deep Reinforcement Learning with Double Q-learning", _AAAI_ — shows single-network DQN systematically overestimates action values on Atari, gives the maximization-bias argument (the max of noisy unbiased estimates is positively biased), and demonstrates that the online-select / target-evaluate decoupling reduces both the value error and the resulting suboptimality. The dueling architecture is **Wang, Schaul, Hessel, van Hasselt, Lanctot, and de Freitas** (2016), "Dueling Network Architectures for Deep Reinforcement Learning", _ICML_.
[^rainbow]: **Hessel, Modayil, van Hasselt, Schaul, Ostrovski, Dabney, Horgan, Piot, Azar, and Silver** (2018), "Rainbow: Combining Improvements in Deep Reinforcement Learning", _AAAI_ — integrates Double DQN, dueling, prioritized replay, multi-step returns, distributional value learning, and noisy networks into one agent and ablates each on the 57-game Atari suite, finding the combination outperforms every individual component and that prioritized replay and multi-step returns contribute the most.
[^distributional]: **Bellemare, Dabney, and Munos** (2017), "A Distributional Perspective on Reinforcement Learning", _ICML_ — the C51 agent, modeling the return distribution $Z(s,a)$ as a categorical over 51 fixed atoms trained by a projected distributional Bellman update; **Dabney, Rowland, Bellemare, and Munos** (2018), "Distributional Reinforcement Learning with Quantile Regression", _AAAI_ — QR-DQN, predicting quantiles instead of fixed-atom probabilities. Noisy exploration: **Fortunato, Azar, Piot, et al.** (2018), "Noisy Networks for Exploration", _ICLR_.