---
title: Deep Q-Networks
module: Deep Reinforcement Learning
moduleNumber: 4
lessonNumber: 1
order: 401
summary: >
  Deep Q-networks replace the linear value function with a neural network
  $Q(s,a;\mathbf{w})$ and confront the fact that a nonlinear approximator, off-policy
  bootstrapping, and correlated online data — the deadly triad — make naive
  Q-learning diverge. DQN counters this empirically with two stabilizers: an experience
  replay buffer that decorrelates and reuses samples, and a periodically-frozen
  target network that fixes the bootstrap target. We derive the DQN loss and
  gradient, walk through the Atari convolutional architecture and its results, and
  then add the three refinements that define modern value-based deep RL — Double
  DQN, dueling networks, and prioritized experience replay.
topics: [Deep RL]
sources:
  - book: Grokking Deep RL
    ref: "Ch. 8 — Introduction to value-based deep RL; §8.3 NFQ; Ch. 9 — DQN and Double DQN; Ch. 10 — Dueling DDQN and PER"
  - book: Sutton & Barto
    ref: "§16.5 — Human-level Video Game Play (the Atari DQN case study)"
---

Every method up to now approximated value with a _linear_ map: pick features
$\mathbf{x}(s,a)$ by hand, and estimate $Q(s,a) \approx \mathbf{w}^\top
\mathbf{x}(s,a)$. That works when a person can design good features, and it comes
with the [projected-Bellman-error geometry](/reinforcement-learning/approximation/off-policy-and-the-deadly-triad)
that makes stable off-policy methods possible. It fails the moment the state is a
raw image. An Atari frame is $210 \times 160$ pixels over three color channels; the
number of distinct states is astronomically larger than the number of atoms in the
observable universe, and no hand-built feature set can cover it.[^gd-sampled]
Deep reinforcement learning instead lets a **neural network** be the feature
extractor _and_ the value head at once, learned end to end from reward.

A **deep Q-network** approximates the action-value function with a neural network
whose weights are $\mathbf{w}$:

$$
Q(s, a; \mathbf{w}) \;\approx\; q_\ast(s, a).
$$

The network takes the state, and outputs one Q-value per action in a single forward
pass. That is the whole idea. The rest of this lesson is about the one thing that
makes it hard: a nonlinear approximator trained by online Q-learning is _unstable_,
and left alone it diverges. DQN made it converge in practice, and is the algorithm
that started modern deep RL.[^sb-dqn]

## From features to a network

A tabular Q-learner keeps a matrix indexed by state and action. Function
approximation replaces that lookup with a parametric function, and the gain is
generalization: adjusting $\mathbf{w}$ to fix the value of one state-action pair also
shifts the value of every _similar_ pair, so the agent learns from states it has
never visited. In a linear model that generalization is limited to what the chosen
features can express; a neural network discovers its own features, and can represent
the intricate value surfaces that raw perception requires.[^gd-funcapprox]

There are two ways to wire a network for $Q$, and the choice matters. One feeds in
both the state _and_ an action and outputs a single scalar; evaluating all actions
then costs one forward pass per action. The efficient alternative — the one DQN uses
— feeds in the state alone and outputs a **vector of Q-values, one per action**. A
single forward pass then scores every action at once, handing an
$\varepsilon$-greedy or softmax policy the full vector it consumes.[^gd-arch]

$$
% caption: The state-in, values-out architecture. The network reads the raw state
% and emits one Q-value per action in a single forward pass, so acting greedily
% costs one evaluation regardless of the number of actions. Weights are $\mathbf{w}$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  u/.style={circle, draw, minimum size=5mm, inner sep=0pt},
  io/.style={draw, minimum width=17mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  % input
  \node[io] (s) at (-2.9,0) {state s\\(raw input)};
  % hidden columns (integer indices; y = 2 - i)
  \foreach \i in {1,2,3,4} \node[u] (ha\i) at (0,{2-\i*0.9}) {};
  \foreach \i in {1,2,3,4} \node[u] (hb\i) at (1.6,{2-\i*0.9}) {};
  % outputs (one per action)
  \node[u, draw=acc] (o1) at (3.7,1.1) {};
  \node[u, draw=acc] (o2) at (3.7,0.0) {};
  \node[u, draw=acc] (o3) at (3.7,-1.1) {};
  \node[anchor=west, acc, font=\scriptsize] at (4.1,1.1) {Q(s, a1)};
  \node[anchor=west, acc, font=\scriptsize] at (4.1,0.0) {Q(s, a2)};
  \node[anchor=west, acc, font=\scriptsize] at (4.1,-1.1) {Q(s, a3)};
  % edges (sparse, representative)
  \foreach \i in {1,2,3,4} \draw[black] (s.east) -- (ha\i);
  \foreach \a in {1,2,3,4} \foreach \b in {1,2,3,4} \draw[black] (ha\a) -- (hb\b);
  \foreach \i in {1,2,3,4} { \draw[acc!60] (hb\i) -- (o1); \draw[acc!60] (hb\i) -- (o2); \draw[acc!60] (hb\i) -- (o3); }
  \node[font=\scriptsize, black] at (0.8,-2.4) {hidden layers (weights w)};
\end{tikzpicture}
$$

Given this network, the greedy action in a state is $\arg\max_a Q(s, a; \mathbf{w})$,
computed from one pass, and the whole apparatus of generalized policy iteration
carries over: evaluate the current policy by regressing $Q$ toward a target, improve
by acting greedily (with exploration) on the new estimates, repeat. The catch is
that with a nonlinear $Q$ there are no convergence guarantees left.

## The ideal objective, and why it is out of reach

Training the network would be ordinary supervised learning if the correct answers
were available. They are not, so the target must be manufactured from the
network's own estimates, at the cost of a non-stationary target.

If we somehow had access to the optimal action-value function $q_\ast$, learning $Q$
would be ordinary supervised regression: minimize the squared distance to the true
values,

$$
L_i(\mathbf{w}) \;=\; \mathbb{E}_{s,a}\!\left[\bigl(q_\ast(s, a) - Q(s, a; \mathbf{w})\bigr)^2\right].
$$

This objective is unavailable.[^gd-objective] We have no
$q_\ast$ to regress against — if we did, there would be nothing to learn — and we
cannot even _sample_ $q_\ast$, because we have neither the optimal policy nor the
environment's model. The fix is the same bootstrapping trick as tabular Q-learning:
replace the unavailable $q_\ast(s, a)$ with a **target built from the network's own
next-state estimate**. For an off-policy Q-learning target,

$$
y \;=\; r + \gamma \max_{a'} Q(s', a'; \mathbf{w}),
$$

so the loss over sampled experience tuples $(s, a, r, s')$ becomes

$$
L(\mathbf{w}) \;=\; \mathbb{E}_{s,a,r,s'}\!\left[\Bigl(\underbrace{r + \gamma \max_{a'} Q(s', a'; \mathbf{w})}_{\text{target } y} - Q(s, a; \mathbf{w})\Bigr)^2\right].
$$

Differentiating, and treating the target as a constant with respect to
$\mathbf{w}$ — the gradient flows only through the _predicted_ value $Q(s,a;\mathbf{w})$,
never through the target — gives the update direction

$$
\nabla_{\mathbf{w}} L(\mathbf{w}) \;=\; -\,\mathbb{E}_{s,a,r,s'}\!\left[\Bigl(r + \gamma \max_{a'} Q(s', a'; \mathbf{w}) - Q(s, a; \mathbf{w})\Bigr) \nabla_{\mathbf{w}} Q(s, a; \mathbf{w})\right].
$$

Detaching the target matters. In supervised learning the labels are fixed
constants; here the "label" is manufactured from the very network being optimized,
so letting the gradient propagate through it optimizes a moving definition of
correctness. Backpropagate through the prediction only.[^gd-target]

> **Definition (Semi-gradient Q-learning target).** The bootstrapped regression
> target $y = r + \gamma \max_{a'} Q(s', a'; \mathbf{w})$ used to train
> $Q(s,a;\mathbf{w})$. It is _off-policy_ (the $\max$ evaluates the greedy policy
> regardless of how the data was collected) and _semi-gradient_ (the target depends
> on $\mathbf{w}$, but its gradient is ignored). This is the same target as tabular
> [Q-learning](/reinforcement-learning/tabular-methods/temporal-difference-learning),
> lifted to a neural network.

## Why the naive version diverges

Assemble the obvious algorithm — a neural $Q$, the Q-learning target above, an
$\varepsilon$-greedy behavior policy, an MSE loss, and a step of RMSprop after each
transition — and it will train for a while and then come apart. This is not a bug in
the implementation; it is the [deadly triad](/reinforcement-learning/approximation/off-policy-and-the-deadly-triad)
in its most dangerous configuration. All three ingredients are present:
**function approximation** (a neural network), **bootstrapping** (the TD target), and
**off-policy training** (the $\max$ learns the greedy policy while behavior explores).
Any two are safe; all three can diverge. Two mechanisms cause the instability.[^gd-wrong]

**The target is non-stationary.** The regression target $y$ is computed from
$Q(\,\cdot\,; \mathbf{w})$, so every gradient step that changes $\mathbf{w}$ also
changes the target for the next step. The network chases a value it is itself
moving. Worse, because the approximator generalizes, updating $Q(s,a;\mathbf{w})$
drags the value of the _next_ state $s'$ along with it — and $s'$ is precisely what
the target for $(s,a)$ depends on. Improve the prediction and you spoil the target
that defined "improvement," which can spiral into divergence.

$$
% caption: Chasing a moving target. Without stabilization, each update shifts the
% network, which shifts the target it was aiming at; the optimizer never arrives
% and can spiral outward. Fixing the target for many steps turns this into a
% sequence of ordinary, stationary regressions.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % --- left: moving target ---
  \node[font=\footnotesize\bfseries] at (1.7,2.7) {moving target};
  \fill[red] (2.9,1.9) circle (2.4pt);
  \node[red, anchor=west, font=\scriptsize] at (3.05,1.9) {target};
  \draw[->, acc, thick] (0.2,0.4) -- (1.5,1.7);
  % target has moved; estimate overshoots
  \fill[red] (0.9,0.1) circle (2.4pt);
  \draw[->, acc, thick] (1.5,1.7) -- (0.6,0.35);
  \draw[->, acc, thick] (0.6,0.35) -- (2.2,1.55);
  \node[acc, anchor=south, font=\scriptsize] at (1.2,1.75) {estimate};
  % --- right: frozen target ---
  \begin{scope}[xshift=6.4cm]
    \node[font=\footnotesize\bfseries] at (1.5,2.7) {frozen target};
    \fill[red] (2.9,1.5) circle (2.4pt);
    \node[red, anchor=west, font=\scriptsize] at (3.05,1.5) {target (f\/ixed)};
    \draw[->, acc, thick] (0.2,0.4) -- (1.2,0.95);
    \draw[->, acc, thick] (1.2,0.95) -- (2.1,1.3);
    \draw[->, acc, thick] (2.1,1.3) -- (2.75,1.47);
    \node[acc, anchor=north, font=\scriptsize] at (1.4,0.85) {stable descent};
  \end{scope}
\end{tikzpicture}
$$

**The data is not IID.** Optimization methods assume samples are independent and
identically distributed; online RL supplies the opposite. Consecutive transitions
come from the same trajectory, so the sample at time $t{+}1$ is highly correlated
with the one at $t$ — the network overfits to whatever local region of the
state space the current episode is wandering through. And because the behavior
policy keeps changing, the sample distribution drifts too, so the data is neither
independent nor identically distributed.[^gd-iid] Both violations push the
optimizer toward instability.

DQN's contribution is two mechanisms that repair exactly these two problems, and
between them make deep value-based RL behave enough like supervised learning to
train reliably.

## Stabilizer one: experience replay

The fix for correlated, non-IID data is to stop training on the online stream. As
the agent acts, it stores each transition $(s, a, r, s')$ in a large
**replay buffer** $D$; training then draws a **mini-batch uniformly at random** from
$D$ rather than using the most recent transitions.[^gd-replay]

> **Definition (Experience replay).** A memory $D = \{e_1, \dots, e_M\}$ of past
> transitions $e_t = (s_t, a_t, r_t, s_{t+1})$. The agent inserts each new
> transition as it acts and trains on mini-batches sampled uniformly from $D$. When
> the buffer is full (capacity $M$, typically $10^4$ to $10^6$), the oldest
> transition is evicted to make room for the newest.

Random sampling from a large buffer breaks the temporal correlation: a mini-batch
now mixes transitions from many different trajectories and many past policies, so
the updates look far more like draws from a fixed distribution — approximately
IID, as the optimizer assumes. The second benefit is
**data efficiency**: each transition can be sampled and learned from many times
rather than being seen once and thrown away, which matters when interaction with the
environment is the expensive part.

$$
% caption: DQN with a replay buffer. The agent acts through an exploration policy,
% stores each transition, and trains the Q-network on a mini-batch sampled uniformly
% at random from the buffer — decorrelating the data and reusing every experience.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=20mm, minimum height=9mm, align=center, font=\scriptsize},
  env/.style={draw, minimum width=20mm, minimum height=13mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % agent cluster (left)
  \node[box, draw=acc, text=acc] (q) at (0,2.0) {Q-network\\Q(s, a; w)};
  \node[box] (exp) at (3.2,2.0) {exploration\\policy};
  \node[box] (buf) at (0,-1.4) {replay bu\/f\/fer D};
  \node[box] (mb) at (0,0.3) {mini-batch};
  \node[env] (env) at (6.6,0.4) {environment};
  % agent internal wiring
  \draw[->, acc] (q) -- node[above, font=\scriptsize] {Q-values} (exp);
  \draw[->, red] (buf) -- node[right, font=\scriptsize] {sample} (mb);
  \draw[->, acc] (mb) -- node[right, font=\scriptsize] {train} (q);
  % agent-environment loop
  \draw[->, thick] (exp.east) -- ++(0.6,0) |- (env.west) node[pos=0.25, right, font=\scriptsize] {action};
  \draw[->, thick] (env.south) |- (buf.east) node[pos=0.75, above, font=\scriptsize] {store (s, a, r, s-next)};
\end{tikzpicture}
$$

The gradient is unchanged in form; only the sampling distribution moves from the
online stream to a uniform draw over the buffer, written $(s,a,r,s') \sim
\mathcal{U}(D)$:

$$
\nabla_{\mathbf{w}} L(\mathbf{w}) \;=\; -\,\mathbb{E}_{(s,a,r,s') \sim \mathcal{U}(D)}\!\left[\Bigl(r + \gamma \max_{a'} Q(s', a'; \mathbf{w}) - Q(s, a; \mathbf{w})\Bigr) \nabla_{\mathbf{w}} Q(s, a; \mathbf{w})\right].
$$

Experience replay predates deep RL — Lin proposed it in 1992 — but pairing it with a
neural Q-function is what enabled stable training from raw experience.[^gd-replay]

## Stabilizer two: the target network

Replay handles the data; it does nothing about the moving target. For that, DQN
keeps a **second copy of the network** whose weights are frozen and used only to
compute targets: the **target network**, with weights $\mathbf{w}^-$.[^gd-targetnet]

> **Definition (Target network).** A periodically-frozen copy of the Q-network,
> with weights $\mathbf{w}^-$, used solely to evaluate the bootstrap target
> $y = r + \gamma \max_{a'} Q(s', a'; \mathbf{w}^-)$. The _online_ network
> $\mathbf{w}$ is updated every step; the target weights are held fixed and only
> refreshed to the online weights every $C$ steps ($\mathbf{w}^- \gets \mathbf{w}$).

The two networks share the same architecture; they differ only in the **age of the
weights**. The online network $\mathbf{w}$ is the one we optimize on every step. The
target network $\mathbf{w}^-$ is a snapshot of the online weights from up to $C$
steps ago, and it defines the target. The DQN gradient differs from the naive one in
exactly one symbol — $\mathbf{w}^-$ where the target's parameters used to be
$\mathbf{w}$:

$$
\nabla_{\mathbf{w}} L(\mathbf{w}) \;=\; -\,\mathbb{E}_{(s,a,r,s') \sim \mathcal{U}(D)}\!\left[\Bigl(r + \gamma \max_{a'} Q(s', a'; \mathbf{w}^-) - Q(s, a; \mathbf{w})\Bigr) \nabla_{\mathbf{w}} Q(s, a; \mathbf{w})\right].
$$

Because $\mathbf{w}^-$ is fixed between refreshes, the target stops moving for $C$
steps at a time. Each such window is an ordinary, stationary regression problem: a
fixed set of labels the online network can descend toward without the goalposts
sliding. Every $C$ steps the labels are updated all at once and a new stationary
problem is set. Freezing the target does not guarantee convergence to the optimum — that
does not exist under nonlinear approximation — but it substantially reduces the risk
of divergence, which is the actual failure mode.

$$
% caption: The two networks and the loss. The frozen target network (weights w-minus,
% a past snapshot) computes the bootstrap target; the online network (weights w) is
% trained to match it, and every $C$ steps its weights are copied into the target.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  net/.style={draw, minimum width=26mm, minimum height=13mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % online network
  \node[net, draw=acc, text=acc, thick] (on) at (0,0) {online network\\Q(s, a; w)};
  % target network
  \node[net] (tg) at (6.6,0) {target network\\Q(s-next, a-next; w-minus)};
  % loss node
  \node[draw, minimum width=24mm, minimum height=9mm, align=center] (loss) at (3.3,-2.7)
    {loss L(w)\\squared TD error};
  % predicted value into loss
  \draw[->, acc, thick] (on.south) |- (loss.west);
  \node[acc, font=\scriptsize, anchor=south] at (0,-1.35) {prediction Q(s, a; w)};
  % target value into loss
  \draw[->, black, thick] (tg.south) |- (loss.east);
  \node[black, font=\scriptsize, anchor=south] at (6.6,-1.35) {target r + gamma max};
  % periodic copy
  \draw[->, red, thick, dashed] (on.north) to[bend left=22] node[above, font=\scriptsize] {copy weights every C steps} (tg.north);
\end{tikzpicture}
$$

How often to refresh depends on the problem. For a small control task like the
cart-pole, copying every 10 to 20 steps works; for the convolutional networks used
on Atari, the target is frozen for on the order of 10,000 steps. (Steps, not
episodes — a common and costly confusion.) A softer variant, **Polyak averaging**,
avoids the abrupt copy entirely: instead of $\mathbf{w}^- \gets \mathbf{w}$ every
$C$ steps, blend a little of the online weights into the target on _every_ step,

$$
\mathbf{w}^- \;\gets\; \tau\, \mathbf{w} + (1 - \tau)\, \mathbf{w}^-,
\qquad \tau \ll 1,
$$

so the target always lags but by a small, smoothly-shrinking gap rather than jumping
in discrete leaps.[^gd-polyak]

## The full DQN algorithm

Replay and the target network are the two additions; everything else is standard
value-based RL. The complete loop:

```algorithm
caption: $\textsc{Deep-Q-Network}$ — value-based control with replay and a target network
input: sync period $C$, discount $\gamma$, exploration $\varepsilon$
initialize online weights $\mathbf{w}$ randomly
$\mathbf{w}^- \gets \mathbf{w}$ // target weights
$D \gets \emptyset$ // empty replay buffer
loop
  observe state $s$
  with probability $\varepsilon$ choose random action $a$
  else $a \gets \arg\max_{a'} Q(s, a'; \mathbf{w})$
  execute $a$, observe reward $r$ and next state $s'$
  store $(s, a, r, s')$ in $D$
  sample a minibatch of transitions from $D$
  for each sampled $(s, a, r, s')$ do
    if $s'$ is terminal then
      $y \gets r$
    else
      $y \gets r + \gamma \max_{a'} Q(s', a'; \mathbf{w}^-)$
  take a gradient step on $\tfrac{1}{2}\,[y - Q(s, a; \mathbf{w})]^2$ w.r.t. $\mathbf{w}$
  every $C$ steps: $\mathbf{w}^- \gets \mathbf{w}$ // sync target
```

Two implementation details matter. **Terminal states must be grounded to
zero**: when $s'$ is terminal there is no future, so the target is just $r$, and
forgetting this lets phantom future value leak in and destabilize training.[^gd-target]
And the network outputs a full vector of Q-values, so the prediction for the update
is the entry $Q(s, a; \mathbf{w})$ at the action actually taken, gathered from that
vector.

The original DQN paper (Mnih et al., 2013) introduced the replay buffer; the 2015
follow-up added the target network, and that two-mechanism version — sometimes
called Nature DQN — is the one described above and the baseline every later
improvement builds on.[^sb-dqn]

## A minibatch of targets, worked end to end

Take $\gamma = 0.99$ and a minibatch of four transitions sampled from the buffer.
For each we need the frozen target network's Q-values at the next state $s'$, the
online network's prediction $Q(s, a; \mathbf{w})$ at the action actually taken, and
a terminal flag. Suppose the target network reports these next-state values, and the
online network these predictions:

| $i$ | $r$ | terminal? | $\max_{a'} Q(s', a'; \mathbf{w}^-)$ | $Q(s, a; \mathbf{w})$ |
| --- | --- | --- | --- | --- |
| 1 | $0$ | no | $8.0$ | $6.5$ |
| 2 | $+1$ | no | $12.0$ | $11.2$ |
| 3 | $-1$ | yes | — | $-0.4$ |
| 4 | $0$ | no | $5.0$ | $5.6$ |

The target for a non-terminal transition is $y = r + \gamma \max_{a'} Q(s', a';
\mathbf{w}^-)$; for the terminal transition it is just $y = r$, with no bootstrap
term. Reading down the table:

$$
\begin{aligned}
y_1 &= 0 + 0.99 \times 8.0 = 7.92, & \delta_1 &= 7.92 - 6.5 = 1.42, \\
y_2 &= 1 + 0.99 \times 12.0 = 12.88, & \delta_2 &= 12.88 - 11.2 = 1.68, \\
y_3 &= -1 \quad(\text{terminal, no bootstrap}), & \delta_3 &= -1 - (-0.4) = -0.60, \\
y_4 &= 0 + 0.99 \times 5.0 = 4.95, & \delta_4 &= 4.95 - 5.6 = -0.65.
\end{aligned}
$$

The minibatch loss is the mean squared TD error, $\tfrac{1}{4}\sum_i \delta_i^2 =
\tfrac{1}{4}(1.42^2 + 1.68^2 + 0.60^2 + 0.65^2) = \tfrac{1}{4}(2.02 + 2.82 + 0.36 +
0.42) = 1.40$. The gradient for each transition is $-\delta_i \nabla_{\mathbf{w}}
Q(s_i, a_i; \mathbf{w})$, so transitions 1 and 2 raise their predicted values (the
target sat above the prediction) while 3 and 4 lower theirs. Two details this makes
concrete: transition 3 shows the terminal grounding — had we bootstrapped a phantom
$+\gamma \max Q$ there, $y_3$ would have carried future value the episode does not
have, and that leak is a classic source of divergence. And every gradient flows only
through the $Q(s, a; \mathbf{w})$ column; the target column is a constant, computed
from $\mathbf{w}^-$ and detached.

$$
% caption: One DQN minibatch. Each transition's frozen-target value and reward form
% the label $y$; the online prediction is subtracted to give the TD error $\delta$,
% whose square is averaged into the loss. The terminal transition (row 3) drops the
% bootstrap term entirely.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  row/.style={draw, minimum width=13mm, minimum height=7mm, align=center, font=\scriptsize},
  lab/.style={font=\scriptsize, text=black}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[lab] at (0,1.7) {y = r + g max};
  \node[lab] at (2.6,1.7) {Q(s, a; w)};
  \node[lab, text=red] at (5.0,1.7) {TD error d};
  \foreach \i/\y/\q/\d in {1/7.92/6.5/1.42, 2/12.88/11.2/1.68, 3/{-1.0}/{-0.4}/{-0.60}, 4/4.95/5.6/{-0.65}} {
    \pgfmathsetmacro{\yy}{1.0 - \i*0.85}
    \node[row, draw=acc, text=acc] at (0,\yy) {\y};
    \node[row] at (2.6,\yy) {\q};
    \node[row, draw=red, text=red] at (5.0,\yy) {\d};
    \draw[->, black] (0.7,\yy) -- (1.9,\yy);
    \draw[->, red] (3.3,\yy) -- (4.3,\yy);
  }
  \node[lab, align=center] at (7.6,-1.7) {mean of d-squared\\loss = 1.40};
  \draw[->, red] (5.7,-1.2) -- (6.5,-1.6);
\end{tikzpicture}
$$

## The Atari architecture and results

The demonstration that made DQN famous is Sutton and Barto's chosen case study: a
single agent, with no game-specific features, learning to play dozens of Atari 2600
games directly from pixels.[^sb-atari] The setup is worth stating precisely, because
its choices recur throughout deep RL.

**Input.** Each raw frame is $210 \times 160$ pixels with 128 colors at 60 Hz. DQN
preprocesses every frame down to an $84 \times 84$ array of luminance values, then
**stacks the four most recent frames** so the network input has dimension
$84 \times 84 \times 4$. Stacking matters: a single frame does not reveal velocity
or direction — is the ball rising or falling? — and four stacked frames restore
enough of that to make the state approximately Markov.

$$
% caption: Frame stacking restores motion. A single Atari frame shows a ball at one
% position but not its velocity; stacking the four most recent luminance frames as
% the network's channels lets a convolution read direction and speed from the
% offsets between them, making the stacked state approximately Markov.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  % four stacked frames, offset ball
  \foreach \i/\bx/\by in {0/0.4/0.4, 1/0.6/0.7, 2/0.8/1.0, 3/1.0/1.3} {
    \begin{scope}[xshift={\i*0.35cm}, yshift={\i*0.35cm}]
      \draw[black, fill=white] (0,0) rectangle (1.6,1.6);
      \fill[acc] (\bx,\by) circle (2.2pt);
    \end{scope}
  }
  \node[font=\scriptsize, black, align=center] at (1.8,-0.55) {4 stacked frames (84x84x4)};
  \draw[->, thick, black] (3.6,1.4) -- (5.0,1.4);
  \node[draw, minimum width=16mm, minimum height=10mm, align=center, font=\scriptsize] (cnn) at (6.0,1.4) {conv reads\\the of\/fsets};
  \node[acc, font=\scriptsize, anchor=west, align=left] at (7.1,1.4) {velocity and\\direction recovered};
\end{tikzpicture}
$$

**Network.** Three convolutional layers extract spatial features, followed by a
fully-connected hidden layer and a linear output. Concretely: a first conv layer
producing 32 feature maps of $20 \times 20$, a second producing 64 of $9 \times 9$,
a third producing 64 of $7 \times 7$, each with a ReLU nonlinearity; the $3{,}136$
units of the last conv layer feed a fully-connected layer of 512 units, which
connects to up to 18 outputs — one Q-value per possible Atari action. This is the
state-in, values-out design at scale.

$$
% caption: The Atari DQN architecture. Four stacked $84\times84$ frames pass through
% three convolutional layers, a fully-connected layer of 512 units, and a linear head
% emitting one Q-value per action — the whole map from raw pixels to $Q(s,a;\mathbf{w})$
% learned end to end.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  layer/.style={draw, minimum width=15mm, minimum height=14mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[layer] (in)   at (0,0)    {input\\84x84x4\\(4 frames)};
  \node[layer] (c1)   at (2.2,0)  {conv 1\\32 maps\\20x20};
  \node[layer] (c2)   at (4.4,0)  {conv 2\\64 maps\\9x9};
  \node[layer] (c3)   at (6.6,0)  {conv 3\\64 maps\\7x7};
  \node[layer] (fc)   at (8.8,0)  {FC\\512\\units};
  \node[layer, draw=acc, text=acc] (out) at (11.2,0) {output\\Q per\\action};
  \draw[->, black, thick] (in) -- (c1);
  \draw[->, black, thick] (c1) -- (c2);
  \draw[->, black, thick] (c2) -- (c3);
  \draw[->, black, thick] (c3) -- (fc);
  \draw[->, acc, thick] (fc) -- (out);
  \node[font=\scriptsize, black] at (3.3,-1.4) {convolutional feature extraction};
  \node[font=\scriptsize, acc] at (10.0,-1.4) {value head};
\end{tikzpicture}
$$

**Training choices.** The reward is clipped to its sign — $+1$ when the game score
rises, $-1$ when it falls, $0$ otherwise — so one step size works across games whose
raw scores span wildly different scales. Behavior is $\varepsilon$-greedy with
$\varepsilon$ decaying linearly over the first million frames and then held low. The
optimizer is RMSprop over mini-batches of 32, and the TD error itself is **clipped to
$[-1, 1]$**, a further stabilizer on top of replay and the target network. The
semi-gradient Q-learning update the paper uses is precisely the one derived above.

**Results.** The same architecture, the same hyperparameters, and the same raw pixel
input were applied to 49 different games, with only the network weights reset per
game.[^sb-atari] Learning ran for 50 million frames per game — roughly 38 days of
game experience. Measured against a professional human tester, DQN played at or above
human level on **29 of the 46** games evaluated, and beat the best prior (linear,
feature-engineered) reinforcement-learning systems on all but 6. What made it a
landmark is the _uniformity_ rather than any single score: one learning system, no
per-game engineering, reaching human-competitive play across games as different as
Breakout, Pong, and Space Invaders. The games it failed — Montezuma's Revenge,
where DQN scored about as well as a random player — were the ones demanding long-horizon
planning beyond what one-step Q-learning was built for, a gap later chapters take up.
This lesson built the core value-based deep agent. The three refinements that make it
the standard modern one — Double DQN for the maximization bias, dueling networks, and
prioritized experience replay, plus the Rainbow agent that combines them — continue in
[DQN Improvements: Double, Dueling, and Prioritized Replay](/reinforcement-learning/deep-rl/dqn-improvements).

[^gd-sampled]: **Morales**, _Grokking Deep Reinforcement Learning_, Ch. 8 — the kind of feedback deep-RL agents deal with: high-dimensional and continuous state spaces (an Atari frame is $210\times160\times3$, a state count larger than the atoms in the observable universe) that make exhaustive tabular sampling impossible and force function approximation.
[^sb-dqn]: **Sutton & Barto**, _Reinforcement Learning: An Introduction_ (2nd ed.), §16.5 — Mnih et al.'s deep Q-network combining Q-learning with a deep convolutional network; and **Morales**, Ch. 9 — DQN as the algorithm that "started a series of research innovations that mark the history of RL," introduced 2013 (replay) and completed 2015 (target network).
[^gd-funcapprox]: **Morales**, §8.2 — Introduction to function approximation for RL: replacing tabular lookups with a parametric function so that updating one state-action value generalizes to similar ones, and the efficiency gain of learning underlying relationships from fewer samples.
[^gd-arch]: **Morales**, §8.3, "Selecting a neural network architecture" — the state-action-in / value-out design versus the more efficient state-in / values-out design, which returns all action values in one forward pass and suits epsilon-greedy and softmax policies.
[^gd-objective]: **Morales**, §8.3, "Selecting what to optimize" — the ideal objective $L_i(\theta) = \mathbb{E}_{s,a}[(q_\ast(s,a) - Q(s,a;\theta))^2]$ and the argument that we cannot use it because we have neither the optimal action-value function nor an optimal policy to sample it from; we must alternate policy evaluation and improvement as in generalized policy iteration.
[^gd-target]: **Morales**, §8.3, the Q-learning target box and "I Speak Python" on the Q-learning target — the off-policy TD target $y = r + \gamma \max_{a'} Q(s', a'; \theta)$, the gradient flowing only through the predicted value (the target must be detached / treated as a constant), and grounding terminal states to zero.
[^gd-wrong]: **Morales**, §8.3, "Things that could (and do) go wrong" — the non-stationary-target problem (targets computed from the network being optimized) and the correlated / non-IID online data, the two instabilities that DQN addresses.
[^gd-iid]: **Morales**, Ch. 9, "Common problems in value-based deep RL" and the IID boil-it-down — samples from a trajectory are correlated (not independent) and their distribution drifts with the improving policy (not identically distributed), violating the assumptions optimization methods rely on.
[^gd-replay]: **Morales**, Ch. 9, "Using experience replay" — the replay buffer $D = \{e_1, \dots, e_M\}$ of transitions, uniform mini-batch sampling to decorrelate data and reuse experience, buffer sizes of $10^4$–$10^6$ with oldest-first eviction, and the history note crediting Lin (1992).
[^gd-targetnet]: **Morales**, Ch. 9, "Using target networks" and the target-network gradient box — the frozen target weights $\theta^-$, the update differing from NFQ only in the age of the weights used for the target, and refresh frequencies (10–20 steps for cart-pole, up to 10,000 for Atari).
[^gd-polyak]: **Morales**, Ch. 10, "Continuously updating the target network" — Polyak averaging $\theta^- \gets \tau\theta + (1-\tau)\theta^-$ as a smooth alternative to freezing and copying the target network every $C$ steps.
[^sb-atari]: **Sutton & Barto**, §16.5 — Human-level Video Game Play: preprocessing to $84\times84$ luminance and stacking four frames ($84\times84\times4$) for Markov-ness; three convolutional layers (32 $20\times20$, 64 $9\times9$, 64 $7\times7$, ReLU), a 512-unit fully-connected layer, and up to 18 action outputs; reward-sign clipping, error clipping to $[-1,1]$, RMSprop, linearly-decayed epsilon; 49 games at 50M frames each with weights reset per game; human-level play on 29 of 46 games.