---
title: Dopamine and the TD Error
module: Reinforcement Learning in Minds and Brains
moduleNumber: 6
lessonNumber: 3
order: 603
summary: >
  The TD error was invented as an algorithm; a decade later it turned out to
  closely describe the firing of the brain's dopamine neurons.
  We follow Schultz's experiments — dopamine fires at an unpredicted reward,
  shifts to the earliest predictive cue, and dips below baseline when a predicted
  reward is withheld — and match each result to the TD error term by term. We
  then read the basal ganglia as a neural actor–critic with dopamine as its
  shared training signal, and close on addiction as a hijacking of that signal.
topics: [Minds and Brains]
sources:
  - book: Sutton & Barto
    ref: "Ch. 15 — Neuroscience; §15.3 The Reward Prediction Error Hypothesis; §15.4 Dopamine"
  - book: Sutton & Barto
    ref: "§15.5 Experimental Support; §15.6 TD Error/Dopamine Correspondence; §15.7 Neural Actor–Critic; §15.12 Addiction"
---

Most of the ideas in these notes were built for machines. The
[TD error](/reinforcement-learning/tabular-methods/temporal-difference-learning)
$\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)$ was derived from optimal
control and dynamic programming, years before anyone connected it to a brain.
Then, in the 1990s, neuroscientists recording from single neurons in the
midbrain of monkeys found a signal that behaves — in trial after trial, detail
after detail — exactly like $\delta_t$. The chemical carrying that signal is
**dopamine**, and the match between a quantity engineers wrote down and a signal
evolution built is one of the sharpest points of contact between machine
learning and biology.[^sb-intro]

A computational idea _predicted_ a biological measurement before the biology was
understood. That the prediction held — and with the precision it did — is the
strongest evidence we have that the reinforcement-learning framing captures
something real about how animals learn from reward.

## The reward prediction error hypothesis

Start with the claim, stated plainly.[^sb-rpe]

> **Definition (Reward prediction error hypothesis).** One function of the
> **phasic** activity of dopamine-producing neurons is to broadcast a **reward
> prediction error** (RPE): the difference between an old and a new estimate of
> the expected future reward. The phasic dopamine signal at time $t$ behaves like
> the TD error $\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)$, not like the
> raw reward $R_{t+1}$.

Two words carry the load. **Phasic** activity is a brief burst of spikes lasting
tens of milliseconds, in contrast to the neuron's slow, steady **tonic**
background firing. A **prediction error** measures the gap between what was
expected and what arrived: positive when reward beats expectation, negative when
it falls short, zero when reality matches the forecast exactly. The hypothesis
says the burst of dopamine is not a report that reward _happened_ — it is a
report that reward happened _more_ (or _less_) than predicted.[^sb-signals]

The distinction matters because it contradicts the older, intuitive view. For
decades the assumption was that dopamine simply signals reward: good things
happen, dopamine fires. The RPE hypothesis says something stricter and stranger.
A fully predicted reward — one your value function already saw coming — produces
_no_ dopamine burst at all, because there is no error to report. The reward is
still delivered, still consumed; the neuron stays silent because $\delta_t = 0$.

$$
% caption: The reward prediction error hypothesis. A dopamine neuron's phasic
% burst tracks the TD error $\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)$, not
% the raw reward $R_{t+1}$; a fully predicted reward produces no burst because the
% error is zero.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=26mm, minimum height=12mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[box] (rew) at (0,1.4)  {reward $R$\\(what arrived)};
  \node[box] (pred) at (0,-1.4) {prediction $V$\\(what was expected)};
  \node[box, draw=acc, text=acc, thick] (err) at (4.8,0) {TD error\\(surprise)};
  \node[box, draw=red, text=red, thick] (dop) at (9.4,0) {dopamine\\burst};
  \draw[->, acc, thick] (rew) -- (err);
  \draw[->, acc, thick] (pred) -- (err) node[midway, below left, font=\scriptsize] {subtract};
  \draw[->, red, thick] (err) -- (dop) node[midway, above, font=\scriptsize] {behaves like};
\end{tikzpicture}
$$

The hypothesis was first stated, in not quite these words, by Montague, Dayan,
and Sejnowski in 1996, who showed that the TD error accounts for many features of
the phasic activity of dopamine neurons. The experiments that inspired it came a
few years earlier, from the laboratory of Wolfram Schultz. Those experiments are
the heart of the story.[^sb-rpe]

## What dopamine neurons do: Schultz's experiments

Schultz and colleagues recorded from individual dopamine neurons — cells whose
bodies sit in two midbrain clusters, the substantia nigra pars compacta and the
ventral tegmental area — while monkeys learned simple tasks for a drop of apple
juice.[^sb-dopamine] The recordings sort into three signature results, and
together they are the fingerprint of the TD error.

**One: unpredicted reward.** Give the monkey a drop of juice it did not expect —
no cue, no warning — and the dopamine neurons fire a sharp phasic burst at the
moment the juice arrives. Reward beats expectation, so $\delta_t > 0$, and the
neuron reports the positive surprise.

**Two: the shift to the earliest predictor.** Now precede the juice, every trial,
with a light or a tone. At first the burst still fires at the juice. But as the
animal learns that the cue predicts reward, the burst _migrates_: it shrinks at
the time of the juice and grows at the time of the cue, until — once learning is
complete — the neuron fires at the **cue** and stays silent at the juice.
Schultz's group ran this across a chain of predictors, adding an instruction cue
a full second before the trigger cue, and the response marched all the way back
to the _earliest_ reliable predictor, dropping off the later ones as it went.
The reward, when it finally comes, is now fully predicted, so it draws no
response. The surprise has moved to the first moment the world tips its hand.

**Three: omission.** After the animal has learned the cue, deliver the cue and
then _withhold_ the juice. At the cue, the dopamine neurons fire as usual. But at
the exact time the juice was due — with no external stimulus to mark that moment —
the neurons' firing **dips below baseline**, a brief pause in the tonic
background. The predicted reward did not arrive, so the prediction error is
negative, $\delta_t < 0$, and the neuron reports it by going quiet. The animal
was somehow keeping time; the dip appears at the moment reward was expected, cued
by nothing but the internal clock.[^sb-experiments]

$$
% caption: Schultz's three signatures of dopamine firing, the fingerprint of the
% TD error. Top: an unpredicted reward drives a burst ($\delta > 0$). Middle:
% once a cue predicts the reward, the burst shifts to the cue and the (now
% predicted) reward draws nothing ($\delta = 0$ at reward). Bottom: a predicted
% reward that is omitted drives the firing below baseline at the expected time
% ($\delta < 0$).
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % ===== PANEL 1: unpredicted reward =====
  \begin{scope}[yshift=0cm]
    \draw[black] (0,0) -- (6.6,0);
    \draw[black, dashed] (0,0) -- (6.6,0);
    % baseline noise
    \foreach \x in {0.3,0.7,1.1,1.6,2.2,2.7,4.9,5.4,5.9,6.3} \draw[black] (\x,0) -- (\x,0.14);
    % burst at reward
    \foreach \x/\h in {3.7/1.3, 3.85/1.55, 4.0/1.1, 4.15/0.7} \draw[red, thick] (\x,0) -- (\x,\h);
    \node[anchor=west, font=\scriptsize] at (6.75,0) {no cue};
    \node[red, anchor=south, font=\scriptsize] at (3.9,1.6) {burst};
    \node[anchor=north, font=\scriptsize] at (3.9,-0.08) {reward};
    \node[anchor=east, align=center, font=\footnotesize, text=black] at (-0.15,0.5) {unpredicted\\reward};
    \node[acc, anchor=west, font=\scriptsize] at (6.75,0.5) {delta $>$ 0};
  \end{scope}
  % ===== PANEL 2: predicted reward =====
  \begin{scope}[yshift=-2.6cm]
    \draw[black] (0,0) -- (6.6,0);
    \foreach \x in {0.3,0.7,2.6,3.1,3.6,5.3,5.8,6.3} \draw[black] (\x,0) -- (\x,0.14);
    % burst at cue
    \foreach \x/\h in {1.5/1.3, 1.65/1.55, 1.8/1.1, 1.95/0.7} \draw[red, thick] (\x,0) -- (\x,\h);
    \node[anchor=north, font=\scriptsize] at (1.7,-0.08) {cue};
    \node[anchor=north, font=\scriptsize] at (4.4,-0.08) {reward};
    \draw[black] (4.4,0) -- (4.4,0.14);
    \node[red, anchor=south, font=\scriptsize] at (1.7,1.6) {burst};
    \node[anchor=east, align=center, font=\footnotesize, text=black] at (-0.15,0.5) {predicted\\reward};
    \node[acc, anchor=west, font=\scriptsize] at (6.75,0.5) {delta = 0 at reward};
  \end{scope}
  % ===== PANEL 3: omitted reward =====
  \begin{scope}[yshift=-5.2cm]
    \draw[black] (0,0) -- (6.6,0);
    \draw[black, dashed] (0,0) -- (6.6,0);
    \foreach \x in {0.3,0.7,2.6,3.1,5.3,5.8,6.3} \draw[black] (\x,0) -- (\x,0.14);
    % burst at cue
    \foreach \x/\h in {1.5/1.3, 1.65/1.55, 1.8/1.1, 1.95/0.7} \draw[red, thick] (\x,0) -- (\x,\h);
    \node[anchor=north, font=\scriptsize] at (1.7,-0.08) {cue};
    % dip below baseline at expected reward time -- a gap in firing
    \draw[acc, thick] (4.05,-0.35) -- (4.05,-0.12) -- (4.75,-0.12) -- (4.75,-0.35);
    \node[acc, anchor=north, font=\scriptsize] at (4.4,-0.4) {pause below baseline};
    \node[anchor=south, font=\scriptsize] at (4.4,0.18) {no reward};
    \node[red, anchor=south, font=\scriptsize] at (1.7,1.6) {burst};
    \node[anchor=east, align=center, font=\footnotesize, text=black] at (-0.15,0.5) {omitted\\reward};
    \node[acc, anchor=west, font=\scriptsize] at (6.75,0.5) {delta $<$ 0};
  \end{scope}
\end{tikzpicture}
$$

Read the three panels together and the pattern is unmistakable. Dopamine fires
for reward only when reward is a surprise; the response chases the surprise
backward in time to the earliest cue that removes it; and when a promised reward
fails to appear, the signal turns negative on schedule. Schultz drew exactly this
conclusion: dopamine neurons respond to unpredicted rewards, to the earliest
predictors of reward, and _decrease below baseline_ when a predicted reward fails
to arrive. The firing pattern is the behavior of the TD error.[^sb-experiments]

## From correlation to cause

Schultz's recordings are correlational. An electrode listens; it does not speak.
The neuron fires like a TD error, but the experiment cannot show that the firing
_causes_ the learning it accompanies — the burst and the learning could both be
downstream of some third thing. Turning the correlation into a causal claim needs
a manipulation: create a dopamine burst on demand and check whether learning
follows.

**Optogenetics** supplies the tool. Introduce a light-sensitive ion channel into
dopamine neurons and a flash of laser light drives them to fire on a
millisecond schedule the experimenter controls. Tsai and colleagues (2009) showed
the basic effect — mice made to receive optogenetic dopamine bursts on one side
of a chamber came to prefer that side — establishing that an artificial burst is
reinforcing. The sharper test came from Steinberg and colleagues (2013), who
aimed the manipulation at a case where the RPE hypothesis makes a
counter-intuitive prediction: **blocking**.[^sb-optogenetic]

Blocking is the phenomenon from the
[psychology lesson](/reinforcement-learning/minds-and-brains/psychology-of-reinforcement).
Train an animal so that cue A fully predicts reward. Then present A together with
a new cue X, still followed by the same reward. The animal learns little or
nothing about X — A already predicted the reward, so there is no prediction error
left for X to soak up. In TD terms the reward on the compound trials is fully
predicted, $\delta \approx 0$, and with no error there is no learning. The RPE
hypothesis makes a strong prediction: if you _add_ a dopamine burst at the
otherwise-predicted reward — manufacturing an error where none existed — the
animal should now learn about X after all. Steinberg's group did exactly this.
Optogenetically bursting dopamine neurons at the time of the fully-predicted
reward **unblocked** learning: animals that received the extra burst acquired a
response to X that control animals did not. The same manipulation, applied on
extinction trials where reward was omitted, sustained responding that would
otherwise have faded.[^sb-optogenetic]

$$
% caption: Optogenetic unblocking (Steinberg et al., 2013). Left: normal
% blocking. Cue A already predicts the reward, so on A+X compound trials the TD
% error at reward is near zero and the animal learns little about X. Right: a
% laser-driven dopamine burst is added at the (predicted) reward, forcing a
% positive error the model says should drive learning about X. The animal now
% learns X -- the burst causes learning.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  cue/.style={draw, minimum width=9mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % ===== LEFT: blocking =====
  \begin{scope}
    \node[anchor=south, font=\footnotesize] at (1.5,2.2) {blocking};
    \node[cue] (a1) at (0.4,1.4) {A};
    \node[cue] (x1) at (1.4,1.4) {X};
    \node[anchor=west, font=\scriptsize] at (2.1,1.4) {compound cue};
    \draw[->, black] (1.5,1.0) -- (1.5,0.35);
    \node[anchor=west, font=\scriptsize] at (0.1,0.05) {reward (fully predicted by A)};
    \draw[black] (0,-0.9) -- (3.0,-0.9);
    \foreach \x in {0.5,0.9,2.4,2.8} \draw[black] (\x,-0.9) -- (\x,-0.78);
    \node[anchor=north, align=center, font=\scriptsize] at (1.5,-1.05) {error near 0\\little learned about X};
  \end{scope}
  % ===== RIGHT: unblocking =====
  \begin{scope}[xshift=6.2cm]
    \node[anchor=south, font=\footnotesize] at (1.5,2.2) {optogenetic unblocking};
    \node[cue] (a2) at (0.4,1.4) {A};
    \node[cue] (x2) at (1.4,1.4) {X};
    \node[anchor=west, font=\scriptsize] at (2.1,1.4) {compound cue};
    \draw[->, black] (1.5,1.0) -- (1.5,0.35);
    \node[anchor=west, font=\scriptsize] at (0.1,0.05) {reward, plus laser burst};
    \draw[black] (0,-0.9) -- (3.0,-0.9);
    \foreach \x/\h in {1.25/0.55, 1.4/0.7, 1.55/0.45} \draw[red, thick] (\x,-0.9) -- (\x,{-0.9+\h});
    \node[red, anchor=west, font=\scriptsize] at (1.75,-0.5) {forced burst};
    \node[anchor=north, align=center, font=\scriptsize] at (1.5,-1.05) {positive error\\X is learned};
  \end{scope}
\end{tikzpicture}
$$

This is the argument the recordings could not make. A burst that the animal's own
value function should have cancelled, supplied artificially, produces learning on
schedule. Dopamine is not a readout of learning happening elsewhere; the
transient _is_ the teaching signal. Bayer and Glimcher (2005) tightened the
correlational side in parallel, showing the burst is not merely present or absent
but graded: across trials the firing rate of dopamine neurons scales with the
_size_ of the reward prediction error, coding a quantitative RPE rather than a
bare sign.[^sb-quantitative] Their data also exposed an asymmetry that matters
later. Positive errors are coded roughly linearly in firing rate, but negative
errors run into a floor — a neuron cannot fire fewer than zero spikes, and the
tonic baseline is already low, so large negative errors are compressed into a
short pause that saturates quickly.

$$
% caption: Dopamine firing as a quantitative code for the RPE (schematic, after
% Bayer & Glimcher, 2005). Firing rate rises roughly linearly with a positive
% prediction error but flattens for negative errors, floored by the low baseline
% firing rate -- the signal has more dynamic range above baseline than below.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[black, ->] (-3.0,0) -- (3.2,0) node[anchor=north, font=\scriptsize] {prediction error};
  \draw[black, ->] (0,-0.6) -- (0,2.9) node[anchor=east, font=\scriptsize] {f\/iring rate};
  % baseline
  \draw[black, dashed] (-3.0,0.55) -- (3.0,0.55);
  \node[anchor=east, font=\scriptsize, text=black] at (-2.55,0.55) {baseline};
  % transfer curve: floored for negative, linear-ish for positive
  \draw[acc, very thick] (-2.6,0.06) .. controls (-1.4,0.12) and (-0.7,0.35) .. (0,0.55)
                          .. controls (0.9,0.85) and (1.8,1.7) .. (2.8,2.55);
  \node[acc, anchor=west, font=\scriptsize] at (1.7,2.15) {linear when positive};
  \node[anchor=west, font=\scriptsize, text=black] at (-2.9,-0.35) {f\/loored when negative};
\end{tikzpicture}
$$

## The correspondence, term by term

If you take the TD-error equation and run it on a
simple version of Schultz's task, the three dopamine patterns are not extra
assumptions — they are what the one equation _does_ at three stages of learning.
Nothing is added; we read $\delta$ off the definition.

Work through an idealized version of Schultz's
task and the TD error reproduces all three signatures from the definition alone.
Assume the animal has already learned to act, so its only job is prediction:
learn a value function $V$ for the fixed sequence of states in a trial. Use
TD(0), a value table initialized to zero, and — following Montague et al. — let
each moment of a trial be its own state, so the reward-predicting stimuli occupy
a run of states leading up to the one where reward is delivered. Take $\gamma
\approx 1$ so we can ignore discounting, and let the reward at the rewarding
state be some positive $R^\star$.[^sb-corr]

Two states matter. The **earliest reward-predicting state** is the first state in
a trial that reliably predicts the reward to come — the analogue of Schultz's
instruction cue. The **latest reward-predicting state** is the one immediately
before the rewarding state. Now trace $\delta$ through learning.

$$
% caption: The TD error $\delta_t$ across learning, reproducing all three
% dopamine signatures. Early: $V \approx 0$, so $\delta$ spikes only at the
% reward. Complete: $V$ has risen to $R^\star$ across the predicting states, so
% $\delta$ spikes at the earliest cue and is zero at the (now predicted) reward.
% Omitted: with $V$ high but no reward, $\delta$ goes negative at the reward's
% expected time.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % time axis references
  \def\xearly{1.4}   % earliest predictor
  \def\xrew{5.4}     % reward time
  % ===== ROW: early in learning =====
  \begin{scope}[yshift=0cm]
    \draw[black] (0,0) -- (6.4,0);
    \foreach \x/\h in {5.4/1.4, 5.55/1.55, 5.7/1.05} \draw[red, thick] (\x,0) -- (\x,\h);
    \node[red, anchor=south, font=\scriptsize] at (5.5,1.6) {spike};
    \node[anchor=east, font=\footnotesize] at (-0.15,0.4) {early};
    \node[anchor=north, font=\scriptsize] at (1.4,-0.08) {cue};
    \node[anchor=north, font=\scriptsize] at (5.4,-0.08) {reward};
  \end{scope}
  % ===== ROW: learning complete =====
  \begin{scope}[yshift=-2.4cm]
    \draw[black] (0,0) -- (6.4,0);
    \foreach \x/\h in {1.4/1.4, 1.55/1.55, 1.7/1.05} \draw[red, thick] (\x,0) -- (\x,\h);
    \node[red, anchor=south, font=\scriptsize] at (1.5,1.6) {spike};
    \node[anchor=south, font=\scriptsize] at (5.4,0.1) {(nothing)};
    \node[anchor=east, font=\footnotesize] at (-0.15,0.4) {complete};
    \node[anchor=north, font=\scriptsize] at (1.4,-0.08) {cue};
    \node[anchor=north, font=\scriptsize] at (5.4,-0.08) {reward};
  \end{scope}
  % ===== ROW: reward omitted =====
  \begin{scope}[yshift=-4.8cm]
    \draw[black] (0,0) -- (6.4,0);
    \foreach \x/\h in {1.4/1.4, 1.55/1.55, 1.7/1.05} \draw[red, thick] (\x,0) -- (\x,\h);
    % negative spike at reward time
    \foreach \x/\h in {5.4/-1.3, 5.55/-1.05} \draw[acc, thick] (\x,0) -- (\x,\h);
    \node[red, anchor=south, font=\scriptsize] at (1.5,1.6) {spike};
    \node[acc, anchor=north, font=\scriptsize] at (5.5,-1.35) {negative};
    \node[anchor=east, font=\footnotesize] at (-0.15,0.4) {omitted};
    \node[anchor=north, font=\scriptsize] at (1.4,-0.08) {cue};
    \node[anchor=south, font=\scriptsize] at (5.4,0.15) {no reward};
  \end{scope}
\end{tikzpicture}
$$

**Early in learning**, every $V$ is still zero. The TD error available at time
$t$ is $\delta_{t-1} = R_t + V_t - V_{t-1}$, which stays zero at every transition
until the reward arrives, where it becomes $R^\star + 0 - 0 = R^\star$. So
$\delta$ spikes only at the reward — the unpredicted-reward panel, a burst for
juice the value function did not see coming.

**As learning proceeds**, TD(0) updates run at every transition, and the value
increases spread _backward_ from the rewarding state to its predecessors, exactly
as bootstrapping propagates information one step at a time. When learning is
complete, every reward-predicting state has value $R^\star$. Now check the TD
error at each transition. From one predicting state to the next,
$\delta = R + V_t - V_{t-1} = 0 + R^\star - R^\star = 0$. From the latest
predicting state to the reward, $\delta = R^\star + 0 - R^\star = 0$ — the reward
is fully predicted, so it cancels, matching the silence at a predicted reward. But
on the transition _into_ the earliest predicting state, whose predecessors still
have value zero, $\delta = R + V_t - V_{t-1} = 0 + R^\star - 0 = R^\star$. The
positive error has migrated to the earliest predictor — the shift Schultz saw.

**If the reward is then omitted** after learning, the value of the latest
predicting state is now too high. At the reward's expected time,
$\delta = R + V_t - V_{t-1} = 0 + 0 - R^\star = -R^\star$: the TD error goes
negative, driven below zero by nothing but the mismatch between a confident
prediction and an empty world. That is the dip below baseline.[^sb-corr]

All three signatures fall out of one equation and one learning rule. The reward
term $R_{t+1}$ carries the raw juice; the bootstrapped term $\gamma V(S_{t+1}) -
V(S_t)$ carries the _prediction_, and it is the interplay of the two — reward
minus expectation — that produces the burst-shift-dip pattern. Phasic dopamine is
not signaling $R_{t+1}$. It is signaling $\delta_t$.

The core claim — that phasic dopamine is well described as a TD-style reward
prediction error — has held up across three decades of increasingly refined
experiments.

## Watching the burst migrate: a worked trace

The term-by-term argument shows what the endpoints of learning look like — the
burst at reward before learning, the burst at the cue after. The trip between them
is the interesting part. The migration is
not a jump but a slow crawl, one state per trial, driven by nothing but
repeated application of a single update. Trace it with real numbers.

Fix a short trial. Four states run in sequence,
$s_1 \to s_2 \to s_3 \to s_4$, and reward $R^\star = 1$ arrives on entering
$s_4$; there is no reward on the earlier transitions. Take $\gamma = 1$,
step size $\alpha = 0.5$, and initialize every value to zero. The prediction
problem is the idealized policy evaluation of the last section, stated as a
procedure.

```algorithm
Algorithm (TD(0) prediction over a fixed trial).
initialize $V(s) \gets 0$ for every state $s$
for each trial do
  observe the start state $s_1$
  for each transition $s_t \to s_{t+1}$ in the trial do
    receive reward $r$ (which is $R^\star$ only on entering $s_4$, else $0$)
    compute $\delta \gets r + \gamma\, V(s_{t+1}) - V(s_t)$
    update $V(s_t) \gets V(s_t) + \alpha\, \delta$
  end for
end for
```

The transition into $s_4$ needs a value for $s_4$ to bootstrap from. Because the
rewarding state does not predict any further reward within the trial, its value
stays pinned at $V(s_4) = 0$ throughout — it is a terminal state. So the only
values that move are $V(s_1)$, $V(s_2)$, $V(s_3)$.

**Trial 1.** Every value is zero. On $s_1 \to s_2$,
$\delta = 0 + V(s_2) - V(s_1) = 0$, so nothing changes; the same on
$s_2 \to s_3$. On the final transition $s_3 \to s_4$, reward appears:
$\delta = 1 + V(s_4) - V(s_3) = 1 + 0 - 0 = 1$. The update lifts
$V(s_3) \gets 0 + 0.5 \times 1 = 0.5$. One trial, one non-zero error, at the
reward — the unpredicted-reward burst. The value bump lands on $s_3$, the latest
predictor, and nowhere else.

**Trial 2.** Now $V(s_3) = 0.5$. On $s_2 \to s_3$ the error is no longer zero:
$\delta = 0 + V(s_3) - V(s_2) = 0 + 0.5 - 0 = 0.5$, which lifts
$V(s_2) \gets 0.5 \times 0.5 = 0.25$. On $s_3 \to s_4$,
$\delta = 1 + 0 - 0.5 = 0.5$, lifting $V(s_3) \gets 0.5 + 0.5 \times 0.5 = 0.75$.
The error has split: still positive at the reward, but now also positive one step
earlier, at the transition into $s_3$. The burst has begun to move.

**Trial 3.** With $V(s_2) = 0.25$, $V(s_3) = 0.75$: on $s_1 \to s_2$,
$\delta = 0.25$, so $V(s_1) \gets 0.125$; on $s_2 \to s_3$,
$\delta = 0.75 - 0.25 = 0.5$, so $V(s_2) \gets 0.25 + 0.25 = 0.5$; on
$s_3 \to s_4$, $\delta = 1 - 0.75 = 0.25$, so $V(s_3) \gets 0.875$. The error at
the reward has shrunk to $0.25$, while a positive error now reaches all the way
back to the entry into $s_2$. Every trial, the reward error gets smaller and the
earliest positive error moves one state to the left.

Continue and the values climb toward their fixed point $V(s_1) = V(s_2) =
V(s_3) = 1$, at which every within-chain error is zero except the one on entering
$s_1$ — the earliest predictor — and the reward error has vanished. The table
carries the first four trials.

$$
% caption: A TD(0) trace of the burst migrating backward. Values $V(s_1), V(s_2),
% V(s_3)$ after each trial (top rows), and the TD error $\delta$ on each
% transition within the trial (bottom rows), for $R^\star = 1$, $\gamma = 1$,
% $\alpha = 0.5$. The reward error shrinks $1 \to 0.5 \to 0.25 \to \dots$ while
% the earliest non-zero error marches leftward, one state per trial, toward
% $s_1$.
\begin{tikzpicture}[font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \def\rw{0.62} % row height
  \def\cw{2.0}  % col width
  % column headers
  \foreach \c/\lab in {1/{after trial 1}, 2/{after trial 2}, 3/{after trial 3}, 4/{after trial 4}}
    \node[anchor=south, font=\scriptsize] at ({\c*\cw},{1.2*\rw}) {\lab};
  \node[anchor=east, font=\scriptsize\bfseries, rotate=90] at (-1.35,{-\rw}) {values};
  % value rows
  \foreach \r/\lab in {0/{V(s1)}, 1/{V(s2)}, 2/{V(s3)}}
    \node[anchor=east, font=\scriptsize] at (0.35,{-\r*\rw}) {\lab};
  % values: rows s1,s2,s3 ; cols trial1..4
  \foreach \c/\a/\b/\cc in {1/0/0/0.5, 2/0/0.25/0.75, 3/0.125/0.5/0.875, 4/0.31/0.69/0.94} {
    \node at ({\c*\cw},0)        {\a};
    \node at ({\c*\cw},{-\rw})   {\b};
    \node[acc] at ({\c*\cw},{-2*\rw}) {\cc};
  }
  % separating line
  \draw[black] (0.4,{-2.6*\rw}) -- ({4.5*\cw},{-2.6*\rw});
  \node[anchor=east, font=\scriptsize\bfseries, rotate=90] at (-1.35,{-4.9*\rw}) {errors};
  \foreach \r/\lab in {0/{on entry s1}, 1/{on entry s2}, 2/{on entry s3}, 3/{at reward}}
    \node[anchor=east, font=\scriptsize] at (0.35,{-3.4*\rw-\r*\rw}) {\lab};
  % errors per trial: rows (entry s1, entry s2, entry s3, reward)
  \foreach \c/\a/\b/\cc/\dd in {1/{n/a}/0/0/1, 2/{n/a}/0/0.5/0.5, 3/0.25/0.5/0.25/0.25, 4/0.375/0.375/0.19/0.12} {
    \node at ({\c*\cw},{-3.4*\rw})        {\a};
    \node at ({\c*\cw},{-4.4*\rw})   {\b};
    \node at ({\c*\cw},{-5.4*\rw}) {\cc};
    \node[red] at ({\c*\cw},{-6.4*\rw}) {\dd};
  }
\end{tikzpicture}
$$

The bottom block is the model's version of Schultz's raster: read any column and
you see where the error — the dopamine burst — sits on that trial. In trial 1 it
sits entirely at the reward. By trial 4 it has spread across the chain, and its
mass has shifted toward the entry into the earliest states while the reward term
has collapsed toward zero. Left to run, the process converges to a single
positive error at the entry into $s_1$ and silence everywhere after. The famous
migration is this table read left to right.

$$
% caption: The same trace as growing value bars. Across trials the value profile
% over states $s_1, s_2, s_3$ fills in from the right (nearest the reward) and
% rises toward $V = R^\star = 1$; the leftmost non-zero TD error (arrow) walks
% back toward $s_1$ one state per trial.
\begin{tikzpicture}[font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \def\bh{1.5} % max bar height for V=1
  % per trial: value of s1,s2,s3 and the x of the leftmost non-zero error arrow
  \foreach \t/\va/\vb/\vc/\errx in {0/0/0/0.5/2.6, 1/0/0.25/0.75/2.05, 2/0.125/0.5/0.875/0.65, 3/0.31/0.66/0.94/0.65} {
    \begin{scope}[xshift={\t*3.7cm}]
      \draw[black] (0,0) -- (3.0,0);
      % bars for s1,s2,s3 centered at 0.65,1.4,2.15
      \foreach \i/\v in {0/\va, 1/\vb, 2/\vc} {
        \fill[acc!18] ({0.4+\i*0.75},0) rectangle ({0.9+\i*0.75},{\v*\bh});
        \draw[acc] ({0.4+\i*0.75},0) rectangle ({0.9+\i*0.75},{\v*\bh});
      }
      % reward marker at the right end
      \draw[red, thick] (2.75,0) -- (2.75,0.75);
      \node[red, anchor=south, font=\scriptsize] at (2.75,0.75) {R};
      % leftmost-error arrow
      \draw[red, ->, thick] ({\errx},{\bh+0.55}) -- ({\errx},{\bh+0.2});
      % per-state tick labels aligned to bar centers
      \foreach \i/\lab in {0/{s1}, 1/{s2}, 2/{s3}}
        \node[anchor=north, font=\scriptsize] at ({0.65+\i*0.75},-0.05) {\lab};
      \node[anchor=south, font=\scriptsize] at (1.4,{\bh+0.55}) {trial \the\numexpr\t+1\relax};
    \end{scope}
  }
\end{tikzpicture}
$$

Two facts about the crawl are worth stating in their own right. The rate is set
by $\alpha$: with $\alpha = 0.5$ the reward error halves each trial
($1, 0.5, 0.25, \dots$), so the values approach $R^\star$ geometrically rather
than in a single leap. And the _direction_ is forced — information can only move
backward, from a state to its predecessor, because the update to $V(s_t)$ reads
the already-updated estimate at $s_{t+1}$. This backward flow is bootstrapping
itself, and it is why the dopamine response, trial after trial, retreats
toward the earliest cue rather than advancing toward the reward.

## When the reward comes early: a known imperfection

The trace above is clean because the reward always arrives on schedule. Push on
the timing and the simple model cracks in an instructive way. Suppose learning
has converged, so the animal expects reward at a fixed moment, and then on a
probe trial the reward is delivered _earlier_ than usual. Two things should
happen if dopamine tracks the TD error with a complete-serial-compound (CSC)
representation, in which each instant of the trial is its own state.[^sb-mismatch]

The early reward is itself unpredicted at the moment it arrives — no state
predicted reward _there_ — so the TD error is positive and dopamine should burst.
It does; this half matches. But the model also predicts a _second_ event. The
state at the originally-expected time still carries a high value, learned over
many on-time trials. When the trial reaches that moment and no reward appears —
it was already collected early — the CSC model computes a negative error there,
the same omission dip as before, and predicts a pause below baseline. Dopamine
neurons show no such pause (Hollerman and Schultz, 1998). The reward was received;
the animal is not disappointed at the old time, but the model predicts it should be.

$$
% caption: The early-reward mismatch. Top: on a normal trial, reward at the
% expected time draws no error (fully predicted). Bottom: on an early-reward
% probe, the TD error with a CSC representation predicts a burst at the early
% reward (correct) AND a spurious dip at the old expected time (an omission the
% animal does not experience). Real dopamine shows the burst but not the dip.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % ===== TOP: normal timing =====
  \begin{scope}
    \draw[black] (0,0) -- (6.6,0);
    \draw[black, dashed] (4.4,0.35) -- (4.4,1.9);
    \foreach \x/\h in {1.4/1.3, 1.55/1.5, 1.7/1.0} \draw[red, thick] (\x,0) -- (\x,\h);
    \node[anchor=north, font=\scriptsize] at (1.55,-0.08) {cue};
    \node[anchor=north east, font=\scriptsize] at (4.25,-0.08) {expected reward};
    \node[anchor=east, font=\scriptsize] at (4.2,0.95) {(nothing)};
    \node[red, anchor=south, font=\scriptsize] at (1.55,1.55) {burst};
    \node[anchor=east, align=center, font=\footnotesize] at (-0.15,0.55) {normal\\trial};
  \end{scope}
  % ===== BOTTOM: early reward =====
  \begin{scope}[yshift=-3.0cm]
    \draw[black] (0,0) -- (6.6,0);
    \draw[black, dashed] (4.4,0.05) -- (4.4,1.9);
    \foreach \x/\h in {1.4/1.3, 1.55/1.5, 1.7/1.0} \draw[red, thick] (\x,0) -- (\x,\h);
    % burst at early reward
    \foreach \x/\h in {2.8/1.25, 2.95/1.45, 3.1/0.95} \draw[red, thick] (\x,0) -- (\x,\h);
    % spurious predicted dip at old time
    \draw[acc, thick] (4.15,-0.6) -- (4.15,-0.12) -- (4.65,-0.12) -- (4.65,-0.6);
    \node[anchor=north, font=\scriptsize] at (1.55,-0.08) {cue};
    \node[red, anchor=south, font=\scriptsize] at (2.95,1.5) {burst};
    \node[anchor=south west, font=\scriptsize] at (3.25,0.02) {early reward};
    \node[acc, anchor=north, align=center, font=\scriptsize] at (4.4,-0.72) {model predicts a dip\\(animal shows none)};
    \node[anchor=east, align=center, font=\footnotesize] at (-0.15,0.55) {early\\reward};
  \end{scope}
\end{tikzpicture}
$$

The mismatch does not overturn the RPE hypothesis; it locates the seam. The TD
error is right; the _state representation_ fed to it is too rigid. A CSC assigns
an indelible marker to the clock time of the original reward, so the machinery
cannot help predicting an event there. The fixes all soften that marker. Suri and
Schultz (1999) let a reward cancel the internal signals set off by earlier
stimuli, erasing the stale prediction. Ludvig, Sutton, and Kehoe (2008) replaced
the CSC with a **microstimulus** representation — a set of overlapping,
temporally-blurred basis functions rather than one crisp state per instant —
and found it fits the early-reward case and several others better than a CSC.
Daw, Courville, and Touretzky (2006) went further and derived the state
representation itself from a statistical model of the sensory stream. Pan,
Schmidt, Wickens, and Hyland (2005) showed that even keeping the CSC, prolonged
eligibility traces improve the fit. The lesson repeats across all of them: the
error signal is the fixed point of the theory, and the representation feeding it
is where the biology adds detail.

## Where the signal goes

We have matched the phasic dopamine signal to the TD error, from Schultz's
recordings through the causal optogenetic tests to a term-by-term derivation of the
burst, the shift, and the dip. What we have not asked is what the signal _does_ once
it fires — where those axons go, what they change, and which algorithm the wiring is
running. That is the subject of the next part: the basal ganglia read as a neural
actor–critic, the corticostriatal synapse where three signals meet, and addiction as
a hijacking of the same learning signal.

This continues in [Dopamine in the Brain: The Neural Actor–Critic](/reinforcement-learning/minds-and-brains/dopamine-in-the-brain).

[^sb-intro]: **Sutton & Barto**, _Reinforcement Learning: An Introduction_ (2nd ed.), Ch. 15 — introduction: the mounting neuroscience evidence that nervous systems implement algorithms corresponding to reinforcement-learning algorithms, dopamine as the most remarkable point of contact, and the reward prediction error hypothesis of dopamine neuron activity as the result of a convergence of computational RL and neuroscience.
[^sb-rpe]: **Sutton & Barto**, §15.3 — The Reward Prediction Error Hypothesis: the hypothesis (Montague, Dayan, and Sejnowski, 1996) that phasic dopamine activity delivers an error between an old and a new estimate of expected future reward, matching the TD error and not the raw reward, previewing the four features of dopamine activity it explains.
[^sb-signals]: **Sutton & Barto**, §15.1–§15.2 — Neuroscience Basics; Reward Signals, Reinforcement Signals, Values, and Prediction Errors: phasic versus tonic activity, neuromodulators and synaptic plasticity, and the distinction between reward signals $R_t$, value signals $V$/$Q$, and prediction errors, with the TD error $\delta_{t-1} = R_t + \gamma V(S_t) - V(S_{t-1})$ as the reinforcement signal for a TD method.
[^sb-dopamine]: **Sutton & Barto**, §15.4 — Dopamine: dopamine produced in the substantia nigra pars compacta (SNpc) and ventral tegmental area (VTA); its roles in motivation, learning, action-selection, and addiction; the reconciliation of the reward-prediction-error view with the older "dopamine signals reward" view; and the huge axonal arbors making roughly 500,000 synaptic contacts that suit dopamine neurons to broadcasting a scalar reinforcement signal.
[^sb-experiments]: **Sutton & Barto**, §15.5 — Experimental Support for the Reward Prediction Error Hypothesis: Romo and Schultz and Schultz, Apicella, and Ljungberg — dopamine neurons respond to unpredicted reward, shift their phasic response to the earliest reward-predicting cue (Figure 15.2), and drop below baseline at the expected reward time when a predicted reward is omitted (Figure 15.3), with the animal internally timing the omission.
[^sb-corr]: **Sutton & Barto**, §15.6 — TD Error/Dopamine Correspondence: the idealized policy-evaluation task with a CSC representation and TD(0), the earliest and latest reward-predicting states, and the term-by-term computation showing $\delta$ spikes only at reward early in learning, migrates to the earliest predictor once learning is complete ($\delta_{t-1} = 0 + R^\star - 0 = R^\star$), and goes negative ($-R^\star$) at the expected time when reward is omitted (Figure 15.4).
[^sb-mismatch]: **Sutton & Barto**, §15.6 — the reward-earlier-than-expected mismatch (Hollerman and Schultz, 1998) and the refinements that address it: alternative stimulus representations (Suri and Schultz, 1999; Daw, Courville, and Touretzky, 2006; Ludvig, Sutton, and Kehoe, 2008) and prolonged eligibility traces (Pan, Schmidt, Wickens, and Hyland, 2005) that improve the TD fit without refuting the core claim.
[^sb-optogenetic]: **Sutton & Barto**, §15.4 — Dopamine (causal evidence): the optogenetic experiments that establish phasic dopamine as a reinforcement signal rather than a correlate — Tsai et al. (2009) conditioning a place preference with dopamine bursts, and Steinberg et al. (2013, _Nature Neuroscience_), "A causal link between prediction errors, dopamine neurons and learning," replacing the omission pause with an artificial burst to sustain extinction responding and, in the blocking paradigm (§14.2.1), to unblock learning about an otherwise-redundant cue.
[^sb-quantitative]: Bayer, H. M., and Glimcher, P. W. (2005), "Midbrain dopamine neurons encode a quantitative reward prediction error signal," _Neuron_ 47(1):129–141 — dopamine firing rate scales with the magnitude of the reward prediction error, with positive errors coded approximately linearly and negative errors compressed against the low baseline firing rate (a floor on the pause duration).