Dopamine and the TD Error

Most of the ideas in these notes were built for machines. The TD error${\delta }_t=R_{t+1}+\gamma V(S_{t+1})-V(S_t)$ was derived from optimal control and dynamic programming, years before anyone connected it to a brain. Then, in the 1990s, neuroscientists recording from single neurons in the midbrain of monkeys found a signal that behaves — in trial after trial, detail after detail — exactly like ${\delta }_t$. The chemical carrying that signal is dopamine, and the match between a quantity engineers wrote down and a signal evolution built is one of the sharpest points of contact between machine learning and biology.¹

A computational idea predicted a biological measurement before the biology was understood. That the prediction held — and with the precision it did — is the strongest evidence we have that the reinforcement-learning framing captures something real about how animals learn from reward.

The reward prediction error hypothesis

Start with the claim, stated plainly.²

Two words carry the load. Phasic activity is a brief burst of spikes lasting tens of milliseconds, in contrast to the neuron's slow, steady tonic background firing. A prediction error measures the gap between what was expected and what arrived: positive when reward beats expectation, negative when it falls short, zero when reality matches the forecast exactly. The hypothesis says the burst of dopamine is not a report that reward happened — it is a report that reward happened more (or less) than predicted.³

The distinction matters because it contradicts the older, intuitive view. For decades the assumption was that dopamine simply signals reward: good things happen, dopamine fires. The RPE hypothesis says something stricter and stranger. A fully predicted reward — one your value function already saw coming — produces no dopamine burst at all, because there is no error to report. The reward is still delivered, still consumed; the neuron stays silent because ${\delta }_t=0$.

The hypothesis was first stated, in not quite these words, by Montague, Dayan, and Sejnowski in 1996, who showed that the TD error accounts for many features of the phasic activity of dopamine neurons. The experiments that inspired it came a few years earlier, from the laboratory of Wolfram Schultz. Those experiments are the heart of the story.²

What dopamine neurons do: Schultz's experiments

Schultz and colleagues recorded from individual dopamine neurons — cells whose bodies sit in two midbrain clusters, the substantia nigra pars compacta and the ventral tegmental area — while monkeys learned simple tasks for a drop of apple juice.⁴ The recordings sort into three signature results, and together they are the fingerprint of the TD error.

One: unpredicted reward. Give the monkey a drop of juice it did not expect — no cue, no warning — and the dopamine neurons fire a sharp phasic burst at the moment the juice arrives. Reward beats expectation, so ${\delta }_t>0$, and the neuron reports the positive surprise.

Two: the shift to the earliest predictor. Now precede the juice, every trial, with a light or a tone. At first the burst still fires at the juice. But as the animal learns that the cue predicts reward, the burst migrates: it shrinks at the time of the juice and grows at the time of the cue, until — once learning is complete — the neuron fires at the cue and stays silent at the juice. Schultz's group ran this across a chain of predictors, adding an instruction cue a full second before the trigger cue, and the response marched all the way back to the earliest reliable predictor, dropping off the later ones as it went. The reward, when it finally comes, is now fully predicted, so it draws no response. The surprise has moved to the first moment the world tips its hand.

Three: omission. After the animal has learned the cue, deliver the cue and then withhold the juice. At the cue, the dopamine neurons fire as usual. But at the exact time the juice was due — with no external stimulus to mark that moment — the neurons' firing dips below baseline, a brief pause in the tonic background. The predicted reward did not arrive, so the prediction error is negative, ${\delta }_t<0$, and the neuron reports it by going quiet. The animal was somehow keeping time; the dip appears at the moment reward was expected, cued by nothing but the internal clock.⁵

Read the three panels together and the pattern is unmistakable. Dopamine fires for reward only when reward is a surprise; the response chases the surprise backward in time to the earliest cue that removes it; and when a promised reward fails to appear, the signal turns negative on schedule. Schultz drew exactly this conclusion: dopamine neurons respond to unpredicted rewards, to the earliest predictors of reward, and decrease below baseline when a predicted reward fails to arrive. The firing pattern is the behavior of the TD error.⁵

From correlation to cause

Schultz's recordings are correlational. An electrode listens; it does not speak. The neuron fires like a TD error, but the experiment cannot show that the firing causes the learning it accompanies — the burst and the learning could both be downstream of some third thing. Turning the correlation into a causal claim needs a manipulation: create a dopamine burst on demand and check whether learning follows.

Optogenetics supplies the tool. Introduce a light-sensitive ion channel into dopamine neurons and a flash of laser light drives them to fire on a millisecond schedule the experimenter controls. Tsai and colleagues (2009) showed the basic effect — mice made to receive optogenetic dopamine bursts on one side of a chamber came to prefer that side — establishing that an artificial burst is reinforcing. The sharper test came from Steinberg and colleagues (2013), who aimed the manipulation at a case where the RPE hypothesis makes a counter-intuitive prediction: blocking.⁶

Blocking is the phenomenon from the psychology lesson. Train an animal so that cue A fully predicts reward. Then present A together with a new cue X, still followed by the same reward. The animal learns little or nothing about X — A already predicted the reward, so there is no prediction error left for X to soak up. In TD terms the reward on the compound trials is fully predicted, $\delta \approx 0$, and with no error there is no learning. The RPE hypothesis makes a strong prediction: if you add a dopamine burst at the otherwise-predicted reward — manufacturing an error where none existed — the animal should now learn about X after all. Steinberg's group did exactly this. Optogenetically bursting dopamine neurons at the time of the fully-predicted reward unblocked learning: animals that received the extra burst acquired a response to X that control animals did not. The same manipulation, applied on extinction trials where reward was omitted, sustained responding that would otherwise have faded.⁶

This is the argument the recordings could not make. A burst that the animal's own value function should have cancelled, supplied artificially, produces learning on schedule. Dopamine is not a readout of learning happening elsewhere; the transient is the teaching signal. Bayer and Glimcher (2005) tightened the correlational side in parallel, showing the burst is not merely present or absent but graded: across trials the firing rate of dopamine neurons scales with the size of the reward prediction error, coding a quantitative RPE rather than a bare sign.⁷ Their data also exposed an asymmetry that matters later. Positive errors are coded roughly linearly in firing rate, but negative errors run into a floor — a neuron cannot fire fewer than zero spikes, and the tonic baseline is already low, so large negative errors are compressed into a short pause that saturates quickly.

The correspondence, term by term

If you take the TD-error equation and run it on a simple version of Schultz's task, the three dopamine patterns are not extra assumptions — they are what the one equation does at three stages of learning. Nothing is added; we read $\delta$ off the definition.

Work through an idealized version of Schultz's task and the TD error reproduces all three signatures from the definition alone. Assume the animal has already learned to act, so its only job is prediction: learn a value function $V$ for the fixed sequence of states in a trial. Use TD(0), a value table initialized to zero, and — following Montague et al. — let each moment of a trial be its own state, so the reward-predicting stimuli occupy a run of states leading up to the one where reward is delivered. Take $\gamma \approx 1$ so we can ignore discounting, and let the reward at the rewarding state be some positive $R^{\star }$.⁸

Two states matter. The earliest reward-predicting state is the first state in a trial that reliably predicts the reward to come — the analogue of Schultz's instruction cue. The latest reward-predicting state is the one immediately before the rewarding state. Now trace $\delta$ through learning.

Early in learning, every $V$ is still zero. The TD error available at time $t$ is ${\delta }_{t-1}=R_t+V_t-V_{t-1}$, which stays zero at every transition until the reward arrives, where it becomes $R^{\star }+0-0=R^{\star }$. So $\delta$ spikes only at the reward — the unpredicted-reward panel, a burst for juice the value function did not see coming.

As learning proceeds, TD(0) updates run at every transition, and the value increases spread backward from the rewarding state to its predecessors, exactly as bootstrapping propagates information one step at a time. When learning is complete, every reward-predicting state has value $R^{\star }$. Now check the TD error at each transition. From one predicting state to the next, $\delta =R+V_t-V_{t-1}=0+R^{\star }-R^{\star }=0$. From the latest predicting state to the reward, $\delta =R^{\star }+0-R^{\star }=0$ — the reward is fully predicted, so it cancels, matching the silence at a predicted reward. But on the transition into the earliest predicting state, whose predecessors still have value zero, $\delta =R+V_t-V_{t-1}=0+R^{\star }-0=R^{\star }$. The positive error has migrated to the earliest predictor — the shift Schultz saw.

If the reward is then omitted after learning, the value of the latest predicting state is now too high. At the reward's expected time, $\delta =R+V_t-V_{t-1}=0+0-R^{\star }=-R^{\star }$: the TD error goes negative, driven below zero by nothing but the mismatch between a confident prediction and an empty world. That is the dip below baseline.⁸

All three signatures fall out of one equation and one learning rule. The reward term $R_{t+1}$ carries the raw juice; the bootstrapped term $\gamma V(S_{t+1})-V(S_t)$ carries the prediction, and it is the interplay of the two — reward minus expectation — that produces the burst-shift-dip pattern. Phasic dopamine is not signaling $R_{t+1}$. It is signaling ${\delta }_t$.

The core claim — that phasic dopamine is well described as a TD-style reward prediction error — has held up across three decades of increasingly refined experiments.

Watching the burst migrate: a worked trace

The term-by-term argument shows what the endpoints of learning look like — the burst at reward before learning, the burst at the cue after. The trip between them is the interesting part. The migration is not a jump but a slow crawl, one state per trial, driven by nothing but repeated application of a single update. Trace it with real numbers.

Fix a short trial. Four states run in sequence, $s_1\to s_2\to s_3\to s_4$, and reward $R^{\star }=1$ arrives on entering $s_4$; there is no reward on the earlier transitions. Take $\gamma =1$, step size $\alpha =0.5$, and initialize every value to zero. The prediction problem is the idealized policy evaluation of the last section, stated as a procedure.

The transition into $s_4$ needs a value for $s_4$ to bootstrap from. Because the rewarding state does not predict any further reward within the trial, its value stays pinned at $V(s_4)=0$ throughout — it is a terminal state. So the only values that move are $V(s_1)$, $V(s_2)$, $V(s_3)$.

Trial 1. Every value is zero. On $s_1\to s_2$, $\delta =0+V(s_2)-V(s_1)=0$, so nothing changes; the same on $s_2\to s_3$. On the final transition $s_3\to s_4$, reward appears: $\delta =1+V(s_4)-V(s_3)=1+0-0=1$. The update lifts $V(s_3)\leftarrow 0+0.5\times 1=0.5$. One trial, one non-zero error, at the reward — the unpredicted-reward burst. The value bump lands on $s_3$, the latest predictor, and nowhere else.

Trial 2. Now $V(s_3)=0.5$. On $s_2\to s_3$ the error is no longer zero: $\delta =0+V(s_3)-V(s_2)=0+0.5-0=0.5$, which lifts $V(s_2)\leftarrow 0.5\times 0.5=0.25$. On $s_3\to s_4$, $\delta =1+0-0.5=0.5$, lifting $V(s_3)\leftarrow 0.5+0.5\times 0.5=0.75$. The error has split: still positive at the reward, but now also positive one step earlier, at the transition into $s_3$. The burst has begun to move.

Trial 3. With $V(s_2)=0.25$, $V(s_3)=0.75$: on $s_1\to s_2$, $\delta =0.25$, so $V(s_1)\leftarrow 0.125$; on $s_2\to s_3$, $\delta =0.75-0.25=0.5$, so $V(s_2)\leftarrow 0.25+0.25=0.5$; on $s_3\to s_4$, $\delta =1-0.75=0.25$, so $V(s_3)\leftarrow 0.875$. The error at the reward has shrunk to $0.25$, while a positive error now reaches all the way back to the entry into $s_2$. Every trial, the reward error gets smaller and the earliest positive error moves one state to the left.

Continue and the values climb toward their fixed point $V(s_1)=V(s_2)=V(s_3)=1$, at which every within-chain error is zero except the one on entering $s_1$ — the earliest predictor — and the reward error has vanished. The table carries the first four trials.

The bottom block is the model's version of Schultz's raster: read any column and you see where the error — the dopamine burst — sits on that trial. In trial 1 it sits entirely at the reward. By trial 4 it has spread across the chain, and its mass has shifted toward the entry into the earliest states while the reward term has collapsed toward zero. Left to run, the process converges to a single positive error at the entry into $s_1$ and silence everywhere after. The famous migration is this table read left to right.

Two facts about the crawl are worth stating in their own right. The rate is set by $\alpha$: with $\alpha =0.5$ the reward error halves each trial ($1,0.5,0.25,\dots$), so the values approach $R^{\star }$ geometrically rather than in a single leap. And the direction is forced — information can only move backward, from a state to its predecessor, because the update to $V(s_t)$ reads the already-updated estimate at $s_{t+1}$. This backward flow is bootstrapping itself, and it is why the dopamine response, trial after trial, retreats toward the earliest cue rather than advancing toward the reward.

When the reward comes early: a known imperfection

The trace above is clean because the reward always arrives on schedule. Push on the timing and the simple model cracks in an instructive way. Suppose learning has converged, so the animal expects reward at a fixed moment, and then on a probe trial the reward is delivered earlier than usual. Two things should happen if dopamine tracks the TD error with a complete-serial-compound (CSC) representation, in which each instant of the trial is its own state.⁹

The early reward is itself unpredicted at the moment it arrives — no state predicted reward there — so the TD error is positive and dopamine should burst. It does; this half matches. But the model also predicts a second event. The state at the originally-expected time still carries a high value, learned over many on-time trials. When the trial reaches that moment and no reward appears — it was already collected early — the CSC model computes a negative error there, the same omission dip as before, and predicts a pause below baseline. Dopamine neurons show no such pause (Hollerman and Schultz, 1998). The reward was received; the animal is not disappointed at the old time, but the model predicts it should be.

The mismatch does not overturn the RPE hypothesis; it locates the seam. The TD error is right; the state representation fed to it is too rigid. A CSC assigns an indelible marker to the clock time of the original reward, so the machinery cannot help predicting an event there. The fixes all soften that marker. Suri and Schultz (1999) let a reward cancel the internal signals set off by earlier stimuli, erasing the stale prediction. Ludvig, Sutton, and Kehoe (2008) replaced the CSC with a microstimulus representation — a set of overlapping, temporally-blurred basis functions rather than one crisp state per instant — and found it fits the early-reward case and several others better than a CSC. Daw, Courville, and Touretzky (2006) went further and derived the state representation itself from a statistical model of the sensory stream. Pan, Schmidt, Wickens, and Hyland (2005) showed that even keeping the CSC, prolonged eligibility traces improve the fit. The lesson repeats across all of them: the error signal is the fixed point of the theory, and the representation feeding it is where the biology adds detail.

Where the signal goes

We have matched the phasic dopamine signal to the TD error, from Schultz's recordings through the causal optogenetic tests to a term-by-term derivation of the burst, the shift, and the dip. What we have not asked is what the signal does once it fires — where those axons go, what they change, and which algorithm the wiring is running. That is the subject of the next part: the basal ganglia read as a neural actor–critic, the corticostriatal synapse where three signals meet, and addiction as a hijacking of the same learning signal.

This continues in Dopamine in the Brain: The Neural Actor–Critic.