Dopamine in the Brain: The Neural Actor–Critic

This builds on Dopamine and the TD Error, which matched the phasic dopamine signal to the TD error term by term — the burst at an unpredicted reward, the shift to the earliest cue, the dip when a predicted reward is withheld. Granting that correspondence, this part asks the next question: what does the signal do? We follow the axons into the basal ganglia, read the wiring as an actor–critic, and close on addiction and on where the scalar picture breaks.

Reward pathways and the basal ganglia

Where does this signal go, and what does it change? Dopamine is a neuromodulator: rather than simply exciting or inhibiting its targets, it alters how other synapses learn. It is well suited to broadcasting a single scalar. Each dopamine neuron has an enormous axonal arbor — the tree of branches through which its output reaches its targets — making on the order of $500,000$ synaptic contacts, one hundred to a thousand times more than a typical neuron. If the signal it carries is a scalar $\delta$, a widely branching arbor is the wiring you would design to deliver that one number to many places at once.¹

Most of those axons target the basal ganglia, a group of nuclei at the base of the forebrain whose main input structure is the striatum. Nearly the whole cerebral cortex feeds the striatum, whose output loops back through other basal ganglia nuclei and the thalamus to the frontal cortex, where it shapes movement, decision, and reward processing. Two subdivisions of the striatum recur in what follows: the dorsal striatum, tied to action selection, and the ventral striatum, tied to reward processing and to assigning value to sensations.²

The input neurons of the striatum are medium spiny neurons, whose dendrites are studded with tiny protrusions called spines. Cortical axons contact the tips of these spines while dopamine axons contact their stems, placing three signals in close company at a single synapse: the presynaptic cortical activity (the state), the postsynaptic activity of the striatal neuron (the action or value), and dopamine (the error). Evidence is mounting that changes in the efficacy of these corticostriatal synapses — the connections from cortex into the striatum — depend critically on appropriately timed dopamine signals. That is a learning rule implemented in tissue.²

The arrangement is worth drawing, because it puts the three quantities a learning rule needs into physical contact at one spine. The cortical axon at the spine tip carries the state — which features are active now. The medium spiny neuron's own firing carries the output — the action taken, or the value asserted. And the dopamine axon at the spine stem carries the error — the scalar $\delta$. A synapse that changes its strength as a function of all three is computing an update of the form (feature) $\times$ (error), gated by whether the postsynaptic cell fired. That is the shape of the actor and critic rules below, and the geometry of the spine is what lets one synapse read all three signals at once.

The neural actor–critic

The RPE hypothesis says dopamine is a reinforcement signal. Which algorithm uses a reinforcement signal that way? An actor–critic, and the fit to the anatomy is close enough to be a serious hypothesis about the brain.³

Recall the structure. The critic learns a value function for the policy the actor is currently following, and computes a TD error $\delta$ from reward and its own value estimates. The actor learns the policy, and adjusts it using $\delta$ as its teaching signal: a positive $\delta$ means the action led somewhere better than expected, so make it more likely; a negative $\delta$ means worse than expected, so make it less likely. The single error $\delta$ trains both halves, but it does different work in each — it tells the critic how to sharpen its predictions and tells the actor how to shift its choices.

Two features support the mapping. First, the actor–critic has two learning components, and the striatum has two subdivisions with the matching division of labor: the ventral striatum, tied to value, plays the critic; the dorsal striatum, tied to action selection, plays the actor. Second, the TD error trains both components, and dopamine axons target both subdivisions of the striatum. One signal, broadcast to two places that need it, is what the algorithm calls for and what the wiring provides.³

The learning rules refine the picture. In the actor–critic policy-gradient form, each transition computes ${\delta }_t$ and then updates the critic's value weights $\mathbf{w}$ and the actor's policy weights $\mathbf{\theta }$ using the same${\delta }_t$, each scaled by its own eligibility trace:

The rules differ only in the eligibility trace each carries.⁴

The critic's trace depends on presynaptic activity alone, so its rule is the TD model of classical conditioning. The actor's trace also depends on whether the postsynaptic neuron fired, carrying the factor $A_t-\pi (A_t\mid S_t,\mathbf{\theta })$ that records which action was taken — an instrumental rule.

That postsynaptic contingency has a biological counterpart. Neuroscientists have found a form of Hebbian plasticity called spike-timing-dependent plasticity, in which a synapse strengthens when its presynaptic spike arrives just before the postsynaptic neuron fires. And they have found a reward-modulated form, in which the change happens only if a neuromodulatory pulse — dopamine — arrives within a time window that can last up to ten seconds after the paired spikes. A synapse that remembers a pre-then-post coincidence and commits the change only when dopamine later confirms it is, in tissue, a contingent eligibility trace waiting on the actor's learning signal.⁴

Addiction: hijacking the learning signal

If dopamine carries the learning signal, then a drug that tampers with dopamine tampers with learning itself. Many drugs of abuse — cocaine among them — raise dopamine, directly or indirectly, in the striatal regions the RPE hypothesis places at the center of reward learning. A model due to Redish (2004) turns the TD account into an account of what goes wrong.⁵

The pathology is a broken cancellation. In normal learning, once a reward is predicted the bootstrapped term drives the TD error at that reward to zero — a fully anticipated reward stops reinforcing. Redish's model assumes an addictive drug produces a dopamine surge that cannot be cancelled by any rise in the value function: the drug adds a non-negative term $D\ge 0$ that floors the error,

so the reward can never be predicted away. With the error-correcting feedback that normally levels off learning gone, the values of states leading to drug-taking rise without bound, and the actions leading there come to be preferred above all others — even against the self-destruction they cause.

Real addiction is far more complicated than one clamped term — dopamine is not central to every form of it, susceptibility varies, and the model omits the many circuit changes that accompany chronic drug use. But the model illustrates a theme that runs through this whole chapter: a computational theory of reward learning gives neuroscience a precise, testable handle on a real disorder. The same reasoning has helped launch computational psychiatry, which aims to sharpen the understanding of mental illness with mathematical models of the learning systems that go awry.⁵

Distributional dopamine and the RPE hypothesis under test

The RPE hypothesis, as stated so far, is a claim about a scalar: one number, $\delta$, broadcast uniformly. Three lines of work published after Sutton and Barto's account push on that picture — one confirming the arithmetic the theory needs, one enriching what the population encodes, and one marking where the model-free reading runs out.

The circuit computes a real subtraction. The TD error is a difference, reward minus expectation, and Eshel and colleagues (2015, Nature) asked whether dopamine neurons implement that subtraction literally or only approximate it. Recording from mouse VTA dopamine neurons while varying both reward size and the animal's expectation, they found that the response to a reward equals the reward's value minus the expected value — a genuine subtraction, not a division or a threshold — and, across the population, that the neurons share a nearly identical scaling. Different dopamine neurons respond to different absolute amounts, but they subtract expectation in the same way and differ only by a common gain. The circuit is doing the arithmetic the TD error asks for, uniformly enough that the population can be read as broadcasting one error.⁶

The population encodes a distribution, not just a mean. A scalar RPE throws away everything about reward except its expected value. Distributional reinforcement learning (Bellemare, Dabney, and Munos, 2017, ICML) had shown that an agent learns better if it tracks the whole distribution of returns rather than its mean, using a population of predictors with different optimism — some tuned to weight good outcomes, others bad. Dabney, Kurth-Nelson, Uchida, Starkweather, Hassabis, Munos, and Botvinick (2020, Nature) imported that idea as a prediction about the brain and tested it in mouse VTA. They found that dopamine neurons are not interchangeable scalar reporters. Different neurons have different reversal points — the reward magnitude at which a given neuron switches from a positive to a negative response — so some are relatively optimistic and others pessimistic. Read as a population, the varied reversal points encode a full distribution over upcoming reward, and the distribution recovered from the neural data matched the one implied by the animals' behavior. The reward prediction error hypothesis, on this account, is the mean of a richer distributional code.⁷

The signal is not purely model-free. A TD error read off a value function is a model-free quantity: it knows returns, not the structure of the world that produces them. Yet dopamine transients also carry information they should not, on a strict model-free reading. Bromberg-Martin, Matsumoto, and Hikosaka (2010, Neuron) found dopamine responses sensitive to sensory and motivational information beyond cached value, and Doll, Simon, and Daw (2012, Current Opinion in Neurobiology) reviewed evidence that model-based knowledge influences the error signal and the choices it drives. The pure model-free RPE is an approximation that holds well for habitual behavior and leaks where planning enters — the seam the next lesson and the neuroscience of model-based control take up in full.⁸

None of this unseats the correspondence. Each result takes the TD error as the starting point and asks a sharper question of the biology than the original recordings could — whether the subtraction is exact, what the population as a whole represents, where the model-free reading stops. The hypothesis holds up under this more precise questioning, and gains structure rather than losing it.

The convergence, and where the course began

The TD error was written down for reasons that had nothing to do with biology — optimal control, dynamic programming, the problem of learning to predict before an outcome is known. Years later, recordings from dopamine neurons revealed a signal that matches it in signature after signature: the burst at unpredicted reward, the migration to the earliest predictor, the dip at an omitted one. The algorithm was not reverse-engineered from the brain; it was built for machines and turned out to describe the brain anyway. That unplanned agreement is the strongest sign we have that reinforcement learning captures something true about how nervous systems learn from reward.⁹

Which returns us to where the course started. We opened by defining reinforcement learning as learning what to do from a reward signal, discovered by trial and error rather than told. Every method since — bandits, MDPs, dynamic programming, Monte Carlo, temporal difference, function approximation, policy gradients, deep RL — was a way of solving that one problem. And the quantity at the hinge of it all, the TD error ${\delta }_t=R_{t+1}+\gamma V(S_{t+1})-V(S_t)$, turns out to be not only the engine of the algorithms but, as far as we can now tell, the currency the brain itself trades in when it learns from reward. The mathematics of trial-and-error learning and the biology of trial-and-error learning appear to be the same subject.