The Neuroscience of Reinforcement

The dopamine=TD-error correspondence is the sharpest single point of contact between reinforcement learning and the brain, but it is one point. It says a chemical signal behaves like ${\delta }_t$; it does not say what a synapse is, how a synapse changes, which brain structures carry the actor and the critic, or whether the brain runs anything besides a model-free error-correcting loop. This lesson supplies the surrounding neuroscience so the mapping is self-contained, and it makes one argument throughout: the brain is not a single reinforcement-learning algorithm faithfully implemented in tissue. It is several learning systems — model-free and model-based, habitual and goal-directed — running at once and arbitrating between themselves.¹

Reinforcement learning is a good lens for that machinery precisely because it gives distinct names to signals the brain keeps distinct. Before any mapping, though, we need the vocabulary of the tissue.

A neuroscience primer

A neuron is a cell specialized to process and transmit information with electrical and chemical signals. The parts that matter here are three. The cell body (soma) integrates incoming signals. Dendrites branch from the cell body to receive input from other neurons, and in sensory neurons to receive signals from the outside world. A single axon carries the neuron's output to its targets. A neuron's output is a sequence of brief electrical pulses called action potentials — also called spikes — that travel along the axon; a neuron is said to fire when it emits one. In neural-network models we usually abstract this spike train to a single number, the firing rate: the average count of spikes per unit of time.²

The axon can branch widely, and this branching structure is the neuron's axonal arbor. Conduction of an action potential is an active, regenerative process, not unlike a burning fuse: when a spike reaches a branch point it lights up spikes on all the outgoing branches (though propagation into any one branch can fail). A neuron with a large arbor can therefore influence a great many target sites at once — a fact that becomes central when we ask how a single chemical signal reaches synapses all over the brain.

Synapses and synaptic plasticity

A synapse is the structure, usually at the end of an axon branch, that transmits a signal from one neuron to the next. On the presynaptic side is the sending neuron's axon; on the postsynaptic side is the receiving neuron's dendrite or cell body. With few exceptions the transmission is chemical: the arrival of an action potential causes the presynaptic terminal to release a neurotransmitter, whose molecules diffuse across the narrow synaptic cleft and bind to receptors on the postsynaptic surface. That binding can excite the postsynaptic neuron (push it toward firing), inhibit it (push it away), or modulate its behavior in other ways. A single neurotransmitter can bind several receptor types, each producing a different effect — dopamine alone acts through at least five distinct receptor types.³

The strength with which a synapse's transmitter drives the postsynaptic neuron is the synapse's efficacy — the biological counterpart of a weight in a network. The ability of that efficacy to change with experience is synaptic plasticity, and it is one of the primary mechanisms responsible for learning. When we adjust a weight with a learning rule, the brain's analog is a change in synaptic efficacy; the parameters an algorithm tunes correspond to the efficacies a brain modifies. Everything the rest of this lesson calls learning ultimately means this.

Neuromodulators and the diffuse broadcast

Not every chemical signal is fast point-to-point excitation or inhibition. A neuromodulator is a neurotransmitter that has effects other than, or in addition to, direct fast signaling — it can change how a circuit functions, and, critically, it can gate synaptic plasticity. The brain contains several distinct neuromodulatory systems, each a cluster of neurons with widely branching axonal arbors, each using a different transmitter, together mediating motivation, arousal, attention, memory, mood, and more. What matters here is a structural fact: because a neuromodulatory system's neurons have enormous arbors, such a system can distribute something like a scalar signal — a single number, such as a reinforcement signal — to synapses at widely distributed sites all at once.⁴

That matches the shape a reinforcement signal needs. An algorithm broadcasts one ${\delta }_t$ to every parameter; a neuromodulatory system broadcasts one chemical level to every synapse in its reach. Dopamine is the leading candidate for this role, and its neurons are unusually well suited to it: each dopamine axon makes on the order of hundreds of thousands of synaptic contacts, reaching 100 to 1,000 times more sites than a typical neuron's axon.

The corticostriatal loop

The structures where reward-based learning is thought to happen form a loop through three regions. The cerebral cortex — the outer sheet of the brain — supplies context: sensory input, current state, candidate actions. That cortical activity projects to the striatum, the input structure of the basal ganglia, a group of nuclei deep in the brain heavily implicated in action selection and learning. The striatum divides into a dorsal part and a ventral part, and this division will matter: one is a better fit for the actor, the other for the critic. The basal ganglia's output loops back, through the thalamus, to the cortex — closing a cortico-striatal-thalamo-cortical loop. Cutting across this loop are the dopamine neurons, whose cell bodies sit in two midbrain nuclei (the substantia nigra pars compacta and the ventral tegmental area) and whose axons blanket the striatum with the phasic reinforcement signal.⁵

A neuron's steady, slow background activity is its tonic firing; a brief burst usually driven by synaptic input is phasic activity. The reinforcement signal of interest rides on the phasic dopamine response — the burst — while the tonic level does other work. Keep the two apart: the RPE signal rides on the burst, not the tonic level.

Four signals that get conflated

Casual usage runs together four ideas that reinforcement learning keeps sharply distinct, and the neuroscience only makes sense once they are separated. When we label a neural signal by its function — calling a dopamine burst a reinforcement signal — we mean it behaves like, and is conjectured to function like, the corresponding theoretical term in an equation. The four terms are reward, reinforcement, value, and prediction error.⁶

Reward is the signal $R_t$. In reinforcement learning it defines the problem: along with the environment, the reward signal specifies what the agent is trying to achieve. It is best read as an abstraction. There is almost certainly no single master reward wire in the brain; $R_t$ summarizes the overall effect of a multitude of neural signals that assess the rewarding or punishing quality of sensations and states, distributed across many sites.

Value is $v(s)$ or $q(s,a)$: an estimate of what is good or bad over the long run. Values are predictions of the total reward an agent can expect to accumulate from a state (or from an action in a state) into the future. Good decisions come from choosing states with high estimated value, or actions with high estimated action value — not from chasing immediate reward.

A prediction error measures a discrepancy between an expected and a received signal. A reward prediction error (RPE) is specifically the gap in reward: positive when the reward signal exceeds expectation, negative when it falls short, zero when the forecast is exact. A TD error is a particular kind of RPE — one that compares current and earlier expectations of reward over the long term, folding the value change $\gamma v(S_{t+1})-v(S_t)$ into the discrepancy rather than looking at $R_t$ alone. When neuroscientists refer to RPEs they usually mean TD RPEs.

Reinforcement is the signal that actually directs the change a learning algorithm makes to its policy, values, or model. This is the one most often misnamed, because it need not be the reward. For a TD method the reinforcement signal at time $t$ is the TD error

the reward signal adjusted by other information — here the value estimates. Reward is a component of $\delta$, but not the whole of it; the extra term $\gamma v(S_{t+1})-v(S_t)$ is the higher-order part. This is why a dopamine burst is a reinforcement signal, not a reward signal: it tracks $\delta$, and even when reward occurs ($R_{t+1}\ne 0$) the burst can be silent if the reward was fully predicted, because $\delta \approx 0$. Reward happened; reinforcement did not.

The distinctions are not pedantry. Neural activity related to reward processing turns up in nearly every part of the brain, and representations of these four signals tend to be strongly correlated, so an experiment that cannot dissociate them cannot say which one it has found. Designing a task that separates a reward signal from a value signal from a prediction error is most of the difficulty in this field.

A task that dissociates the three

To see why dissociation is hard and how it is done, take a cued task with a single outcome. A cue predicts reward with probability $p$; on each trial the outcome is either reward ($R=1$) or nothing ($R=0$). Value is the expected reward the cue forecasts, $v=p$ (with $\gamma$ irrelevant here — the outcome follows the cue by a fixed delay and there is no further state). The reward prediction error at the outcome is $\delta =R-v$. If an experiment used only trials with $p=1$, reward and value and RPE would move together on every trial and no recording could tell them apart: reward is always $1$, value is always $1$, RPE is always $0$. Varying $p$ across cues, and reading out $R$ and $\delta$ separately, pulls the three apart.

Reading down the table: the same reward $R=1$ produces three different phasic responses ($0$, $+0.5$, $+1.0$) depending on the value the cue set up, so the burst cannot be a reward signal. The same value $v=0.5$ produces both a positive and a negative response depending on the outcome, so it cannot be a value signal either. Only $\delta =R-v$ predicts every cell. This is the logic behind the classic dopamine recordings: fix the reward, vary what the animal expects, and watch the burst track the difference. Bayer and Glimcher (2005) pushed it quantitatively, showing that the firing rate of midbrain dopamine neurons scales with the numerical size of the prediction error rather than the reward.⁷

Actor and critic learning rules

The dopamine lesson read the basal ganglia as a neural actor-critic with dopamine as the shared training signal. Here we spell out the learning rules and ask whether they are neurally plausible. The setup is a linear critic unit and a small network of actor units, all receiving the same feature-vector representation of the state.⁸

Let $\mathbf{x}(S_t)=(x_1(S_t),\dots ,x_n(S_t){)}^{\top }$ be the feature vector for the current state — think of each $x_i$ as the presynaptic signal arriving at one of the unit's synapses. The critic is a single linear neuron-like unit with weight vector $\mathbf{w}$; its estimate is $\hat{v}(s,\mathbf{w})={\mathbf{w}}^{\top }\mathbf{x}(s)$. The actor is a one-layer network of $k$ identical units, each with its own weight vector ${\mathbf{\theta }}_j$; taking two actions, an actor unit is a Bernoulli-logistic unit whose output $A_t\in \{0,1\}$ (fire or not) has probability $\pi (1\mid s,\mathbf{\theta })=1/(1+\exp (-{\mathbf{\theta }}^{\top }\mathbf{x}(s)))$. Both are updated by the same broadcast signal ${\delta }_t$ acting as reinforcement:

The two rules differ only in their eligibility traces ${\mathbf{z}}_t$, and that difference is the whole point. The critic's trace accumulates the presynaptic activity alone, ${\mathbf{z}}_t^{\mathbf{w}}={\lambda }^{\mathbf{w}}{\mathbf{z}}_{t-1}^{\mathbf{w}}+\nabla \hat{v}(S_t,\mathbf{w})={\lambda }^{\mathbf{w}}{\mathbf{z}}_{t-1}^{\mathbf{w}}+\mathbf{x}(S_t)$: a synapse becomes eligible for modification whenever it carries presynaptic input, independent of whether the postsynaptic neuron fires. This is a non-contingent eligibility trace. The actor's trace, by contrast, includes the postsynaptic factor: ${\mathbf{z}}_t^{\mathbf{\theta }}={\lambda }^{\mathbf{\theta }}{\mathbf{z}}_{t-1}^{\mathbf{\theta }}+\nabla \ln \pi (A_t\mid S_t,\mathbf{\theta })$, and that gradient works out to $(A_t-\pi (A_t\mid S_t,\mathbf{\theta }))\mathbf{x}(S_t)$ — a product of a presynaptic factor $\mathbf{x}(S_t)$ and a postsynaptic factor $(A_t-\pi (\cdot ))$ that depends on whether the unit actually fired. This is a contingent eligibility trace.

Both rules are descendants of Hebb's classic proposal — that when a presynaptic signal repeatedly helps fire a postsynaptic neuron, the synapse strengthens. But each adds structure Hebb did not have. The critic rule involves two interacting factors: the reinforcement signal $\delta$ and a presynaptic-only trace; neuroscientists call this a two-factor learning rule. The actor rule involves three: $\delta$, a presynaptic factor, and a postsynaptic factor — a three-factor learning rule. In each case the reinforcement signal $\delta$ arrives to modulate a synapse whose eligibility trace was set by local pre/post activity, possibly long before; the trace bridges the delay between the eligible event and the arrival of reinforcement.

One update, worked through

The rules are abstract until real numbers pass through them. Take a two-state, two-action task with a one-hot state representation: $\mathbf{x}(s_1)=(1,0{)}^{\top }$ and $\mathbf{x}(s_2)=(0,1{)}^{\top }$, so each state activates a single synapse on the critic and on each actor unit. Start both learners blank: critic weights $\mathbf{w}=(0,0{)}^{\top }$ and actor weights $\mathbf{\theta }=(0,0{)}^{\top }$. With $\mathbf{\theta }=0$ the actor's firing probability is $\pi (1\mid s_1)=1/(1+e^0)=0.5$: an unbiased coin, exactly what a fresh policy should be. Set $\gamma =0.9$ and step sizes ${\alpha }^{\mathbf{w}}={\alpha }^{\mathbf{\theta }}=0.2$; take the traces at their simplest, $\lambda =0$, so $\mathbf{z}$ is just the current gradient.

Now run one transition. In $s_1$ the actor fires, $A=1$ (it came up heads); the action carries the agent to $s_2$ and yields reward $R=1$. Compute the value estimates from the still-zero weights: $\hat{v}(s_1)={\mathbf{w}}^{\top }\mathbf{x}(s_1)=0$ and $\hat{v}(s_2)=0$. The reinforcement signal is

Critic update (two-factor). The non-contingent trace is the presynaptic input alone, ${\mathbf{z}}^{\mathbf{w}}=\mathbf{x}(s_1)=(1,0{)}^{\top }$. So

Only the synapse carrying $s_1$'s input moved, and it moved up: $\hat{v}(s_1)$ rises from $0$ to $0.2$, a step toward the reward that just arrived. The critic did not need to know which action was taken — it is learning to predict, not to choose.

Actor update (three-factor). The contingent trace multiplies the presynaptic input by the postsynaptic factor $(A-\pi (A\mid s_1))=(1-0.5)=0.5$, giving ${\mathbf{z}}^{\mathbf{\theta }}=0.5\cdot \mathbf{x}(s_1)=(0.5,0{)}^{\top }$. So

The $s_1$ weight rises to $0.1$, which lifts the firing probability from $0.5$ to $\pi (1\mid s_1)=1/(1+e^{-0.1})\approx 0.525$: the action that was taken becomes a little more likely, because the reinforcement that followed it was positive. Had the same $\delta >0$ arrived after the unit stayed silent ($A=0$), the postsynaptic factor would have been $(0-0.5)=-0.5$ and the weight would have moved the other way, making firing less likely — the trace remembers what the unit did and signs the update accordingly.

The single number $\delta$ did two jobs at once. Broadcast unchanged to both learners, it pushed the critic's estimate toward the reward it under-predicted and pushed the actor's policy toward the action it had just tried. Nothing in the wiring distinguishes the two updates except the trace each synapse keeps — non-contingent for the predictor, contingent for the controller.

The timing problem, and STDP

Real neurons force a subtlety the idealized units hide. When a presynaptic spike arrives, the postsynaptic neuron does not respond instantly; the chemistry of transmission takes tens of milliseconds. For the contingent actor rule to assign credit correctly, the presynaptic factor that sets the trace must be a genuine cause of the postsynaptic firing that defines it — which means a realistic actor unit has to respect this activation time. (Do not confuse activation time, tens of milliseconds, with the much longer interval an eligibility trace must span to bridge a delayed reward.)

The brain has a plasticity mechanism with exactly the right shape: spike-timing-dependent plasticity (STDP). STDP is Hebbian, but the direction of the change depends on the relative timing of pre- and postsynaptic spikes: a synapse strengthens if its presynaptic spike arrives shortly before the postsynaptic neuron fires, and weakens if the order is reversed. STDP thus takes activation time into account — one of the ingredients an actor-like rule needs. Write $\Delta t=t_{\text{post}}-t_{\text{pre}}$ for the timing offset. The canonical STDP curve, mapped out in cortical and hippocampal slices, is a sharp asymmetric window: $\Delta w>0$ for $\Delta t>0$ (pre before post, the causal order), $\Delta w<0$ for $\Delta t<0$ (post before pre), with both lobes decaying over tens of milliseconds and a discontinuity at $\Delta t=0$. Markram and colleagues (1997) demonstrated the timing dependence between connected cortical pyramidal neurons; Bi and Poo (1998) mapped the full window in hippocampal culture, fixing the tens-of-milliseconds width that has anchored the field since.⁹

And there is more: neuroscientists have found reward-modulated STDP, a three-factor form in which the STDP change occurs only if a neuromodulatory input (dopamine) arrives within a time window after the paired pre/post spikes. The STDP window sets which synapses become eligible; the dopamine pulse decides whether the eligible change is actually committed. Reynolds and Wickens (2002) proposed exactly this three-factor corticostriatal rule — an STDP-eligible synapse held in a plastic state until dopamine gates the change.¹⁰ Yagishita and colleagues (2014) gave it a mechanism: at the dendritic spines of striatal medium spiny neurons, the spine enlarges (the structural correlate of potentiation) only when dopamine arrives in a narrow window, on the order of $0.3$ to $2$ seconds, after paired pre/post activity. Experiments have shown lasting changes in corticostriatal synapses when dopamine follows a pre-then-post pairing within a window that can last up to about ten seconds — direct, if indirect, evidence for contingent eligibility traces with long time courses, at exactly the synapses where the actor's learning is hypothesized to live.¹¹

The two rules differ in which factors must line up in time. The critic is two-factor: a presynaptic-marked synapse, then dopamine. The actor is three-factor: a presynaptic spike, then a postsynaptic spike (this pairing sets the contingent trace), then dopamine within the window. Laying the signals on a timeline makes the difference legible.

Beyond the single loop

We now have the tissue-level vocabulary and the two learning rules: a two-factor critic and a three-factor actor, both driven by a broadcast $\delta$, both grounded in reward-modulated STDP at corticostriatal synapses. That is one model-free loop. But the brain is not one loop. The next part asks where the actor rule came from — Klopf's hedonistic neuron — whether teams of such neurons can learn together, and where the model-based systems live that the single loop leaves out. The recurring finding is that the brain is not one reinforcement-learning algorithm but several interacting systems.

This continues in The Brain's Several Learning Systems.