---
title: The Neuroscience of Reinforcement
module: Reinforcement Learning in Minds and Brains
moduleNumber: 6
lessonNumber: 7
order: 607
summary: >
  The dopamine story is one contact point between reinforcement learning and the
  brain; this lesson fills in the surrounding neuroscience so the mapping stands on
  its own. We build a working primer of neurons, synapses, and neuromodulation;
  separate four signals that casual usage conflates — reward, reinforcement, value,
  and prediction error; and read the actor and critic as corticostriatal synapses
  updated by two- and three-factor rules, grounded in spike-timing-dependent and
  reward-modulated plasticity.
topics: [Minds and Brains]
sources:
  - book: Sutton & Barto
    ref: "Ch. 15 — Neuroscience; §15.1 Neuroscience Basics; §15.2 Reward Signals, Reinforcement Signals, Values, and Prediction Errors"
  - book: Sutton & Barto
    ref: "§15.8 Actor and Critic Learning Rules"
---

The [dopamine=TD-error correspondence](/reinforcement-learning/minds-and-brains/dopamine-and-td-error)
is the sharpest single point of contact between reinforcement learning and the
brain, but it is one point. It says a chemical signal behaves like $\delta_t$; it
does not say what a synapse is, how a synapse changes, which brain structures
carry the actor and the critic, or whether the brain runs anything besides a
model-free error-correcting loop. This lesson supplies the surrounding
neuroscience so the mapping is self-contained, and it makes one argument
throughout: the brain is not a single reinforcement-learning algorithm faithfully
implemented in tissue. It is several learning systems — model-free and
model-based, habitual and goal-directed — running at once and arbitrating between
themselves.[^sb-intro]

Reinforcement learning is a good lens for that machinery precisely because it
gives distinct names to signals the brain keeps distinct. Before any mapping,
though, we need the vocabulary of the tissue.

## A neuroscience primer

A **neuron** is a cell specialized to process and transmit information with
electrical and chemical signals. The parts that matter here are three. The **cell
body** (soma) integrates incoming signals. **Dendrites** branch from the cell body
to receive input from other neurons, and in sensory neurons to receive signals
from the outside world. A single **axon** carries the neuron's output to its
targets. A neuron's output is a sequence of brief electrical pulses called **action
potentials** — also called **spikes** — that travel along the axon; a neuron is
said to **fire** when it emits one. In neural-network models we usually abstract
this spike train to a single number, the **firing rate**: the average count of
spikes per unit of time.[^sb-neuron]

$$
% caption: A neuron. Dendrites collect input at the cell body, which fires action
% potentials down a single axon; the axon branches into an axonal arbor, so one
% action potential can reach many downstream targets.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  % soma
  \draw[fill=acc!8, draw=acc, thick] (0,0) circle (7mm);
  \node[font=\footnotesize] at (0,0) {soma};
  % dendrites (left, fanning in)
  \foreach \a in {150,170,190,210} \draw[black, thick] (0,0) ++(\a:7mm) -- ++(\a:9mm);
  \node[black, font=\footnotesize, anchor=east] at (-2.15,0.55) {dendrites};
  \draw[->, black] (-1.55,0.55) -- (-1.0,0.35);
  % axon to the right
  \draw[acc, thick] (0.7,0) -- (3.2,0);
  \node[acc, font=\footnotesize, anchor=south] at (1.9,0.05) {axon};
  % axonal arbor branches
  \draw[acc, thick] (3.2,0) -- (4.2,0.8);
  \draw[acc, thick] (3.2,0) -- (4.3,0.2);
  \draw[acc, thick] (3.2,0) -- (4.2,-0.8);
  \foreach \p in {(4.2,0.8),(4.3,0.2),(4.2,-0.8)} \fill[acc] \p circle (1.6pt);
  \node[black, font=\footnotesize, anchor=west] at (4.4,-0.85) {to targets};
  \node[align=center, font=\footnotesize, anchor=north] at (3.5,-1.4) {axonal arbor\\(one spike, many targets)};
  % spike glyph on axon
  \draw[acc, very thick] (2.55,0.15) -- (2.65,0.6) -- (2.75,0.15);
\end{tikzpicture}
$$

The axon can branch widely, and this branching structure is the neuron's **axonal
arbor**. Conduction of an action potential is an active, regenerative process,
not unlike a burning fuse: when a spike reaches a branch point it "lights up"
spikes on all the outgoing branches (though propagation into any one branch can
fail). A neuron with a large arbor can therefore influence a great many target
sites at once — a fact that becomes central when we ask how a single chemical
signal reaches synapses all over the brain.

### Synapses and synaptic plasticity

A **synapse** is the structure, usually at the end of an axon branch, that
transmits a signal from one neuron to the next. On the **presynaptic** side is the
sending neuron's axon; on the **postsynaptic** side is the receiving neuron's
dendrite or cell body. With few exceptions the transmission is chemical: the
arrival of an action potential causes the presynaptic terminal to release a
**neurotransmitter**, whose molecules diffuse across the narrow **synaptic cleft**
and bind to receptors on the postsynaptic surface. That binding can excite the
postsynaptic neuron (push it toward firing), inhibit it (push it away), or modulate
its behavior in other ways. A single neurotransmitter can bind several receptor
types, each producing a different effect — dopamine alone acts through at least
five distinct receptor types.[^sb-synapse]

$$
% caption: A synapse. A presynaptic spike releases neurotransmitter into the cleft;
% it binds postsynaptic receptors, exciting or inhibiting the receiving neuron. The
% synapse's efficacy — how strongly the presynaptic signal drives the postsynaptic
% neuron — is the adjustable weight of the brain.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  % presynaptic terminal
  \draw[fill=acc!8, draw=acc, thick] (-3.4,0) -- (-1.2,0) -- (-1.2,1.4) -- (-3.4,1.4) -- cycle;
  \node[align=center, font=\footnotesize] at (-2.3,0.7) {presynaptic\\terminal};
  \draw[acc, thick] (-4.6,0.7) -- (-3.4,0.7);
  \node[acc, font=\footnotesize, anchor=south] at (-4.0,0.75) {spike};
  % cleft with transmitter dots
  \foreach \p in {(-0.9,0.4),(-0.6,0.9),(-0.4,0.3),(-0.2,0.7),(-0.7,0.6)} \fill[black] \p circle (1.4pt);
  \node[black, font=\footnotesize, anchor=south] at (-0.55,1.15) {transmitter};
  \node[black, font=\scriptsize, anchor=north] at (-0.55,-0.05) {cleft};
  % postsynaptic
  \draw[fill=black!4, draw=black, thick] (0.2,0) -- (2.4,0) -- (2.4,1.4) -- (0.2,1.4) -- cycle;
  \node[align=center, font=\footnotesize] at (1.3,0.7) {postsynaptic\\neuron};
  % receptors
  \foreach \y in {0.35,0.7,1.05} \draw[black, thick] (0.2,\y) ++(-1.5pt,0) -- ++(3pt,0);
  \node[black, font=\footnotesize, anchor=west] at (2.55,0.7) {receptors};
  % efficacy label
  \node[acc, font=\footnotesize, anchor=north, align=center] at (-0.5,-0.65) {e\/f\/f\/icacy = weight};
\end{tikzpicture}
$$

The strength with which a synapse's transmitter drives the postsynaptic neuron is
the synapse's **efficacy** — the biological counterpart of a weight in a network.
The ability of that efficacy to change with experience is **synaptic plasticity**,
and it is one of the primary mechanisms responsible for learning. When we adjust a
weight with a learning rule, the brain's analog is a change in synaptic efficacy;
the parameters an algorithm tunes correspond to the efficacies a brain modifies.
Everything the rest of this lesson calls "learning" ultimately means _this_.

### Neuromodulators and the diffuse broadcast

Not every chemical signal is fast point-to-point excitation or inhibition. A
**neuromodulator** is a neurotransmitter that has effects other than, or in
addition to, direct fast signaling — it can change how a circuit functions, and,
critically, it can gate synaptic plasticity. The brain contains several distinct
**neuromodulatory systems**, each a cluster of neurons with widely branching
axonal arbors, each using a different transmitter, together mediating motivation,
arousal, attention, memory, mood, and more. What matters here is a structural
fact: because a neuromodulatory system's neurons have enormous arbors, such a
system can distribute something like a _scalar_ signal — a single number, such as a
reinforcement signal — to synapses at widely distributed sites all at once.[^sb-neuromod]

That matches the shape a reinforcement signal needs. An algorithm broadcasts one
$\delta_t$ to every parameter; a neuromodulatory system broadcasts one chemical
level to every synapse in its reach. Dopamine is the leading candidate for this
role, and its neurons are unusually well suited to it: each dopamine axon makes on
the order of hundreds of thousands of synaptic contacts, reaching 100 to 1,000
times more sites than a typical neuron's axon.

### The corticostriatal loop

The structures where reward-based learning is thought to happen form a loop through
three regions. The **cerebral cortex** — the outer sheet of the brain — supplies
context: sensory input, current state, candidate actions. That cortical activity
projects to the **striatum**, the input structure of the **basal ganglia**, a group
of nuclei deep in the brain heavily implicated in action selection and learning.
The striatum divides into a **dorsal** part and a **ventral** part, and this
division will matter: one is a better fit for the actor, the other for the critic.
The basal ganglia's output loops back, through the thalamus, to the cortex — closing
a cortico-striatal-thalamo-cortical loop. Cutting across this loop are the dopamine
neurons, whose cell bodies sit in two midbrain nuclei (the substantia nigra pars
compacta and the ventral tegmental area) and whose axons blanket the striatum with
the phasic reinforcement signal.[^sb-loop]

$$
% caption: The corticostriatal loop. Cortex sends state and candidate actions to the
% striatum (the basal ganglia's input); the basal ganglia loop back to cortex via the
% thalamus. Midbrain dopamine neurons broadcast the phasic reinforcement signal to the
% dorsal and ventral striatum, gating plasticity at corticostriatal synapses.
\begin{tikzpicture}[>=stealth, font=\small,
  reg/.style={draw, thick, minimum width=22mm, minimum height=11mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[reg, draw=acc, text=acc] (ctx)  at (0,2.2)   {cortex\\(state, actions)};
  \node[reg] (str)  at (0,0)     {striatum\\(dorsal / ventral)};
  \node[reg] (bg)   at (4.6,0)   {basal ganglia\\output};
  \node[reg] (thal) at (4.6,2.2) {thalamus};
  \node[reg, draw=red, text=red] (da) at (-4.6,0) {midbrain\\dopamine};
  % loop
  \draw[->, acc, thick] (ctx) -- (str) node[midway, left, font=\scriptsize] {project};
  \draw[->, black, thick] (str) -- (bg);
  \draw[->, black, thick] (bg) -- (thal);
  \draw[->, black, thick] (thal) -- (ctx);
  % dopamine broadcast
  \draw[->, red, thick] (da) -- (str) node[midway, above, font=\scriptsize] {phasic delta};
  \node[red, font=\scriptsize, anchor=north, align=center] at (-4.6,-0.75) {reinforcement\\broadcast};
\end{tikzpicture}
$$

A neuron's steady, slow background activity is its **tonic** firing; a brief burst
usually driven by synaptic input is **phasic** activity. The reinforcement signal
of interest rides on the phasic dopamine response — the burst — while the tonic
level does other work. Keep the two apart: the RPE signal rides on the burst, not
the tonic level.

## Four signals that get conflated

Casual usage runs together four ideas that reinforcement learning keeps sharply
distinct, and the neuroscience only makes sense once they are separated. When we
label a neural signal by its _function_ — calling a dopamine burst a "reinforcement
signal" — we mean it behaves like, and is conjectured to function like, the
corresponding theoretical term in an equation. The four terms are reward,
reinforcement, value, and prediction error.[^sb-fourfold]

$$
% caption: Four distinct signals. Reward $R_t$ scores the moment; a value $v(s)$ or
% $q(s,a)$ predicts long-run return; a prediction error is expected minus received;
% the reinforcement signal is whatever actually drives the weight update — for a TD
% method, the TD error, not the raw reward.
\begin{tikzpicture}[>=stealth, font=\small,
  sig/.style={draw, minimum width=30mm, minimum height=13mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[sig] (rew)  at (0,0)     {reward $R_t$\\scores the moment};
  \node[sig] (val)  at (4.4,0)   {value $v(s)$\\predicts return};
  \node[sig] (pe)   at (0,-2.2)  {prediction error\\expected vs. got};
  \node[sig, draw=red, text=red] (reinf) at (4.4,-2.2) {reinforcement\\drives the update};
  % relationships
  \draw[->, black, thick] (rew) -- (pe) node[midway, left, font=\scriptsize] {input to};
  \draw[->, black, thick] (val) -- (pe) node[midway, right, font=\scriptsize] {input to};
  \draw[->, red, thick] (pe) -- (reinf) node[midway, below, font=\scriptsize] {is the};
\end{tikzpicture}
$$

**Reward** is the signal $R_t$. In reinforcement learning it defines the _problem_:
along with the environment, the reward signal specifies what the agent is trying to
achieve. It is best read as an abstraction. There is almost certainly no single
master reward wire in the brain; $R_t$ summarizes the overall effect of a multitude
of neural signals that assess the rewarding or punishing quality of sensations and
states, distributed across many sites.

**Value** is $v(s)$ or $q(s,a)$: an estimate of what is good or bad over the _long
run_. Values are predictions of the total reward an agent can expect to accumulate
from a state (or from an action in a state) into the future. Good decisions come
from choosing states with high estimated value, or actions with high estimated
action value — not from chasing immediate reward.

A **prediction error** measures a discrepancy between an expected and a received
signal. A **reward prediction error** (RPE) is specifically the gap in _reward_:
positive when the reward signal exceeds expectation, negative when it falls short,
zero when the forecast is exact. A TD error is a particular kind of RPE — one that
compares current and earlier expectations of reward over the long term, folding the
value change $\gamma v(S_{t+1}) - v(S_t)$ into the discrepancy rather than looking
at $R_t$ alone. When neuroscientists refer to RPEs they usually mean TD RPEs.

**Reinforcement** is the signal that actually _directs the change_ a learning
algorithm makes to its policy, values, or model. This is the one most often
misnamed, because it need not be the reward. For a TD method the reinforcement
signal at time $t$ is the TD error

$$
\delta_{t} \;=\; R_{t+1} + \gamma\, v(S_{t+1}) - v(S_t),
$$

the reward signal _adjusted_ by other information — here the value estimates.
Reward is a component of $\delta$, but not the whole of it; the extra term
$\gamma v(S_{t+1}) - v(S_t)$ is the higher-order part. This is why a dopamine burst
is a **reinforcement signal, not a reward signal**: it tracks $\delta$, and even
when reward occurs ($R_{t+1} \ne 0$) the burst can be silent if the reward was
fully predicted, because $\delta \approx 0$. Reward happened; reinforcement did
not.

| Signal | Symbol | What it is | In the brain |
| --- | --- | --- | --- |
| Reward | $R_t$ | scores the current moment; defines the problem | no single wire; a distributed summary |
| Value | $v(s),\, q(s,a)$ | prediction of long-run return | striatal / cortical estimates |
| Prediction error | RPE, TD error | expected minus received reward | conjectured in phasic dopamine |
| Reinforcement | $\delta$ (for TD) | signal that drives the weight update | phasic dopamine burst |

The distinctions are not pedantry. Neural activity related to reward processing
turns up in nearly every part of the brain, and representations of these four
signals tend to be strongly correlated, so an experiment that cannot _dissociate_
them cannot say which one it has found. Designing a task that separates a reward
signal from a value signal from a prediction error is most of the difficulty in
this field.

### A task that dissociates the three

To see why dissociation is hard and how it is done, take a cued task with a single
outcome. A cue predicts reward with probability $p$; on each trial the outcome is
either reward ($R=1$) or nothing ($R=0$). Value is the expected reward the cue
forecasts, $v = p$ (with $\gamma$ irrelevant here — the outcome follows the cue by a
fixed delay and there is no further state). The reward prediction error at the
outcome is $\delta = R - v$. If an experiment used only trials with $p = 1$, reward
and value and RPE would move together on every trial and no recording could tell
them apart: reward is always $1$, value is always $1$, RPE is always $0$. Varying
$p$ across cues, and reading out $R$ and $\delta$ separately, pulls the three apart.

| Cue $p$ | Outcome $R$ | Value $v=p$ | RPE $\delta=R-v$ | What the burst should do |
| --- | --- | --- | --- | --- |
| $1.0$ | $1$ (got it) | $1.0$ | $0.0$ | nothing — fully predicted |
| $0.5$ | $1$ (got it) | $0.5$ | $+0.5$ | fire — better than expected |
| $0.5$ | $0$ (nothing) | $0.5$ | $-0.5$ | dip — worse than expected |
| $0.0$ | $1$ (surprise) | $0.0$ | $+1.0$ | fire hard — pure surprise |

Reading down the table: the same reward $R=1$ produces three different phasic
responses ($0$, $+0.5$, $+1.0$) depending on the value the cue set up, so the
burst cannot be a reward signal. The same value $v=0.5$ produces both a positive
and a negative response depending on the outcome, so it cannot be a value signal
either. Only $\delta = R - v$ predicts every cell. This is the logic behind the
classic dopamine recordings: fix the reward, vary what the animal expects, and
watch the burst track the _difference_. Bayer and Glimcher (2005) pushed it
quantitatively, showing that the firing rate of midbrain dopamine neurons scales
with the numerical size of the prediction error rather than the reward.[^bayer]

## Actor and critic learning rules

The [dopamine lesson](/reinforcement-learning/minds-and-brains/dopamine-and-td-error)
read the basal ganglia as a neural actor-critic with dopamine as the shared
training signal. Here we spell out the _learning rules_ and ask whether they are
neurally plausible. The setup is a linear critic unit and a small network of
actor units, all receiving the same feature-vector representation of the state.[^sb-actorcritic]

Let $\mathbf{x}(S_t) = (x_1(S_t), \dots, x_n(S_t))^\top$ be the feature vector for
the current state — think of each $x_i$ as the presynaptic signal arriving at one
of the unit's synapses. The **critic** is a single linear neuron-like unit with
weight vector $\mathbf{w}$; its estimate is $\hat v(s, \mathbf{w}) = \mathbf{w}^\top
\mathbf{x}(s)$. The **actor** is a one-layer network of $k$ identical units, each
with its own weight vector $\boldsymbol{\theta}_j$; taking two actions, an actor
unit is a Bernoulli-logistic unit whose output $A_t \in \{0, 1\}$ (fire or not)
has probability $\pi(1 \mid s, \boldsymbol{\theta}) = 1 / (1 + \exp(-\boldsymbol{\theta}^\top \mathbf{x}(s)))$.
Both are updated by the same broadcast signal $\delta_t$ acting as reinforcement:

$$
\delta_t = R_{t+1} + \gamma\, \hat v(S_{t+1}, \mathbf{w}) - \hat v(S_t, \mathbf{w}),
\qquad
\mathbf{w} \gets \mathbf{w} + \alpha^{\mathbf{w}}\, \delta_t\, \mathbf{z}^{\mathbf{w}}_t,
\qquad
\boldsymbol{\theta} \gets \boldsymbol{\theta} + \alpha^{\boldsymbol{\theta}}\, \delta_t\, \mathbf{z}^{\boldsymbol{\theta}}_t .
$$

The two rules differ only in their **eligibility traces** $\mathbf{z}_t$, and that
difference is the whole point. The critic's trace accumulates the presynaptic
activity alone, $\mathbf{z}^{\mathbf{w}}_t = \lambda^{\mathbf{w}} \mathbf{z}^{\mathbf{w}}_{t-1} + \nabla \hat v(S_t, \mathbf{w}) = \lambda^{\mathbf{w}} \mathbf{z}^{\mathbf{w}}_{t-1} + \mathbf{x}(S_t)$:
a synapse becomes eligible for modification whenever it carries presynaptic input,
independent of whether the postsynaptic neuron fires. This is a **non-contingent**
eligibility trace. The actor's trace, by contrast, includes the postsynaptic factor:
$\mathbf{z}^{\boldsymbol{\theta}}_t = \lambda^{\boldsymbol{\theta}} \mathbf{z}^{\boldsymbol{\theta}}_{t-1} + \nabla \ln \pi(A_t \mid S_t, \boldsymbol{\theta})$,
and that gradient works out to $\bigl(A_t - \pi(A_t \mid S_t, \boldsymbol{\theta})\bigr)\, \mathbf{x}(S_t)$
— a product of a presynaptic factor $\mathbf{x}(S_t)$ and a postsynaptic factor
$\bigl(A_t - \pi(\cdot)\bigr)$ that depends on whether the unit actually fired. This is
a **contingent** eligibility trace.

> **Definition (Contingent vs. non-contingent eligibility).** A **non-contingent**
> eligibility trace increments on **presynaptic** activity alone; a **contingent**
> trace increments on the coincidence of **pre- and postsynaptic** activity. The
> postsynaptic contingency is the _only_ difference between the actor and critic
> rules: the critic learns to predict (no action taken), so it needs no record of
> what it did; the actor learns to control, so it must remember _which action_ it
> emitted in order to credit or blame it.

$$
% caption: The only difference between the two rules. The critic synapse's trace
% depends on presynaptic input alone (non-contingent); the actor synapse's trace
% depends on the coincidence of presynaptic input and the unit's own firing
% (contingent), so it records which action was taken and can assign credit to it.
\begin{tikzpicture}[>=stealth, font=\small,
  cell/.style={draw, circle, minimum size=8mm, inner sep=0pt, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % --- critic (left) ---
  \node[cell, draw=acc] (cpre) at (0,0.9) {pre};
  \node[cell, draw=acc] (cpost) at (2.2,0.9) {V};
  \draw[->, acc, thick] (cpre) -- (cpost);
  \node[acc, font=\footnotesize, anchor=south, align=center] at (1.1,1.55) {critic (non-contingent)};
  \node[black, font=\scriptsize, anchor=north, align=center] at (1.1,0.3) {trace uses pre only};
  % --- actor (right) ---
  \begin{scope}[xshift=6.2cm]
    \node[cell, draw=red] (apre) at (0,0.9) {pre};
    \node[cell, draw=red] (apost) at (2.2,0.9) {A};
    \draw[->, red, thick] (apre) -- (apost);
    \node[red, font=\footnotesize, anchor=south, align=center] at (1.1,1.55) {actor (contingent)};
    \node[black, font=\scriptsize, anchor=north, align=center] at (1.1,0.3) {trace uses pre x post};
    % feedback loop showing postsynaptic firing feeds the trace
    \draw[->, red, thick] (apost) to[bend right=45] (1.1,0.15);
  \end{scope}
  % --- broadcast delta to both ---
  \node[black, font=\footnotesize] (delta) at (4.1,-1.4) {broadcast delta (dopamine)};
  \draw[->, black, thick] (delta) to[bend left=15] (1.1,0.05);
  \draw[->, black, thick] (delta) to[bend right=15] (7.3,0.05);
\end{tikzpicture}
$$

Both rules are descendants of Hebb's classic proposal — that when a presynaptic
signal repeatedly helps fire a postsynaptic neuron, the synapse strengthens. But
each adds structure Hebb did not have. The critic rule involves two interacting
factors: the reinforcement signal $\delta$ and a presynaptic-only trace;
neuroscientists call this a **two-factor learning rule**. The actor rule involves
three: $\delta$, a presynaptic factor, and a postsynaptic factor — a **three-factor
learning rule**. In each case the reinforcement signal $\delta$ arrives to modulate
a synapse whose eligibility trace was set by _local_ pre/post activity, possibly
long before; the trace bridges the delay between the eligible event and the arrival
of reinforcement.

> **Definition (Three-factor learning rule).** A synaptic update driven by the
> conjunction of three signals: **presynaptic** activity, **postsynaptic** activity,
> and a diffuse **neuromodulatory** reinforcement signal (dopamine, standing in for
> $\delta$). Unlike Hebb's two-factor product of simultaneous pre- and postsynaptic
> activity, the relative _timing_ of the three factors is essential: an eligibility
> trace set by pre/post coincidence must still be open when reinforcement arrives.

### One update, worked through

The rules are abstract until real numbers pass through them. Take a two-state,
two-action task with a one-hot state representation: $\mathbf{x}(s_1) = (1, 0)^\top$
and $\mathbf{x}(s_2) = (0, 1)^\top$, so each state activates a single synapse on the
critic and on each actor unit. Start both learners blank: critic weights
$\mathbf{w} = (0, 0)^\top$ and actor weights $\boldsymbol{\theta} = (0, 0)^\top$.
With $\boldsymbol{\theta} = \mathbf{0}$ the actor's firing probability is
$\pi(1 \mid s_1) = 1/(1 + e^{0}) = 0.5$: an unbiased coin, exactly what a fresh
policy should be. Set $\gamma = 0.9$ and step sizes $\alpha^{\mathbf{w}} =
\alpha^{\boldsymbol{\theta}} = 0.2$; take the traces at their simplest, $\lambda = 0$,
so $\mathbf{z}$ is just the current gradient.

Now run one transition. In $s_1$ the actor fires, $A = 1$ (it came up heads); the
action carries the agent to $s_2$ and yields reward $R = 1$. Compute the value
estimates from the still-zero weights: $\hat v(s_1) = \mathbf{w}^\top \mathbf{x}(s_1) = 0$
and $\hat v(s_2) = 0$. The reinforcement signal is

$$
\delta = R + \gamma\, \hat v(s_2) - \hat v(s_1) = 1 + 0.9 \cdot 0 - 0 = 1.0 .
$$

**Critic update (two-factor).** The non-contingent trace is the presynaptic input
alone, $\mathbf{z}^{\mathbf{w}} = \mathbf{x}(s_1) = (1, 0)^\top$. So

$$
\mathbf{w} \gets \mathbf{w} + \alpha^{\mathbf{w}}\, \delta\, \mathbf{z}^{\mathbf{w}}
= (0,0)^\top + 0.2 \cdot 1.0 \cdot (1,0)^\top = (0.2,\; 0)^\top .
$$

Only the synapse carrying $s_1$'s input moved, and it moved _up_: $\hat v(s_1)$ rises
from $0$ to $0.2$, a step toward the reward that just arrived. The critic did not
need to know which action was taken — it is learning to predict, not to choose.

**Actor update (three-factor).** The contingent trace multiplies the presynaptic
input by the postsynaptic factor $\bigl(A - \pi(A \mid s_1)\bigr) = (1 - 0.5) = 0.5$,
giving $\mathbf{z}^{\boldsymbol{\theta}} = 0.5 \cdot \mathbf{x}(s_1) = (0.5, 0)^\top$.
So

$$
\boldsymbol{\theta} \gets \boldsymbol{\theta} + \alpha^{\boldsymbol{\theta}}\, \delta\, \mathbf{z}^{\boldsymbol{\theta}}
= (0,0)^\top + 0.2 \cdot 1.0 \cdot (0.5,0)^\top = (0.1,\; 0)^\top .
$$

The $s_1$ weight rises to $0.1$, which lifts the firing probability from $0.5$ to
$\pi(1 \mid s_1) = 1/(1 + e^{-0.1}) \approx 0.525$: the action that was taken becomes
a little more likely, because the reinforcement that followed it was positive. Had
the same $\delta > 0$ arrived after the unit stayed _silent_ ($A = 0$), the
postsynaptic factor would have been $(0 - 0.5) = -0.5$ and the weight would have
moved the other way, making firing _less_ likely — the trace remembers what the unit
did and signs the update accordingly.

$$
% caption: One shared reinforcement signal, two updates. With $\delta = 1$, the
% critic synapse for $s_1$ moves to $0.2$ (value toward accuracy) and the actor
% synapse for $s_1$ moves to $0.1$ (policy toward the taken action). The critic's
% trace is the presynaptic input; the actor's is that input scaled by $A - \pi = 0.5$.
\begin{tikzpicture}[>=stealth, font=\small,
  box/.style={draw, thick, minimum width=44mm, minimum height=17mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[font=\footnotesize] (delta) at (0,1.7) {shared delta = 1.0};
  \node[box, draw=acc, text=acc] (crit) at (-3.1,0) {critic (two-factor)\\trace = x = (1, 0)\\w: 0 to 0.2};
  \node[box, draw=red, text=red] (act)  at (3.1,0)  {actor (three-factor)\\trace = (A - pi) x = (0.5, 0)\\theta: 0 to 0.1};
  \draw[->, acc, thick] (delta) to[bend right=18] (crit.north);
  \draw[->, red, thick] (delta) to[bend left=18] (act.north);
  \node[acc, font=\scriptsize, anchor=north, align=center] at (-3.1,-1.15) {value toward accuracy};
  \node[red, font=\scriptsize, anchor=north, align=center] at (3.1,-1.15) {policy toward taken action};
\end{tikzpicture}
$$

The single number $\delta$ did two jobs at once. Broadcast unchanged to both
learners, it pushed the critic's estimate toward the reward it under-predicted and
pushed the actor's policy toward the action it had just tried. Nothing in the wiring
distinguishes the two updates except the trace each synapse keeps — non-contingent
for the predictor, contingent for the controller.

### The timing problem, and STDP

Real neurons force a subtlety the idealized units hide. When a presynaptic spike
arrives, the postsynaptic neuron does not respond instantly; the chemistry of
transmission takes tens of milliseconds. For the contingent actor rule to assign
credit correctly, the presynaptic factor that sets the trace must be a genuine
_cause_ of the postsynaptic firing that defines it — which means a realistic actor
unit has to respect this activation time. (Do not confuse activation time, tens of
milliseconds, with the much longer interval an eligibility trace must span to
bridge a delayed reward.)

The brain has a plasticity mechanism with exactly the right shape:
**spike-timing-dependent plasticity** (STDP). STDP is Hebbian, but the direction of
the change depends on the relative _timing_ of pre- and postsynaptic spikes: a
synapse strengthens if its presynaptic spike arrives shortly _before_ the
postsynaptic neuron fires, and weakens if the order is reversed. STDP thus takes
activation time into account — one of the ingredients an actor-like rule needs.
Write $\Delta t = t_{\text{post}} - t_{\text{pre}}$ for the timing offset. The
canonical STDP curve, mapped out in cortical and hippocampal slices, is a sharp
asymmetric window: $\Delta w > 0$ for $\Delta t > 0$ (pre before post, the causal
order), $\Delta w < 0$ for $\Delta t < 0$ (post before pre), with both lobes
decaying over tens of milliseconds and a discontinuity at $\Delta t = 0$. Markram
and colleagues (1997) demonstrated the timing dependence between connected cortical
pyramidal neurons; Bi and Poo (1998) mapped the full window in hippocampal culture,
fixing the tens-of-milliseconds width that has anchored the field since.[^stdp]

$$
% caption: The spike-timing-dependent plasticity window. Plotting the change in
% synaptic weight against the timing offset $\Delta t = t_{\text{post}} - t_{\text{pre}}$:
% pre before post ($\Delta t > 0$) potentiates the synapse, post before pre
% ($\Delta t < 0$) depresses it, and both lobes decay over tens of milliseconds. The
% causal order — presynaptic input helping cause the postsynaptic spike — is the one
% that strengthens the synapse.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, black, thick] (-3.6,0) -- (3.9,0);
  \draw[->, black, thick] (0,-2.0) -- (0,2.2);
  \node[black, font=\scriptsize, anchor=north east] at (3.9,-0.08) {dt (ms)};
  \node[black, font=\scriptsize, anchor=south, align=center] at (0.0,2.2) {dw};
  % potentiation lobe: pre before post, dt>0, decaying positive
  \draw[acc, very thick] plot[domain=0.05:3.4, samples=60] (\x, {1.8*exp(-\x/1.15)});
  % depression lobe: post before pre, dt<0, decaying negative
  \draw[red, very thick] plot[domain=-3.4:-0.05, samples=60] (\x, {-1.5*exp(\x/1.15)});
  % labels for lobes
  \node[acc, font=\scriptsize, anchor=west, align=left] at (1.1,1.15) {pre before post\\strengthen};
  \node[red, font=\scriptsize, anchor=east, align=right] at (-1.1,-1.05) {post before pre\\weaken};
  \node[black, font=\scriptsize, anchor=north] at (1.7,-0.12) {dt $>$ 0};
  \node[black, font=\scriptsize, anchor=north] at (-1.7,-0.12) {dt $<$ 0};
\end{tikzpicture}
$$

And there is more: neuroscientists have found **reward-modulated STDP**, a
three-factor form in which the STDP change occurs _only if_ a neuromodulatory input
(dopamine) arrives within a time window after the paired pre/post spikes. The STDP
window sets which synapses become _eligible_; the dopamine pulse decides whether the
eligible change is actually committed. Reynolds and Wickens (2002) proposed exactly
this three-factor corticostriatal rule — an STDP-eligible synapse held in a plastic
state until dopamine gates the change.[^reynolds] Yagishita and colleagues (2014)
gave it a mechanism: at the dendritic spines of striatal medium spiny neurons, the
spine enlarges (the structural correlate of potentiation) only when dopamine arrives
in a narrow window, on the order of $0.3$ to $2$ seconds, after paired pre/post
activity. Experiments have shown lasting changes in corticostriatal synapses when
dopamine follows a pre-then-post pairing within a window that can last up to about
ten seconds — direct, if indirect, evidence for contingent eligibility traces with
long time courses, at exactly the synapses where the actor's learning is
hypothesized to live.[^sb-stdp]

The two rules differ in _which_ factors must line up in time. The critic is
two-factor: a presynaptic-marked synapse, then dopamine. The actor is three-factor:
a presynaptic spike, then a postsynaptic spike (this pairing sets the contingent
trace), then dopamine within the window. Laying the signals on a timeline makes the
difference legible.

$$
% caption: Two-factor versus three-factor timing. The critic (top) needs only a
% presynaptic-marked synapse and a later dopamine pulse. The actor (bottom) needs a
% presynaptic spike, then a postsynaptic spike setting the contingent trace, then
% dopamine arriving inside the eligibility window (Yagishita: ~0.3-2 s, up to ~10 s)
% to commit the change. Time runs left to right.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % --- critic timeline (top) ---
  \draw[black, thick] (0,1.6) -- (7.4,1.6);
  \node[acc, font=\footnotesize, anchor=east] at (-0.15,1.6) {critic};
  \draw[acc, very thick] (0.6,1.6) -- (0.6,2.15);
  \node[acc, font=\scriptsize, anchor=south] at (0.6,2.15) {pre};
  \draw[red, very thick] (5.6,1.6) -- (5.6,2.15);
  \node[red, font=\scriptsize, anchor=south] at (5.6,2.15) {dopamine};
  % eligibility span critic
  \draw[acc!55, line width=2.4pt, opacity=0.5] (0.6,1.6) -- (5.6,1.6);
  % --- actor timeline (bottom) ---
  \draw[black, thick] (0,0) -- (7.4,0);
  \node[red, font=\footnotesize, anchor=east] at (-0.15,0) {actor};
  \draw[acc, very thick] (0.6,0) -- (0.6,0.55);
  \node[acc, font=\scriptsize, anchor=south] at (0.6,0.55) {pre};
  \draw[black, very thick] (1.5,0) -- (1.5,0.55);
  \node[black, font=\scriptsize, anchor=south] at (1.5,0.55) {post};
  \draw[red, very thick] (5.6,0) -- (5.6,0.55);
  \node[red, font=\scriptsize, anchor=south] at (5.6,0.55) {dopamine};
  % contingent trace span
  \draw[red!55, line width=2.4pt, opacity=0.5] (1.5,0) -- (5.6,0);
  \node[black, font=\scriptsize, anchor=north, align=center] at (3.55,-0.1) {contingent trace open (window)};
  \node[black, font=\scriptsize, anchor=north east] at (7.4,-0.55) {time};
\end{tikzpicture}
$$

## Beyond the single loop

We now have the tissue-level vocabulary and the two learning rules: a two-factor
critic and a three-factor actor, both driven by a broadcast $\delta$, both grounded
in reward-modulated STDP at corticostriatal synapses. That is one model-free loop.
But the brain is not one loop. The next part asks where the actor rule came from — 
Klopf's hedonistic neuron — whether teams of such neurons can learn together, and
where the model-based systems live that the single loop leaves out. The recurring
finding is that the brain is not one reinforcement-learning algorithm but several
interacting systems.

This continues in [The Brain's Several Learning Systems](/reinforcement-learning/minds-and-brains/several-learning-systems).

[^sb-intro]: **Sutton & Barto**, _Reinforcement Learning: An Introduction_ (2nd ed.), Ch. 15 — Neuroscience: the chapter's framing that reinforcement-learning theory and the neuroscience of reward-based learning form a fruitful two-way channel, without claiming the brain implements any single algorithm.
[^sb-neuron]: **Sutton & Barto**, §15.1 — Neuroscience Basics: neurons, cell body, dendrites, axon, action potentials/spikes, firing rate, and the axonal arbor whose active conduction lets one spike reach many targets.
[^sb-synapse]: **Sutton & Barto**, §15.1 — Neuroscience Basics: synapse, presynaptic/postsynaptic sides, neurotransmitter, synaptic cleft, receptors, synaptic efficacy, and synaptic plasticity as a primary mechanism of learning (dopamine acting through at least five receptor types).
[^sb-neuromod]: **Sutton & Barto**, §15.1 — Neuroscience Basics: neuromodulators and neuromodulatory systems as clusters of widely branching neurons able to distribute a scalar signal (like a reinforcement signal) to distributed sites; §15.4 for the scale of dopamine axonal arbors (roughly 500,000 contacts per axon).
[^sb-loop]: **Sutton & Barto**, §15.1; §15.7 — the cortico-striatal loop, the striatum as the basal ganglia's input structure with dorsal and ventral subdivisions, and midbrain dopamine (SNpc, VTA) broadcasting phasic reinforcement to the striatum.
[^sb-fourfold]: **Sutton & Barto**, §15.2 — Reward Signals, Reinforcement Signals, Values, and Prediction Errors: the careful separation of $R_t$ (reward, defining the problem), value estimates $V$/$Q$, prediction errors (RPEs, with the TD error a special case), and the reinforcement signal that drives the update — the TD error, not the raw reward.
[^sb-actorcritic]: **Sutton & Barto**, §15.8 — Actor and Critic Learning Rules: the linear critic unit $\hat v(s,\mathbf{w}) = \mathbf{w}^\top\mathbf{x}(s)$, the Bernoulli-logistic actor units, the shared reinforcement $\delta_t$, and the non-contingent (critic) versus contingent (actor) eligibility traces, with the actor's $\bigl(A_t - \pi(A_t\mid S_t,\boldsymbol{\theta})\bigr)\mathbf{x}(S_t)$ trace. The postsynaptic contingency is the only difference between the rules.
[^sb-stdp]: **Sutton & Barto**, §15.8 — Actor and Critic Learning Rules: two-factor versus three-factor learning rules, the activation-time subtlety, spike-timing-dependent plasticity (STDP), and reward-modulated STDP at corticostriatal synapses with dopamine as the modulator within a window up to ~10 seconds (Yagishita et al., 2014).
[^bayer]: **Bayer, H. M., & Glimcher, P. W.** (2005), "Midbrain dopamine neurons encode a quantitative reward prediction error signal," _Neuron_ 47(1), 129-141: the firing rate of midbrain dopamine neurons scales with the numerical magnitude of the reward prediction error, and does so asymmetrically for positive versus negative errors.
[^stdp]: **Markram, H., Lübke, J., Frotscher, M., & Sakmann, B.** (1997), "Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs," _Science_ 275(5297), 213-215 — timing dependence of plasticity between connected cortical pyramidal neurons; and **Bi, G., & Poo, M.** (1998), "Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type," _J. Neurosci._ 18(24), 10464-10472 — the full asymmetric STDP window in hippocampal culture, with tens-of-milliseconds lobes.
[^reynolds]: **Reynolds, J. N. J., & Wickens, J. R.** (2002), "Dopamine-dependent plasticity of corticostriatal synapses," _Neural Networks_ 15(4-6), 507-521: a three-factor corticostriatal plasticity rule in which appropriately-timed pre/post activity is committed to a lasting change only when dopamine is present.