---
title: "The Brain's Several Learning Systems"
module: Reinforcement Learning in Minds and Brains
moduleNumber: 6
lessonNumber: 8
order: 608
summary: >
  The actor's three-factor rule has an ancestor in Klopf's hedonistic neuron — a
  single cell as a reinforcement-seeking agent — and a bacterium's run-and-twiddle
  shows the Law of Effect with no synapses at all. Teams of such neurons implement
  policy gradient collectively, the broadcast reward replacing backpropagation. And
  the brain is not only model-free: outcome devaluation, prefrontal value coding,
  and hippocampal forward sweeps localize a model-based system. The recurring
  conclusion is that the brain is several interacting learning systems, not one
  algorithm.
topics: [Minds and Brains]
sources:
  - book: Sutton & Barto
    ref: "Ch. 15 — Neuroscience; §15.9 Hedonistic Neurons; §15.10 Collective Reinforcement Learning"
  - book: Sutton & Barto
    ref: "§15.11 Model-based Methods in the Brain; §15.13 Summary"
---

This builds on [The Neuroscience of Reinforcement](/reinforcement-learning/minds-and-brains/neuroscience-of-reinforcement),
which built the tissue-level vocabulary and the actor and critic learning rules — a
two-factor critic and a three-factor actor, both driven by a broadcast dopamine
signal. That was one model-free loop. This part asks where the actor rule came from,
whether teams of neuron-like agents can learn together, and where the model-based
systems live that a single loop leaves out — and arrives at the module's recurring
conclusion: the brain is several interacting systems, not one algorithm.

## Hedonistic neurons

The neuron-like actor unit, with its Law-of-Effect learning rule, has an
intellectual ancestor: the **hedonistic-neuron hypothesis** of A. H. Klopf (1972,
1982). Klopf conjectured that individual neurons are themselves reinforcement-seeking
agents — that a single neuron seeks to maximize the difference between synaptic input
treated as _rewarding_ and synaptic input treated as _punishing_, adjusting the
efficacies of its synapses on the basis of the rewarding or punishing _consequences
of its own action potentials_. A neuron, on this view, can be trained by
response-contingent reinforcement the way an animal is trained in an instrumental
conditioning task.[^sb-klopf]

The mechanism Klopf proposed is the three-factor rule in embryo. When a neuron
fires, all the synapses that were active in contributing to that spike become
**eligible** for change — Klopf introduced the term. If the spike is followed within
some window by increased reward, the eligible synapses strengthen; if by punishment,
they weaken. That is a contingent eligibility trace: triggered by the coincidence of
presynaptic activity with the postsynaptic firing that the presynaptic activity
helped cause. Klopf's traces were richer than the ones we use. He imagined the shape
of a trace as a histogram of the durations of the many feedback loops the neuron sits
in — some closing within the brain and body, others reaching out through the
environment via the animal's muscles and senses. The exponentially decaying traces of
our algorithms are simplified stand-ins for this richer conception.

$$
% caption: Klopf's hedonistic neuron. A neuron's own action potential closes feedback
% loops of varying length — through the body, out through the environment — that
% return reward or punishment; the neuron marks the synapses that fired it as eligible
% and adjusts them by the returning signal, learning like an animal in an instrumental
% task.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[fill=acc!8, draw=acc, thick] (0,0) circle (7mm);
  \node[font=\footnotesize] at (0,0) {neuron};
  % action out (top)
  \draw[->, acc, thick] (0.55,0.45) -- (2.7,0.45);
  \node[acc, font=\footnotesize, anchor=south] at (1.6,0.55) {f\/ires};
  % short internal loop back
  \draw[->, black, thick] (2.7,0.6) to[bend left=45] (-0.6,0.35);
  \node[black, font=\scriptsize, anchor=south] at (1.05,1.5) {short loop};
  % long environment loop (below)
  \draw[->, red, thick] (2.7,0.3) to[out=-45,in=-25, looseness=1.7] (-0.55,-0.35);
  \node[red, font=\scriptsize, anchor=north] at (1.1,-1.7) {long loop (through the environment)};
  % returning reward
  \node[red, font=\footnotesize, anchor=east, align=right] at (-0.9,-0.3) {reward /\\punishment};
\end{tikzpicture}
$$

The hypothesis is less far-fetched than it sounds. A single-celled organism can
already seek some stimuli and avoid others: the bacterium _Escherichia coli_ swims by
rotating flagella, and it biases its tumbles so that it persists up gradients of
useful molecules and away from harmful ones — a strategy the AI pioneer Oliver
Selfridge called "run and twiddle," keep going the same way if things are improving,
otherwise move around.[^koshland] The mechanism is memoryless minimization made into
a search: the bacterium has no map of the gradient and cannot steer toward the
source, but it can compare the concentration it senses now against the recent past,
and it lets that comparison set the odds of tumbling. Written as a loop, the strategy
is short.

```algorithm
Run-and-twiddle (E. coli chemotaxis)
repeat
  swim straight ahead for a short interval (a "run")
  let $\Delta c \gets c_{\text{now}} - c_{\text{recent}}$   // change in attractant sensed
  if $\Delta c > 0$ then                                    // things improving
    keep the current heading (suppress tumbling)
  else                                                       // not improving
    tumble: pick a new heading at random ("twiddle")
until the cell dies or the gradient is exhausted
```

Nothing in the loop knows the gradient's direction. Suppressing the tumble whenever
things improve, and re-randomizing whenever they do not, is enough to bias the
undirected walk uphill: runs up the gradient are extended, runs down it are cut
short, and the net drift climbs toward the attractant. It is the Law of Effect
running in a bacterium — reinforcement biasing the persistence of a behavior — with
no synapses and no lasting memory.

$$
% caption: Run-and-twiddle as a biased random walk. Each run is a straight swim; when
% the sensed attractant is rising the cell keeps its heading, when it is falling the
% cell tumbles to a random new direction. Extending good runs and cutting short bad
% ones drifts the undirected walk up the gradient toward the source.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % gradient shading suggestion: source at right
  \node[red, font=\footnotesize, anchor=west] at (4.0,1.55) {attractant source};
  \fill[red] (4.7,1.1) circle (2.2pt);
  \node[black, font=\scriptsize, anchor=west] at (-3.5,-1.7) {higher concentration to the right};
  % biased walk path: long runs rightish, short runs elsewhere, tumbles at joints
  \draw[acc, very thick] (-3.2,-0.9) -- (-1.6,-0.4); % good run
  \draw[red, thick] (-1.6,-0.4) -- (-1.9,0.15);      % short bad run then tumble
  \draw[acc, very thick] (-1.9,0.15) -- (-0.2,0.55); % good run
  \draw[red, thick] (-0.2,0.55) -- (-0.35,0.05);     % short bad run
  \draw[acc, very thick] (-0.35,0.05) -- (1.5,0.65); % good run
  \draw[red, thick] (1.5,0.65) -- (1.3,1.05);        % short bad run
  \draw[acc, very thick] (1.5,0.65) -- (3.4,0.95);   % good run toward source
  % tumble markers
  \foreach \p in {(-1.6,-0.4),(-1.9,0.15),(-0.2,0.55),(-0.35,0.05),(1.5,0.65)}
    \fill[black] \p circle (1.4pt);
  \node[acc, font=\scriptsize, anchor=south] at (-2.4,-0.75) {run (improving)};
  \node[red, font=\scriptsize, anchor=north] at (0.9,1.0) {twiddle};
\end{tikzpicture}
$$

A neuron, embedded in its own dense web of feedback loops, might "swim" the same way
through the space of its inputs, unlike the bacterium retaining, in its synaptic
strengths, a lasting record of what it learned. The discovery of STDP, and especially
of reward-modulated STDP, suggests Klopf's ideas may not have been far off the mark.

## Collective reinforcement learning

Klopf pushed the idea one step further: if individual neurons are reinforcement-seeking
agents, then much of intelligent behavior might be the _collective_ behavior of a
population of such self-interested agents interacting in the immense society that is a
nervous system. This reframes the actor of the previous section. The actor is not one
unit but $k$ units, each a hedonistic neuron, each receiving the _same_ broadcast
$\delta$. Because each unit tries to make $\delta$ as large as possible, $\delta$
effectively acts as a common reward signal for the whole team — a **cooperative game**,
or **team problem**, in which every agent seeks to increase the _same_ signal.[^sb-collective]

$$
% caption: A team of hedonistic neurons. Each actor unit receives the same broadcast
% reinforcement delta and outputs one component of the team's collective action; the
% shared signal evaluates the whole pattern, so no unit can move it alone and credit
% must be apportioned across the team — the structural credit-assignment problem.
\begin{tikzpicture}[>=stealth, font=\small,
  unit/.style={draw, circle, minimum size=8mm, inner sep=0pt, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % state input
  \node[black, font=\footnotesize] (state) at (-3.4,0) {state x(s)};
  % three actor units
  \node[unit, draw=acc] (u1) at (0,1.5)  {a1};
  \node[unit, draw=acc] (u2) at (0,0)    {a2};
  \node[unit, draw=acc] (u3) at (0,-1.5) {a3};
  \foreach \u in {u1,u2,u3} \draw[->, black, thick] (state) -- (\u);
  % collective action
  \node[black, font=\footnotesize, align=center] (act) at (3.4,0) {collective\\action};
  \foreach \u in {u1,u2,u3} \draw[->, acc, thick] (\u) -- (act);
  % common reward signal from evaluation, back to all
  \node[red, font=\footnotesize] (delta) at (0,-3.0) {common delta (shared reward)};
  \draw[->, red, thick] (act) to[bend left=25] (delta);
  \foreach \u in {u1,u2,u3} \draw[->, red!70, thick, dashed] (delta) to[bend right=12] (\u);
\end{tikzpicture}
$$

Two features make the team problem hard. First, the common signal $\delta$ evaluates
the _pattern_ produced by the entire population — the **collective action** — so any
single agent contributes just one component of what is scored. Learning therefore
requires solving a **structural credit-assignment problem**: which members deserve
credit for a favorable signal, or blame for an unfavorable one? Second, each agent is
embedded in the others: from any one unit's vantage the other units are part of its
environment, since both its input and the reward it sees depend on how everyone else
behaves. Each unit partially observes the true state, cannot see the others' actions,
and has its reward buried in noise created by the rest.

To see the difficulty concretely, take the smallest team that has it: two
Bernoulli-logistic units, $U_1$ and $U_2$, each firing $A_i \in \{0, 1\}$, sharing a
single reinforcement signal $\delta$ that scores their _joint_ output. Suppose the
team is rewarded ($\delta > 0$) exactly when both fire together, $A_1 = A_2 = 1$, and
penalized otherwise. On a trial where both fire and $\delta$ comes back positive, what
should $U_1$ conclude? From its own vantage it fired and got rewarded, so its
contingent trace $(A_1 - \pi_1)\mathbf{x} = (1 - \pi_1)\mathbf{x} > 0$ tells it to
make firing more likely. But $U_1$ cannot see $A_2$; it has no way to know whether the
reward came because _it_ fired or because its teammate happened to fire at the same
time. If on the next such trial $U_1$ fires while $U_2$ stays silent, the reward
vanishes, and $U_1$'s trace now points the update the other way. A single unit's
action and the team's reward are related only as a **statistical correlation**, one
that has to be estimated across many trials — this is the structural credit-assignment
problem, and no single trial resolves it.

$$
% caption: Structural credit assignment in a two-unit team. Both units fire and the
% shared delta comes back positive; from unit 1's local view it fired and was
% rewarded, but it cannot see unit 2, so it cannot tell whether its own action or its
% teammate's caused the reward. Only the average over many trials disentangles them.
\begin{tikzpicture}[>=stealth, font=\small,
  unit/.style={draw, circle, minimum size=9mm, inner sep=0pt, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[unit, draw=acc] (u1) at (0,0.9)  {A1=1};
  \node[unit, draw=acc] (u2) at (0,-0.9) {A2=1};
  \node[draw, thick, minimum width=20mm, minimum height=16mm, align=center, font=\footnotesize] (eval) at (3.2,0) {evaluate\\joint output};
  \draw[->, acc, thick] (u1) -- (eval);
  \draw[->, acc, thick] (u2) -- (eval);
  \node[red, font=\footnotesize, anchor=west] (d) at (5.0,0) {delta $>$ 0};
  \draw[->, red, thick] (eval) -- (d);
  \draw[->, red!70, thick, dashed] (d.north) to[out=110,in=20] (u1.east);
  \draw[->, red!70, thick, dashed] (d.south) to[out=250,in=340] (u2.east);
  \node[black, font=\scriptsize, anchor=north, align=center] at (0,-1.9) {who earned the credit?};
\end{tikzpicture}
$$

Two ingredients let a team learn anyway. Each member needs a **contingent** eligibility
trace to correlate its own action with the reward that follows — non-contingent traces
support prediction (the critic) but not control. And each member needs **variability** in
its output, so the team explores the space of collective actions; a Bernoulli-logistic
unit supplies this automatically, its firing probability $\pi_i$ nudged but never pinned
to $\{0,1\}$. One result closes the loop:
a team of Bernoulli-logistic REINFORCE units — even interconnected into a multilayer
network, even with each unit's reward corrupted by large noise and lacking full state
information — implements a policy-gradient algorithm _as a whole_, ascending the average
reward rate of the team's common signal.[^sb-williams] Where a conventional network would
need error backpropagation to train its hidden units, here the broadcast reward signal
replaces backprop. Slower in practice, but far more plausible as a neural mechanism —
especially in light of reward-modulated STDP.

## Model-based methods in the brain

Everything so far has been **model-free**: the actor-critic loop learns values and a
policy by prediction error, with no explicit model of how the world transitions or
where reward comes from. But the brain is not only model-free. Reinforcement learning's
distinction between model-free and model-based methods maps onto a distinction psychology
draws between **habitual** and **goal-directed** behavior — and the neural evidence says
the brain runs _both_, in different structures, and arbitrates between them.[^sb-modelbased]

The cleanest evidence comes from **outcome-devaluation** experiments. Train a rat to
press a lever for food, then devalue the food (feed the rat to satiety, or pair the food
with illness). A **goal-directed** (model-based) system, representing the outcome and
consulting its current value, immediately stops pressing. A **habitual** (model-free)
system, having cached a stimulus-response value with no link to the outcome, keeps
pressing. Inactivating specific regions and watching which behavior survives localizes
the two systems:[^sb-devaluation]

| Inactivated region | Impairs | Spared control | Reading |
| --- | --- | --- | --- |
| **Dorsolateral striatum** (DLS) | habit learning | goal-directed | model-free |
| **Dorsomedial striatum** (DMS) | goal-directed | habit | model-based |

The DLS looks more model-free, the DMS more model-based — a sign that the simple "actor
in the dorsal striatum" picture is too coarse.

$$
% caption: Two systems, dissociated by outcome devaluation. Inactivating the dorsolateral
% striatum spares goal-directed control (model-based, sensitive to devaluation);
% inactivating the dorsomedial striatum spares habit (model-free, insensitive). The
% behaviors, and the structures, come apart.
\begin{tikzpicture}[>=stealth, font=\small,
  sys/.style={draw, minimum width=34mm, minimum height=15mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[sys, draw=acc, text=acc] (mf) at (0,0)   {model-free / habit\\dorsolateral striatum\\keeps pressing};
  \node[sys, draw=red, text=red] (mb) at (5.6,0) {model-based / goal\\dorsomedial striatum\\stops pressing};
  \node[black, font=\footnotesize, anchor=south, align=center] at (2.8,1.35) {after outcome devaluation};
  \draw[<->, black, thick] (mf) -- (mb) node[midway, below, font=\scriptsize] {arbitration};
\end{tikzpicture}
$$

Model-based behavior reaches beyond the striatum, and the two components of a model
localize to distinct structures:[^sb-hippocampus]

| Model component | Structure | Signature |
| --- | --- | --- |
| Reward $r(s,a)$ | orbitofrontal cortex (OFC) | tracks subjective and expected reward value |
| Transition $p(s'\mid s,a)$ | hippocampus | forward sweeps at choice points |

The **prefrontal cortex** — seat of executive function, planning, and decision — is
implicated, and within it the **orbitofrontal cortex** (OFC): recordings in monkeys and
human neuroimaging show OFC activity tracking the _subjective_ reward value of stimuli and
the reward _expected_ from actions. The **hippocampus** supplies the transition component:
its place-learning led Tolman to posit cognitive maps, and when a rat pauses at a choice
point, decoding the hippocampal population reveals the representation of space **sweeping
forward** along the candidate paths — not backward — predicting the rat's subsequent
choice.

$$
% caption: Hippocampal forward sweeps. At a choice point the decoded representation of
% space runs ahead down each candidate path (not backward), simulating possible futures
% before the rat commits — a neural rollout, and evidence that the hippocampus carries the
% transition model used in planning.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % T-maze corridor
  \draw[black, thick] (-0.35,-1.6) -- (-0.35,0.2) -- (-2.4,0.2) -- (-2.4,1.0);
  \draw[black, thick] (0.35,-1.6) -- (0.35,0.2) -- (2.4,0.2) -- (2.4,1.0);
  \draw[black, thick] (-2.4,0.9) -- (-1.7,0.9);
  \draw[black, thick] (2.4,0.9) -- (1.7,0.9);
  % rat at choice point
  \fill[black] (0,-1.1) circle (2.4pt);
  \node[black, font=\footnotesize, anchor=west] at (0.2,-1.1) {rat (paused)};
  % forward sweep left
  \draw[->, acc, very thick] (0,-0.6) to[bend right=20] (-2.0,0.65);
  % forward sweep right
  \draw[->, red, very thick] (0,-0.6) to[bend left=20] (2.0,0.65);
  \node[acc, font=\scriptsize, anchor=east] at (-1.5,1.2) {sweep left};
  \node[red, font=\scriptsize, anchor=west] at (1.5,1.2) {sweep right};
  \node[black, font=\footnotesize, anchor=north, align=center] at (0,-1.85) {simulate futures, then choose};
\end{tikzpicture}
$$

Forward sweeps are the neural signature of **rollouts**: the system uses a model to
simulate possible future state sequences and assess where they lead, a form of planning
done at decision time. Pfeiffer and Foster (2013) sharpened the evidence, showing that
these forward hippocampal trajectories are not idle simulation — they preferentially
depict the path to the animal's upcoming goal, predicting where the rat will actually
go.[^pfeiffer] But decision-time rollout is not the only way a model can be put to
work. Beyond the choice point, the hippocampus **replays** trajectories during quiet
rest and sleep, reactivating sequences of place cells offline. Replay is a candidate
substrate for a different model-based computation: refining cached values in the
background from remembered experience, the way the [Dyna architecture](/reinforcement-learning/tabular-methods/planning-and-learning)
interleaves planning with acting by replaying stored transitions. The distinction
matters for the theory. Forward sweeps look like planning _at_ decision time; replay
looks like planning _between_ decisions, updating a value function offline so the fast
system can act on it later. Which the brain relies on, and when, is open.

$$
% caption: Two ways a model gets used. Decision-time rollout (top): at a choice point,
% forward sweeps simulate candidate futures before committing. Background replay
% (bottom): during rest, remembered trajectories are reactivated offline to refine
% cached values, Dyna-style, for the fast system to use later.
\begin{tikzpicture}[>=stealth, font=\small,
  mode/.style={draw, thick, minimum width=34mm, minimum height=13mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[mode, draw=acc, text=acc] (dt) at (0,0.9)   {decision-time rollout\\forward sweeps};
  \node[mode, draw=red, text=red] (bg) at (0,-0.9)  {background replay\\re\/f\/ine cached values};
  \node[black, font=\scriptsize, anchor=west] at (2.5,0.9)  {at the choice point};
  \node[black, font=\scriptsize, anchor=west] at (2.5,-0.9) {during rest and sleep};
  \draw[<->, black, thick] (dt) -- (bg) node[midway, left, font=\scriptsize, align=right] {same\\model};
\end{tikzpicture}
$$

Open questions remain sharp. Is _all_ planning done at decision
time by forward simulation, or does the brain sometimes refine cached values in the
background between decisions, the way the [Dyna architecture](/reinforcement-learning/tabular-methods/planning-and-learning)
interleaves planning and acting? Are there separate structures for the transition and
reward components of a model? How does the brain decide, moment to moment, whether to
trust the fast habitual system or the slow deliberative one — and is there even a clean
separation between their neural substrates? The evidence points against a clean split:
model-based influences appear more or less wherever the brain processes reward
information, including in the dopamine signals themselves, which carry model-based
information alongside the reward prediction errors long thought to be purely model-free.

## Meta-RL, deep RL as a model of cortex, and successor maps

Sutton and Barto's account divides labor between a slow, dopamine-trained model-free loop and
a separate model-based system. Three lines of work published around and after its
second edition complicate that division.

| Line of work | Claim | Effect on the two-system split |
| --- | --- | --- |
| **Meta-RL** (Wang et al., 2018) | slow model-free training installs a fast learner in recurrent PFC activity | one system trained on top of the other, not two circuits |
| **Distributional dopamine** (Dabney et al., 2020) | dopamine population encodes a distribution of returns | deep-RL construct anticipates a neural finding |
| **Successor representation** (Dayan, 1993; Stachenfeld et al., 2017) | store discounted future state occupancy; values factor as map $\times$ reward | a middle ground, neither purely model-free nor model-based |

Each is worth stating precisely.

**Meta-RL in prefrontal cortex.** Wang and colleagues (2018), in _Prefrontal cortex
as a meta-reinforcement learning system_, trained a recurrent network with a slow,
model-free, dopamine-like RL algorithm on families of related tasks. They reported
that after this slow training, the network's _recurrent activity_ came to implement a
second, faster learning process: presented with a new task from the family, the
network adapted within a single episode using only its activation dynamics, its
weights fixed. They proposed a mapping in which the dopamine-based algorithm that
trains the slow system corresponds to what this lesson has called the model-free loop,
while the emergent fast learner it installs in recurrent prefrontal activity accounts
for the flexible, model-based-looking behavior — recasting the two systems as one
trained on top of the other rather than two independent circuits.[^wang]

**Deep RL and neuroscience, two ways.** Botvinick and colleagues (2020), in _Deep
reinforcement learning and its neuroscientific implications_, catalogued constructs
from deep RL — distributed representations, experience replay, auxiliary prediction
tasks, and meta-learning among them — as explicit hypotheses about neural mechanism,
arguing the traffic runs both directions between the fields. The sharpest instance is
distributional RL. Dabney and colleagues (2020), in _A distributional code for value
in dopamine-based reinforcement learning_, took the machine-learning idea that an agent
should learn the full _distribution_ of returns rather than its mean, predicted that a
population of dopamine neurons would encode a spread of reward predictions with
systematically different optimism, and reported recordings consistent with that
prediction — a case where a deep-RL construct anticipated a neural finding.[^dabney]

**The successor representation as a middle ground.** Sutton and Barto frame the brain as
model-free _or_ model-based; a third option sits between them. Dayan (1993) introduced
the **successor representation** (SR): rather than caching a scalar value or storing a
full transition model, an agent stores, for each state, the discounted expected future
occupancy $M(s,s') = \mathbb{E}\big[\sum_{t} \gamma^t \mathbb{1}[S_t = s'] \mid S_0 = s\big]$
of every other state. Values then factor as $v(s) = \sum_{s'} M(s,s')\, r(s')$ — a
predictive map times a reward vector — so a change in reward re-computes values without
relearning dynamics: partial model-based flexibility at model-free cost. Stachenfeld, Botvinick and Gershman
(2017), in _The hippocampus as a predictive map_, argued that hippocampal place-cell
firing fields are better explained as encoding an SR than as encoding raw location, and
that entorhinal grid cells resemble its low-dimensional components. On this account the
hippocampus this lesson credited with a transition model may instead carry a predictive
map — a substrate that is neither purely model-free nor fully model-based, which is
exactly the kind of intermediate system Sutton and Barto's "several systems" thesis invites.[^sr]

## The brain is several systems

Pull the threads together and one conclusion recurs. There is no single place in the
brain where "the reinforcement-learning algorithm" runs. There is a model-free
actor-critic loop, plausibly spanning corticostriatal synapses gated by phasic dopamine,
trained by a two-factor rule in the critic and a three-factor rule in the actor. There is
a model-based system in prefrontal cortex and hippocampus that builds and consults an
environment model, simulating futures at the choice point. There are neuromodulatory
systems broadcasting scalar signals to distributed synapses, and populations of neuron-like
agents whose collective behavior may itself be a reinforcement-learning process. These
systems overlap, share signals, and arbitrate — the dorsal striatum leans habitual, the
dorsomedial goal-directed, the dopamine burst carries both error and model information.

Reinforcement learning earns its place as a lens not because the brain implements one of
its algorithms, but because it supplies the right _distinctions_ — reward versus value
versus prediction error versus reinforcement; model-free versus model-based; prediction
versus control — to name the parts and see how they fit. A sharper account of these
interacting neural systems is where the theory of reinforcement learning and the
neuroscience of reward-based learning are most likely to inform each other next, perhaps
yielding algorithms that combine model-free and model-based methods in ways computational
reinforcement learning has not yet explored.

[^sb-klopf]: **Sutton & Barto**, §15.9 — Hedonistic Neurons: Klopf's (1972, 1982) hypothesis that individual neurons are reinforcement-seeking agents adjusting synapses by the consequences of their own action potentials, his introduction of synaptic eligibility, the feedback-loop shape of his traces, and the _E. coli_ / "run and twiddle" analogy.
[^sb-collective]: **Sutton & Barto**, §15.10 — Collective Reinforcement Learning: teams of hedonistic neurons sharing a common reinforcement signal as a cooperative game / team problem, the structural credit-assignment problem, and the two requirements — contingent eligibility traces and output variability.
[^sb-williams]: **Sutton & Barto**, §15.10 — Collective Reinforcement Learning: Williams's (1992) result that teams of Bernoulli-logistic REINFORCE units implement a policy-gradient algorithm as a whole, even when interconnected into a multilayer network, with the broadcast reward signal replacing error backpropagation.
[^sb-modelbased]: **Sutton & Barto**, §15.11 — Model-based Methods in the Brain: the model-free/model-based distinction mapped onto habitual/goal-directed behavior, and the finding (Doll, Simon, and Daw, 2012) that model-based influences appear wherever the brain processes reward, including within dopamine signals.
[^sb-devaluation]: **Sutton & Barto**, §15.11 — Model-based Methods in the Brain: outcome-devaluation experiments dissociating habitual from goal-directed control, with the dorsolateral striatum (DLS) more model-free and the dorsomedial striatum (DMS) more model-based.
[^sb-hippocampus]: **Sutton & Barto**, §15.11 — Model-based Methods in the Brain: prefrontal cortex and the orbitofrontal cortex (OFC) tracking subjective and expected reward value, and hippocampal forward sweeps at choice points (Johnson and Redish, 2007; Pfeiffer and Foster, 2013) as evidence of the transition model and decision-time rollouts.
[^koshland]: **Sutton & Barto**, §15.9 — Hedonistic Neurons: the _E. coli_ chemotaxis / klinokinesis example and Selfridge's (1978, 1984) "run and twiddle" characterization ("keep going the same way if things are getting better, and otherwise move around"), drawing on Koshland's (1980) study of bacterial chemotaxis.
[^pfeiffer]: **Pfeiffer, B. E., & Foster, D. J.** (2013), "Hippocampal place-cell sequences depict future paths to remembered goals," _Nature_ 497, 74-79: forward hippocampal trajectories at rest preferentially represent the path to the animal's upcoming goal and predict its subsequent navigation.
[^wang]: **Wang, J. X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., Hassabis, D., & Botvinick, M.** (2018), "Prefrontal cortex as a meta-reinforcement learning system," _Nature Neuroscience_ 21, 860-868: a recurrent network trained by a model-free, dopamine-like RL algorithm develops, in its recurrent activity dynamics, a second faster learning process that adapts to new tasks within an episode without weight changes.
[^dabney]: **Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J., & Kurth-Nelson, Z.** (2020), "Deep reinforcement learning and its neuroscientific implications," _Neuron_ 107(4), 603-616 — deep-RL constructs as hypotheses about neural mechanism; and **Dabney, W., Kurth-Nelson, Z., Uchida, N., Starkweather, C. K., Hassabis, D., Munos, R., & Botvinick, M.** (2020), "A distributional code for value in dopamine-based reinforcement learning," _Nature_ 577, 671-675 — dopamine neurons encode a spread of reward predictions with systematically varying optimism, as predicted by distributional RL.
[^sr]: **Dayan, P.** (1993), "Improving generalization for temporal difference learning: the successor representation," _Neural Computation_ 5(4), 613-624 — the successor representation as discounted expected future state occupancy; and **Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J.** (2017), "The hippocampus as a predictive map," _Nature Neuroscience_ 20, 1643-1653 — hippocampal place fields and entorhinal grid cells modeled as encoding a successor representation.