The Brain's Several Learning Systems

This builds on The Neuroscience of Reinforcement, which built the tissue-level vocabulary and the actor and critic learning rules — a two-factor critic and a three-factor actor, both driven by a broadcast dopamine signal. That was one model-free loop. This part asks where the actor rule came from, whether teams of neuron-like agents can learn together, and where the model-based systems live that a single loop leaves out — and arrives at the module's recurring conclusion: the brain is several interacting systems, not one algorithm.

Hedonistic neurons

The neuron-like actor unit, with its Law-of-Effect learning rule, has an intellectual ancestor: the hedonistic-neuron hypothesis of A. H. Klopf (1972, 1982). Klopf conjectured that individual neurons are themselves reinforcement-seeking agents — that a single neuron seeks to maximize the difference between synaptic input treated as rewarding and synaptic input treated as punishing, adjusting the efficacies of its synapses on the basis of the rewarding or punishing consequences of its own action potentials. A neuron, on this view, can be trained by response-contingent reinforcement the way an animal is trained in an instrumental conditioning task.¹

The mechanism Klopf proposed is the three-factor rule in embryo. When a neuron fires, all the synapses that were active in contributing to that spike become eligible for change — Klopf introduced the term. If the spike is followed within some window by increased reward, the eligible synapses strengthen; if by punishment, they weaken. That is a contingent eligibility trace: triggered by the coincidence of presynaptic activity with the postsynaptic firing that the presynaptic activity helped cause. Klopf's traces were richer than the ones we use. He imagined the shape of a trace as a histogram of the durations of the many feedback loops the neuron sits in — some closing within the brain and body, others reaching out through the environment via the animal's muscles and senses. The exponentially decaying traces of our algorithms are simplified stand-ins for this richer conception.

The hypothesis is less far-fetched than it sounds. A single-celled organism can already seek some stimuli and avoid others: the bacterium Escherichia coli swims by rotating flagella, and it biases its tumbles so that it persists up gradients of useful molecules and away from harmful ones — a strategy the AI pioneer Oliver Selfridge called run and twiddle, keep going the same way if things are improving, otherwise move around.² The mechanism is memoryless minimization made into a search: the bacterium has no map of the gradient and cannot steer toward the source, but it can compare the concentration it senses now against the recent past, and it lets that comparison set the odds of tumbling. Written as a loop, the strategy is short.

Nothing in the loop knows the gradient's direction. Suppressing the tumble whenever things improve, and re-randomizing whenever they do not, is enough to bias the undirected walk uphill: runs up the gradient are extended, runs down it are cut short, and the net drift climbs toward the attractant. It is the Law of Effect running in a bacterium — reinforcement biasing the persistence of a behavior — with no synapses and no lasting memory.

A neuron, embedded in its own dense web of feedback loops, might swim the same way through the space of its inputs, unlike the bacterium retaining, in its synaptic strengths, a lasting record of what it learned. The discovery of STDP, and especially of reward-modulated STDP, suggests Klopf's ideas may not have been far off the mark.

Collective reinforcement learning

Klopf pushed the idea one step further: if individual neurons are reinforcement-seeking agents, then much of intelligent behavior might be the collective behavior of a population of such self-interested agents interacting in the immense society that is a nervous system. This reframes the actor of the previous section. The actor is not one unit but $k$ units, each a hedonistic neuron, each receiving the same broadcast $\delta$. Because each unit tries to make $\delta$ as large as possible, $\delta$ effectively acts as a common reward signal for the whole team — a cooperative game, or team problem, in which every agent seeks to increase the same signal.³

Two features make the team problem hard. First, the common signal $\delta$ evaluates the pattern produced by the entire population — the collective action — so any single agent contributes just one component of what is scored. Learning therefore requires solving a structural credit-assignment problem: which members deserve credit for a favorable signal, or blame for an unfavorable one? Second, each agent is embedded in the others: from any one unit's vantage the other units are part of its environment, since both its input and the reward it sees depend on how everyone else behaves. Each unit partially observes the true state, cannot see the others' actions, and has its reward buried in noise created by the rest.

To see the difficulty concretely, take the smallest team that has it: two Bernoulli-logistic units, $U_1$ and $U_2$, each firing $A_i\in \{0,1\}$, sharing a single reinforcement signal $\delta$ that scores their joint output. Suppose the team is rewarded ($\delta >0$) exactly when both fire together, $A_1=A_2=1$, and penalized otherwise. On a trial where both fire and $\delta$ comes back positive, what should $U_1$ conclude? From its own vantage it fired and got rewarded, so its contingent trace $(A_1-{\pi }_1)\mathbf{x}=(1-{\pi }_1)\mathbf{x}>0$ tells it to make firing more likely. But $U_1$ cannot see $A_2$; it has no way to know whether the reward came because it fired or because its teammate happened to fire at the same time. If on the next such trial $U_1$ fires while $U_2$ stays silent, the reward vanishes, and $U_1$'s trace now points the update the other way. A single unit's action and the team's reward are related only as a statistical correlation, one that has to be estimated across many trials — this is the structural credit-assignment problem, and no single trial resolves it.

Two ingredients let a team learn anyway. Each member needs a contingent eligibility trace to correlate its own action with the reward that follows — non-contingent traces support prediction (the critic) but not control. And each member needs variability in its output, so the team explores the space of collective actions; a Bernoulli-logistic unit supplies this automatically, its firing probability ${\pi }_i$ nudged but never pinned to $\{0,1\}$. One result closes the loop: a team of Bernoulli-logistic REINFORCE units — even interconnected into a multilayer network, even with each unit's reward corrupted by large noise and lacking full state information — implements a policy-gradient algorithm as a whole, ascending the average reward rate of the team's common signal.⁴ Where a conventional network would need error backpropagation to train its hidden units, here the broadcast reward signal replaces backprop. Slower in practice, but far more plausible as a neural mechanism — especially in light of reward-modulated STDP.

Model-based methods in the brain

Everything so far has been model-free: the actor-critic loop learns values and a policy by prediction error, with no explicit model of how the world transitions or where reward comes from. But the brain is not only model-free. Reinforcement learning's distinction between model-free and model-based methods maps onto a distinction psychology draws between habitual and goal-directed behavior — and the neural evidence says the brain runs both, in different structures, and arbitrates between them.⁵

The cleanest evidence comes from outcome-devaluation experiments. Train a rat to press a lever for food, then devalue the food (feed the rat to satiety, or pair the food with illness). A goal-directed (model-based) system, representing the outcome and consulting its current value, immediately stops pressing. A habitual (model-free) system, having cached a stimulus-response value with no link to the outcome, keeps pressing. Inactivating specific regions and watching which behavior survives localizes the two systems:⁶

The DLS looks more model-free, the DMS more model-based — a sign that the simple actor in the dorsal striatum picture is too coarse.

Model-based behavior reaches beyond the striatum, and the two components of a model localize to distinct structures:⁷

The prefrontal cortex — seat of executive function, planning, and decision — is implicated, and within it the orbitofrontal cortex (OFC): recordings in monkeys and human neuroimaging show OFC activity tracking the subjective reward value of stimuli and the reward expected from actions. The hippocampus supplies the transition component: its place-learning led Tolman to posit cognitive maps, and when a rat pauses at a choice point, decoding the hippocampal population reveals the representation of space sweeping forward along the candidate paths — not backward — predicting the rat's subsequent choice.

Forward sweeps are the neural signature of rollouts: the system uses a model to simulate possible future state sequences and assess where they lead, a form of planning done at decision time. Pfeiffer and Foster (2013) sharpened the evidence, showing that these forward hippocampal trajectories are not idle simulation — they preferentially depict the path to the animal's upcoming goal, predicting where the rat will actually go.⁸ But decision-time rollout is not the only way a model can be put to work. Beyond the choice point, the hippocampus replays trajectories during quiet rest and sleep, reactivating sequences of place cells offline. Replay is a candidate substrate for a different model-based computation: refining cached values in the background from remembered experience, the way the Dyna architecture interleaves planning with acting by replaying stored transitions. The distinction matters for the theory. Forward sweeps look like planning at decision time; replay looks like planning between decisions, updating a value function offline so the fast system can act on it later. Which the brain relies on, and when, is open.

Open questions remain sharp. Is all planning done at decision time by forward simulation, or does the brain sometimes refine cached values in the background between decisions, the way the Dyna architecture interleaves planning and acting? Are there separate structures for the transition and reward components of a model? How does the brain decide, moment to moment, whether to trust the fast habitual system or the slow deliberative one — and is there even a clean separation between their neural substrates? The evidence points against a clean split: model-based influences appear more or less wherever the brain processes reward information, including in the dopamine signals themselves, which carry model-based information alongside the reward prediction errors long thought to be purely model-free.

Meta-RL, deep RL as a model of cortex, and successor maps

Sutton and Barto's account divides labor between a slow, dopamine-trained model-free loop and a separate model-based system. Three lines of work published around and after its second edition complicate that division.

Each is worth stating precisely.

Meta-RL in prefrontal cortex. Wang and colleagues (2018), in Prefrontal cortex as a meta-reinforcement learning system, trained a recurrent network with a slow, model-free, dopamine-like RL algorithm on families of related tasks. They reported that after this slow training, the network's recurrent activity came to implement a second, faster learning process: presented with a new task from the family, the network adapted within a single episode using only its activation dynamics, its weights fixed. They proposed a mapping in which the dopamine-based algorithm that trains the slow system corresponds to what this lesson has called the model-free loop, while the emergent fast learner it installs in recurrent prefrontal activity accounts for the flexible, model-based-looking behavior — recasting the two systems as one trained on top of the other rather than two independent circuits.⁹

Deep RL and neuroscience, two ways. Botvinick and colleagues (2020), in Deep reinforcement learning and its neuroscientific implications, catalogued constructs from deep RL — distributed representations, experience replay, auxiliary prediction tasks, and meta-learning among them — as explicit hypotheses about neural mechanism, arguing the traffic runs both directions between the fields. The sharpest instance is distributional RL. Dabney and colleagues (2020), in A distributional code for value in dopamine-based reinforcement learning, took the machine-learning idea that an agent should learn the full distribution of returns rather than its mean, predicted that a population of dopamine neurons would encode a spread of reward predictions with systematically different optimism, and reported recordings consistent with that prediction — a case where a deep-RL construct anticipated a neural finding.¹⁰

The successor representation as a middle ground. Sutton and Barto frame the brain as model-free or model-based; a third option sits between them. Dayan (1993) introduced the successor representation (SR): rather than caching a scalar value or storing a full transition model, an agent stores, for each state, the discounted expected future occupancy $M(s,s^{\prime })=\mathbb{E}[{\sum }_t{\gamma }^{t}1[S_t=s^{\prime }]\mid S_0=s]$ of every other state. Values then factor as $v(s)={\sum }_{s^{\prime }}M(s,s^{\prime })r(s^{\prime })$ — a predictive map times a reward vector — so a change in reward re-computes values without relearning dynamics: partial model-based flexibility at model-free cost. Stachenfeld, Botvinick and Gershman (2017), in The hippocampus as a predictive map, argued that hippocampal place-cell firing fields are better explained as encoding an SR than as encoding raw location, and that entorhinal grid cells resemble its low-dimensional components. On this account the hippocampus this lesson credited with a transition model may instead carry a predictive map — a substrate that is neither purely model-free nor fully model-based, which is exactly the kind of intermediate system Sutton and Barto's several systems thesis invites.¹¹

The brain is several systems

Pull the threads together and one conclusion recurs. There is no single place in the brain where the reinforcement-learning algorithm runs. There is a model-free actor-critic loop, plausibly spanning corticostriatal synapses gated by phasic dopamine, trained by a two-factor rule in the critic and a three-factor rule in the actor. There is a model-based system in prefrontal cortex and hippocampus that builds and consults an environment model, simulating futures at the choice point. There are neuromodulatory systems broadcasting scalar signals to distributed synapses, and populations of neuron-like agents whose collective behavior may itself be a reinforcement-learning process. These systems overlap, share signals, and arbitrate — the dorsal striatum leans habitual, the dorsomedial goal-directed, the dopamine burst carries both error and model information.

Reinforcement learning earns its place as a lens not because the brain implements one of its algorithms, but because it supplies the right distinctions — reward versus value versus prediction error versus reinforcement; model-free versus model-based; prediction versus control — to name the parts and see how they fit. A sharper account of these interacting neural systems is where the theory of reinforcement learning and the neuroscience of reward-based learning are most likely to inform each other next, perhaps yielding algorithms that combine model-free and model-based methods in ways computational reinforcement learning has not yet explored.