The Psychology of Reinforcement: Instrumental Control

This builds on The Psychology of Reinforcement, which followed the prediction half of the correspondence: classical conditioning, the Rescorla–Wagner rule, and its real-time TD extension. This part takes up the other half — control. Where classical conditioning taught the animal what to expect, instrumental conditioning teaches it what to do, and that maps onto the control algorithms of this course: the Law of Effect is trial-and-error control, and the habitual/goal-directed distinction is the model-free/model-based split.

Instrumental conditioning and the Law of Effect

Instrumental conditioning is control. The reinforcer is contingent on behavior, and the animal learns to produce behavior that pays. Its founding observation is Thorndike's: cats placed in puzzle boxes, able to escape by the right sequence of actions, escaped faster and faster over repeated trials. Successful actions were stamped in, unsuccessful ones stamped out.¹ From this Thorndike drew the Law of Effect.

The Law of Effect is a compact description of a reinforcement learning algorithm. Two features line up exactly, and each distinguishes it from a neighboring learning paradigm:²

Selection plus association is search plus memory: search by trying actions, memory by linking situations to the actions that worked. Both are essential to every control algorithm here. Search requires exploration — the animal must try alternatives — which is why the control methods carry $\epsilon$-greedy or upper-confidence-bound action selection.³ Thorndike's cats selected from their instinctual impulses, not every conceivable action: the animal-learning image of an admissible action set $\mathcal{A}(s)$.

Skinner extended this line with shaping: reinforcing successive approximations of a target behavior. Waiting for the full behavior to occur by chance can take forever; instead you reward anything resembling it, then progressively raise the bar. Shaping is directly useful for training reinforcement learning agents when reward is too sparse for the agent ever to stumble onto it — start with an easier version of the problem and increase difficulty as the agent improves.⁴

Skinner also formalized the rule by which the apparatus delivers reinforcement, the reinforcement schedule — part of the environment's reward function, fixing the mapping from behavior to reward. The animal's steady-state response rate is a policy adapted to it.⁵

Partial reinforcement is an analogue of the stochastic-reward setting the control methods already face: a variable-ratio schedule delivers reward intermittently and unpredictably, so the learner acts under high return variance — the condition that makes exploration and value averaging matter. Behavior trained under partial reinforcement also extinguishes more slowly, since intermittent reward looks, locally, like the training regime.

Habitual and goal-directed behavior

The last correspondence is the closest. Psychologists distinguish two modes of controlling learned behavior, and they map cleanly onto the model-free / model-based split from planning and learning.⁶

Habitual behavior is triggered by antecedent stimuli and runs off automatically: decisions read off cached state-action values built up over many trials. Goal-directed behavior consults an internal model, simulates the consequences of candidate action sequences, and chooses the best outcome — slower, but able to change the moment the model's knowledge of the goal changes, no re-experience required.

The experimental test that separates the two is outcome devaluation. Train a rat to press a lever for food pellets. Then devalue the pellets — pair them with a nausea-inducing injection so the rat no longer wants them — without the rat ever pressing the lever while sick. Now put the lever back and see whether the rat still presses.⁷ A goal-directed (model-based) rat presses less right from the start: it knows lever pressing leads to pellets, and it now knows pellets are bad, so it combines the two and stops — without ever having experienced lever-pressing-then-sickness. A purely habitual (model-free) rat keeps pressing, because its cached lever-press value was never updated by direct experience.

The logic hangs on the fact that devaluation happens in a context separate from lever pressing. Reading it as a timeline makes the dissociation clear: only the model-based agent has a stored link from lever to pellet that it can re-evaluate when the pellet's value changes; the model-free agent has only a cached number on the lever action, and that number was set before devaluation and never revisited.

Both systems are present, and behavior shifts between them. Adams showed that extended training converts goal-directed behavior into habit: modestly trained rats stayed sensitive to devaluation, overtrained rats kept pressing.⁸ Daw, Niv, and Dayan model the animal as running both processes and deferring to whichever is more trustworthy, measured by the uncertainty each maintains — control follows $\arg \min$ over the two posterior uncertainties. Early on the model-based process is more reliable, chaining short-term predictions that sharpen quickly; with experience the model-free values become well-estimated and cheaper to consult, and control shifts toward habit. This is arbitration between model-free and model-based control, expressed in the animal.⁹

Delayed reinforcement

One thread runs through both paradigms: reinforcement is often delayed. In instrumental conditioning the reward can arrive well after the action that earned it; in classical conditioning the US can follow CS offset by a gap. This is the credit-assignment problem — how to distribute credit for an outcome among the earlier decisions that led to it.¹⁰

Animal-learning theory proposed two mechanisms for bridging the gap, each with an exact counterpart here:¹¹

A stimulus leaves a decaying trace in the nervous system, and learning occurs while the CS trace still overlaps the US — the ancestor of the eligibility trace. A stimulus that reliably predicts reward becomes reinforcing itself, supplying immediate reinforcement when the primary reward is delayed — the role of a value function. In an actor–critic architecture the critic's TD error ${\delta }_t=R_{t+1}+\gamma \hat{v}(S_{t+1})-\hat{v}(S_t)$ acts as exactly such a conditioned reinforcer for the actor, giving immediate feedback even when primary reward is far off.

The two mechanisms are the two answers this course gives to delayed reinforcement, and animal-learning theory arrived at both first: the eligibility trace was proposed for rats before it was proposed for algorithms, and the value function is the formal version of a secondary reinforcer.

From conditioning models to deep RL and meta-learning

The line from Rescorla–Wagner to the algorithms of this course does not stop at the classical-conditioning models; it runs directly into modern deep reinforcement learning, and it loops back into experiments on human choice.

The behavioral ancestor of the training signal in deep RL is the prediction error these models formalized. In the Deep Q-Network of Mnih and colleagues (2015, Nature), which learned to play Atari games from pixels, the quantity that drives every weight update is the temporal-difference error $R_t+\gamma {\max }_{a}Q(S_{t+1},a)-Q(S_t,a)$ — the same ${\delta }_t$ that the TD model of conditioning uses, now applied to a deep network's value estimate. Actor–critic deep methods carry the same TD error as the critic's teaching signal for the actor. The Rescorla–Wagner rule and its real-time TD extension are the behavioral lineage of that signal, not a loose metaphor for it.¹²

The habit/goal-directed distinction this lesson draws has been measured directly in people. Daw, Gershman, Seymour, Dayan, and Dolan (2011, Neuron) introduced the two-step task, a sequential choice in which model-free and model-based control predict different patterns of how a reward changes the next choice. Human choices showed a mixture of the two strategies rather than either alone — the empirical counterpart of the two-system picture above. Earlier, Gläscher, Daw, Dayan, and O'Doherty (2010, Neuron) used fMRI to dissociate two error signals during learning: a state-prediction error that tracks the learning of the transition model (the model-based component) and a separate reward-prediction error (the model-free component), each with its own neural correlate. The habit/goal-directed contrast is not only a behavioral description; it corresponds to distinct, measurable learning signals.¹³

The two modes have also been shown to emerge from one another. Wang and colleagues (2018, Nature Neuroscience), in Prefrontal cortex as a meta-reinforcement learning system, trained a recurrent network with a plain model-free RL algorithm across a distribution of related tasks. The slow model-free training shaped the network's recurrent dynamics into a faster inner learning loop that behaved in a model-based-like way on each new task — model-based behavior arising from model-free learning, a meta-reinforcement-learning account of how a flexible controller can be built from an inflexible learning rule. Botvinick and colleagues (2020, Nature Reviews Neuroscience) survey the broader set of contact points between deep reinforcement learning and neuroscience that this lineage opened up.¹⁴

Why this matters

Reinforcement learning was designed for engineering, and it remains a theory of effective learning algorithms, not a catalogue of animal behavior. But the correspondences are not superficial. The prediction/control split is the classical/instrumental split. The Rescorla–Wagner prediction error is the TD error with the time removed. The Law of Effect is trial-and-error control. Habit is model-free, deliberation is model-based, and the shift between them is arbitration. Eligibility traces are stimulus traces, and value functions are secondary reinforcers.

None of this means the brain runs our exact equations. It means the abstract structure of the learning problem is the same whether the learner is silicon or tissue, and that the theory built to solve it computationally keeps rediscovering the mechanisms that evolution found. The next lesson follows this contact point into the brain itself, where the dopamine signal turns out to behave like a TD error.