The Psychology of Reinforcement

The algorithms in this course were built to solve an engineering problem: an agent in an unknown environment, maximizing reward. But the ideas were not invented in a vacuum. Trial-and-error learning, prediction of upcoming reward, the notion that surprise drives learning — all of these were studied first by psychologists working with animals, decades before the computational theory existed. The correspondence between the two bodies of work is not an accident of vocabulary. It is close enough that the same equations describe a rat pressing a lever and a value-function update, and close enough that reinforcement learning has fed new models back into experimental psychology.¹

This lesson argues that reinforcement learning is a theory of learning in animals, not merely a method that borrows animal-learning words. The argument rests on a single alignment: the two families of algorithms in this book — those for prediction and those for control — correspond to the two kinds of conditioning that animal-learning psychology has studied for over a century.

Prediction and control, classical and instrumental

Every algorithm we have built estimates something or improves something. Prediction algorithms estimate a quantity that depends on how the environment is expected to unfold — a value function, the expected discounted return from a state. Control algorithms find a policy that collects more reward. The two are coupled (policy evaluation is prediction; policy improvement is control), but the split is real, and it lines up with the two experimental paradigms of animal learning.²

In classical conditioning (also called Pavlovian conditioning), a stimulus predicts a biologically significant event, and the animal comes to respond to the predictor. The reinforcing event arrives whether or not the animal does anything; the animal has no control over it. What it learns is a prediction. In instrumental conditioning (also called operant conditioning), the reinforcing event is contingent on what the animal does — the apparatus delivers reward or punishment depending on the animal's behavior. What the animal learns is a policy: which action to take. This is control.

The alignment is a convenient first approximation rather than a clean partition. There is more to classical conditioning than pure prediction — the conditioned response is itself an action, so classical conditioning also involves a kind of control, sometimes called Pavlovian control. And the two paradigms interact in most real experiments. But as a way to connect the computational picture to the animal-learning picture, prediction-to-classical and control-to-instrumental is the right first cut.³

Classical conditioning

Pavlov's dogs are the origin. Studying salivation, Pavlov found that an innate reflex — salivating to food in the mouth — could be triggered by an initially neutral stimulus, the sound of a metronome, after the metronome had repeatedly preceded the food.⁴ The vocabulary that survives from his work:

• The food is an unconditioned stimulus (US): it triggers a response innately, with no learning.
• Salivation to food is the unconditioned response (UR).
• The metronome, initially neutral, becomes a conditioned stimulus (CS) once it reliably predicts the US.
• Salivation to the metronome alone is the conditioned response (CR).

The US is called a reinforcer because it reinforces the production of a CR to the CS. The CR is anticipatory: in a much-studied preparation, a tone CS predicts a puff of air to a rabbit's eye (the US), and the rabbit learns to close a protective membrane before the puff arrives, timed to peak just when the puff is due. Acting in anticipation of important events, rather than merely reacting to them, is adaptive, which is why this ability is widespread across the animal kingdom.⁵

Two arrangements of the CS and US in time are standard, and the difference matters for the real-time model below. In delay conditioning the CS stays on until the US begins, overlapping it. In trace conditioning the CS ends before the US begins, leaving a gap — the trace interval — that the animal must bridge with some persisting internal signal.

Four phenomena a model must explain

Decades of experiments established properties that any account of classical conditioning has to reproduce. Four are central.⁶

Acquisition. Over repeated CS–US pairings the CR appears and strengthens, approaching an asymptote set by the US.

Extinction. Present the CS repeatedly without the US and the CR weakens and disappears. Prediction without confirmation decays.

Blocking. This one is the pivotal test. First condition the animal to a tone CS so it predicts the US. Then present a compound CS — tone plus a new light — paired with the same US, over many trials. Finally test the light alone. The animal shows little or no CR to the light: learning about the light has been blocked by the prior learning about the tone.⁷ Blocking is devastating for the naive theory that conditioning depends only on temporal contiguity — on the CS and US simply co-occurring. The light co-occurs with the US just as reliably as the tone does, yet almost nothing is learned about it. What the animal already predicts, it does not need to learn again.

Higher-order conditioning. A previously conditioned CS can itself act as a US for a new, neutral stimulus. Pavlov conditioned a dog to salivate to a metronome predicting food, then paired a black square with the metronome — never with food. Within a few trials the dog salivated to the black square. The metronome, now a secondary reinforcer, transferred its predictive value to the square. This is second-order conditioning; chains of it are third-order, and so on.⁸

Blocking is the phenomenon that reshaped the field, because it says plainly that learning is driven by prediction error, not co-occurrence. An animal learns about a stimulus only to the extent that the outcome is surprising — not already predicted by everything else present.

The Rescorla–Wagner model

Rescorla and Wagner built their model mainly to account for blocking. The core idea: an animal learns only when events violate its expectations — only when it is surprised.⁹

Each component stimulus of a compound CS carries an associative strength $V$, a number for how strongly and reliably that component predicts the US. Consider a compound CS $\mathrm{AX}$ made of components $A$ and $X$, with strengths $V_A$ and $V_X$. What matters on a trial is the aggregate associative strength of everything present, not each component alone. On a trial where the compound is followed by the US, the components update by

with the key assumption that the aggregate strength is the sum, $V_{\mathrm{AX}}=V_A+V_X$. The symbols:

• $\alpha$ is the salience of the CS component (how noticeable it is),
• $\beta$ is a rate parameter tied to the US,
• $\lambda$ is the asymptote — the level of associative strength the US can support (Sutton and Barto write $R$ here, the magnitude of the reward signal, to avoid confusion with the eligibility-trace $\lambda$; we keep the classical $\lambda$ in this section),
• and the bracket $(\lambda -\sum V)$ is the prediction error.

The rule is a prediction-error update: each associative strength moves toward a target $\lambda$, by an amount proportional to how far the summed prediction $\sum V$ falls short of it. When the aggregate prediction already matches the US, the error is zero and nothing changes. Learning stops when the prediction is right.

A worked acquisition trace

Run numbers through the rule. Take a single CS $A$, so the aggregate strength is just $V_A$. Fix the product $\alpha \beta =0.3$ (a salience of $0.3$, a US rate of $1$), the asymptote $\lambda =1$, and start from $V_0=0$. The update on every reinforced trial is

Carrying it forward, each trial closes $30\%$ of the remaining gap to $\lambda$:

The strength climbs quickly at first, then slows as the error shrinks, tracing a decaying exponential toward $\lambda$. In closed form the residual gap follows $\lambda -V_n=(1-\alpha \beta {)}^n(\lambda -V_0)=0.7^n$, so $V_n=1-0.7^n$ — a negatively accelerated acquisition curve, the classic learning-curve shape. Extinction is the mirror image: present $A$ with no US, set $\lambda =0$, and the same rule drives $V$ back down by $30\%$ of its remaining height each trial.

How it explains blocking

Blocking falls straight out of the summed-error term. After the tone $A$ has been fully conditioned, $V_A\approx \lambda$: the tone alone already predicts the US. Now add the light $X$ (initially $V_X=0$) to form the compound. The prediction error on the very first compound trial is

The error is already zero because the tone alone accounts for the US. With no error, $\Delta V_X=\alpha \beta \cdot 0=0$, and the light learns nothing, trial after trial. The US is already predicted, so its arrival is no surprise, and surprise is what drives learning. That is blocking, in one line of algebra.

To see the flat line rather than assert it, run phase 2 the same way as the acquisition trace. Pretraining leaves the tone at $V_A=1.0$ (its asymptote). Now every compound trial presents $A$ and $X$ together, so the error uses the sum $V_A+V_X$, and both components share it. With $\alpha \beta =0.3$ each:

The error is pinned at zero from the first compound trial, so $\Delta V_X=0$ on every trial and the light never acquires strength. Contrast this with a control group that sees $X$ paired with the US without a pretrained tone: there $V_X$ would run the acquisition trace above, climbing $0\to 0.30\to 0.51\to \dots$ The only difference is the pretrained predictor that zeroes the shared error, which is the summed-error term at work.

From the machine-learning side, the Rescorla–Wagner rule is an error-correction supervised learning rule — essentially the Least-Mean-Square (Widrow–Hoff) rule, a regression algorithm that drives the mean-squared prediction error toward zero.¹⁰ Its historical importance was to show that a mechanistic rule, needing no explicit reasoning about which stimulus was added, could account for blocking, when everyone had assumed blocking demanded a cognitive story.

Rescorla–Wagner is the LMS rule, precisely

The animal is doing linear regression: fitting weights so a sum of present cues predicts the US, adjusting those weights by the same error-times-input step a statistician uses. The equivalence to Least-Mean-Square turns the psychology into optimization. Collect the component strengths into a weight vector $\mathbf{w}$ and describe the compound present on a trial by a binary presence vector $\mathbf{x}$, with $x_i=1$ when component $i$ is present. The aggregate prediction is the dot product ${\mathbf{w}}^{\top }\mathbf{x}={\sum }_i{V}_i$, exactly the summed associative strength. Define the per-trial squared error against the US level $\lambda$:

Its gradient is ${\nabla }_{\mathbf{w}}L=-(\lambda -{\mathbf{w}}^{\top }\mathbf{x})\mathbf{x}$, so one step of gradient descent with rate $\alpha \beta$ gives

which is the Rescorla–Wagner update, component by component: the factor $\mathbf{x}$ is why only present components change, and $(\lambda -{\mathbf{w}}^{\top }\mathbf{x})$ is the shared prediction error. Rescorla and Wagner had, without naming it, written down stochastic gradient descent on squared prediction error — the same rule Widrow and Hoff published for adaptive filters in 1960.¹¹ In this light blocking is a familiar regression fact: the added feature $X$ is collinear with a feature already fit to the target, so once $A$ explains the US the residual $(\lambda -{\mathbf{w}}^{\top }\mathbf{x})$ is zero and least-squares assigns the redundant feature zero weight. Overexpectation, below, is the same picture read in the opposite direction.

Overexpectation: a prediction contiguity cannot make

Because the error term sums over everything present, Rescorla–Wagner predicts a phenomenon that a pure co-occurrence account rules out. Train $A$ alone to the asymptote, so $V_A\approx 1$. Separately train $X$ alone to the asymptote, so $V_X\approx 1$. Each has been paired with exactly the same US $\lambda =1$ and neither is over-predicting anything on its own. Now present the compound $AX$, still followed by the same single US. The summed prediction is now $V_A+V_X\approx 2$, but the US supports only $\lambda =1$, so the shared error goes negative:

Both strengths fall below the asymptote each earned alone, settling toward $V_A=V_X=0.5$ where the sum matches $\lambda$. The compound over-predicted the US, so the model unlearns some of what each component had — even though each component is still perfectly contiguous with the US. This is overexpectation, and it is a signature of the summed-error rule: contiguity predicts the strengths should hold, the summed error predicts they drop, and the animal drops them.¹²

Limitations

Rescorla–Wagner is a trial-level model. It treats a whole trial as one event — a set of components present, one prediction error, one update — and says nothing about what happens within a trial or between trials. That is a serious restriction:¹³

• It cannot represent the timing of stimuli, so it cannot explain why the CR appears before the US, or how the inter-stimulus interval controls the strength of conditioning.
• It has no mechanism for higher-order conditioning — its updates only ever push strengths toward the US level $\lambda$, and a stimulus never present with the US can never gain strength.
• It cannot address trace conditioning, serial compounds, or the shape of the conditioned response over time.

Every one of these gaps is about time within the trial. The fix is to make the model real-time.

The TD model of classical conditioning

The TD model extends Rescorla–Wagner from a trial-level model to a real-time one. Now $t$ labels a small time step within or between trials, not a whole trial. A trial becomes a sequence of states, one per time step, and the animal experiences an endless stream of states $s$, each described by a feature vector $\mathbf{x}(s)$. With this move the trial-level restrictions dissolve, and the connection to the rest of this course becomes exact.¹⁴

Instead of one update per whole trial, the animal now makes a small update at every instant, comparing what it predicts now to what it predicts a moment later. The three equations below write out that comparison — a value estimate, an error, and a memory of what was recently active.

Let the associative-strength vector be $\mathbf{w}$. The aggregate associative strength of a state — the US prediction — is the linear function

which is precisely a value estimate with linear function approximation. The learning update is

where the scalar ${\delta }_t$ is the TD error

and ${\mathbf{z}}_t$ is a vector of eligibility traces that decays at rate $\gamma \lambda$,

Here $R_t$ is the prediction target (the US magnitude), $\gamma$ is a discount factor, and $\lambda$ is the eligibility-trace decay. This reproduces the backward view of semi-gradient $\text{TD}(\lambda )$ with linear function approximation, the prediction algorithm from temporal-difference learning and eligibility traces — the only twist being that $R_t$ here is a US intensity rather than a reward signal.¹⁵

The single difference between the two models is the error term. Rescorla–Wagner uses the trial-level prediction error $\lambda -\sum V$. The TD model uses the temporal-difference error ${\delta }_t=R_t+\gamma \hat{v}(S_{t+1})-\hat{v}(S_t)$, which compares the prediction at one instant with the discounted prediction at the next instant. Set $\gamma =0$ and the TD model collapses back to Rescorla–Wagner. The bootstrapping term $\gamma \hat{v}(S_{t+1})$ is what buys everything the trial-level model lacked.

Because the TD error carries the term $\gamma \hat{v}(S_{t+1})-\hat{v}(S_t)$, a change in prediction from one instant to the next is treated exactly like the arrival of a US. This is bootstrapping — a guess corrected toward a later guess — and it is what gives the TD model two things Rescorla–Wagner cannot produce:

• Timing. With a fine-grained state representation, the US prediction grows through the interval between CS and US and peaks at the moment the US is due, and the CR can anticipate the US. The inter-stimulus interval now controls learning.
• Higher-order conditioning. A previously conditioned CS raises $\hat{v}$, so when it is paired with a new neutral stimulus, the temporal difference in prediction is itself non-zero and drives learning about the new stimulus — with no US present at all. Second- and higher-order conditioning is a direct consequence of bootstrapping.

The TD model is not a perfect account of classical conditioning, but it is a normative one: it says what the animal's nervous system is trying to do during conditioning — form accurate long-term predictions, given the limits of how stimuli are represented and how time is handled. That reframing, from immediate to long-term prediction, is the model's central claim.¹⁶

How time gets represented: presence, CSC, and microstimuli

The feature vector $\mathbf{x}(s)$ is where all the timing lives, and the TD model behaves very differently depending on how a stimulus is turned into features over time. Three representations recur in the literature, and they line up along a temporal generalization gradient — how much a feature active at one instant also speaks for nearby instants.¹⁷

The presence representation is the coarsest: one feature per CS, equal to $1$ for the whole time the CS is on and $0$ otherwise. It carries no information about when within the CS you are, so it forces complete generalization across all times the CS is present. It is not a plausible brain code, yet it is enough to produce most of the timing phenomena below.

The complete serial compound (CSC) is the finest: the onset of a CS starts a sequence of short, precisely-timed internal signals, one distinct feature per (CS, elapsed-time) pair — a tapped delay line, as if the nervous system ran a clock. Nearby time points share no features, so there is no generalization between them; the model can learn a separate prediction for each moment. It is also unrealistic as a brain code (Ludvig and colleagues call it a useful fiction), but it isolates the model's behavior from the representation, and it is the representation used in most TD models of dopamine neurons.

The microstimulus (MS) representation sits in the middle. A CS onset launches a cascade of internal features that are extended and overlapping: as time elapses, later microstimuli grow wider in time and reach a lower peak. Overlap means partial generalization between nearby times — coarser than the CSC clock, finer than the single presence feature — and its smeared, decaying shape is closer to what a population of neurons might actually produce.¹⁸

Why the CR anticipates the US

With the CSC representation the model learns a US prediction that is not flat across the interval but rising. Discounting is the reason: the value at each state is the discounted prediction of the value one step later, so at asymptote the prediction at time $t$ before the US is roughly ${\gamma }^k$ times the US level, where $k$ is the number of steps still to go. That grows toward the US level as the US approaches and peaks exactly at US time. The presence representation, lacking any within-CS timing, gives a near-flat prediction; MS gives a smoothed version of the CSC ramp.

The rising prediction explains a fact the trial-level model cannot touch: the rabbit's protective membrane closes before the air puff, timed to peak when the puff is due. Closing too early wastes vision, closing too late gives no protection, so a CR locked to US time is the adaptive one, and the ramp-and-peak prediction is what supports it.¹⁹

ISI tuning and temporal primacy over blocking

Two further predictions of the real-time model have no trial-level analogue. The first is ISI dependence. Conditioning strength depends on the inter-stimulus interval — the gap between CS onset and US onset — in a characteristic inverted-U: it is negligible at a zero or negative ISI (US at or before CS onset), rises to a maximum at some positive ISI, then falls off as the interval grows too long to bridge. The TD model reproduces this shape, with the peak and width set by its discounting and trace parameters.²⁰

The second is sharper, because the model made it before anyone had looked. In ordinary blocking the added CS is simultaneous with the pretrained one. But if the added CS is moved earlier, so its onset precedes the pretrained CS, the TD model predicts that blocking reverses: the earlier stimulus is no longer blocked, and as training continues it gains strength while the later, pretrained stimulus loses it. The reason is bootstrapping. Updates push each state's prediction toward the prediction of the state that follows, so an earlier predictor takes precedence over a later one — temporal primacy overriding blocking. Sutton and Barto reported this prediction in 1981; Kehoe, Schreurs, and Graham confirmed it in 1987 in the rabbit nictitating-membrane preparation, noting that non-TD models struggle with the data.²¹

From prediction to control

We have followed the prediction half of the correspondence to its end: classical conditioning is prediction, the Rescorla–Wagner rule is a trial-level prediction error, and its real-time TD extension gives the prediction algorithm of this course, with timing, higher-order conditioning, and temporal primacy falling out of bootstrapping. What remains is the other half — control. Instrumental conditioning is the animal learning what to do, not just what to expect, and it maps onto the control side of the theory: the Law of Effect is trial-and-error control, and the habitual/goal-directed distinction is the model-free/model-based split.

This continues in The Psychology of Reinforcement: Instrumental Control.