Animal Learning and Cognition

The psychology of reinforcement set out the correspondence: prediction is classical conditioning, control is instrumental conditioning, and the Rescorla–Wagner and TD models turn associative learning into prediction-error updates. That lesson compressed three phenomena. Each deserves a full treatment, because each is a case where a mechanism this course derives from the optimization problem was first measured in an animal, decades earlier, by someone with no computer in mind.¹

Take the three in order. Blocking is the experiment that forced the field to say learning is driven by prediction error, not by two events co-occurring in time; alongside it, higher-order conditioning and conditioned reinforcement are how a learned prediction becomes a stand-in reward — the behavioral face of a value function. Delayed reinforcement is the credit-assignment problem in an animal, and the two mechanisms animal-learning theory proposed to solve it, stimulus traces and secondary reinforcement, are eligibility traces and TD-learned values. Cognitive maps are internal models: Tolman's rats learned the layout of a maze with no reward at all, which is model-based reinforcement learning, and the way they learned it is what control engineers call system identification.

Blocking

Beyond the anticipatory nature of the conditioned response, two properties of classical conditioning figure most prominently in models of it: blocking and higher-order conditioning. Blocking is the pivotal one. It occurs when an animal fails to learn a conditioned response to a potential conditioned stimulus, because that stimulus is presented in compound with another one that was already conditioned to produce the response.²

Kamin's experiment has three phases. Take the rabbit nictitating-membrane preparation, where a tone conditioned stimulus predicts an air puff to the eye (the unconditioned stimulus), and the rabbit learns to close its protective membrane in anticipation.

• Phase 1. Condition the rabbit to a tone alone, paired with the air puff, until the tone reliably produces the membrane-closure conditioned response. The tone now predicts the puff.
• Phase 2. Present a compound stimulus — the same tone plus an added light — paired with the same air puff, for many trials. The light co-occurs with the puff on every one of these trials.
• Phase 3. Test the light alone. The rabbit produces very few, or no, conditioned responses to the light. Learning to the light was blocked by the prior learning to the tone.³

Blocking breaks the simplest theory of conditioning, the one that says a necessary and sufficient condition for conditioning is temporal contiguity — the unconditioned stimulus frequently following a conditioned stimulus closely in time.⁴ By that theory the light should condition just fine: in phase 2 it precedes the air puff on every trial, contiguous with it exactly as the tone is. Yet almost nothing is learned about the light. Contiguity is present and learning is absent. What the animal already predicts, it does not learn again.

The Rescorla–Wagner model was built mainly to account for this. It reads out of the summed-error term in one line. After phase 1 the tone's associative strength has reached the level the unconditioned stimulus supports, $V_{\text{tone}}\approx R$. When the light (initially $V_{\text{light}}=0$) joins the compound, the prediction error on the first compound trial is

because the model's key assumption is that the aggregate strength is the sum of the components present. With the error already at zero, $\Delta V_{\text{light}}=\alpha \beta \cdot 0=0$, and the light learns nothing, trial after trial. The occurrence of the unconditioned stimulus is already predicted nearly perfectly, so little or no error — no surprise — is introduced by the new component, and surprise is the only thing that drives learning.⁵

The historical weight of the result is that it made a mechanistic rule beat a cognitive one. Blocking looks like it needs a cognitive story — the animal noticing that a component was added and reasoning that the old one already explains the outcome. Rescorla and Wagner showed that no such reasoning is required: a traditional contiguity theory, adjusted so the error term uses the aggregate strength of everything present, accounts for blocking with a simple error-driven update and nothing else.⁶ From the machine-learning side that update is the Least-Mean-Square (Widrow–Hoff) rule, a regression algorithm that drives mean-squared prediction error toward zero. Blocking is what a regression fit does when a new feature is collinear with one already fit: the residual is gone, so the new feature gets no weight.

Blocking, trial by trial

The one-line argument is easy to accept and easy to under-feel. Running the numbers shows the mechanism at work. Take $\alpha \beta =0.3$, an unconditioned stimulus that supports strength $R=1$, and both components starting at $V=0$. In phase 1 only the tone is present, so each trial updates $V_{\text{tone}}\leftarrow V_{\text{tone}}+0.3(1-V_{\text{tone}})$. The strength climbs and the increments shrink as the error shrinks:

After a handful more trials the tone sits near $0.95$; take it into phase 2 at $V_{\text{tone}}\approx 0.95$, with the light joining fresh at $V_{\text{light}}=0$. Now the compound's aggregate strength is $V_{\text{tone}}+V_{\text{light}}$, and the error every component sees is $1-(V_{\text{tone}}+V_{\text{light}})$. The tone has already all but closed the gap, so that error is tiny, and the light's share of a tiny error is tinier still:

The light gains about $0.024$ over four compound trials and is still crawling toward a ceiling under $0.03$. It will never do much better: the tone and light split whatever error remains, and the tone entered the phase already near the ceiling, so there is almost nothing to split. Contrast a control group that skips phase 1 entirely and meets the compound cold, both components at $0$. There the first-trial error is the full $1-(0+0)=1$, the light takes its half-share $\Delta V_{\text{light}}=0.15$, and across the same four trials it climbs to roughly $0.28$ — an order of magnitude above the blocked group. The light is physically identical in the two groups and equally contiguous with the puff; the only difference is what the animal already predicted walking in. Prediction, not contiguity, gates the learning.

The collinear-feature view

The regression reading of blocking is the exact machine-learning mirror of the phenomenon, and it explains why the added stimulus gets no weight, not merely that it does. Treat each trial as a supervised example: the input is a feature vector marking which stimuli are present, the target is the unconditioned-stimulus magnitude $R$, and the associative strengths are the regression weights. Least-mean-square regression moves the weights down the gradient of squared error, which is precisely the Rescorla–Wagner update.⁷

In phase 1 the tone feature $x_1$ alone is switched on, and its weight $w_1$ is driven until $w_1{x}_1$ matches the target — the residual on those trials goes to zero. Phase 2 switches on a second feature $x_2$ (the light) that, across every compound trial, is perfectly correlated with the first: whenever the tone fires the light fires, and both fire on exactly the trials where the target is $R$. A perfectly collinear feature adds no independent information about the target. The residual it would be fit against is already zero, so the gradient with respect to $w_2$ is zero, and $w_2$ stays put. Least squares cannot separate the contributions of two features that always move together; it credits the one it saw first and leaves the redundant one at whatever it started, which here is nothing.

The identity is not loose analogy. Widrow and Hoff (1960) derived the least-mean-square rule as an adaptive filter; Rescorla and Wagner (1972) derived their conditioning rule from animal data; the update equation is the same, and blocking falls out of the collinearity structure either way. The animal-learning result and the regression fact are one computation seen from two disciplines.

Higher-order conditioning and conditioned reinforcement

The second prominent property is higher-order conditioning: a previously-conditioned conditioned stimulus can itself act as an unconditioned stimulus for a new, initially neutral stimulus.⁸

Pavlov's demonstration is the cleanest. His assistant first conditioned a dog to salivate to a metronome that predicted food. Then, in a second stage, a black square — to which the dog was indifferent — was placed in the dog's line of vision and followed by the sound of the metronome, and this was never followed by food. Within about ten trials the dog began to salivate merely on seeing the black square, even though the sight of it had never been paired with food. The metronome, though it is not food, acted as an unconditioned stimulus in conditioning a salivation response to the square. This is second-order conditioning. If the black square were then used to condition yet another neutral stimulus, that would be third-order, and so on.⁹

Higher-order conditioning is hard to demonstrate above the second order, because a higher-order reinforcer loses its reinforcing value if it is not itself occasionally re-paired with the original unconditioned stimulus. Present the black square enough times without the metronome ever leading back to food, and its borrowed value extinguishes. Under favorable conditions — intermixing first-order trials with higher-order ones, or adding a general energizing stimulus — orders beyond the second can be shown.¹⁰ The Rescorla–Wagner model, which only ever pushes strengths toward the unconditioned-stimulus level, has no mechanism for this at all; the TD model produces it for free, because a change in prediction from one instant to the next has the same status as the unconditioned stimulus, and the metronome supplies just that to the square.

Why higher-order conditioning rises then falls

The TD model does more than allow higher-order conditioning; it predicts the peculiar shape of its time course, and the shape is why third and higher orders are so hard to get. Run the second-order simulation the way Sutton and Barto do. In a first phase, CSB (the metronome) is conditioned to the unconditioned stimulus until its strength saturates — say $V_{\text{B}}\approx 1.65$ in the units of one published run. Then the second phase begins: CSA (the square) is presented followed by CSB, and the unconditioned stimulus is never delivered. Two things happen at once on every second-order trial.

CSA learns from the drop in prediction between seeing CSA and then seeing CSB, because in the TD error the term $\hat{v}(S_{t+1})-\hat{v}(S_t)$ has the same status as a reward. So long as CSB still carries strength, CSA sees a positive target and climbs. But CSB is now in extinction — it predicts an unconditioned stimulus that no longer comes, so its own strength decays trial by trial. The reinforcer CSA is feeding on is draining. CSA therefore rises while CSB is still strong, reaches a peak, and then falls as CSB's collapse pulls the target it feeds on down toward zero.

The numbers are illustrative of the published trajectory, not a fit to any one animal, but the qualitative pattern — CSA up then down, tracking CSB's decay with a lag — is what both the TD model and animal experiments show.¹¹

This is the whole difficulty with orders beyond the second, stated mechanically. Each higher order feeds on the strength of the one below it, and each order below is being extinguished the moment you stop pairing it with the real unconditioned stimulus. The borrowed strength decays faster than the next order can accumulate it, so the chain starves. The fix animal researchers use follows straight from the mechanism: interleave occasional first-order trials to refresh the bottom of the chain before it extinguishes. Finch and Culler (1934) reached as high as fifth-order withdrawal conditioning in dogs by keeping motivation high and periodically renewing the lower-order links — evidence that the ceiling on order is set by extinction of the borrowed reinforcer, not by any hard limit on bootstrapping itself.¹²

Conditioned reinforcement

The same phenomenon occurs on the instrumental side, and there it has a name that connects straight to value functions. A stimulus that consistently predicts primary reinforcement becomes a reinforcer itself. Primary reinforcement is the innately rewarding or penalizing kind, built into the animal by evolution; the predicting stimulus becomes a secondary, or more generally a higher-order or conditioned reinforcer.¹³ A conditioned reinforcer delivers conditioned reinforcement: it acts like primary reinforcement, increasing an animal's tendency to produce behavior that leads to conditioned reward and decreasing the tendency to produce behavior that leads to conditioned penalty.

This is why an animal — or a person — will work for money. Money's worth derives solely from what having it predicts. It is a conditioned reinforcer, and the behavior it sustains matches what a value function, not a primary reward, would drive.¹⁴

The tie to this course is precise, and it is the argument of the whole module in one line: a value estimate is a secondary reinforcer. In an actor–critic architecture the critic uses a TD method to evaluate the actor's policy, and its value estimates provide conditioned reinforcement to the actor, letting the actor improve its policy.¹⁵ The critic's TD error is a conditioned reinforcer in the technical, animal-learning sense: it acts like reward, it was learned from prediction rather than given, and it lets the actor be trained even when primary reward is absent from the current step. Higher-order conditioning in the lab and bootstrapping in the algorithm are the same move — a learned prediction standing in for the outcome it predicts.

Delayed reinforcement

The Law of Effect requires a backward effect: a consequence must reach back and strengthen the connection that produced it. Early critics could not conceive how the present could affect something already in the past, and the worry sharpened when experiments showed learning still occurs with a considerable delay between an action and its consequent reward. The same holds in classical conditioning, where learning occurs even when the unconditioned stimulus follows conditioned- stimulus offset by a non-negligible interval. This is the problem of delayed reinforcement, and it is Minsky's (1961) credit-assignment problem for learning systems: how do you distribute credit for success among the many decisions that may have been involved in producing it?¹⁶

The algorithms in this course address delayed reinforcement with two mechanisms, and both correspond to mechanisms proposed in theories of animal learning. The first is eligibility traces. The second is TD-learned value functions that provide nearly immediate evaluations of actions, in instrumental tasks, or nearly immediate prediction targets, in classical ones.¹⁷

Stimulus traces

Pavlov pointed out that every stimulus must leave a trace in the nervous system that persists for some time after the stimulus ends, and he proposed that these stimulus traces make learning possible when there is a temporal gap between the conditioned-stimulus offset and the unconditioned-stimulus onset. This is why the gapped arrangement is called trace conditioning: assuming a trace of the conditioned stimulus remains when the unconditioned stimulus arrives, learning occurs through the simultaneous presence of the trace and the unconditioned stimulus.¹⁸

Hull built decaying stimulus traces into his influential learning theory to account for the goal gradient — his term for the way the maximum strength of an instrumentally conditioned response falls off as the delay of reinforcement increases. He hypothesized that an action leaves an internal stimulus whose trace decays exponentially as a function of the time since the action was taken, and, reading the animal data available to him, he estimated the traces effectively reach zero after 30 to 40 seconds.¹⁹

The eligibility traces this course uses are like Hull's: decaying traces of past state visitations, or of past state–action pairs, that mark recent experiences as eligible for a learning update when reinforcement finally arrives. Klopf (1972) introduced eligibility traces in a neuronal theory, where they are temporally-extended traces of past activity at synapses — the connections between neurons; Klopf's traces are more elaborate than the exponentially-decaying ones our algorithms use, but the idea is his.²⁰ The eligibility-traces mechanism, the vector ${\mathbf{z}}_t$ that decays at rate $\gamma \lambda$ and gates each component's update, is Pavlov's and Hull's stimulus trace made precise.

The trace interval sets the credit

The quantitative content of the trace idea is that credit assigned to a conditioned stimulus is proportional to its trace height at the moment reinforcement arrives. Model the trace of a stimulus that ended at time $0$ as $z(\Delta )=e^{-k\Delta }$, decaying with the interval $\Delta$ since offset. If the unconditioned stimulus arrives after a trace interval $\Delta$, the update the stimulus receives scales with $z(\Delta )$. Two trace intervals then predict two learning rates from one curve. With a decay constant $k=0.15{\text{ s}}^{-1}$, a stimulus reinforced $2$ s after offset is updated in proportion to $e^{-0.3}\approx 0.74$; one reinforced $8$ s after offset is updated in proportion to $e^{-1.2}\approx 0.30$. The near stimulus gets roughly two and a half times the credit of the far one, and conditioning is correspondingly weaker at the longer trace interval — the standard finding that trace conditioning degrades as the gap grows, read straight off the trace's height.

Value functions as the longer gradient

Stimulus traces alone cannot span very long delays — Hull's reach zero in half a minute. To account for goal gradients over longer periods, Hull (1943) proposed that longer gradients result from conditioned reinforcement passing backwards from the goal, acting together with his molar stimulus traces. Animal experiments had shown that if conditions favor conditioned reinforcement during a delay — if there are stimuli that regularly occur during the delay interval — then learning does not decrease with increased delay as much as it does under conditions that obstruct secondary reinforcement. It is as if the reward is not actually delayed, because there is more immediate conditioned reinforcement. Hull envisioned a primary gradient based on the delay of primary reinforcement, mediated by stimulus traces, progressively modified and lengthened by conditioned reinforcement.²¹

Hull's goal gradient has a definite functional form worth writing down. If a response leaves a trace that decays exponentially, response strength as a function of the delay $d$ to reinforcement is $g(d)\propto e^{-kd}$, the same exponential the trace itself follows. Reading the animal data of his day, Hull put the constant so that the gradient was already near zero by $30$ to $40$ s of delay: with $k\approx 0.1{\text{ s}}^{-1}$, strength at $30$ s is $e^{-3}\approx 0.05$, effectively gone. That short reach is the problem. Many tasks put reward far more than half a minute past the actions that earn it, and a raw trace simply cannot span the gap. Hull's answer was that stimuli regularly occurring during the delay pick up conditioned-reinforcement value and hand credit backwards in stages, so the effective gradient is the primary exponential lengthened by secondary reinforcement — a shallow, far-reaching tail grafted onto the steep near part.

A TD-learned value function does just this, and the actor–critic architecture illustrates the correspondence most clearly. The critic uses a TD algorithm to learn a value function for the current policy — to predict the current policy's return — and the actor updates its policy on changes in the critic's predictions. The TD error produced by the critic acts as a conditioned reinforcement signal for the actor, providing an immediate evaluation of performance even when the primary reward signal is considerably delayed. Action-value methods like Q-learning and Sarsa use the same TD principle to enable learning with delayed reinforcement by means of conditioned reinforcement.²² Hull's primary gradient lengthened by secondary reinforcement is a value function bootstrapping credit back across a gap that a raw trace could never cover — the two mechanisms of this course, working together, are the two mechanisms he proposed.

From associations to maps

Blocking, higher-order conditioning, and delayed reinforcement are all mechanisms of association — cues acquiring predictive strength, predictions standing in for reward, traces carrying credit across a gap. None of them requires the animal to hold a model of its world. The next part turns to the phenomenon that does: Tolman's rats learned the layout of a maze with no reward at all, then used that knowledge the moment reward appeared. That is a cognitive map — model-based reinforcement learning, measured in a rat — and it is where this thread continues.

This continues in Cognitive Maps and Model-Based Learning.