---
title: The Psychology of Reinforcement
module: Reinforcement Learning in Minds and Brains
moduleNumber: 6
lessonNumber: 1
order: 601
summary: >
  Reinforcement learning is both an engineering method and a theory
  of how animals learn. The prediction/control split of the algorithms mirrors
  the psychologist's split between classical and instrumental conditioning. We
  trace the correspondence: the Rescorla–Wagner model as a prediction-error rule
  that explains blocking, its real-time TD extension, Thorndike's Law of Effect
  behind trial-and-error control, and the habitual/goal-directed distinction that
  maps onto model-free versus model-based learning.
topics: [Minds and Brains]
sources:
  - book: Sutton & Barto
    ref: "Ch. 14 — Psychology; §14.1 Prediction and Control; §14.2 Classical Conditioning"
  - book: Sutton & Barto
    ref: "§14.2.2 The Rescorla–Wagner Model; §14.2.3 The TD Model; §14.3 Instrumental Conditioning; §14.6 Habitual and Goal-directed Behavior"
---

The algorithms in this course were built to solve an engineering problem: an
agent in an unknown environment, maximizing reward. But the ideas were not
invented in a vacuum. Trial-and-error learning, prediction of upcoming reward,
the notion that surprise drives learning — all of these were studied first by
psychologists working with animals, decades before the computational theory
existed. The correspondence between the two bodies of work is not an accident of
vocabulary. It is close enough that the same equations describe a rat pressing a
lever and a value-function update, and close enough that reinforcement learning
has fed new models back into experimental psychology.[^rl-not-vacuum]

This lesson argues that reinforcement learning is a _theory of learning
in animals_, not merely a method that borrows animal-learning words. The argument
rests on a single alignment: the two families of algorithms in this
book — those for **prediction** and those for **control** — correspond to the two
kinds of conditioning that animal-learning psychology has studied for over a
century.

## Prediction and control, classical and instrumental

Every algorithm we have built estimates something or improves something.
**Prediction** algorithms estimate a quantity that depends on how the environment
is expected to unfold — a value function, the expected discounted return from a
state. **Control** algorithms find a policy that collects more reward. The two
are coupled (policy evaluation is prediction; policy improvement is control), but
the split is real, and it lines up with the two experimental paradigms of animal
learning.[^pred-control]

In **classical conditioning** (also called _Pavlovian_ conditioning), a stimulus
predicts a biologically significant event, and the animal comes to respond to the
predictor. The reinforcing event arrives whether or not the animal does anything;
the animal has no control over it. What it learns is a _prediction_. In
**instrumental conditioning** (also called _operant_ conditioning), the
reinforcing event is _contingent_ on what the animal does — the apparatus
delivers reward or punishment depending on the animal's behavior. What the animal
learns is a _policy_: which action to take. This is _control_.

$$
% caption: The two families of algorithms in this course map onto the two paradigms
% of animal-learning psychology: prediction to classical (Pavlovian) conditioning,
% control to instrumental (operant) conditioning.
\begin{tikzpicture}[>=stealth, font=\small,
  box/.style={draw, minimum width=34mm, minimum height=12mm, align=center, font=\footnotesize},
  lab/.style={font=\scriptsize, text=black}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box, draw=acc, text=acc, thick] (pred) at (0,0)   {prediction\\(estimate a value)};
  \node[box, draw=acc, text=acc, thick] (ctrl) at (0,-2.2) {control\\(improve a policy)};
  \node[box] (cc) at (6,0)   {classical conditioning\\(Pavlovian)};
  \node[box] (ic) at (6,-2.2) {instrumental conditioning\\(operant)};
  \draw[<->, acc, thick] (pred) -- (cc) node[midway, above, lab] {predict a stimulus};
  \draw[<->, acc, thick] (ctrl) -- (ic) node[midway, above, lab] {act on consequences};
\end{tikzpicture}
$$

The alignment is a convenient first approximation rather than a clean partition.
There is more to classical conditioning than pure prediction — the conditioned
response is itself an action, so classical conditioning also involves a kind of
control, sometimes called _Pavlovian control_. And the two paradigms interact in
most real experiments. But as a way to connect the computational picture to the
animal-learning picture, prediction-to-classical and control-to-instrumental is
the right first cut.[^first-approx]

> **Definition (Classical vs. instrumental conditioning).** In **classical
> (Pavlovian)** conditioning the reinforcing stimulus is delivered independently
> of the animal's behavior; the animal learns to _predict_ it. In **instrumental
> (operant)** conditioning the reinforcing stimulus is _contingent_ on behavior;
> the animal learns which behavior _produces_ it.

## Classical conditioning

Pavlov's dogs are the origin. Studying salivation, Pavlov found that an innate
reflex — salivating to food in the mouth — could be triggered by an initially
neutral stimulus, the sound of a metronome, after the metronome had repeatedly
preceded the food.[^pavlov] The vocabulary that survives from his work:

- The food is an **unconditioned stimulus** (US): it triggers a response
  innately, with no learning.
- Salivation to food is the **unconditioned response** (UR).
- The metronome, initially neutral, becomes a **conditioned stimulus** (CS) once
  it reliably predicts the US.
- Salivation to the metronome _alone_ is the **conditioned response** (CR).

The US is called a **reinforcer** because it reinforces the production of a CR to
the CS. The CR is anticipatory: in a much-studied preparation, a tone CS predicts
a puff of air to a rabbit's eye (the US), and the rabbit learns to close a
protective membrane _before_ the puff arrives, timed to peak just when the puff is
due. Acting in anticipation of important events, rather than merely reacting to
them, is adaptive, which is why this ability is widespread across the
animal kingdom.[^anticipatory]

Two arrangements of the CS and US in time are standard, and the difference
matters for the real-time model below. In **delay conditioning** the CS stays on
until the US begins, overlapping it. In **trace conditioning** the CS ends before
the US begins, leaving a gap — the _trace interval_ — that the animal must bridge
with some persisting internal signal.

$$
% caption: Delay vs. trace conditioning. In delay conditioning the CS overlaps
% the US; in trace conditioning the CS ends before the US, leaving a trace
% interval that a persisting internal signal must bridge.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  % --- delay conditioning (top) ---
  \node[anchor=west, font=\small] at (-0.2,3.3) {delay conditioning};
  \node[anchor=east] at (-0.1,2.6) {CS};
  \draw[acc, thick] (0,2.35) -- (0.6,2.35) -- (0.6,2.85) -- (3.2,2.85) -- (3.2,2.35) -- (4.4,2.35);
  \node[anchor=east] at (-0.1,1.6) {US};
  \draw[black, thick] (0,1.35) -- (3.0,1.35) -- (3.0,1.85) -- (3.6,1.85) -- (3.6,1.35) -- (4.4,1.35);
  % --- trace conditioning (bottom) ---
  \node[anchor=west, font=\small] at (-0.2,0.7) {trace conditioning};
  \node[anchor=east] at (-0.1,0.0) {CS};
  \draw[acc, thick] (0,-0.25) -- (0.6,-0.25) -- (0.6,0.25) -- (2.0,0.25) -- (2.0,-0.25) -- (4.4,-0.25);
  \node[anchor=east] at (-0.1,-1.0) {US};
  \draw[black, thick] (0,-1.25) -- (3.0,-1.25) -- (3.0,-0.75) -- (3.6,-0.75) -- (3.6,-1.25) -- (4.4,-1.25);
  % time axis
  \draw[->, black] (0,-1.7) -- (4.6,-1.7) node[anchor=west, black] {t};
  % trace-interval bracket
  \draw[black, <->] (2.05,-0.45) -- (2.95,-0.45);
  \node[black, anchor=north, font=\scriptsize] at (2.5,-0.5) {trace};
\end{tikzpicture}
$$

### Four phenomena a model must explain

Decades of experiments established properties that any account of classical
conditioning has to reproduce. Four are central.[^blocking-hoc]

**Acquisition.** Over repeated CS–US pairings the CR appears and strengthens,
approaching an asymptote set by the US.

**Extinction.** Present the CS repeatedly _without_ the US and the CR weakens and
disappears. Prediction without confirmation decays.

**Blocking.** This one is the pivotal test. First condition the animal to a tone
CS so it predicts the US. Then present a _compound_ CS — tone plus a new light —
paired with the same US, over many trials. Finally test the light alone. The
animal shows little or no CR to the light: learning about the light has been
**blocked** by the prior learning about the tone.[^blocking] Blocking is
devastating for the naive theory that conditioning depends only on _temporal
contiguity_ — on the CS and US simply co-occurring. The light co-occurs with the
US just as reliably as the tone does, yet almost nothing is learned about it. What
the animal already predicts, it does not need to learn again.

**Higher-order conditioning.** A previously conditioned CS can itself act as a US
for a new, neutral stimulus. Pavlov conditioned a dog to salivate to a metronome
predicting food, then paired a black square with the metronome — never with food.
Within a few trials the dog salivated to the black square. The metronome, now a
_secondary reinforcer_, transferred its predictive value to the square. This is
second-order conditioning; chains of it are third-order, and so on.[^hoc]

Blocking is the phenomenon that reshaped the field, because it says plainly that
learning is driven by _prediction error_, not co-occurrence. An animal learns
about a stimulus only to the extent that the outcome is _surprising_ — not already
predicted by everything else present.

## The Rescorla–Wagner model

Rescorla and Wagner built their model mainly to account for blocking. The core
idea: an animal learns only when events violate its expectations —
only when it is _surprised_.[^rw]

Each component stimulus of a compound CS carries an **associative strength** $V$,
a number for how strongly and reliably that component predicts the US. Consider a
compound CS $\mathrm{AX}$ made of components $A$ and $X$, with strengths $V_A$ and
$V_X$. What matters on a trial is the **aggregate** associative strength of
everything present, not each component alone. On a trial where the compound is
followed by the US, the components update by

$$
\Delta V_A = \alpha\beta\,(\lambda - V_{\mathrm{AX}}),
\qquad
\Delta V_X = \alpha\beta\,(\lambda - V_{\mathrm{AX}}),
$$

with the key assumption that the aggregate strength is the sum,
$V_{\mathrm{AX}} = V_A + V_X$. The symbols:

- $\alpha$ is the **salience** of the CS component (how noticeable it is),
- $\beta$ is a **rate** parameter tied to the US,
- $\lambda$ is the **asymptote** — the level of associative strength the US can
  support (Sutton and Barto write $R$ here, the magnitude of the reward signal, to
  avoid confusion with the eligibility-trace $\lambda$; we keep the classical
  $\lambda$ in this section),
- and the bracket $(\lambda - \sum V)$ is the **prediction error**.

The rule is a prediction-error update: each associative
strength moves toward a target $\lambda$, by an amount proportional to how far the
_summed_ prediction $\sum V$ falls short of it. When the aggregate prediction
already matches the US, the error is zero and nothing changes. Learning stops when
the prediction is right.

> **Algorithm (Rescorla–Wagner update).** On each trial, adjust every present
> component's associative strength toward the US level $\lambda$ in proportion to
> the _shared_ prediction error: $\Delta V_i = \alpha_i\beta\,(\lambda - \sum_j
> V_j)$, where the sum runs over all components present on the trial. Absent
> components do not change.

### A worked acquisition trace

Run numbers through the rule. Take a single CS
$A$, so the aggregate strength is just $V_A$. Fix the product $\alpha\beta = 0.3$
(a salience of $0.3$, a US rate of $1$), the asymptote $\lambda = 1$, and start
from $V_0 = 0$. The update on every reinforced trial is

$$
V \leftarrow V + 0.3\,(1 - V).
$$

Carrying it forward, each trial closes $30\%$ of the remaining gap to $\lambda$:

| trial | $V$ before | error $\lambda - V$ | $\Delta V = 0.3\,(\lambda-V)$ | $V$ after |
|:-----:|:----------:|:-------------------:|:-----------------------------:|:---------:|
| 1 | $0.000$ | $1.000$ | $0.300$ | $0.300$ |
| 2 | $0.300$ | $0.700$ | $0.210$ | $0.510$ |
| 3 | $0.510$ | $0.490$ | $0.147$ | $0.657$ |
| 4 | $0.657$ | $0.343$ | $0.103$ | $0.760$ |
| 5 | $0.760$ | $0.240$ | $0.072$ | $0.832$ |
| 6 | $0.832$ | $0.168$ | $0.050$ | $0.882$ |

The strength climbs quickly at first, then slows as the error shrinks, tracing a
decaying exponential toward $\lambda$. In closed form the residual gap follows
$\lambda - V_n = (1-\alpha\beta)^n(\lambda - V_0) = 0.7^{\,n}$, so $V_n = 1 -
0.7^{\,n}$ — a negatively accelerated acquisition curve, the classic learning-curve
shape. Extinction is the mirror image: present $A$ with no US, set $\lambda = 0$,
and the same rule drives $V$ back down by $30\%$ of its remaining height each trial.

$$
% caption: Rescorla-Wagner acquisition for a single CS with $\alpha\beta=0.3$,
% $\lambda=1$, $V_0=0$. The associative strength follows $V_n=1-0.7^{\,n}$, closing
% $30\%$ of the gap to $\lambda$ each trial: a negatively accelerated approach to
% the asymptote.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  % axes
  \draw[->, black] (0,0) -- (7.2,0) node[anchor=west, black, font=\scriptsize] {trial};
  \draw[->, black] (0,0) -- (0,3.2) node[anchor=south, black, font=\scriptsize] {strength V};
  % asymptote
  \draw[black, dashed] (0,2.7) -- (7.0,2.7);
  \node[anchor=east, black, font=\scriptsize] at (-0.05,2.7) {level};
  % data points V_n = 1 - 0.7^n scaled by 2.7; x at 0.9 unit spacing
  \foreach \n/\v in {1/0.300, 2/0.510, 3/0.657, 4/0.760, 5/0.832, 6/0.882}{
    \fill[acc] (\n*0.95, {\v*2.7}) circle (1.6pt);
  }
  % connecting curve
  \draw[acc, very thick] (0,0)
    .. controls (0.5,0.62) and (0.75,0.81) .. (0.95,0.81)
    .. controls (1.5,1.20) and (1.9,1.38) .. (2.85,1.77)
    .. controls (3.8,2.05) and (4.8,2.25) .. (5.7,2.38);
  \node[acc, anchor=west, font=\scriptsize] at (5.75,2.30) {V};
\end{tikzpicture}
$$

### How it explains blocking

Blocking falls straight out of the summed-error term. After the tone $A$ has been
fully conditioned, $V_A \approx \lambda$: the tone alone already predicts the US.
Now add the light $X$ (initially $V_X = 0$) to form the compound. The prediction
error on the very first compound trial is

$$
\lambda - V_{\mathrm{AX}} = \lambda - (V_A + V_X) \approx \lambda - \lambda = 0.
$$

The error is already zero because the tone alone accounts for the US. With no
error, $\Delta V_X = \alpha\beta \cdot 0 = 0$, and the light learns nothing, trial
after trial. The US is already predicted, so its arrival is no surprise, and
surprise is what drives learning. That is blocking, in one line of algebra.

$$
% caption: Blocking in the Rescorla–Wagner model. Tone $A$ is pretrained to the
% asymptote $\lambda$; when the light $X$ is added, the summed prediction $V_A+V_X$
% already equals $\lambda$, so the error $\lambda-\sum V$ is zero and $V_X$ stays
% flat while $V_A$ holds.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, black] (0,0) -- (8.2,0) node[anchor=west, black, font=\scriptsize] {trials};
  \draw[->, black] (0,0) -- (0,3.4) node[anchor=south, black, font=\scriptsize] {assoc. strength};
  % asymptote line
  \draw[black, dashed] (0,2.7) -- (8.0,2.7);
  \node[anchor=east, black, font=\scriptsize] at (-0.05,2.7) {level};
  % phase divider
  \draw[black, dotted] (3.8,0) -- (3.8,3.2);
  \node[anchor=south, font=\scriptsize, text=black] at (1.9,3.15) {phase 1: A alone};
  \node[anchor=south, font=\scriptsize, text=black] at (5.9,3.15) {phase 2: A + X};
  % V_A curve: rises then holds
  \draw[acc, very thick] (0,0) .. controls (1.2,2.2) and (2.4,2.65) .. (3.8,2.7) -- (8.0,2.7);
  \node[acc, anchor=west, font=\scriptsize] at (8.05,2.7) {V(A)};
  % V_X curve: stays at zero
  \draw[red, very thick] (3.8,0.08) -- (8.0,0.08);
  \node[red, anchor=west, font=\scriptsize] at (8.05,0.15) {V(X)};
\end{tikzpicture}
$$

To see the flat line rather than assert it, run phase 2 the same way as the
acquisition trace. Pretraining leaves the tone at $V_A = 1.0$ (its asymptote).
Now every compound trial presents $A$ and $X$ together, so the error uses the
_sum_ $V_A + V_X$, and both components share it. With $\alpha\beta = 0.3$ each:

| trial | $V_A$ | $V_X$ | error $\lambda-(V_A+V_X)$ | $\Delta V_A=\Delta V_X$ | $V_A'$ | $V_X'$ |
|:-----:|:-----:|:-----:|:-------------------------:|:-----------------------:|:------:|:------:|
| 1 | $1.00$ | $0.00$ | $1-1.00=0.00$ | $0.00$ | $1.00$ | $0.00$ |
| 2 | $1.00$ | $0.00$ | $1-1.00=0.00$ | $0.00$ | $1.00$ | $0.00$ |
| 3 | $1.00$ | $0.00$ | $1-1.00=0.00$ | $0.00$ | $1.00$ | $0.00$ |

The error is pinned at zero from the first compound trial, so $\Delta V_X = 0$ on
every trial and the light never acquires strength. Contrast this with a control
group that sees $X$ paired with the US _without_ a pretrained tone: there $V_X$
would run the acquisition trace above, climbing $0 \to 0.30 \to 0.51 \to \dots$
The only difference is the pretrained predictor that zeroes the shared error, which
is the summed-error term at work.

From the machine-learning side, the Rescorla–Wagner rule is an error-correction
supervised learning rule — essentially the Least-Mean-Square (Widrow–Hoff) rule,
a regression algorithm that drives the mean-squared prediction error toward
zero.[^lms] Its historical importance was to show that a _mechanistic_ rule,
needing no explicit reasoning about which stimulus was added, could account for
blocking, when everyone had assumed blocking demanded a cognitive story.

### Rescorla–Wagner is the LMS rule, precisely

The animal is doing linear regression: fitting weights so a
sum of present cues predicts the US, adjusting those weights by the same
error-times-input step a statistician uses. The equivalence to Least-Mean-Square
turns the psychology into optimization. Collect the component strengths into a weight vector
$\mathbf{w}$ and describe the compound present on a trial by a binary presence
vector $\mathbf{x}$, with $x_i = 1$ when component $i$ is present. The aggregate
prediction is the dot product $\mathbf{w}^\top\mathbf{x} = \sum_i V_i$, exactly
the summed associative strength. Define the per-trial squared error against the
US level $\lambda$:

$$
L(\mathbf{w}) = \tfrac{1}{2}\bigl(\lambda - \mathbf{w}^\top\mathbf{x}\bigr)^2.
$$

Its gradient is $\nabla_{\mathbf{w}} L = -(\lambda - \mathbf{w}^\top\mathbf{x})\,
\mathbf{x}$, so one step of gradient descent with rate $\alpha\beta$ gives

$$
\mathbf{w} \leftarrow \mathbf{w} + \alpha\beta\,(\lambda - \mathbf{w}^\top
\mathbf{x})\,\mathbf{x},
$$

which is the Rescorla–Wagner update, component by component: the factor
$\mathbf{x}$ is why only _present_ components change, and $(\lambda -
\mathbf{w}^\top\mathbf{x})$ is the shared prediction error. Rescorla and Wagner
had, without naming it, written down stochastic gradient descent on squared
prediction error — the same rule Widrow and Hoff published for adaptive filters
in 1960.[^lms-gradient] In this light blocking is a familiar regression fact: the
added feature $X$ is collinear with a feature already fit to the target, so once
$A$ explains the US the residual $(\lambda - \mathbf{w}^\top\mathbf{x})$ is zero
and least-squares assigns the redundant feature zero weight. Overexpectation,
below, is the same picture read in the opposite direction.

$$
% caption: The Rescorla-Wagner update is stochastic gradient descent on the
% squared prediction error $L=\tfrac12(\lambda-\mathbf{w}^\top\mathbf{x})^2$: the
% gradient $-(\lambda-\mathbf{w}^\top\mathbf{x})\mathbf{x}$ gives the update
% $\mathbf{w}\gets\mathbf{w}+\alpha\beta(\lambda-\mathbf{w}^\top\mathbf{x})\mathbf{x}$,
% identical to the Widrow-Hoff LMS rule.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  bx/.style={draw, minimum width=30mm, minimum height=10mm,
    align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[bx, draw=acc, text=acc, thick] (rw) at (0,1.4)
    {Rescorla-Wagner\\update present V by\\rate x (level minus sum V)};
  \node[bx] (lms) at (0,-1.4)
    {LMS / Widrow-Ho\/f\/f\\gradient step on\\squared prediction error};
  \node[anchor=west, font=\scriptsize, text=black, align=left] (mid) at (4.2,0)
    {same rule:\\w gets w + rate\\times (target minus w.x) x};
  \draw[<->, acc, thick] (rw) to[bend left=18] (mid.west);
  \draw[<->, black, thick] (lms) to[bend right=18] (mid.west);
\end{tikzpicture}
$$

### Overexpectation: a prediction contiguity cannot make

Because the error term sums over everything present, Rescorla–Wagner predicts a
phenomenon that a pure co-occurrence account rules out. Train $A$ alone to the
asymptote, so $V_A \approx 1$. Separately train $X$ alone to the asymptote, so
$V_X \approx 1$. Each has been paired with exactly the same US $\lambda = 1$ and
neither is over-predicting anything on its own. Now present the _compound_ $AX$,
still followed by the same single US. The summed prediction is now $V_A + V_X
\approx 2$, but the US supports only $\lambda = 1$, so the shared error goes
_negative_:

| trial | $V_A$ | $V_X$ | error $\lambda-(V_A+V_X)$ | $\Delta V=0.3\cdot\text{err}$ | $V_A'$ | $V_X'$ |
|:-----:|:-----:|:-----:|:-------------------------:|:-----------------------------:|:------:|:------:|
| 1 | $1.000$ | $1.000$ | $1-2.000=-1.000$ | $-0.300$ | $0.700$ | $0.700$ |
| 2 | $0.700$ | $0.700$ | $1-1.400=-0.400$ | $-0.120$ | $0.580$ | $0.580$ |
| 3 | $0.580$ | $0.580$ | $1-1.160=-0.160$ | $-0.048$ | $0.532$ | $0.532$ |

Both strengths _fall below_ the asymptote each earned alone, settling toward $V_A
= V_X = 0.5$ where the sum matches $\lambda$. The compound over-predicted the US,
so the model unlearns some of what each component had — even though each component
is still perfectly contiguous with the US. This is **overexpectation**, and it is
a signature of the summed-error rule: contiguity predicts the strengths should
hold, the summed error predicts they drop, and the animal drops them.[^overexpect]

$$
% caption: Overexpectation. $A$ and $X$ are each trained alone to $\lambda=1$;
% presenting the compound $AX$ with the same single US makes the summed prediction
% $V_A+V_X\approx2$ exceed $\lambda$, driving a negative error so both strengths
% decline toward $0.5$.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (0,0) -- (8.2,0) node[anchor=west, black, font=\scriptsize] {trials};
  \draw[->, black] (0,0) -- (0,3.4) node[anchor=south, black, font=\scriptsize] {assoc. strength};
  \draw[black, dashed] (0,2.7) -- (8.0,2.7);
  \node[anchor=east, black, font=\scriptsize] at (-0.05,2.7) {level};
  \draw[black, dashed] (0,1.35) -- (8.0,1.35);
  \node[anchor=east, black, font=\scriptsize] at (-0.05,1.35) {half};
  \draw[black, dotted] (3.8,0) -- (3.8,3.2);
  \node[anchor=south, font=\scriptsize, text=black] at (1.9,3.15) {phase 1: A, X separate};
  \node[anchor=south, font=\scriptsize, text=black] at (5.9,3.15) {phase 2: compound AX};
  % both trained to level in phase 1, then decline in phase 2 toward half
  \draw[acc, very thick] (0,0.2) .. controls (1.4,2.5) .. (3.8,2.7)
    .. controls (5.0,2.0) and (6.4,1.55) .. (8.0,1.42);
  \node[acc, anchor=west, font=\scriptsize] at (8.05,1.55) {V(A)};
  \draw[red, very thick, densely dashed] (0,0.4) .. controls (1.4,2.4) .. (3.8,2.62)
    .. controls (5.0,1.9) and (6.4,1.45) .. (8.0,1.30);
  \node[red, anchor=west, font=\scriptsize] at (8.05,1.15) {V(X)};
\end{tikzpicture}
$$

### Limitations

Rescorla–Wagner is a **trial-level** model. It treats a whole trial as one event —
a set of components present, one prediction error, one update — and says nothing
about what happens _within_ a trial or _between_ trials. That is a serious
restriction:[^rw-limits]

- It cannot represent the **timing** of stimuli, so it cannot explain why the CR
  appears _before_ the US, or how the inter-stimulus interval controls the
  strength of conditioning.
- It has no mechanism for **higher-order conditioning** — its updates only ever
  push strengths toward the US level $\lambda$, and a stimulus never present with
  the US can never gain strength.
- It cannot address trace conditioning, serial compounds, or the shape of the
  conditioned response over time.

Every one of these gaps is about _time within the trial_. The fix is to make the
model real-time.

## The TD model of classical conditioning

The **TD model** extends Rescorla–Wagner from a trial-level model to a _real-time_
one. Now $t$ labels a small time step _within or between_ trials, not a whole
trial. A trial becomes a sequence of states, one per time step, and the animal
experiences an endless stream of states $s$, each described by a feature vector
$\mathbf{x}(s)$. With this move the trial-level restrictions dissolve, and the
connection to the rest of this course becomes exact.[^td-model]

Instead of one update per
whole trial, the animal now makes a small update at every instant, comparing what it
predicts now to what it predicts a moment later. The three equations below write out
that comparison — a value estimate, an error, and a memory of what was
recently active.

Let the associative-strength vector be $\mathbf{w}$. The aggregate associative
strength of a state — the US prediction — is the linear function

$$
\hat v(s, \mathbf{w}) = \mathbf{w}^\top \mathbf{x}(s),
$$

which is precisely a **value estimate** with linear function approximation. The
learning update is

$$
\mathbf{w}_{t+1} = \mathbf{w}_t + \alpha\,\delta_t\,\mathbf{z}_t,
$$

where the scalar $\delta_t$ is the **TD error**

$$
\delta_t = R_t + \gamma\,\hat v(S_{t+1}, \mathbf{w}_t) - \hat v(S_t, \mathbf{w}_t),
$$

and $\mathbf{z}_t$ is a vector of **eligibility traces** that decays at rate
$\gamma\lambda$,

$$
\mathbf{z}_{t+1} = \gamma\lambda\,\mathbf{z}_t + \mathbf{x}(S_t).
$$

Here $R_t$ is the prediction target (the US magnitude), $\gamma$ is a discount
factor, and $\lambda$ is the eligibility-trace decay. This reproduces the backward
view of semi-gradient $\text{TD}(\lambda)$ with linear function approximation, the
prediction algorithm from
[temporal-difference learning](/reinforcement-learning/tabular-methods/temporal-difference-learning)
and [eligibility traces](/reinforcement-learning/approximation/eligibility-traces)
— the only twist being that $R_t$ here is a US intensity rather than a reward
signal.[^td-equiv]

The single difference between the two models is the error term. Rescorla–Wagner
uses the trial-level prediction error $\lambda - \sum V$. The TD model uses the
_temporal-difference_ error $\delta_t = R_t + \gamma\hat v(S_{t+1}) - \hat
v(S_t)$, which compares the prediction at one instant with the discounted
prediction at the _next_ instant. Set $\gamma = 0$ and the TD model collapses back
to Rescorla–Wagner. The bootstrapping term $\gamma\hat v(S_{t+1})$ is what buys
everything the trial-level model lacked.

$$
% caption: Rescorla–Wagner and the TD model differ only in the error term. RW
% compares the summed prediction against the US level $\lambda$ once per trial; the
% TD model compares the prediction at $t$ against the discounted prediction at
% $t+1$ every time step. With $\gamma=0$ the TD model reduces to RW.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=40mm, minimum height=15mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box] (rw) at (0,0) {Rescorla-Wagner (trial-level)\\error = level minus sum V\\one update per trial};
  \node[box, draw=acc, text=acc, thick] (td) at (6.5,0) {TD model (real-time)\\error = R + discounted v(next) - v(now)\\one update per time step};
  \draw[->, acc, thick] (rw) -- (td) node[midway, above, font=\scriptsize, text=black] {add time};
  \draw[->, black, thick] (td) to[bend left=32] node[midway, below, font=\scriptsize, text=black] {set discount = 0} (rw);
\end{tikzpicture}
$$

Because the TD error carries the term $\gamma\hat v(S_{t+1}) - \hat v(S_t)$, a
change in prediction from one instant to the next is treated _exactly like_ the
arrival of a US. This is bootstrapping — a guess corrected toward a later guess —
and it is what gives the TD model two things Rescorla–Wagner cannot produce:

- **Timing.** With a fine-grained state representation, the US prediction grows
  through the interval between CS and US and peaks at the moment the US is due, and
  the CR can anticipate the US. The inter-stimulus interval now controls learning.
- **Higher-order conditioning.** A previously conditioned CS raises $\hat v$, so
  when it is paired with a new neutral stimulus, the _temporal difference_ in
  prediction is itself non-zero and drives learning about the new stimulus — with
  no US present at all. Second- and higher-order conditioning is a direct
  consequence of bootstrapping.

The TD model is not a perfect account of classical conditioning, but it is a
_normative_ one: it says what the animal's nervous system is _trying to do_ during
conditioning — form accurate _long-term_ predictions, given the limits of how
stimuli are represented and how time is handled. That reframing, from immediate to
long-term prediction, is the model's central claim.[^normative]

### How time gets represented: presence, CSC, and microstimuli

The feature vector $\mathbf{x}(s)$ is where all the timing lives, and the TD model
behaves very differently depending on how a stimulus is turned into features over
time. Three representations recur in the literature, and they line up along a
_temporal generalization gradient_ — how much a feature active at one instant also
speaks for nearby instants.[^stim-reps]

The **presence representation** is the coarsest: one feature per CS, equal to $1$
for the whole time the CS is on and $0$ otherwise. It carries no information about
_when_ within the CS you are, so it forces complete generalization across all
times the CS is present. It is not a plausible brain code, yet it is enough to
produce most of the timing phenomena below.

The **complete serial compound** (CSC) is the finest: the onset of a CS starts a
sequence of short, precisely-timed internal signals, one distinct feature per
(CS, elapsed-time) pair — a "tapped delay line," as if the nervous system ran a
clock. Nearby time points share _no_ features, so there is no generalization
between them; the model can learn a separate prediction for each moment. It is
also unrealistic as a brain code (Ludvig and colleagues call it a "useful
fiction"), but it isolates the model's behavior from the representation, and it is
the representation used in most TD models of dopamine neurons.

The **microstimulus** (MS) representation sits in the middle. A CS onset launches
a cascade of internal features that are _extended_ and _overlapping_: as time
elapses, later microstimuli grow wider in time and reach a lower peak. Overlap
means partial generalization between nearby times — coarser than the CSC clock,
finer than the single presence feature — and its smeared, decaying shape is closer
to what a population of neurons might actually produce.[^ms-realistic]

$$
% caption: Three stimulus representations along a temporal generalization
% gradient. CSC (left): one narrow non-overlapping feature per time step, no
% generalization. Microstimuli (center): overlapping features widening and
% lowering over time, partial generalization. Presence (right): a single feature
% held at 1 while the CS is on, complete generalization. Time runs left to right
% within each column.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  % ----- column titles -----
  \node[font=\small] at (1.6,3.35) {CSC};
  \node[font=\small] at (5.4,3.35) {microstimuli};
  \node[font=\small] at (9.2,3.35) {presence};
  % ===== CSC: five narrow non-overlapping bumps =====
  \begin{scope}[xshift=0cm]
    \draw[->, black] (0,0) -- (3.3,0) node[anchor=west, black, font=\scriptsize] {t};
    \foreach \x in {0.35,0.9,1.45,2.0,2.55}{
      \draw[acc, thick] (\x,0) -- (\x,0) .. controls (\x+0.06,1.9) and (\x+0.12,1.9) .. (\x+0.18,0);
      \fill[acc!18] (\x,0) .. controls (\x+0.06,1.9) and (\x+0.12,1.9) .. (\x+0.18,0) -- cycle;
    }
  \end{scope}
  % ===== MS: overlapping bumps, widening and lowering =====
  \begin{scope}[xshift=3.8cm]
    \draw[->, black] (0,0) -- (3.3,0) node[anchor=west, black, font=\scriptsize] {t};
    \draw[acc, thick] (0.2,0) .. controls (0.55,1.85) and (0.85,1.85) .. (1.2,0);
    \draw[acc, thick] (0.7,0) .. controls (1.15,1.5) and (1.55,1.5) .. (2.0,0);
    \draw[acc, thick] (1.3,0) .. controls (1.9,1.2) and (2.5,1.2) .. (3.1,0);
  \end{scope}
  % ===== presence: single wide flat-top feature =====
  \begin{scope}[xshift=7.6cm]
    \draw[->, black] (0,0) -- (3.3,0) node[anchor=west, black, font=\scriptsize] {t};
    \draw[acc, thick] (0.3,0) -- (0.3,1.6) -- (2.7,1.6) -- (2.7,0);
    \fill[acc!14] (0.3,0) -- (0.3,1.6) -- (2.7,1.6) -- (2.7,0) -- cycle;
  \end{scope}
  % gradient annotation
  \draw[black, ->] (0.4,-0.55) -- (10.4,-0.55);
  \node[anchor=north, font=\scriptsize, text=black] at (1.6,-0.55) {no generalization};
  \node[anchor=north, font=\scriptsize, text=black] at (9.0,-0.55) {full generalization};
\end{tikzpicture}
$$

### Why the CR anticipates the US

With the CSC representation the model learns a US prediction that is not flat
across the interval but _rising_. Discounting is the reason: the value at each
state is the discounted prediction of the value one step later, so at asymptote
the prediction at time $t$ before the US is roughly $\gamma^{\,k}$ times the US
level, where $k$ is the number of steps still to go. That grows toward the US
level as the US approaches and peaks exactly at US time. The presence
representation, lacking any within-CS timing, gives a near-flat prediction; MS
gives a smoothed version of the CSC ramp.

$$
% caption: US-prediction time course across the CS-US interval. With the CSC
% representation the learned prediction $\hat v$ rises like $\gamma^{k}$ toward the
% US level and peaks at US onset; presence gives a near-flat prediction; MS
% smooths the CSC ramp. This ramp is why the conditioned response is timed to
% anticipate the US.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (0,0) -- (7.4,0) node[anchor=west, black, font=\scriptsize] {time in interval};
  \draw[->, black] (0,0) -- (0,3.2) node[anchor=south, black, font=\scriptsize] {prediction v};
  % US marker
  \draw[black, dashed] (6.0,0) -- (6.0,2.9);
  \node[anchor=south, font=\scriptsize, text=black] at (6.0,2.9) {US};
  \node[anchor=north, font=\scriptsize, text=black] at (0.0,-0.05) {CS on};
  % CSC exponential ramp toward US level 2.6 at x=6
  \draw[acc, very thick] (0.2,0.35)
    .. controls (2.6,0.55) and (4.4,1.1) .. (6.0,2.6);
  \node[acc, anchor=south east, font=\scriptsize] at (5.7,2.15) {CSC};
  % MS smoothed
  \draw[red, very thick, densely dashed] (0.2,0.5)
    .. controls (2.8,0.95) and (4.6,1.5) .. (6.0,2.35);
  \node[red, anchor=south east, font=\scriptsize] at (2.4,2.15) {MS};
  \draw[red, thin] (2.35,2.1) -- (2.0,1.0);
  % presence near-flat
  \draw[black, very thick, dotted] (0.2,1.0) -- (6.0,1.15);
  \node[anchor=south, font=\scriptsize, text=black] at (2.4,1.05) {presence};
\end{tikzpicture}
$$

The rising prediction explains a fact the trial-level model cannot touch: the
rabbit's protective membrane closes _before_ the air puff, timed to peak when the
puff is due. Closing too early wastes vision, closing too late gives no
protection, so a CR locked to US time is the adaptive one, and the ramp-and-peak
prediction is what supports it.[^cr-timing]

### ISI tuning and temporal primacy over blocking

Two further predictions of the real-time model have no trial-level analogue. The
first is **ISI dependence**. Conditioning strength depends on the inter-stimulus
interval — the gap between CS onset and US onset — in a characteristic
inverted-U: it is negligible at a zero or negative ISI (US at or before CS onset),
rises to a maximum at some positive ISI, then falls off as the interval grows too
long to bridge. The TD model reproduces this shape, with the peak and width set by
its discounting and trace parameters.[^isi]

$$
% caption: The inter-stimulus-interval tuning curve. Conditioning is negligible
% at zero or negative ISI, rises to a peak at a positive ISI where the CS best
% predicts the US, then declines as the interval grows too long. The TD model
% reproduces this inverted-U.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \draw[->, black] (-1.4,0) -- (6.6,0) node[anchor=west, black, font=\scriptsize] {ISI};
  \draw[->, black] (0,0) -- (0,3.0) node[anchor=south, black, font=\scriptsize] {conditioning};
  % zero marker
  \draw[black, dotted] (0,0) -- (0,2.7);
  \node[anchor=north east, font=\scriptsize, text=black] at (0,-0.05) {0};
  \node[anchor=north, font=\scriptsize, text=black] at (-0.9,-0.05) {negative};
  \node[anchor=north, font=\scriptsize, text=black] at (4.6,-0.05) {longer};
  % inverted-U: flat near zero, rise, peak, decline
  \draw[acc, very thick] (-1.3,0.06) .. controls (-0.3,0.06) and (0.2,0.1) .. (0.6,0.4)
    .. controls (1.3,1.4) and (1.8,2.5) .. (2.4,2.55)
    .. controls (3.2,2.6) and (4.4,1.1) .. (6.0,0.25);
  \node[acc, anchor=south, font=\scriptsize] at (2.4,2.55) {peak ISI};
\end{tikzpicture}
$$

The second is sharper, because the model made it before anyone had looked. In
ordinary blocking the added CS is _simultaneous_ with the pretrained one. But if
the added CS is moved _earlier_, so its onset precedes the pretrained CS, the TD
model predicts that blocking **reverses**: the earlier stimulus is no longer
blocked, and as training continues it _gains_ strength while the later, pretrained
stimulus _loses_ it. The reason is bootstrapping. Updates push each state's
prediction toward the prediction of the state that follows, so an earlier
predictor takes precedence over a later one — temporal primacy overriding
blocking. Sutton and Barto reported this prediction in 1981; Kehoe, Schreurs, and
Graham confirmed it in 1987 in the rabbit nictitating-membrane preparation, noting
that non-TD models struggle with the data.[^primacy]

$$
% caption: Temporal primacy overriding blocking. Standard blocking (top): the
% added CS X starts at the same time as the pretrained CS A and stays blocked. If
% X is moved earlier (bottom) so its onset precedes A, bootstrapping lets X gain
% strength while pretrained A loses it - blocking reverses.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % ===== top: standard, X blocked =====
  \node[anchor=west, font=\small] at (-0.2,3.0) {X with A: blocked};
  \node[anchor=east, font=\scriptsize] at (-0.1,2.35) {A};
  \draw[acc, thick] (0.3,2.35) -- (3.4,2.35);
  \node[anchor=east, font=\scriptsize] at (-0.1,1.75) {X};
  \draw[red, thick] (0.3,1.75) -- (3.4,1.75);
  \node[anchor=east, font=\scriptsize] at (-0.1,1.15) {US};
  \draw[black, thick] (3.6,1.15) -- (4.2,1.15);
  \node[anchor=west, font=\scriptsize, text=red] at (4.5,1.9) {V(X) stays low};
  % ===== bottom: X earlier, not blocked =====
  \node[anchor=west, font=\small] at (-0.2,0.35) {X earlier: not blocked};
  \node[anchor=east, font=\scriptsize] at (-0.1,-0.3) {X};
  \draw[red, thick] (0.3,-0.3) -- (3.4,-0.3);
  \node[anchor=east, font=\scriptsize] at (-0.1,-0.9) {A};
  \draw[acc, thick] (1.6,-0.9) -- (3.4,-0.9);
  \node[anchor=east, font=\scriptsize] at (-0.1,-1.5) {US};
  \draw[black, thick] (3.6,-1.5) -- (4.2,-1.5);
  \node[anchor=west, font=\scriptsize, text=red] at (4.5,-0.3) {V(X) rises,};
  \node[anchor=west, font=\scriptsize, text=acc] at (4.5,-0.9) {V(A) falls};
\end{tikzpicture}
$$

## From prediction to control

We have followed the prediction half of the correspondence to its end: classical
conditioning is prediction, the Rescorla–Wagner rule is a trial-level prediction
error, and its real-time TD extension gives the prediction algorithm of this
course, with timing, higher-order conditioning, and temporal primacy falling out of
bootstrapping. What remains is the other half — _control_. Instrumental conditioning
is the animal learning what to _do_, not just what to expect, and it maps onto the
control side of the theory: the Law of Effect is trial-and-error control, and the
habitual/goal-directed distinction is the model-free/model-based split.

This continues in [The Psychology of Reinforcement: Instrumental Control](/reinforcement-learning/minds-and-brains/instrumental-conditioning-and-control).

[^rl-not-vacuum]: **Sutton & Barto**, _Reinforcement Learning_, Ch. 14 — Psychology: the correspondences between reinforcement learning and animal learning are striking because they connect ideas arising from different objectives, and psychology's influence on reinforcement learning is real though the two fields' terminologies do not always coincide.
[^pred-control]: **Sutton & Barto**, §14.1 — Prediction and Control: their algorithms fall into prediction and control, which correspond respectively to classical (Pavlovian) and instrumental (operant) conditioning; prediction algorithms are policy-evaluation algorithms that estimate how features of the environment will unfold.
[^first-approx]: **Sutton & Barto**, §14.1 — Prediction and Control: aligning the classical/instrumental distinction with prediction/control is a convenient first approximation; classical conditioning also involves action ("Pavlovian control"), and the two paradigms interact in most experiments.
[^pavlov]: **Sutton & Barto**, §14.2 — Classical Conditioning: Pavlov found an innate salivary reflex could be triggered by an initially neutral stimulus (a metronome) after repeated pairing with food; a neutral stimulus becomes a CS as the animal learns it predicts the US.
[^anticipatory]: **Sutton & Barto**, §14.2 — Classical Conditioning: the CR is anticipatory, e.g. a rabbit learns to close its nictitating membrane before an air puff, timed to peak when the puff is due; acting in anticipation of important events is beneficial and widespread across the animal kingdom.
[^blocking-hoc]: **Sutton & Barto**, §14.2.1 — Blocking and Higher-order Conditioning: beyond the anticipatory nature of CRs, blocking and higher-order conditioning are two widely-studied properties that figure prominently in classical-conditioning models.
[^blocking]: **Sutton & Barto**, §14.2.1 — Blocking: an animal fails to learn a CR to a stimulus presented in compound with a previously-conditioned stimulus; e.g. a light added to a pretrained tone produces few or no CRs when tested alone, challenging the idea that conditioning depends on temporal contiguity.
[^hoc]: **Sutton & Barto**, §14.2.1 — Higher-order Conditioning: a previously-conditioned CS acts as a US for a new neutral stimulus (Pavlov's black square paired with a food-predicting metronome), producing second-order conditioning and, in principle, higher orders.
[^rw]: **Sutton & Barto**, §14.2.2 — The Rescorla–Wagner Model: the model was created mainly to account for blocking; its core idea is that an animal learns only when events violate its expectations, i.e. only when surprised, with the aggregate associative strength of the whole compound (not each component alone) determining the update.
[^lms]: **Sutton & Barto**, §14.2.2 — The Rescorla–Wagner Model: from the machine-learning perspective the rule is an error-correction supervised learning rule, essentially the Least-Mean-Square (Widrow–Hoff, 1960) rule that minimizes mean-squared error.
[^rw-limits]: **Sutton & Barto**, §14.2.2–14.2.3: the Rescorla–Wagner model is a trial-level model that deals with how associative strengths change from trial to trial without considering within-trial or between-trial timing, and it includes no mechanism for higher-order conditioning; the TD model extends it to address these.
[^td-model]: **Sutton & Barto**, §14.2.3 — The TD Model: the TD model is a real-time model where $t$ labels time steps within or between trials; a trial is a sequence of states each represented by a feature vector, and the aggregate associative strength $\hat v(s,\mathbf{w}) = \mathbf{w}^\top\mathbf{x}(s)$ is a value estimate corresponding to the US prediction.
[^td-equiv]: **Sutton & Barto**, §14.2.3 — The TD Model: the update $\mathbf{w}_{t+1} = \mathbf{w}_t + \alpha\delta_t\mathbf{z}_t$ with TD error $\delta_t = R_t + \gamma\hat v(S_{t+1},\mathbf{w}_t) - \hat v(S_t,\mathbf{w}_t)$ and eligibility trace $\mathbf{z}_{t+1} = \gamma\lambda\mathbf{z}_t + \mathbf{x}(S_t)$ is equivalent to the backward view of semi-gradient TD($\lambda$) with linear function approximation, except $R_t$ need not be a reward signal; with $\gamma=0$ it reduces to Rescorla–Wagner.
[^normative]: **Sutton & Barto**, §14.2.4 / §14.3 preamble: the TD model is a normative account suggesting the nervous system is trying to form accurate long-term predictions; the temporal-difference term $\gamma\hat v(S_{t+1}) - \hat v(S_t)$ has the same status as the US, which is why bootstrapping yields second- and higher-order conditioning.
[^lms-gradient]: **Widrow & Hoff (1960)**, _Adaptive switching circuits_, IRE WESCON Convention Record — the Least-Mean-Square (LMS) rule adjusts weights by $\Delta\mathbf{w} = \alpha(\text{target} - \mathbf{w}^\top\mathbf{x})\mathbf{x}$, stochastic gradient descent on squared error. Rescorla–Wagner is the same rule with the presence vector as input and the US level $\lambda$ as target; the differences noted by **Sutton & Barto** (§14.2.2) are only that LMS typically uses real-valued inputs and a separate learning rate. Blocking is then the standard fact that a feature collinear with one already fit to the target receives zero weight.
[^overexpect]: **Sutton & Barto**, §14.2.2 — The Rescorla–Wagner Model: because the error uses the aggregate strength $\sum V$ of all components present, training two CSs separately to $\lambda$ and then presenting them in compound produces a summed prediction above $\lambda$, a negative error, and a decrease in both strengths ("overexpectation"). This follows from the summed-error term and is not predicted by contiguity-only accounts.
[^stim-reps]: **Sutton & Barto**, §14.2.4 — TD Model Simulations, and **Ludvig, Sutton & Kehoe (2012)**, _Learning & Behavior_ 40, "Evaluating the TD model of classical conditioning": the presence, complete serial compound (CSC), and microstimulus (MS) representations lie along a temporal generalization gradient — full generalization for presence, none for the CSC tapped-delay-line, intermediate for MS — and the degree of temporal generalization sets the granularity of the learned US prediction.
[^ms-realistic]: **Sutton & Barto**, §14.2.4, following **Ludvig, Sutton & Kehoe (2008/2012)**: microstimuli are internal signals launched by CS (and US) onset that grow wider and lower over time and overlap, giving partial temporal generalization; they are more realistic than presence or CSC as hypotheses about neural stimulus representations and let the TD model connect to a broader range of conditioning phenomena.
[^cr-timing]: **Sutton & Barto**, §14.2.4 — TD Model Simulations: with a fine-grained representation the learned US prediction $\hat v(S_t,\mathbf{w})$ rises across the CS–US interval and peaks at US time; in the rabbit nictitating-membrane preparation the anticipatory closure grows over the interval and reaches maximal closure at the expected US time, since covering the eye too early impairs vision and too late gives no protection.
[^isi]: **Sutton & Barto**, §14.2.4 — TD Model Simulations: conditioning strength typically depends on the inter-stimulus interval (CS-onset to US-onset) as an inverted-U — negligible at zero or negative ISI, maximal at a positive ISI, declining at longer intervals; the TD model reproduces this dependence, with the peak and width set by its discounting and eligibility-trace parameters.
[^primacy]: **Sutton & Barto**, §14.2.4, Fig. 14.2 (Temporal primacy overriding blocking): the TD model predicts that if the added CS begins _before_ the pretrained CS, blocking is reversed — the earlier CS gains strength and the later pretrained CS loses it — because bootstrapping shifts each state's prediction toward that of later states. Predicted by **Sutton & Barto (1981)** and confirmed in the rabbit nictitating-membrane preparation by **Kehoe, Schreurs & Graham (1987)**, who noted non-TD models have difficulty with the result.