---
title: "Cognitive Maps and Model-Based Learning"
module: Reinforcement Learning in Minds and Brains
moduleNumber: 6
lessonNumber: 6
order: 606
summary: >
  Tolman's rats learned the layout of a maze with no reward, then used it the moment
  food appeared — latent learning, a cognitive map, and the behavioral face of
  model-based reinforcement learning. The map is learned by system identification
  (stimulus–stimulus associations), which fills in whether or not reward is present,
  and queried by planning, which re-solves a route from a single changed reward. The
  successor representation sits between cache and model, and hippocampal predictive
  maps and scaled-up world models carry the same idea into brain and machine.
topics: [Minds and Brains]
sources:
  - book: Sutton & Barto
    ref: "Ch. 14 — Psychology; §14.5 Cognitive Maps; §14.6 Habitual and Goal-directed Behavior"
---

This builds on [Animal Learning and Cognition](/reinforcement-learning/minds-and-brains/animal-learning-and-cognition),
which treated three _associative_ phenomena — blocking, higher-order conditioning,
and delayed reinforcement — as reinforcement-learning mechanisms seen in behavior.
None of them needs the animal to hold a model of its world. This part takes up the
phenomenon that does: the cognitive map, learned without reward and queried by
planning, which is model-based reinforcement learning measured in a rat.

## Cognitive maps

Model-based reinforcement-learning algorithms use environment models that have
elements in common with what psychologists call **cognitive maps**. Recall from
[planning and learning](/reinforcement-learning/tabular-methods/planning-and-learning)
that by an environment model we mean anything an agent can use to predict how its
environment will respond to its actions — in terms of state transitions and rewards
— and by planning we mean any process that computes a policy from such a model. An
environment model has two parts: a **state-transition** part, encoding the effect
of actions on state changes, and a **reward-model** part, encoding the reward
signals expected for each state or state–action pair.[^cog-map-sb]

Whether animals use environment models, and if so what the models are like and how
they are learned, has driven a long argument in animal-learning research. Some
researchers challenged the then-prevailing **stimulus–response (S–R)** view of
learning — which corresponds to the simplest model-free way of learning policies —
by demonstrating **latent learning**.[^latent-sb]

### Latent learning

In the earliest latent-learning experiment, two groups of rats were run in a maze.
For the experimental group there was no reward during the first stage of the
experiment, but food was suddenly introduced into the goal box at the start of the
second stage. For the control group, food was in the goal box throughout both
stages. The question was whether the experimental rats would have learned anything
during the first, unrewarded stage.[^blodgett]

$$
% caption: Latent learning. The control group is rewarded throughout and its
% error count falls steadily. The experimental group is unrewarded until stage 2,
% shows no improvement while unrewarded, then drops sharply to the control level
% once food appears — revealing a map learned silently during the reward-free
% stage.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, black] (0,0) -- (7.4,0) node[anchor=west, black, font=\scriptsize] {days};
  \draw[->, black] (0,0) -- (0,3.4) node[anchor=south, black, font=\scriptsize] {errors in maze};
  % reward-introduction line for experimental group
  \draw[black, dotted] (4.4,0) -- (4.4,3.2);
  \node[anchor=south, font=\scriptsize, text=black] at (4.4,3.15) {food introduced};
  % control curve: steady decline
  \draw[acc, very thick] (0.2,3.0) .. controls (2.0,1.9) and (4.0,1.0) .. (7.0,0.55);
  \node[acc, anchor=west, font=\scriptsize] at (7.05,0.55) {control};
  % experimental curve: flat high, then sharp drop
  \draw[red, very thick] (0.2,3.05) .. controls (2.2,2.75) and (3.6,2.6) .. (4.4,2.55)
    .. controls (5.0,1.4) and (5.6,0.85) .. (7.0,0.6);
  \node[red, anchor=west, font=\scriptsize] at (7.05,0.75) {experimental};
\end{tikzpicture}
$$

Although the experimental rats did not _appear_ to learn much during the first,
unrewarded stage, as soon as they discovered the food in the second stage they
rapidly caught up with the control rats. The conclusion was that during the
non-reward period the experimental rats were developing a **latent learning** of the
maze, which they were able to use as soon as reward was introduced.[^blodgett-quote]
The rats had learned the maze's layout while it earned them nothing; the reward only
revealed a knowledge that was already there.

Latent learning is most closely associated with Edward Tolman, who interpreted
results like these as showing that animals learn a **cognitive map** of the
environment in the absence of rewards or penalties, and can use the map later when
motivated to reach a goal.[^tolman] A cognitive map could let a rat plan a route to
the goal that is different from the route it used in its initial exploration — the
signature of a model you can query for paths you never walked. In modern terms,
cognitive maps are not restricted to spatial layouts but are more generally
environment models, models of an animal's "task space." That a rat's spatial map
has a physical substrate is now familiar: the hippocampus contains place cells that
fire selectively when the animal is at a particular location, a neural correlate of
the map Tolman inferred from behavior alone. The cognitive-map explanation of latent
learning is the claim that **animals use model-based algorithms**, and that
environment models can be learned even without explicit rewards or penalties, then
used for planning once reward appears.[^cog-model-based]

> **Definition (Cognitive map / latent learning).** A cognitive map is an internal
> model of the environment — its transitions and, where relevant, its rewards —
> that an animal can build without reward and later use to plan. Latent learning is
> the acquisition of such a model in the absence of reinforcement, invisible in
> behavior until reward makes it worth acting on.

### How the map is learned: system identification

Tolman's account of _how_ animals learn cognitive maps is that they learn
**stimulus–stimulus (S–S) associations** by experiencing successions of stimuli as
they explore. In psychology this is called **expectancy theory**: given the S–S
associations, the occurrence of one stimulus generates an expectation about the
stimulus to come next.[^expectancy] This is much like what control engineers call
**system identification** — learning a model of a system with unknown dynamics from
labeled training examples. In the simplest discrete-time version the training
examples are $\text{S}\!-\!\text{S}'$ pairs, where $\text{S}$ is a state and
$\text{S}'$ the subsequent state, and the model learns to expect $\text{S}'$ after
$\text{S}$.[^sysid]

The S–S account is the exact contrast to the stimulus–response view it was raised
against, and the contrast is what latent learning turns on. An **S–R** learner caches,
for each state, the action that has paid off there: a lookup table from situation to
response, built only when reward stamps a response in. Take away reward and there is
nothing to stamp in, so an S–R learner should acquire nothing during an unrewarded
stage. An **S–S** learner instead records what follows what — state to next state — a
model of the world that reward never enters. It fills in during exploration whether or
not anything is paid, and stands ready to be queried once a goal is supplied. The
experimental rats behaved like S–S learners: silent during the unrewarded stage, then
immediately competent when food appeared, because the map was already built.

$$
% caption: Stimulus-response versus stimulus-stimulus learning. S-R caches, per state,
% the rewarded action - a policy table that only forms under reward. S-S records what
% state follows what - a transition model that forms during exploration with no reward.
% Latent learning is S-S: the map is built unrewarded, then queried once a goal appears.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  nd/.style={draw, minimum width=9mm, minimum height=7mm, font=\scriptsize},
  cap2/.style={font=\scriptsize, text=black, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % ==== S-R (left) ====
  \node[font=\small, anchor=south] at (1.5,2.5) {S-R (policy cache)};
  \node[nd, draw=red, text=red] (sr_s) at (0,1.4) {state};
  \node[nd] (sr_a) at (3.0,1.4) {action};
  \draw[->, red, thick] (sr_s) -- (sr_a) node[midway, above, font=\scriptsize] {cached};
  \node[cap2] at (1.5,0.5) {needs reward to form;\\nothing learned unrewarded};
  % ==== S-S (right) ====
  \begin{scope}[xshift=6.0cm]
    \node[font=\small, anchor=south] at (1.5,2.5) {S-S (world model)};
    \node[nd, draw=acc, text=acc] (ss_s) at (0,1.4) {state};
    \node[nd, draw=acc, text=acc] (ss_s2) at (3.0,1.4) {next state};
    \draw[->, acc, thick] (ss_s) -- (ss_s2) node[midway, above, font=\scriptsize] {predicts};
    \node[cap2] at (1.5,0.5) {forms during exploration;\\built with no reward};
  \end{scope}
\end{tikzpicture}
$$

Models useful for planning involve actions too, so examples become
$\text{SA}\!-\!\text{S}'$: state $\text{S}'$ is expected when action $\text{A}$ is
taken in state $\text{S}$. And to learn where the rewards are, examples take the form
$\text{S}\!-\!R$ or $\text{SA}\!-\!R$, with $R$ the reward signal associated with the
state or the state–action pair. These are all forms of **supervised learning**, by
which an agent can acquire cognitive-like maps whether or not it receives any
non-zero reward signal while exploring, and that is how the experimental rats
could build their map through a stage that paid them nothing.

$$
% caption: A cognitive map is learned by system identification: supervised
% examples of what follows what. State-transition examples S-S' or SA-S' give the
% dynamics; reward examples S-R or SA-R give the reward model. None of this
% requires reward to be present while learning, which is why latent learning works.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  ex/.style={draw, minimum width=20mm, minimum height=8mm, font=\scriptsize, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[ex, draw=acc, text=acc] (t1) at (0,1.0) {S, A  $\Rightarrow$  S'};
  \node[ex, draw=acc, text=acc] (t2) at (0,-0.2) {SA  $\Rightarrow$  R};
  \node[ex] (model) at (5.2,0.4) {environment\\model};
  \node[ex] (plan) at (5.2,-1.6) {planning\\(a policy)};
  \draw[->, acc, thick] (t1) -- (model.west) node[midway, above, font=\scriptsize, text=black] {transition};
  \draw[->, acc, thick] (t2) -- (model.west);
  \draw[->, thick] (model) -- (plan) node[midway, right, font=\scriptsize] {simulate paths};
  \node[anchor=north, font=\scriptsize, text=black, align=center] at (0,-1.0)
    {supervised examples\\(system identif\/ication)};
\end{tikzpicture}
$$

### Planning over the map: a worked maze

The payoff of holding a model rather than cached values is what happens when the
world changes. Consider a rat in a maze with two turns and four distinctive goal
boxes, each delivering a fixed reward. From the start state $S_1$ the rat chooses
left or right to reach $S_2$ or $S_3$, then chooses left or right again to land in
one of the four goal boxes.[^maze-fig]

A **model-free** rat solves this with stored action values, one number per
state–action pair, learned over many runs of the maze. Once those estimates are
good enough, it just picks the largest-valued action at each state. A **model-based**
rat instead learns a transition model (the branching structure of the maze) and a
reward model (which goal box pays what), and decides by simulating action sequences
to find the path with the highest return.

$$
% caption: A model-free rat caches an action value per state-action pair (left);
% a model-based rat holds a transition tree plus a reward model and simulates
% paths to the best goal box (right). With rewards 0, 4, 2, 3, both pick L then R
% from S1 for a return of 4.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  st/.style={circle, draw, minimum size=7mm, inner sep=0pt, font=\scriptsize},
  gb/.style={draw, minimum width=8mm, minimum height=6mm, font=\scriptsize},
  vc/.style={draw, minimum width=8mm, minimum height=6mm, font=\scriptsize},
  kc/.style={draw, minimum width=14mm, minimum height=6mm, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % ===== left: model-free cached values =====
  \node[font=\small, anchor=south] at (1.5,3.15) {model-free};
  \foreach \row/\lab/\val in {0/{S1,L}/4, 1/{S1,R}/3, 2/{S2,L}/0, 3/{S2,R}/4, 4/{S3,L}/2, 5/{S3,R}/3}{
    \node[kc, anchor=north west] (k\row) at (0, 2.7-\row*0.62) {\lab};
    \node[vc, draw=acc, text=acc, anchor=north west] at (1.45, 2.7-\row*0.62) {\val};
  }
  \node[anchor=north, font=\scriptsize, text=black, align=center] at (1.5,-1.15)
    {cached action values,\\pick the largest};
  % ===== right: model-based tree =====
  \begin{scope}[xshift=5.4cm]
    \node[font=\small, anchor=south] at (2.6,3.15) {model-based};
    \node[st] (s1) at (0,1.2) {S1};
    \node[st] (s2) at (2.2,2.4) {S2};
    \node[st] (s3) at (2.2,0.0) {S3};
    \node[gb] (g1) at (4.6,3.0) {=0};
    \node[gb] (g2) at (4.6,1.9) {=4};
    \node[gb] (g3) at (4.6,0.6) {=2};
    \node[gb] (g4) at (4.6,-0.5) {=3};
    \draw[->, acc] (s1) -- (s2) node[midway, above, font=\scriptsize] {L};
    \draw[->, acc] (s1) -- (s3) node[midway, below, font=\scriptsize] {R};
    \draw[->, red] (s2) -- (g1) node[near end, above, font=\scriptsize] {L};
    \draw[->, red] (s2) -- (g2) node[near end, below, font=\scriptsize] {R};
    \draw[->, red] (s3) -- (g3) node[near end, above, font=\scriptsize] {L};
    \draw[->, red] (s3) -- (g4) node[near end, below, font=\scriptsize] {R};
    \node[anchor=north, font=\scriptsize, text=black, align=center] at (2.3,-1.15)
      {transitions + reward model,\\simulate to best goal};
  \end{scope}
\end{tikzpicture}
$$

With rewards $0, 4, 2, 3$ across the four goal boxes, both strategies converge on
the same choice: go left to $S_2$, then right, for a return of $4$. The difference
shows only under change. Suppose the rat is placed directly in the goal box right of
$S_2$ and finds the reward there is now $1$ instead of $4$, without ever running the
maze. A model-based rat updates its reward model from that single experience;
planning then brings the new value to bear on maze-running with no further
experience in the maze, changing the policy to right turns at both $S_1$ and $S_3$
for a return of $3$. A model-free rat cannot do this. To change the action its
policy specifies at a state, or the value it has cached for a state, it has to move
_to_ that state, act from it, and experience the consequences — possibly many
times.[^devaluation-tie] This is the mechanism behind the outcome-devaluation
result from the [previous
lesson](/reinforcement-learning/minds-and-brains/psychology-of-reinforcement): the
goal-directed rat re-plans over its map the instant the goal's value changes, while
the habitual rat must relearn by acting.

Trace the replan explicitly. Before the change, the model-based rat backs values up the
tree by taking the max at each choice. The right goal box of $S_2$ pays $4$, so
$S_2$'s value is $\max(0, 4) = 4$; the goal boxes of $S_3$ pay $2$ and $3$, so $S_3$'s
value is $\max(2, 3) = 3$. From $S_1$ the rat compares going left ($4$) with going right
($3$) and turns left, then right, for a return of $4$. Now devalue: the rat is dropped
into the right box of $S_2$ and finds it pays $1$, not $4$, and it updates that one
reward-model entry. Nothing else in the maze was touched, but the backup changes
wholesale. $S_2$ is now worth $\max(0, 1) = 1$; $S_3$ is untouched at $3$; from $S_1$
the comparison is now left ($1$) versus right ($3$), so the plan flips to right, then
right, for a return of $3$.

| Value | before | after devaluation |
| :-- | ---: | ---: |
| right box of $S_2$ (reward model) | 4 | 1 |
| $v(S_2) = \max(0, \cdot)$ | 4 | 1 |
| $v(S_3) = \max(2, 3)$ | 3 | 3 |
| choice at $S_1$ | L (4 vs 3) | **R** (1 vs 3) |
| return | 4 | 3 |

The model-based rat did no maze-running to produce the new plan; it re-solved the tree
from one changed number. The model-free rat, holding only the cached action values
$\langle S_1,L\rangle = 4$, $\langle S_1,R\rangle = 3$, and so on, has no such move. The
value it cached for $\langle S_1,L\rangle$ was learned from full runs ending at the box
that used to pay $4$; a single visit to that box while _not_ running the maze from $S_1$
gives it no update to $\langle S_1, L\rangle$ at all. To fix the stale cache it must
physically go left from $S_1$, walk to the now-devalued box, take the disappointing
reward, and let the error propagate back — and repeat until $\langle S_1,L\rangle$ falls
below $\langle S_1,R\rangle$. Same maze, same single fact learned; the map-holder acts on
it at once, the cache-holder cannot act on it until it re-runs.

$$
% caption: The devaluation replan. Left: before, both agents pick L then R for return 4.
% Right: the model-based rat learns the S2-right box now pays 1 (not 4) from one direct
% visit, re-backs the tree - S2 becomes max(0,1)=1, S3 stays 3 - and flips to R then R
% for return 3. The changed numbers and the new best path are highlighted in red.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  st/.style={circle, draw, minimum size=7mm, inner sep=0pt, font=\scriptsize},
  gb/.style={draw, minimum width=8mm, minimum height=6mm, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[font=\small, anchor=south] at (2.3,3.3) {after devaluation: re-plan from one changed reward};
  \node[st] (s1) at (0,1.1) {S1};
  \node[st] (s2) at (2.4,2.2) {S2};
  \node[st] (s3) at (2.4,0.0) {S3};
  \node[gb] (g1) at (4.9,3.0) {=0};
  \node[gb, draw=red, text=red] (g2) at (4.9,1.7) {=1};
  \node[gb] (g3) at (4.9,0.6) {=2};
  \node[gb, draw=red, text=red] (g4) at (4.9,-0.6) {=3};
  % chosen path highlighted
  \draw[->, red, very thick] (s1) -- (s3) node[midway, below, font=\scriptsize] {R};
  \draw[->, black] (s1) -- (s2) node[midway, above, font=\scriptsize] {L};
  \draw[->, black] (s2) -- (g1) node[near end, above, font=\scriptsize] {L};
  \draw[->, red, thick] (s2) -- (g2) node[near end, below, font=\scriptsize] {R};
  \draw[->, black] (s3) -- (g3) node[near end, above, font=\scriptsize] {L};
  \draw[->, red, very thick] (s3) -- (g4) node[near end, below, font=\scriptsize] {R};
  % value labels
  \node[anchor=east, font=\scriptsize, text=black] at (2.15,2.55) {v=1};
  \node[anchor=east, font=\scriptsize, text=black] at (2.15,-0.35) {v=3};
  \node[anchor=north, font=\scriptsize, text=black, align=center] at (2.3,-1.1)
    {new plan: R then R, return 3};
\end{tikzpicture}
$$

The connection runs forward as well as back. The same model-based machinery — a
learned transition model, a learned reward model, and planning by simulated
rollouts — is what
[modern model-based reinforcement learning](/reinforcement-learning/modern-deep-rl/model-based-rl)
scales up with function approximation, learning the model with neural networks and
planning with search or imagined trajectories. Tolman's rat, silently mapping a maze
it was not paid to explore, was doing in tissue what a model-based agent does in
silicon.

## The successor representation and evidence for maps

Sutton and Barto leave the model-free/model-based split as a clean dichotomy — cache versus
tree, habit versus plan. Work since has filled in the middle and pinned the split to
brain and behavior. Three lines are worth stating carefully, each for exactly what its
authors showed.

**A predictive map between the two systems.** Dayan (1993, _Neural Computation_)
introduced the **successor representation**: instead of caching a scalar value per
state, or holding a full one-step transition model to plan over, cache for each state a
vector of _expected future occupancy_ — how often, discounted, each other state will be
visited if you follow the current policy from here. Value is then a dot product of this
predictive map with the reward vector. The successor representation sits between
model-free and model-based control. Like a cached value it is fast to read, needing no
search; like a model it factors the world's dynamics apart from reward, so a change in
reward re-values states without re-running the policy — though a change in the
_transitions_ still requires relearning the map, which a full model would not. It is a
partial cognitive map: predictive structure without a plannable model.

Stachenfeld, Botvinick and Gershman (2017, _Nature Neuroscience_) argued that the
hippocampus encodes a map of this kind. They showed that place-cell firing fields, and
the periodic fields of entorhinal grid cells, are well described as components of a
successor representation of the animal's environment — place fields skew and stretch
along frequently-taken routes, and grid patterns emerge as a low-dimensional
decomposition of the predictive map. This gives Tolman's cognitive map, inferred from
behavior in the 1940s, a concrete computational form and a candidate neural substrate:
not a static layout but a _predictive_ map of where the animal is headed.

$$
% caption: The successor representation as a middle ground. Model-free caches a scalar
% value per state (fast, but reward and dynamics are fused). The SR caches expected
% future occupancy (a predictive map, re-valued instantly when reward changes).
% Model-based holds the transition model and plans (flexible, but must search).
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  bx/.style={draw, minimum width=26mm, minimum height=13mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[bx] (mf) at (0,0) {model-free\\value per state};
  \node[bx, draw=acc, text=acc, thick] (sr) at (4.4,0) {successor rep.\\future occupancy};
  \node[bx] (mb) at (8.8,0) {model-based\\transition model};
  \draw[->, black] (mf.east) -- (sr.west);
  \draw[->, black] (sr.east) -- (mb.west);
  \node[font=\scriptsize, text=black, anchor=north] at (0,-0.95) {fast, in\/f\/lexible};
  \node[font=\scriptsize, text=acc, anchor=north] at (4.4,-0.95) {predictive, re-values on reward change};
  \node[font=\scriptsize, text=black, anchor=north] at (8.8,-0.95) {f\/lexible, must plan};
\end{tikzpicture}
$$

**Both systems in one human choice.** Daw, Gershman, Seymour, Dayan and Dolan (2011,
_Neuron_) built a **two-step Markov task** that separates the two controllers in a
single stream of choices. A first-stage choice leads probabilistically to one of two
second-stage states, and reward there drifts over time. A pure model-free learner
repeats whatever first-stage action preceded reward, ignoring _how_ the transition went;
a model-based learner uses its knowledge of the transition structure, so after a rare
transition it does the opposite. Human choices showed a _mixture_ of the two signatures,
and the balance shifted with circumstance — tilting model-free under cognitive load or
time pressure. This is the behavioral fingerprint of the habit/goal-directed distinction
this lesson maps onto model-free and model-based control, measured in people rather than
inferred. Gläscher, Daw, Dayan and O'Doherty (2010, _Neuron_) added the learning-signal
half: alongside the reward-prediction error that trains values, they found a distinct
**state-prediction error** — the surprise when a transition differs from what the model
expected — with its own neural correlate, the error signal that would train the model
itself rather than its cached values.

**Scaled-up maps in machines.** The engineering forward-pointer is real systems that
learn a world model and plan inside it. Ha and Schmidhuber (2018, "World Models") trained
an agent to compress its environment into a learned latent dynamics model and then
learn a policy largely by "dreaming" — rolling the model forward instead of acting. Schrittwieser
et al. (2020, _Nature_), in MuZero, learned a model whose only job is to support
value-accurate planning, and combined it with Monte-Carlo tree search to reach top play
in Go, chess, shogi, and Atari without being told the rules. These are Tolman's map plus
plan built at scale: a learned model of consequences, queried by simulated rollouts, to
choose actions in situations never directly experienced. The rat mapping a maze it was
not paid to explore and the agent planning over a learned latent model are running the
same idea at very different sizes.

## The one claim

Three phenomena, one point. Blocking is prediction error: an animal learns about a
stimulus only to the extent that its outcome is surprising, which is the error term
in every update in this course. Higher-order conditioning and conditioned
reinforcement are bootstrapping: a learned prediction becomes a reward in its own
right, which is what a value function is and what an actor–critic critic supplies.
Delayed reinforcement is credit assignment, and the stimulus traces and lengthened
goal gradients that Pavlov and Hull proposed for it are eligibility traces and
TD-learned values. Cognitive maps are environment models, learned by system
identification with or without reward, and queried by planning — model-based
reinforcement learning, measured in a rat.

None of this asserts that the brain runs these exact equations. It asserts something
narrower and more durable: the structure of the learning problem is the same whether
the learner is tissue or silicon, so a theory built to solve it computationally keeps
landing on mechanisms that animal-learning researchers had already isolated in
behavior. The [next module thread](/reinforcement-learning/minds-and-brains/dopamine-and-td-error)
follows the contact point one level deeper, into the neuron, where the dopamine
signal turns out to behave like the TD error that all of this has been circling.

[^cog-map-sb]: **Sutton & Barto**, §14.5 — Cognitive Maps: model-based reinforcement learning algorithms use environment models with elements in common with psychologists' cognitive maps; an environment model has a state-transition part (effect of actions on state changes) and a reward-model part (reward signals expected for each state or state–action pair), and planning computes a policy by simulating imagined sequences of decisions.
[^latent-sb]: **Sutton & Barto**, §14.5: some researchers challenged the prevailing stimulus–response (S–R) view — corresponding to the simplest model-free way of learning policies — by demonstrating latent learning.
[^blodgett]: **Sutton & Barto**, §14.5: in the earliest latent-learning experiment, two groups of rats ran a maze; the experimental group had no reward in stage 1 with food suddenly introduced into the goal box at the start of stage 2, while the control group had food throughout; the question was whether the experimental group learned anything during the unrewarded first stage.
[^blodgett-quote]: **Sutton & Barto**, §14.5: although the experimental rats did not appear to learn much during the unrewarded first stage, once they discovered the stage-2 food they rapidly caught up to the control rats; Blodgett (1929) concluded that during the non-reward period the rats were developing a latent learning of the maze they could utilize as soon as reward was introduced.
[^tolman]: **Sutton & Barto**, §14.5: latent learning is most closely associated with Edward Tolman (1948), who interpreted such results as animals learning a "cognitive map" of the environment in the absence of rewards or penalties, usable later when motivated to reach a goal; a cognitive map could let a rat plan a route different from the one used in initial exploration.
[^cog-model-based]: **Sutton & Barto**, §14.5: in modern terms cognitive maps are not restricted to spatial layouts but are more generally environment models, models of an animal's "task space" (e.g. Wilson, Takahashi, Schoenbaum, and Niv, 2014); the cognitive-map explanation of latent learning is the claim that animals use model-based algorithms and that environment models can be learned even without explicit rewards or penalties, then used for planning when reward appears.
[^expectancy]: **Sutton & Barto**, §14.5: Tolman's account is that animals learn stimulus–stimulus (S–S) associations by experiencing successions of stimuli while exploring; in psychology this is expectancy theory — given S–S associations, the occurrence of a stimulus generates an expectation about the stimulus to come next.
[^sysid]: **Sutton & Barto**, §14.5: learning S–S associations is much like what control engineers call system identification — learning a model of a system with unknown dynamics from labeled examples; the simplest discrete-time examples are S–S′ pairs, action-inclusive examples are SA–S′, and reward examples are S–R or SA–R, all forms of supervised learning by which an agent can acquire cognitive-like maps whether or not it receives non-zero reward while exploring.
[^maze-fig]: **Sutton & Barto**, §14.6 / Figure 14.5: a rat navigates a maze with distinctive goal boxes each delivering an associated reward; from $S_1$ it selects L or R to reach $S_2$ or $S_3$, then L or R again to reach a goal box. A model-free strategy relies on stored action values for state–action pairs; a model-based strategy learns a state-transition model (a decision tree) and a reward model associating goal-box features with rewards, deciding by simulating action sequences to find the highest-return path.
[^devaluation-tie]: **Sutton & Barto**, §14.6: with rewards 0, 4, 2, 3 both strategies select L then R from $S_1$ for return 4; a model-based agent placed directly in a goal box whose reward changed (e.g. 4 to 1) updates its reward model and re-plans without maze experience, whereas a model-free agent must move to that state and act, possibly many times, to update — the logic underlying outcome-devaluation experiments.