--- title: "The Psychology of Reinforcement: Instrumental Control" module: Reinforcement Learning in Minds and Brains moduleNumber: 6 lessonNumber: 2 order: 602 summary: > Classical conditioning was prediction; instrumental conditioning is control. Thorndike's Law of Effect is trial-and-error control — selection plus association, search plus memory — and Skinner's shaping and schedules are reward engineering. The habitual/goal-directed distinction maps onto model-free versus model-based control, dissociated by outcome devaluation and arbitrated by uncertainty. Delayed reinforcement is the credit-assignment problem, and the stimulus traces and secondary reinforcers of animal-learning theory are eligibility traces and value functions. topics: [Minds and Brains] sources: - book: Sutton & Barto ref: "Ch. 14 — Psychology; §14.3 Instrumental Conditioning; §14.4 Delayed Reinforcement" - book: Sutton & Barto ref: "§14.6 Habitual and Goal-directed Behavior" --- This builds on [The Psychology of Reinforcement](/reinforcement-learning/minds-and-brains/psychology-of-reinforcement), which followed the _prediction_ half of the correspondence: classical conditioning, the Rescorla–Wagner rule, and its real-time TD extension. This part takes up the other half — _control_. Where classical conditioning taught the animal what to expect, instrumental conditioning teaches it what to _do_, and that maps onto the control algorithms of this course: the Law of Effect is trial-and-error control, and the habitual/goal-directed distinction is the model-free/model-based split. ## Instrumental conditioning and the Law of Effect Instrumental conditioning is control. The reinforcer is contingent on behavior, and the animal learns to produce behavior that pays. Its founding observation is Thorndike's: cats placed in "puzzle boxes," able to escape by the right sequence of actions, escaped faster and faster over repeated trials. Successful actions were "stamped in," unsuccessful ones "stamped out."[^thorndike] From this Thorndike drew the **Law of Effect**. > **Definition (Law of Effect).** Actions followed by satisfaction become more > likely in the situation that preceded them; actions followed by discomfort > become less likely. Learning is trial and error: try alternatives, keep what > works, connected to the situation in which it worked. The Law of Effect is a compact description of a reinforcement learning algorithm. Two features line up exactly, and each distinguishes it from a neighboring learning paradigm:[^loe-features] | Feature | Law of Effect | Contrast | | --- | --- | --- | | **Selectional** | tries alternatives, selects by consequence | supervised learning is told the correct answer | | **Associative** | binds selected actions to the states where they worked | natural selection selects but does not associate | Selection plus association is **search** plus **memory**: search by trying actions, memory by linking situations to the actions that worked. Both are essential to every control algorithm here. Search requires _exploration_ — the animal must try alternatives — which is why the control methods carry $\varepsilon$-greedy or upper-confidence-bound action selection.[^exploration] Thorndike's cats selected from their _instinctual impulses_, not every conceivable action: the animal-learning image of an admissible action set $\mathcal{A}(s)$. Skinner extended this line with **shaping**: reinforcing successive approximations of a target behavior. Waiting for the full behavior to occur by chance can take forever; instead you reward anything resembling it, then progressively raise the bar. Shaping is directly useful for training reinforcement learning agents when reward is too sparse for the agent ever to stumble onto it — start with an easier version of the problem and increase difficulty as the agent improves.[^shaping] Skinner also formalized the _rule_ by which the apparatus delivers reinforcement, the **reinforcement schedule** — part of the environment's reward function, fixing the mapping from behavior to reward. The animal's steady-state response rate is a policy adapted to it.[^schedules] | Schedule | Reinforces | Variants | | --- | --- | --- | | **Continuous** | every target response | — | | **Partial — ratio** | after a count of responses | fixed / variable | | **Partial — interval** | first response after an elapsed time | fixed / variable | Partial reinforcement is an analogue of the stochastic-reward setting the control methods already face: a variable-ratio schedule delivers reward intermittently and unpredictably, so the learner acts under high return variance — the condition that makes exploration and value averaging matter. Behavior trained under partial reinforcement also extinguishes more slowly, since intermittent reward looks, locally, like the training regime. ## Habitual and goal-directed behavior The last correspondence is the closest. Psychologists distinguish two modes of controlling learned behavior, and they map cleanly onto the model-free / model-based split from [planning and learning](/reinforcement-learning/tabular-methods/planning-and-learning).[^hab-gd] | | **Habitual** | **Goal-directed** | | --- | --- | --- | | RL analogue | model-free | model-based | | Control by | antecedent stimulus | consequence / outcome value | | Reads | cached $q(s,a)$ or policy | learned model $p(s'\mid s,a)$, $r(s,a)$ | | Decision | look up value | simulate action sequences, pick best return | | Execution | fast | slow | | Adapts to a payoff shift | slowly, only by re-experience | immediately, on model update | **Habitual** behavior is triggered by antecedent stimuli and runs off automatically: decisions read off cached state-action values built up over many trials. **Goal-directed** behavior consults an internal model, simulates the consequences of candidate action sequences, and chooses the best outcome — slower, but able to change the moment the model's knowledge of the goal changes, no re-experience required. $$ % caption: The psychologist's habitual/goal-directed distinction maps onto the % model-free/model-based split. Model-free control reads a cached action value % v(state, action); model-based control simulates action sequences over a learned % transition-and-reward model to find the best-return path. \begin{tikzpicture}[>=stealth, font=\footnotesize, node distance=4mm, st/.style={circle, draw, minimum size=6.5mm, inner sep=0pt, font=\scriptsize}, gb/.style={draw, minimum width=8mm, minimum height=6mm, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % ===== left: model-free (habitual) ===== \node[font=\small, anchor=south] at (1.9,3.15) {habitual / model-free}; % cached action-value table \foreach \row/\lab/\val in {0/{S1,L}/4, 1/{S1,R}/3, 2/{S2,L}/2, 3/{S2,R}/4}{ \node[draw, minimum width=15mm, minimum height=6mm, anchor=north west, font=\scriptsize] (c\row) at (0, 2.6-\row*0.72) {\lab}; \node[draw, acc, minimum width=7mm, minimum height=6mm, anchor=north west, text=acc, font=\scriptsize] at (1.5, 2.6-\row*0.72) {\val}; } \node[anchor=north, font=\scriptsize, text=black, align=center] at (1.9,-0.55) {cached values\\read o\/f\/f, no model}; % ===== right: model-based (goal-directed) ===== \begin{scope}[xshift=6.2cm] \node[font=\small, anchor=south] at (2.0,3.15) {goal-directed / model-based}; \node[st] (s1) at (0,1.1) {S1}; \node[st] (s2) at (2.0,2.1) {S2}; \node[st] (s3) at (2.0,0.1) {S3}; \node[gb] (g1) at (4.0,2.6) {=0}; \node[gb] (g2) at (4.0,1.7) {=4}; \node[gb] (g3) at (4.0,0.6) {=2}; \node[gb] (g4) at (4.0,-0.3) {=3}; \draw[->, acc] (s1) -- (s2) node[midway, above, font=\scriptsize] {L}; \draw[->, acc] (s1) -- (s3) node[midway, below, font=\scriptsize] {R}; \draw[->, red] (s2) -- (g1) node[midway, above, font=\scriptsize] {L}; \draw[->, red] (s2) -- (g2) node[midway, below, font=\scriptsize] {R}; \draw[->, red] (s3) -- (g3) node[midway, above, font=\scriptsize] {L}; \draw[->, red] (s3) -- (g4) node[midway, below, font=\scriptsize] {R}; \node[anchor=north, font=\scriptsize, text=black, align=center] at (2.0,-0.85) {simulate paths,\\pick best return}; \end{scope} \end{tikzpicture} $$ The experimental test that separates the two is **outcome devaluation**. Train a rat to press a lever for food pellets. Then _devalue_ the pellets — pair them with a nausea-inducing injection so the rat no longer wants them — without the rat ever pressing the lever while sick. Now put the lever back and see whether the rat still presses.[^devaluation] A goal-directed (model-based) rat presses _less right from the start_: it knows lever pressing leads to pellets, and it now knows pellets are bad, so it combines the two and stops — without ever having experienced lever-pressing-then-sickness. A purely habitual (model-free) rat keeps pressing, because its cached lever-press value was never updated by direct experience. The logic hangs on the fact that devaluation happens in a context _separate_ from lever pressing. Reading it as a timeline makes the dissociation clear: only the model-based agent has a stored link from lever to pellet that it can re-evaluate when the pellet's value changes; the model-free agent has only a cached number on the lever action, and that number was set before devaluation and never revisited. $$ % caption: The outcome-devaluation timeline. Train: lever produces pellets, valued % (+). Devalue: pellets paired with illness while the lever is absent, so the % pellet becomes aversive (-), but the lever action is never taken. Test: the % model-based (goal-directed) agent re-chains lever to now-bad pellet and stops; % the model-free (habitual) agent presses on, its cached lever value untouched. \begin{tikzpicture}[>=stealth, font=\footnotesize, ph/.style={draw, minimum width=30mm, minimum height=15mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[ph] (train) at (0,0) {TRAIN\\lever produces pellet\\pellet valued (+)}; \node[ph] (dev) at (4,0) {DEVALUE\\pellet plus illness\\pellet aversive (-)\\lever not pressed}; \node[ph, draw=acc, thick] (test) at (8,0) {TEST\\lever available}; \draw[->, black, thick] (train) -- (dev); \draw[->, black, thick] (dev) -- (test); \node[anchor=west, text=acc, font=\scriptsize, align=left] at (10.0,0.45) {model-based:\\stops pressing}; \node[anchor=west, text=red, font=\scriptsize, align=left] at (10.0,-0.55) {model-free:\\keeps pressing}; \end{tikzpicture} $$ _Both_ systems are present, and behavior shifts between them. Adams showed that _extended_ training converts goal-directed behavior into habit: modestly trained rats stayed sensitive to devaluation, overtrained rats kept pressing.[^overtraining] Daw, Niv, and Dayan model the animal as running both processes and deferring to whichever is more _trustworthy_, measured by the uncertainty each maintains — control follows $\arg\min$ over the two posterior uncertainties. Early on the model-based process is more reliable, chaining short-term predictions that sharpen quickly; with experience the model-free values become well-estimated and cheaper to consult, and control shifts toward habit. This is arbitration between model-free and model-based control, expressed in the animal.[^arbitration] $$ % caption: Uncertainty-based arbitration (Daw, Niv, Dayan 2005). A model-free and % a model-based valuation run in parallel; each reports an action value with a % confidence (inverse uncertainty), and control follows whichever is more % confident. Early training favors the model-based system; extended experience % sharpens cached values and shifts control toward the habitual system. \begin{tikzpicture}[>=stealth, font=\footnotesize, sys/.style={draw, minimum width=34mm, minimum height=13mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[sys, draw=acc, text=acc, thick] (mb) at (0,1.2) {model-based\\value plus con\/f\/idence}; \node[sys, draw=red, text=red, thick] (mf) at (0,-1.2) {model-free\\value plus con\/f\/idence}; \node[draw, minimum width=24mm, minimum height=13mm, align=center, font=\scriptsize] (arb) at (5.0,0) {arbitrate:\\pick the more\\con\/f\/ident system}; \node[draw, minimum width=18mm, minimum height=10mm, align=center, font=\scriptsize] (act) at (9.2,0) {action}; \draw[->, acc, thick] (mb) -- (arb); \draw[->, red, thick] (mf) -- (arb); \draw[->, black, thick] (arb) -- (act); \end{tikzpicture} $$ ## Delayed reinforcement One thread runs through both paradigms: reinforcement is often _delayed_. In instrumental conditioning the reward can arrive well after the action that earned it; in classical conditioning the US can follow CS offset by a gap. This is the **credit-assignment problem** — how to distribute credit for an outcome among the earlier decisions that led to it.[^delayed] Animal-learning theory proposed two mechanisms for bridging the gap, each with an exact counterpart here:[^two-mechanisms] | Animal-learning mechanism | RL counterpart | Role | | --- | --- | --- | | **Stimulus trace** (Pavlov, Hull's goal gradient) | eligibility trace $\mathbf{z}_t$ in $\text{TD}(\lambda)$ | decaying record of recent state visitations | | **Conditioned (secondary) reinforcer** | value function $\hat{v}(s)$ | moment-by-moment stand-in for delayed reward | A stimulus leaves a decaying trace in the nervous system, and learning occurs while the CS trace still overlaps the US — the ancestor of the eligibility trace. A stimulus that reliably predicts reward becomes reinforcing itself, supplying _immediate_ reinforcement when the primary reward is delayed — the role of a value function. In an actor–critic architecture the critic's TD error $\delta_t = R_{t+1} + \gamma\hat{v}(S_{t+1}) - \hat{v}(S_t)$ acts as exactly such a conditioned reinforcer for the actor, giving immediate feedback even when primary reward is far off. The two mechanisms are the two answers this course gives to delayed reinforcement, and animal-learning theory arrived at both first: the eligibility trace was proposed for rats before it was proposed for algorithms, and the value function is the formal version of a secondary reinforcer. ## From conditioning models to deep RL and meta-learning The line from Rescorla–Wagner to the algorithms of this course does not stop at the classical-conditioning models; it runs directly into modern deep reinforcement learning, and it loops back into experiments on human choice. The behavioral ancestor of the training signal in deep RL is the prediction error these models formalized. In the Deep Q-Network of Mnih and colleagues (2015, _Nature_), which learned to play Atari games from pixels, the quantity that drives every weight update is the temporal-difference error $R_t + \gamma\max_a Q(S_{t+1},a) - Q(S_t,a)$ — the same $\delta_t$ that the TD model of conditioning uses, now applied to a deep network's value estimate. Actor–critic deep methods carry the same TD error as the critic's teaching signal for the actor. The Rescorla–Wagner rule and its real-time TD extension are the behavioral lineage of that signal, not a loose metaphor for it.[^bb-dqn] The habit/goal-directed distinction this lesson draws has been measured directly in people. Daw, Gershman, Seymour, Dayan, and Dolan (2011, _Neuron_) introduced the **two-step task**, a sequential choice in which model-free and model-based control predict different patterns of how a reward changes the next choice. Human choices showed a _mixture_ of the two strategies rather than either alone — the empirical counterpart of the two-system picture above. Earlier, Gläscher, Daw, Dayan, and O'Doherty (2010, _Neuron_) used fMRI to dissociate two error signals during learning: a **state-prediction error** that tracks the learning of the transition model (the model-based component) and a separate reward-prediction error (the model-free component), each with its own neural correlate. The habit/goal-directed contrast is not only a behavioral description; it corresponds to distinct, measurable learning signals.[^bb-twostep] The two modes have also been shown to _emerge_ from one another. Wang and colleagues (2018, _Nature Neuroscience_), in "Prefrontal cortex as a meta-reinforcement learning system," trained a recurrent network with a plain model-free RL algorithm across a distribution of related tasks. The slow model-free training shaped the network's recurrent dynamics into a faster inner learning loop that behaved in a model-based-like way on each new task — model-based behavior arising from model-free learning, a meta-reinforcement-learning account of how a flexible controller can be built from an inflexible learning rule. Botvinick and colleagues (2020, _Nature Reviews Neuroscience_) survey the broader set of contact points between deep reinforcement learning and neuroscience that this lineage opened up.[^bb-metarl] ## Why this matters Reinforcement learning was designed for engineering, and it remains a theory of _effective_ learning algorithms, not a catalogue of animal behavior. But the correspondences are not superficial. The prediction/control split is the classical/instrumental split. The Rescorla–Wagner prediction error is the TD error with the time removed. The Law of Effect is trial-and-error control. Habit is model-free, deliberation is model-based, and the shift between them is arbitration. Eligibility traces are stimulus traces, and value functions are secondary reinforcers. None of this means the brain runs our exact equations. It means the abstract structure of the learning problem is the same whether the learner is silicon or tissue, and that the theory built to solve it computationally keeps rediscovering the mechanisms that evolution found. The next lesson follows this contact point into the brain itself, where the [dopamine signal](/reinforcement-learning/minds-and-brains/dopamine-and-td-error) turns out to behave like a TD error. [^thorndike]: **Sutton & Barto**, §14.3 — Instrumental Conditioning: Thorndike's cats escaped puzzle boxes faster over repeated trials; successful actions were "stamped in" and unsuccessful ones "stamped out," leading to the Law of Effect, the classic statement of learning by trial and error. [^loe-features]: **Sutton & Barto**, §14.3 — Instrumental Conditioning: reinforcement learning algorithms correspond to the Law of Effect in being selectional (trying alternatives and selecting by consequence) and associative (connecting selected actions to the states in which they worked), unlike supervised learning (associative not selectional) or natural selection (selectional not associative). [^exploration]: **Sutton & Barto**, §14.3 — Instrumental Conditioning: a reinforcement learning algorithm's need to search means it must explore; the $\varepsilon$-greedy and upper-confidence-bound methods used here are among the simplest such schemes, and Thorndike's cats selected from their instinctual impulses, echoing an admissible action set $\mathcal{A}(s)$. [^shaping]: **Sutton & Barto**, §14.3 — Instrumental Conditioning: Skinner's shaping reinforces successive approximations of a target behavior; it is an effective, sometimes indispensable, tool for training reinforcement learning agents when reward is too sparse to be encountered by initial behavior. [^schedules]: **Sutton & Barto**, §14.3 — Instrumental Conditioning: Skinner studied reinforcement schedules (continuous vs. partial, the latter split into ratio and interval, fixed or variable) and their effect on steady-state response rate. The schedule is part of the environment's reward contingency; a variable-ratio schedule delivers reward intermittently and unpredictably, a stochastic-reward setting analogous to the high-variance returns that make exploration and value averaging matter in this course's control methods. [^hab-gd]: **Sutton & Barto**, §14.6 — Habitual and Goal-directed Behavior: the model-free/model-based distinction corresponds to the psychologists' habitual/goal-directed distinction; habits are controlled by antecedent stimuli, goal-directed behavior by knowledge of the value of goals and the action–outcome relationship. [^devaluation]: **Sutton & Barto**, §14.6 — Habitual and Goal-directed Behavior: in the Adams and Dickinson (1981) outcome-devaluation experiment, rats trained to lever-press for pellets, then made to find the pellets aversive (via lithium chloride) without pressing while sick, pressed less from the start of extinction — evidence of a cognitive map linking lever-pressing to pellets to nausea. [^overtraining]: **Sutton & Barto**, §14.6 — Habitual and Goal-directed Behavior: Adams (1982) showed extended (overtraining) reduces sensitivity to outcome devaluation — overtrained rats kept lever-pressing after devaluation while modestly-trained rats did not — evidence that extended training converts goal-directed behavior into a habit. [^arbitration]: **Sutton & Barto**, §14.6 — Habitual and Goal-directed Behavior: Daw, Niv, and Dayan (2005) proposed animals use both model-free and model-based processes, executing the action from whichever is judged more trustworthy by maintained confidence measures, predicting a shift from goal-directed to habitual control as experience accumulates. [^delayed]: **Sutton & Barto**, §14.4 — Delayed Reinforcement: learning can occur with a long delay between action and consequent reward or between CS offset and US onset; this is the delayed-reinforcement problem, related to Minsky's (1961) credit-assignment problem of distributing credit among many decisions. [^two-mechanisms]: **Sutton & Barto**, §14.4 — Delayed Reinforcement: their algorithms address delayed reinforcement with eligibility traces (corresponding to the stimulus traces of Pavlov and Hull) and with TD-learned value functions (corresponding to conditioned/secondary reinforcement), the actor–critic critic's TD error acting as a conditioned reinforcer that gives immediate feedback when primary reward is delayed. [^bb-dqn]: **Mnih et al. (2015)**, _Nature_ 518, "Human-level control through deep reinforcement learning": the Deep Q-Network learns Atari games from pixels, trained by minimizing the temporal-difference error $R_t + \gamma\max_a Q(S_{t+1},a;\theta^-) - Q(S_t,a;\theta)$ — the same TD error the TD model of conditioning uses, applied to a deep value network. Actor–critic deep methods likewise use the critic's TD error as the actor's teaching signal. [^bb-twostep]: **Daw, Gershman, Seymour, Dayan & Dolan (2011)**, _Neuron_ 69, "Model-based influences on humans' choices and striatal prediction errors": the two-step task shows human choices reflect a mixture of model-free and model-based control. **Gläscher, Daw, Dayan & O'Doherty (2010)**, _Neuron_ 66, dissociated a model-based state-prediction error from the model-free reward-prediction error with fMRI. [^bb-metarl]: **Wang et al. (2018)**, _Nature Neuroscience_ 21, "Prefrontal cortex as a meta-reinforcement learning system": a recurrent network trained by model-free RL across related tasks develops a faster, model-based-like inner learning loop in its recurrent dynamics. **Botvinick et al. (2020)**, _Nature Reviews Neuroscience_ 21, "Deep reinforcement learning and its neuroscientific implications," survey the contact points between deep RL and neuroscience.