---
title: "Dopamine in the Brain: The Neural Actor–Critic"
module: Reinforcement Learning in Minds and Brains
moduleNumber: 6
lessonNumber: 4
order: 604
summary: >
  If phasic dopamine is a TD error, where does it go and what does it change? We
  follow the axons into the basal ganglia, read the corticostriatal synapse as the
  place where state, action, and error meet, and map the ventral and dorsal
  striatum onto the critic and the actor of an actor–critic. Addiction becomes a
  broken cancellation in the same learning signal, and distributional dopamine
  extends the scalar RPE into a population code.
topics: [Minds and Brains]
sources:
  - book: Sutton & Barto
    ref: "Ch. 15 — Neuroscience; §15.4 Dopamine; §15.7 Neural Actor–Critic"
  - book: Sutton & Barto
    ref: "§15.8 Actor and Critic Learning Rules; §15.12 Addiction; §15.13 Summary"
---

This builds on [Dopamine and the TD Error](/reinforcement-learning/minds-and-brains/dopamine-and-td-error),
which matched the phasic dopamine signal to the TD error term by term — the burst at
an unpredicted reward, the shift to the earliest cue, the dip when a predicted reward
is withheld. Granting that correspondence, this part asks the next question: what
does the signal _do_? We follow the axons into the basal ganglia, read the wiring as
an actor–critic, and close on addiction and on where the scalar picture breaks.

## Reward pathways and the basal ganglia

Where does this signal go, and what does it change? Dopamine is a
**neuromodulator**: rather than simply exciting or inhibiting its targets, it
alters how _other_ synapses learn. It is well suited to broadcasting a single
scalar. Each dopamine neuron has an enormous **axonal arbor** — the tree of
branches through which its output reaches its targets — making on the order of
$500{,}000$ synaptic contacts, one hundred to a thousand times more than a
typical neuron. If the signal it carries is a scalar $\delta$, a widely branching
arbor is the wiring you would design to deliver that one number to many
places at once.[^sb-dopamine]

Most of those axons target the **basal ganglia**, a group of nuclei at the base
of the forebrain whose main input structure is the **striatum**. Nearly the whole
cerebral cortex feeds the striatum, whose output loops back through other basal
ganglia nuclei and the thalamus to the frontal cortex, where it shapes movement,
decision, and reward processing. Two subdivisions of the striatum recur in what
follows: the **dorsal striatum**, tied to action selection, and the **ventral
striatum**, tied to reward processing and to assigning value to sensations.[^sb-basal]

$$
% caption: The reward circuit at a high level. Cortex conveys states to the
% striatum; dopamine neurons in the midbrain (VTA and SNpc) broadcast the scalar
% error over a vast axonal arbor to both the dorsal and ventral striatum, where it
% governs how corticostriatal synapses change.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=24mm, minimum height=11mm, align=center},
  small/.style={draw, minimum width=22mm, minimum height=10mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[box] (ctx) at (0,0) {cortex\\(states)};
  \node[small] (dstr) at (4.3,1.3) {dorsal striatum\\(action)};
  \node[small] (vstr) at (4.3,-1.3) {ventral striatum\\(value)};
  \node[box, draw=red, text=red, thick] (da) at (9.0,0) {dopamine\\(VTA, SNpc)};
  \draw[->, acc, thick] (ctx) -- (dstr);
  \draw[->, acc, thick] (ctx) -- (vstr);
  \draw[->, red, thick] (da) -- (dstr) node[midway, above, font=\scriptsize] {delta};
  \draw[->, red, thick] (da) -- (vstr) node[midway, below, font=\scriptsize] {delta};
  \draw[->, black, thick] (vstr) to[bend right=20] (da);
  \node[anchor=north, font=\scriptsize, text=black] at (6.6,-1.55) {reward info};
\end{tikzpicture}
$$

The input neurons of the striatum are **medium spiny neurons**, whose dendrites
are studded with tiny protrusions called spines. Cortical axons contact the tips
of these spines while dopamine axons contact their stems, placing three signals in
close company at a single synapse: the presynaptic cortical activity (the state),
the postsynaptic activity of the striatal neuron (the action or value), and
dopamine (the error). Evidence is mounting that changes in the efficacy of these
**corticostriatal synapses** — the connections from cortex into the striatum —
depend critically on appropriately timed dopamine signals. That is a learning
rule implemented in tissue.[^sb-basal]

The arrangement is worth drawing, because it puts the three quantities a learning
rule needs into physical contact at one spine. The cortical axon at the spine tip
carries the **state** — which features are active now. The medium spiny neuron's
own firing carries the **output** — the action taken, or the value asserted. And
the dopamine axon at the spine stem carries the **error** — the scalar $\delta$.
A synapse that changes its strength as a function of all three is computing an
update of the form (feature) $\times$ (error), gated by whether the postsynaptic
cell fired. That is the shape of the actor and critic rules below, and the
geometry of the spine is what lets one synapse read all three signals at once.

$$
% caption: Three signals converge at a single corticostriatal spine. The cortical
% axon (spine tip) delivers the state feature $x_i(s)$; the medium spiny neuron's
% firing is the postsynaptic output; the dopamine axon (spine stem) delivers the
% scalar error $\delta$. A plasticity rule reading all three implements a
% $\delta$-gated update on the synapse's efficacy $w_i$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  sig/.style={draw, minimum width=27mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % the spine / synapse node in the middle
  \node[draw, circle, minimum size=13mm, thick, align=center, font=\scriptsize] (syn) at (0,0) {synapse\\$w_i$};
  % cortical input (state) from the left
  \node[sig, draw=acc, text=acc] (ctx) at (-4.4,1.5) {cortical axon\\(state feature)};
  % postsynaptic output up-right
  \node[sig] (msn) at (4.4,1.5) {MSN f\/iring\\(output)};
  % dopamine from below
  \node[sig, draw=red, text=red] (da) at (0,-2.3) {dopamine axon\\(error delta)};
  \draw[->, acc, thick] (ctx) to[out=-30,in=150] node[above, font=\scriptsize] {tip} (syn);
  \draw[->, black, thick] (syn) to[out=30,in=210] node[above, font=\scriptsize] {f\/ires} (msn);
  \draw[->, red, thick] (da) -- node[right, font=\scriptsize] {stem} (syn);
\end{tikzpicture}
$$

## The neural actor–critic

The RPE hypothesis says dopamine _is_ a reinforcement signal. Which algorithm
uses a reinforcement signal that way? An
[actor–critic](/reinforcement-learning/approximation/policy-gradient-methods), and
the fit to the anatomy is close enough to be a serious hypothesis about the brain.[^sb-ac]

Recall the structure. The **critic** learns a value function for the policy the
actor is currently following, and computes a TD error $\delta$ from reward and its
own value estimates. The **actor** learns the policy, and adjusts it using
$\delta$ as its teaching signal: a positive $\delta$ means the action led
somewhere better than expected, so make it more likely; a negative $\delta$ means
worse than expected, so make it less likely. The single error $\delta$ trains
both halves, but it does different work in each — it tells the critic how to
sharpen its predictions and tells the actor how to shift its choices.

$$
% caption: The actor–critic and its proposed neural map. The critic learns the
% value function and computes the TD error; the actor uses that same error to
% update the policy. Under the hypothesis the ventral striatum is the critic, the
% dorsal striatum is the actor, and dopamine broadcasts the shared $\delta$ to
% both.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=24mm, minimum height=12mm, align=center},
  small/.style={draw, minimum width=26mm, minimum height=11mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % ===== LEFT: the algorithm =====
  \begin{scope}
    \node[box] (env) at (0,0) {environment};
    \node[box, draw=acc, text=acc, thick] (critic) at (0,-2.4) {critic (value)};
    \node[box] (actor) at (0,-4.8) {actor (policy)};
    \draw[->, black, thick] (env.west) to[out=180,in=180] node[left, font=\scriptsize] {reward, state} (critic.west);
    \draw[->, red, thick] (critic) -- (actor) node[midway, right, font=\scriptsize] {delta};
    \draw[->, black, thick] (actor.east) to[out=0,in=0] node[right, font=\scriptsize] {action} (env.east);
    \node[anchor=south, font=\footnotesize] at (0,0.75) {algorithm};
  \end{scope}
  % ===== RIGHT: the brain =====
  \begin{scope}[xshift=7.6cm]
    \node[small] (ctx) at (0,0) {cortex};
    \node[small, draw=acc, text=acc, thick] (vs) at (0,-2.4) {ventral striatum\\= critic};
    \node[small] (ds) at (0,-4.8) {dorsal striatum\\= actor};
    \node[box, draw=red, text=red, thick] (da) at (3.6,-2.4) {dopamine};
    \draw[->, black, thick] (ctx) -- (vs);
    \draw[->, black, thick] (ctx.west) to[out=180,in=180] (ds.west);
    \draw[->, red, thick] (vs) -- (da);
    \draw[->, red, thick] (da) to[out=-90,in=0] node[right, font=\scriptsize] {delta} (ds.east);
    \draw[->, red, thick, dashed] (da) to[out=90,in=0] (vs.east);
    \node[anchor=south, font=\footnotesize] at (0,0.75) {brain};
  \end{scope}
\end{tikzpicture}
$$

Two features support the mapping. First, the actor–critic has two learning
components, and the striatum has two subdivisions with the matching division of
labor: the **ventral striatum**, tied to value, plays the critic; the **dorsal
striatum**, tied to action selection, plays the actor. Second, the TD error
trains _both_ components, and dopamine axons target _both_
subdivisions of the striatum. One signal, broadcast to two places that need it,
is what the algorithm calls for and what the wiring provides.[^sb-ac]

The learning rules refine the picture. In the actor–critic policy-gradient form,
each transition computes $\delta_t$ and then updates the critic's value weights
$\mathbf{w}$ and the actor's policy weights $\boldsymbol{\theta}$ using the _same_
$\delta_t$, each scaled by its own eligibility trace:

$$
\begin{aligned}
\delta_t &= R_{t+1} + \gamma\, \hat{v}(S_{t+1}, \mathbf{w}) - \hat{v}(S_t, \mathbf{w}), \\
\mathbf{w} &\gets \mathbf{w} + \alpha^{\mathbf{w}}\, \delta_t\, \mathbf{z}_t^{\mathbf{w}}, \\
\boldsymbol{\theta} &\gets \boldsymbol{\theta} + \alpha^{\boldsymbol{\theta}}\, \delta_t\, \mathbf{z}_t^{\boldsymbol{\theta}}.
\end{aligned}
$$

The rules differ only in the eligibility trace each carries.[^sb-rules]

| | **Critic** ($\mathbf{w}$) | **Actor** ($\boldsymbol{\theta}$) |
| --- | --- | --- |
| Trace | non-contingent (presynaptic only) | contingent (pre- and postsynaptic) |
| Extra factor | — | $A_t - \pi(A_t \mid S_t, \boldsymbol{\theta})$ |
| Conditioning analogue | classical (prediction) | instrumental (control) |
| Objective | drive $\delta \to 0$, predict accurately | make $\delta$ large, choose better |

The critic's trace depends on presynaptic activity alone, so its rule is the TD model of
classical conditioning. The actor's trace also depends on whether the postsynaptic neuron
fired, carrying the factor $A_t - \pi(A_t \mid S_t, \boldsymbol{\theta})$ that records
_which action was taken_ — an instrumental rule.

That postsynaptic contingency has a biological counterpart. Neuroscientists
have found a form of Hebbian plasticity called **spike-timing-dependent
plasticity**, in which a synapse strengthens when its presynaptic spike arrives
just before the postsynaptic neuron fires. And they have found a **reward-modulated**
form, in which the change happens only if a neuromodulatory pulse — dopamine —
arrives within a time window that can last up to ten seconds after the paired
spikes. A synapse that remembers a pre-then-post coincidence and commits the
change only when dopamine later confirms it is, in tissue, a contingent
eligibility trace waiting on the actor's learning signal.[^sb-rules]

## Addiction: hijacking the learning signal

If dopamine carries the learning signal, then a drug that tampers with dopamine
tampers with learning itself. Many drugs of abuse — cocaine among them — raise
dopamine, directly or indirectly, in the striatal regions the RPE hypothesis
places at the center of reward learning. A model due to Redish (2004) turns the
TD account into an account of what goes wrong.[^sb-addiction]

The pathology is a broken cancellation. In normal learning, once a reward is
predicted the bootstrapped term drives the TD error at that reward to zero — a fully
anticipated reward stops reinforcing. Redish's model assumes an addictive drug produces
a dopamine surge that _cannot_ be cancelled by any rise in the value function: the drug
adds a non-negative term $D \geq 0$ that floors the error,
$$
\delta_t = \max\!\big(R_{t+1} + \gamma V(S_{t+1}) - V(S_t) + D,\; D\big) \geq 0,
$$
so the reward can never be "predicted away." With the error-correcting feedback that
normally levels off learning gone, the values of states leading to drug-taking rise
without bound, and the actions leading there come to be preferred above all others —
even against the self-destruction they cause.

$$
% caption: Addiction as a broken TD error. For a natural reward (left), value
% rises until the TD error at the reward returns to zero — the reward is predicted
% and stops reinforcing. For an addictive drug (right), the surge cannot be
% cancelled, so the TD error is floored above zero and the value climbs without
% bound.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % ===== LEFT: natural reward =====
  \begin{scope}
    \draw[black, ->] (0,0) -- (0,3.2) node[anchor=east, font=\scriptsize] {TD error};
    \draw[black, ->] (0,0) -- (4.6,0) node[anchor=north, font=\scriptsize] {trials};
    \draw[black, dashed] (0,0) -- (4.4,0);
    % decaying curve toward zero
    \draw[acc, very thick] (0.2,2.6) .. controls (1.6,0.6) and (2.6,0.1) .. (4.2,0.05);
    \node[acc, anchor=west, font=\scriptsize] at (2.2,0.65) {settles to 0};
    \node[anchor=north, font=\footnotesize] at (2.3,-0.35) {natural reward};
  \end{scope}
  % ===== RIGHT: addictive drug =====
  \begin{scope}[xshift=6.4cm]
    \draw[black, ->] (0,0) -- (0,3.2) node[anchor=east, font=\scriptsize] {value};
    \draw[black, ->] (0,0) -- (4.6,0) node[anchor=north, font=\scriptsize] {trials};
    % floored error -> value climbs without bound
    \draw[red, very thick, ->] (0.2,0.2) .. controls (2.2,0.8) and (3.2,2.2) .. (4.1,3.0);
    \node[red, anchor=south east, font=\scriptsize] at (4.1,2.9) {climbs without bound};
    \node[anchor=north, font=\footnotesize] at (2.3,-0.35) {addictive drug};
  \end{scope}
\end{tikzpicture}
$$

Real addiction is far more complicated than one clamped term — dopamine is not
central to every form of it, susceptibility varies, and the model omits the many
circuit changes that accompany chronic drug use. But the model illustrates a
theme that runs through this whole chapter: a computational theory of reward
learning gives neuroscience a precise, testable handle on a real disorder. The
same reasoning has helped launch **computational psychiatry**, which aims to
sharpen the understanding of mental illness with mathematical models of the
learning systems that go awry.[^sb-addiction]

## Distributional dopamine and the RPE hypothesis under test

The RPE hypothesis, as stated so far, is a claim about a scalar: one number,
$\delta$, broadcast uniformly. Three lines of work published after Sutton and Barto's
account push on that picture — one confirming the arithmetic the theory needs, one
enriching what the population encodes, and one marking where the model-free reading
runs out.

**The circuit computes a real subtraction.** The TD error is a difference,
reward minus expectation, and Eshel and colleagues (2015, _Nature_) asked whether
dopamine neurons implement that subtraction literally or only approximate it.
Recording from mouse VTA dopamine neurons while varying both reward size and the
animal's expectation, they found that the response to a reward equals the reward's
value minus the expected value — a genuine subtraction, not a division or a
threshold — and, across the population, that the neurons share a nearly identical
scaling. Different dopamine neurons respond to different absolute amounts, but they
subtract expectation in the same way and differ only by a common gain. The circuit
is doing the arithmetic the TD error asks for, uniformly enough that the population
can be read as broadcasting one error.[^dist-eshel]

**The population encodes a distribution, not just a mean.** A scalar RPE throws
away everything about reward except its expected value. Distributional
reinforcement learning (Bellemare, Dabney, and Munos, 2017, _ICML_) had shown that
an agent learns better if it tracks the whole distribution of returns rather than
its mean, using a population of predictors with different optimism — some tuned to
weight good outcomes, others bad. Dabney, Kurth-Nelson, Uchida, Starkweather,
Hassabis, Munos, and Botvinick (2020, _Nature_) imported that idea as a prediction
about the brain and tested it in mouse VTA. They found that dopamine neurons are
_not_ interchangeable scalar reporters. Different neurons have different **reversal
points** — the reward magnitude at which a given neuron switches from a positive to
a negative response — so some are relatively optimistic and others pessimistic.
Read as a population, the varied reversal points encode a full distribution over
upcoming reward, and the distribution recovered from the neural data matched the
one implied by the animals' behavior. The reward prediction error hypothesis, on
this account, is the mean of a richer distributional code.[^dist-dabney]

$$
% caption: Distributional dopamine (Dabney et al., 2020). Instead of one common
% zero-crossing, individual dopamine neurons have different reversal points -- the
% reward size where the response switches sign. Optimistic neurons (blue) cross at
% larger rewards; pessimistic neurons (red) cross earlier. The spread of reversal
% points across the population encodes a distribution over reward, not just a mean.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[black, ->] (0,-1.6) -- (0,1.9) node[anchor=east, font=\scriptsize] {response};
  \draw[black, ->] (0,0) -- (5.6,0) node[anchor=north, font=\scriptsize] {reward size};
  % pessimistic neuron: crosses early
  \draw[red, very thick] (0.2,-1.3) -- (1.6,0) -- (5.0,1.1);
  \node[red, anchor=west, font=\scriptsize] at (5.05,1.05) {pessimistic};
  \node[red, anchor=north, font=\scriptsize] at (1.6,-0.08) {early};
  % optimistic neuron: crosses late
  \draw[acc, very thick] (0.2,-1.0) -- (3.8,0) -- (5.2,0.55);
  \node[acc, anchor=west, font=\scriptsize] at (5.05,0.5) {optimistic};
  \node[acc, anchor=south, font=\scriptsize] at (3.8,0.08) {late};
  \node[anchor=north, font=\scriptsize, text=black] at (2.6,-1.35) {reversal points spread across neurons};
\end{tikzpicture}
$$

**The signal is not purely model-free.** A TD error read off a value function is a
model-free quantity: it knows returns, not the structure of the world that
produces them. Yet dopamine transients also carry information they should not, on a
strict model-free reading. Bromberg-Martin, Matsumoto, and Hikosaka (2010,
_Neuron_) found dopamine responses sensitive to sensory and motivational
information beyond cached value, and Doll, Simon, and Daw (2012, _Current Opinion
in Neurobiology_) reviewed evidence that model-based knowledge influences the
error signal and the choices it drives. The pure model-free RPE is an
approximation that holds well for habitual behavior and leaks where planning
enters — the seam the
[next lesson](/reinforcement-learning/minds-and-brains/animal-learning-and-cognition)
and the
[neuroscience of model-based control](/reinforcement-learning/minds-and-brains/neuroscience-of-reinforcement)
take up in full.[^dist-modelbased]

None of this unseats the correspondence. Each result takes the TD error as the
starting point and asks a sharper question of the biology than the original
recordings could — whether the subtraction is exact, what the population as a whole
represents, where the model-free reading stops. The hypothesis holds up under this
more precise questioning, and gains structure rather than losing it.

## The convergence, and where the course began

The TD error was written down for reasons
that had nothing to do with biology — optimal control, dynamic programming, the
problem of learning to predict before an outcome is known. Years later, recordings
from dopamine neurons revealed a signal that matches it in signature after
signature: the burst at unpredicted reward, the migration to the earliest
predictor, the dip at an omitted one. The algorithm was not
reverse-engineered from the brain; it was built for machines and turned out to
describe the brain anyway. That unplanned agreement is the strongest sign we have
that reinforcement learning captures something true about how nervous systems
learn from reward.[^sb-summary]

Which returns us to where the course started. We opened by defining
[reinforcement learning](/reinforcement-learning/foundations/what-is-reinforcement-learning)
as learning what to do from a reward signal, discovered by trial and error rather
than told. Every method since — bandits, MDPs, dynamic programming, Monte Carlo,
temporal difference, function approximation, policy gradients, deep RL — was a way
of solving that one problem. And the quantity at the hinge of it all, the TD error
$\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)$, turns out to be not only the
engine of the algorithms but, as far as we can now tell, the currency the brain
itself trades in when it learns from reward. The mathematics of trial-and-error
learning and the biology of trial-and-error learning appear to be the same subject.

[^sb-dopamine]: **Sutton & Barto**, §15.4 — Dopamine: dopamine produced in the substantia nigra pars compacta (SNpc) and ventral tegmental area (VTA); its roles in motivation, learning, action-selection, and addiction; the reconciliation of the reward-prediction-error view with the older "dopamine signals reward" view; and the huge axonal arbors making roughly 500,000 synaptic contacts that suit dopamine neurons to broadcasting a scalar reinforcement signal.
[^sb-basal]: **Sutton & Barto**, §15.4 — Dopamine (basal ganglia and striatum): the basal ganglia as forebrain nuclei with the striatum as main input structure; cortex projecting to the striatum; the dorsal striatum implicated in action selection and the ventral striatum in reward processing; medium spiny neurons whose dendritic spines receive cortical input at the tips and dopamine at the stems (Figure 15.1); and corticostriatal plasticity depending on appropriately-timed dopamine.
[^sb-ac]: **Sutton & Barto**, §15.7 — Neural Actor–Critic: the actor learns policies and the critic learns the value function, computing a TD error $\delta$ used by both; the two-component structure mapping onto the dorsal (actor) and ventral (critic) striatum, and the dual role of $\delta$ as the reinforcement signal for both fitting the fact that dopamine axons target both subdivisions (Figure 15.5, following Takahashi, Schoenbaum, and Niv, 2008).
[^sb-rules]: **Sutton & Barto**, §15.8 — Actor and Critic Learning Rules: the actor–critic equations with the shared TD error $\delta_t$ updating the critic weights $\mathbf{w}$ and actor weights $\boldsymbol{\theta}$, each scaled by an eligibility trace; the non-contingent (presynaptic-only) critic trace versus the contingent (pre- and postsynaptic) actor trace with factor $A_t - \pi(A_t \mid S_t, \boldsymbol{\theta})$; and spike-timing-dependent plasticity and its reward-modulated form as biological realizations.
[^sb-addiction]: **Sutton & Barto**, §15.12 — Addiction: Redish's (2004) model in which an addictive drug produces a dopamine surge that increases $\delta$ in a way that cannot be cancelled by rises in the value function, so drug rewards "cannot be predicted away," $\delta$ never becomes negative for drug-associated states, and their values increase without bound; the model's limits; and the connection to the field of computational psychiatry.
[^sb-summary]: **Sutton & Barto**, §15.13 — Summary: the striking correspondences between the brain's reward system and reinforcement-learning theory, the reward prediction error hypothesis and its experimental support, the actor–critic reading of the striatum with dopamine as the shared reinforcement signal, and the observation that RL algorithms developed for purely computational reasons are turning out to be consistent with neuroscience data.
[^dist-eshel]: Eshel, N., Bukwich, M., Rao, V., Hemmelder, V., Tian, J., and Uchida, N. (2015), "Arithmetic and local circuitry underlying dopamine prediction errors," _Nature_ 525:243–246 — VTA dopamine neurons compute reward prediction error as a subtraction of expected from received value, with a common, near-identical scaling across the population.
[^dist-dabney]: Dabney, W., Kurth-Nelson, Z., Uchida, N., Starkweather, C. K., Hassabis, D., Munos, R., and Botvinick, M. (2020), "A distributional code for value in dopamine-based reinforcement learning," _Nature_ 577:671–675 — individual dopamine neurons have distinct reversal points (relative optimism/pessimism), so the population encodes a distribution over reward rather than only its mean; the prediction was imported from distributional RL (Bellemare, Dabney, and Munos, 2017, _ICML_, "A distributional perspective on reinforcement learning") and confirmed in mouse VTA recordings.
[^dist-modelbased]: Bromberg-Martin, E. S., Matsumoto, M., and Hikosaka, O. (2010), "Dopamine in motivational control: rewarding, aversive, and alerting," _Neuron_ 68(5):815–834; and Doll, B. B., Simon, D. A., and Daw, N. D. (2012), "The ubiquity of model-based reinforcement learning," _Current Opinion in Neurobiology_ 22(6):1075–1081 — evidence that dopamine transients carry sensory and model-based information beyond a purely model-free cached-value prediction error.