---
title: "Actor-Critic Methods and Continuous Actions"
module: Approximate Solution Methods
moduleNumber: 3
lessonNumber: 10
order: 310
summary: >
  REINFORCE with a baseline learns a value function but never bootstraps; this
  lesson adds the bootstrapping critic that completes the actor-critic
  architecture. The critic scores each transition into a single TD error that
  steers both the actor's policy step and its own value step, trading a little
  bias for much lower variance and fully online, continuing-task learning. The
  policy gradient theorem carries over unchanged to the average-reward setting,
  a Gaussian policy handles real-valued actions with self-tuning exploration,
  and the natural policy gradient leads straight to TRPO, PPO, and the deep
  actor-critic methods that train today's agents.
topics: [Approximation]
sources:
  - book: Sutton & Barto
    ref: "Ch. 13 — Policy Gradient Methods; §13.5 Actor–Critic Methods; §13.6 Policy Gradient for Continuing Problems"
  - book: Sutton & Barto
    ref: "§13.7 Policy Parameterization for Continuous Actions"
---

This builds on
[policy gradient methods](/reinforcement-learning/approximation/policy-gradient-methods),
which parameterized the policy directly, proved the policy gradient theorem, and
derived REINFORCE and its variance-cutting baseline. REINFORCE with a baseline
already learns both a policy and a value function — but the value function only
recenters the Monte Carlo return; it never bootstraps. This lesson adds the
bootstrapping step that turns the baseline into a true critic, then extends the
architecture to continuing tasks and continuous actions.

## Actor-critic methods

REINFORCE with baseline learns both a policy and a value function, yet it is **not**
an actor-critic method, because its value function never **bootstraps**. It supplies
a baseline for one state, but the target for the policy update is still the full
Monte Carlo return $G_t$. An actor-critic method uses the learned value estimate to
**bootstrap** — to replace the full return with a one-step estimate — and that is
what earns the name **critic**.[^sb-ac]

The move repeats the one that took us from Monte Carlo to
[temporal-difference learning](/reinforcement-learning/tabular-methods/temporal-difference-learning).
Replace the full return $G_t$ in the baselined update with the one-step return
$R_{t+1} + \gamma \hat{v}(S_{t+1}, \mathbf{w})$, and the difference between that
target and the current estimate is precisely the **TD error**:

$$
\begin{aligned}
\boldsymbol{\theta}_{t+1}
&\doteq \boldsymbol{\theta}_t + \alpha\,\big(R_{t+1} + \gamma \hat{v}(S_{t+1}, \mathbf{w}) - \hat{v}(S_t, \mathbf{w})\big)\,\nabla \ln \pi(A_t \mid S_t, \boldsymbol{\theta}_t) \\
&= \boldsymbol{\theta}_t + \alpha\,\delta_t\,\nabla \ln \pi(A_t \mid S_t, \boldsymbol{\theta}_t),
\qquad \delta_t = R_{t+1} + \gamma \hat{v}(S_{t+1}, \mathbf{w}) - \hat{v}(S_t, \mathbf{w}).
\end{aligned}
$$

Bootstrapping introduces **bias** and an asymptotic dependence on the quality of
the value approximation, a cost that REINFORCE-with-baseline avoids. The tradeoff
is usually favorable: bootstrapping substantially reduces variance and lets
the method run **online and incrementally**, updating every step with no wait for
the episode to end. It works for continuing tasks, where Monte Carlo returns do not
even exist.

The architecture splits cleanly into two learners driven by the same TD error.

$$
% caption: The actor-critic loop. The critic scores the transition into a TD error
% $\delta_t$; the same $\delta_t$ steers the actor's policy step and the critic's
% own value step.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=24mm, minimum height=12mm, align=center},
  env/.style={draw, minimum width=24mm, minimum height=12mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[box, draw=acc, text=acc, thick] (actor)  at (0,1.6)  {actor\\pi(a:s;theta)};
  \node[box] (critic) at (0,-1.6) {critic\\v-hat(s,w)};
  \node[env] (env)    at (5.4,0)  {environment};
  \node[red] (td)     at (2.7,0)  {TD error};
  % actor acts on environment
  \draw[->, acc, thick] (actor) to[out=0,in=120] node[midway, above, font=\scriptsize] {action} (env);
  % environment returns reward + next state
  \draw[->, thick] (env) to[out=-120,in=0] node[midway, below, font=\scriptsize] {reward, state} (critic);
  % critic produces TD error
  \draw[->, red, thick] (critic) -- (td);
  % TD error drives actor and critic
  \draw[->, red, thick] (td) -- (actor) node[midway, left, font=\scriptsize] {steer policy};
  \draw[->, red, thick] (td) to[out=-90,in=90] node[midway, right, font=\scriptsize] {steer value} (critic.east);
\end{tikzpicture}
$$

The **actor** is the policy $\pi(a \mid s, \boldsymbol{\theta})$ — it selects
actions. The **critic** is the value function $\hat{v}(s, \mathbf{w})$ — it
evaluates them, producing $\delta_t$. A **positive** TD error means the action led
somewhere better than the critic expected, so the actor pushes to make that action
more likely and the critic revises its estimate upward; a negative error does the
reverse. Both learners consume the single scalar $\delta_t$.

```algorithm
caption: $\textsc{One-Step-Actor-Critic}$ — episodic, estimate $\pi_{\boldsymbol{\theta}} \approx \pi_\ast$
input: a differentiable policy $\pi(a \mid s, \boldsymbol{\theta})$, a differentiable $\hat v(s, \mathbf{w})$
parameters: step sizes $\alpha^{\boldsymbol{\theta}} > 0$, $\alpha^{\mathbf{w}} > 0$
$\boldsymbol{\theta}, \mathbf{w} \gets$ arbitrary (e.g. $\mathbf{0}$)
for each episode do
  initialize $S$ (first state of episode)
  $I \gets 1$
  while $S$ is non-terminal do
    sample $A \sim \pi(\cdot \mid S, \boldsymbol{\theta})$, take it, observe $R$, $S'$
    $\delta \gets R + \gamma\,\hat v(S', \mathbf{w}) - \hat v(S, \mathbf{w})$ // $\hat v(S',\mathbf{w}) \doteq 0$ if $S'$ terminal
    $\mathbf{w} \gets \mathbf{w} + \alpha^{\mathbf{w}}\,\delta\, \nabla \hat v(S, \mathbf{w})$
    $\boldsymbol{\theta} \gets \boldsymbol{\theta} + \alpha^{\boldsymbol{\theta}}\,I\,\delta\, \nabla \ln \pi(A \mid S, \boldsymbol{\theta})$
    $I \gets \gamma I$
    $S \gets S'$
```

This one-step method is the analog of TD(0). Generalizing it to $n$-step returns or
to a $\lambda$-return with separate **eligibility traces** for actor and critic is
mechanical — swap the one-step target for the longer one — and gives a family that
trades bias against variance by the degree of bootstrapping, exactly as in the
value-based case. The critic's own weights $\mathbf{w}$ are learned by
semi-gradient TD, paired to the actor's update through their shared $\delta_t$.

## Policy gradient for continuing problems

Episodic performance is the value of a start state, but continuing problems have no
episode boundary and no start state to anchor $J$. For them, performance is the
**average rate of reward** per step under the policy:[^sb-continuing]

$$
J(\boldsymbol{\theta}) \;\doteq\; r(\pi) \;\doteq\; \lim_{h \to \infty} \frac{1}{h} \sum_{t=1}^{h} \mathbb{E}[R_t \mid S_0, A_{0:t-1} \sim \pi] \;=\; \sum_s \mu(s) \sum_a \pi(a \mid s)\sum_{s',r} p(s',r \mid s,a)\,r,
$$

where $\mu$ is the **steady-state distribution** under $\pi$, defined by
$\mu(s) \doteq \lim_{t \to \infty} \Pr\{S_t = s \mid A_{0:t} \sim \pi\}$ and assumed
to exist independently of the start state (an ergodicity assumption). It is the
special distribution in which staying with $\pi$ keeps you: $\sum_s \mu(s) \sum_a
\pi(a \mid s, \boldsymbol{\theta})\,p(s' \mid s,a) = \mu(s')$ for every $s'$.

Values are then defined against the **differential return**, rewards measured
relative to the average rate,

$$
G_t \;\doteq\; R_{t+1} - r(\pi) + R_{t+2} - r(\pi) + R_{t+3} - r(\pi) + \cdots,
$$

and with these definitions the policy gradient theorem holds **unchanged** — the
same $\nabla J \propto \sum_s \mu(s) \sum_a q_\pi(s,a)\,\nabla \pi(a \mid s,
\boldsymbol{\theta})$, now with the continuing $\mu$ and $q_\pi$. The proof runs
parallel to the episodic one. In the actor-critic algorithm the only practical
change is that the TD error subtracts a running estimate $\bar{R}$ of the average
reward: $\delta \leftarrow R - \bar{R} + \hat{v}(S', \mathbf{w}) - \hat{v}(S,
\mathbf{w})$, with $\bar{R}$ itself updated toward $R$ by the same error. No
discounting appears; the average-reward setting replaces $\gamma$ with the
differential value.

## Continuous actions: the Gaussian policy

The softmax parameterization enumerates a probability per action, which is hopeless
when there are infinitely many. For continuous actions the policy instead learns
the **statistics of a distribution** and samples from it. The standard choice on a
real-valued scalar action is a **Gaussian policy**: output a state-dependent mean
and standard deviation, then draw the action from the normal density.[^sb-continuous]

$$
\pi(a \mid s, \boldsymbol{\theta}) \;\doteq\; \frac{1}{\sigma(s, \boldsymbol{\theta})\sqrt{2\pi}}\,\exp\!\left(-\frac{\big(a - \mu(s, \boldsymbol{\theta})\big)^2}{2\,\sigma(s, \boldsymbol{\theta})^2}\right),
$$

where $\mu : \mathcal{S} \times \mathbb{R}^{d'} \to \mathbb{R}$ and
$\sigma : \mathcal{S} \times \mathbb{R}^{d'} \to \mathbb{R}^{+}$ are two
parameterized function approximators. The mean sets where the policy aims; the
standard deviation sets **how much it explores** — a large $\sigma$ spreads mass
widely and tries varied actions, a small $\sigma$ concentrates near the mean and
commits. Exploration is now a learned quantity, not a fixed $\varepsilon$.

$$
% caption: A Gaussian policy on a scalar action: the mean $\mu(s,\boldsymbol{\theta})$
% sets where the policy aims and the standard deviation $\sigma(s,\boldsymbol{\theta})$
% sets how widely it explores; the sampled action $A_t$ is a draw from this density.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axis
  \draw[black, ->] (-3.4,0) -- (3.6,0) node[anchor=north east, text=black] {action a};
  % bell curve
  \draw[acc, very thick] plot[domain=-3.2:3.2, samples=60] (\x, {2.6*exp(-((\x)^2)/1.6)});
  % mean line
  \draw[black, dashed] (0,0) -- (0,2.75);
  \node[acc, anchor=south] at (0,2.65) {mean mu(s,theta)};
  % sigma span
  \draw[<->, red, thick] (-1.4,0.7) -- (1.4,0.7);
  \node[red, anchor=west] at (1.7,1.35) {spread sigma(s,theta)};
  % sampled action
  \fill[red] (0.9,0) circle (2pt);
  \node[red, anchor=north] at (0.9,-0.05) {At};
\end{tikzpicture}
$$

To finish the construction, split the parameter vector into two blocks,
$\boldsymbol{\theta} = [\boldsymbol{\theta}_\mu, \boldsymbol{\theta}_\sigma]^\top$,
one for each statistic. The mean is a linear function of state features; the
standard deviation, which must stay positive, is the exponential of a linear
function:

$$
\mu(s, \boldsymbol{\theta}) \;\doteq\; \boldsymbol{\theta}_\mu^\top \mathbf{x}_\mu(s),
\qquad
\sigma(s, \boldsymbol{\theta}) \;\doteq\; \exp\!\big(\boldsymbol{\theta}_\sigma^\top \mathbf{x}_\sigma(s)\big).
$$

Both are differentiable in $\boldsymbol{\theta}$, so the eligibility vector
$\nabla \ln \pi(a \mid s, \boldsymbol{\theta})$ is available in closed form and
every algorithm in this lesson — REINFORCE, baselined REINFORCE, actor-critic —
applies verbatim to real-valued control.

Differentiating $\ln \pi$ under the normal density (13.19) splits the eligibility
vector into a mean part and a spread part:

$$
\nabla_{\boldsymbol{\theta}_\mu} \ln \pi(a \mid s, \boldsymbol{\theta}) = \frac{a - \mu(s,\boldsymbol{\theta})}{\sigma(s,\boldsymbol{\theta})^2}\,\mathbf{x}_\mu(s),
\qquad
\nabla_{\boldsymbol{\theta}_\sigma} \ln \pi(a \mid s, \boldsymbol{\theta}) = \left(\frac{\big(a - \mu(s,\boldsymbol{\theta})\big)^2}{\sigma(s,\boldsymbol{\theta})^2} - 1\right)\mathbf{x}_\sigma(s).
$$

Each has a plain meaning. The mean part pushes $\mu$ toward the sampled action when
the return was good — the size of the push is the standardized surprise
$(a-\mu)/\sigma$, scaled again by $1/\sigma$ so that a confident (small-$\sigma$)
policy reacts more sharply to the same deviation. The spread part widens $\sigma$
when the action landed far from the mean (the squared standardized deviation
exceeds $1$) and narrows it when actions cluster near the mean (below $1$).
Exploration tunes itself: consistently rewarded actions near the mean shrink
$\sigma$ and sharpen the policy; surprising high-return outliers grow $\sigma$ and
keep it searching.

**A worked step.** Suppose $\mu(s) = 2.0$, $\sigma(s) = 0.5$, and the agent samples
$a = 2.4$ with a positive advantage $G - b = +3$. The standardized deviation is
$(2.4 - 2.0)/0.5 = 0.8$. The mean eligibility is
$(0.4/0.25)\,\mathbf{x}_\mu = 1.6\,\mathbf{x}_\mu$, so the update moves $\mu$ toward
$2.4$. The spread eligibility is $(0.8^2 - 1)\,\mathbf{x}_\sigma = -0.36\,\mathbf{x}_\sigma$;
because the sampled action sat inside one standard deviation, the positive
advantage times a negative eligibility shrinks $\sigma$, tightening the policy
around a region with high return.

$$
% caption: The Gaussian policy tuning itself. A rewarded action near the mean (top)
% grows the mean toward it and shrinks $\sigma$, sharpening the policy; a rewarded
% action in the far tail (bottom) grows $\sigma$, keeping exploration wide.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % top: narrow after reward near mean
  \begin{scope}[yshift=2.6cm]
    \draw[black, ->] (-3.0,0) -- (3.3,0);
    \draw[black, thick] plot[domain=-2.8:2.8, samples=50] (\x, {1.4*exp(-((\x)^2)/1.2)});
    \draw[acc, very thick] plot[domain=-2.8:2.8, samples=50] (\x, {1.7*exp(-((\x)^2)/0.6)});
    \fill[red] (0.55,0) circle (1.8pt);
    \node[red, anchor=south west, font=\scriptsize] at (0.55,0.05) {action near mean};
    \node[acc, anchor=west, font=\scriptsize] at (1.9,1.3) {sigma shrinks};
  \end{scope}
  % bottom: widen after reward in tail
  \draw[black, ->] (-3.0,0) -- (3.3,0) node[anchor=north east, font=\scriptsize] {action a};
  \draw[black, thick] plot[domain=-2.8:2.8, samples=50] (\x, {1.7*exp(-((\x)^2)/0.6)});
  \draw[acc, very thick] plot[domain=-2.8:2.8, samples=50] (\x, {1.4*exp(-((\x)^2)/1.6)});
  \fill[red] (1.8,0) circle (1.8pt);
  \node[red, anchor=south, font=\scriptsize] at (1.8,0.06) {action in tail};
  \node[acc, anchor=west, font=\scriptsize] at (2.0,1.1) {sigma grows};
\end{tikzpicture}
$$

## Natural policy gradients and the modern lineage

The plain gradient $\nabla J(\boldsymbol{\theta})$ ascends performance in the
geometry of the raw parameter space, treating every coordinate of
$\boldsymbol{\theta}$ as equally scaled. That geometry is arbitrary: rescaling one
weight, or reparameterizing the policy, changes the gradient direction even though
the policy — the actual distribution over actions — is unchanged. As a result,
vanilla policy gradient can be slow. A small step in $\boldsymbol{\theta}$ can
produce a large, destabilizing change in the action distribution in one region and
a negligible change in another.

The **natural policy gradient** fixes this by measuring distance between policies in
the space of distributions rather than of parameters, using the Fisher information
matrix $\mathbf{F}(\boldsymbol{\theta}) = \mathbb{E}_\pi[\nabla \ln \pi\,\nabla \ln \pi^\top]$
as the local metric. The natural gradient premultiplies the ordinary gradient by
its inverse,

$$
\widetilde{\nabla} J(\boldsymbol{\theta}) \;=\; \mathbf{F}(\boldsymbol{\theta})^{-1}\,\nabla J(\boldsymbol{\theta}),
$$

which points in the direction of steepest ascent per unit of change in the policy's
output distribution, invariant to how the policy is parameterized. Amari (1998,
_Neural Computation_) established that this natural gradient is the correct steepest
descent direction on a statistical manifold;[^ng-amari] Kakade (2002, _NeurIPS_, "A
Natural Policy Gradient") brought it to reinforcement learning and showed it moves
toward the greedy policy and often converges far faster than the vanilla
gradient.[^ng-kakade] Sutton and Barto flag natural-gradient methods (Amari 1998;
Kakade 2002; Peters and Schaal 2008) as the main further development past their own
treatment.

$$
% caption: Vanilla versus natural gradient on an anisotropic performance surface.
% The vanilla gradient (red) is perpendicular to raw-parameter contours and zig-zags
% up a stretched valley; the natural gradient (blue) uses the Fisher metric to head
% straight for the optimum, invariant to the parameterization.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % elongated contours (ellipses)
  \foreach \r in {0.6,1.2,1.8,2.4} \draw[black] (3.2,1.4) ellipse ({\r*1.7} and {\r*0.55});
  \fill[black] (3.2,1.4) circle (1.6pt) node[anchor=south west, font=\scriptsize] {optimum};
  % start point
  \fill[black] (0.5,0.5) circle (1.8pt) node[anchor=north, font=\scriptsize] {start};
  % vanilla: zig-zag
  \draw[red, thick] (0.5,0.5) -- (1.5,1.5) -- (2.0,0.9) -- (2.7,1.7) -- (3.0,1.25);
  \node[red, anchor=east, font=\scriptsize] at (1.45,1.65) {vanilla};
  % natural: straight
  \draw[acc, very thick, ->] (0.55,0.55) -- (3.1,1.35);
  \node[acc, anchor=north west, font=\scriptsize] at (1.5,0.95) {natural};
\end{tikzpicture}
$$

This line of work leads to the algorithms that train today's agents. **Trust
region policy optimization** (Schulman et al. 2015, _ICML_) constrains each update
to a small KL-divergence step — a practical approximation of the natural-gradient
idea, capping how far the action distribution may move per update.[^ng-trpo]
**Proximal policy optimization** (Schulman et al. 2017, "Proximal Policy
Optimization Algorithms") replaces the hard constraint with a clipped surrogate
objective that is cheaper and more stable, and is the actor-critic loop of this
lesson with the raw $\delta_t$ swapped for a **generalized advantage estimate**
(Schulman et al. 2016, _ICLR_) and the policy step clipped.[^ng-ppo]
**Deterministic policy gradients** (Silver et al. 2014, _ICML_) take the opposite
turn from the Gaussian policy: they learn a deterministic $\mu(s)$ and differentiate
through a critic, the basis for DDPG in continuous control. And **off-policy
policy-gradient methods** (Degris, White, and Sutton 2012, _ICML_) reweight the
gradient by importance ratios so the policy can be improved from data a different
behavior policy generated[^ng-dpg] — the same
off-policy correction that runs through the eligibility-trace lessons. All rest on
the policy gradient theorem; they differ in the metric of the step and the
estimator of the advantage.

## The bridge to deep RL

Every method here left the function approximator abstract: the preferences $h$, the
value $\hat{v}$, the Gaussian's mean and spread are "some differentiable function of
$\boldsymbol{\theta}$." Make that function a deep neural network and the same three
equations underlie modern reinforcement learning. The eligibility
vector $\nabla \ln \pi$ is computed by backpropagation; the actor and critic become
two networks (or two heads of one); the TD error $\delta_t$ is the training signal
for both. Advantage actor-critic and PPO — the algorithms that train game-playing
agents and align language models — are the actor-critic loop of this lesson with a
neural policy, a clipped or trust-region step to keep the update stable, and an
**advantage** estimate in place of the raw $\delta_t$. We follow that thread into
[actor-critic and PPO](/reinforcement-learning/deep-rl/actor-critic-and-ppo).

The larger shift is conceptual. Before this chapter, learning meant estimating
values and reading a policy off them. Policy gradient methods invert that: the
policy is the object of learning, performance is a differentiable function of its
parameters, and everything reduces to gradient ascent on
$J(\boldsymbol{\theta})$. The policy gradient theorem makes this possible without
a model of the environment, and the actor-critic architecture makes it fast.

[^sb-ac]: **Sutton & Barto**, §13.5 — Actor–Critic Methods: why REINFORCE-with-baseline is not actor-critic (no bootstrapping), the one-step actor-critic updates (13.12)–(13.14) with the TD error $\delta_t$, the bias/variance trade of bootstrapping, and the boxed one-step and eligibility-trace algorithms pairing a semi-gradient TD critic with the policy actor.
[^sb-continuing]: **Sutton & Barto**, §13.6 — Policy Gradient for Continuing Problems: the average-reward performance $r(\pi)$ (13.15), the steady-state distribution $\mu$ (13.16), the differential return (13.17), the invariance of the policy gradient theorem in the continuing case, and the continuing backward-view actor-critic box with the running average-reward estimate $\bar{R}$.
[^sb-continuous]: **Sutton & Barto**, §13.7 — Policy Parameterization for Continuous Actions: the normal density parameterization $\pi(a \mid s, \boldsymbol{\theta})$ (13.18)–(13.19), splitting $\boldsymbol{\theta}$ into mean and standard-deviation blocks with $\mu = \boldsymbol{\theta}_\mu^\top \mathbf{x}_\mu(s)$ and $\sigma = \exp(\boldsymbol{\theta}_\sigma^\top \mathbf{x}_\sigma(s))$ (13.20), and the closed-form eligibility vectors of Exercise 13.4.
[^ng-amari]: **Amari (1998)**, "Natural gradient works efficiently in learning", _Neural Computation_ 10(2): the natural gradient as the steepest-ascent direction on a statistical manifold, obtained by premultiplying the ordinary gradient by the inverse Fisher information matrix; invariant to reparameterization.
[^ng-kakade]: **Kakade (2002)**, "A Natural Policy Gradient", _Advances in Neural Information Processing Systems (NeurIPS)_: applies the natural gradient to policy search, using the Fisher information of the policy as metric; shows the natural policy gradient moves toward the greedy policy and typically converges faster than the vanilla gradient. Sutton & Barto (§13.8, Bibliographical Remarks) cite Amari (1998) and Kakade (2002) as the primary further development past their chapter.
[^ng-trpo]: **Schulman, Levine, Abbeel, Jordan, Moritz (2015)**, "Trust Region Policy Optimization", _ICML_: constrains each policy update to a bounded KL-divergence trust region, a practical approximation to the natural-gradient step; **Schulman et al. (2016)**, "High-Dimensional Continuous Control Using Generalized Advantage Estimation", _ICLR_.
[^ng-ppo]: **Schulman, Wolski, Dhariwal, Radford, Klimov (2017)**, "Proximal Policy Optimization Algorithms", arXiv:1707.06347: replaces the TRPO constraint with a clipped surrogate objective; a first-order actor-critic method that is the basis for much modern policy-gradient training.
[^ng-dpg]: **Silver, Lever, Heess, Degris, Wierstra, Riedmiller (2014)**, "Deterministic Policy Gradient Algorithms", _ICML_: derives a policy-gradient theorem for deterministic policies $\mu(s)$ that differentiates through the critic, the basis for DDPG in continuous control. **Degris, White, Sutton (2012)**, "Off-Policy Actor-Critic", _ICML_: reweights the policy gradient by importance ratios so a target policy can be improved from behavior-policy data.