---
title: "Continuous Control: DDPG and TD3"
module: Modern Deep Reinforcement Learning
moduleNumber: 5
lessonNumber: 3
order: 503
summary: >
  When actions are real-valued, the $\arg\max_a Q(s,a)$ in Q-learning
  becomes an optimization problem on every step. This lesson builds the off-policy
  actor-critic family that sidesteps it: the deterministic policy gradient and
  DDPG, which replaces the max with a learned actor, and the three fixes of TD3 that
  counter the value overestimation DDPG inherits. A companion lesson takes up SAC's
  maximum-entropy objective and the methods built on this template.
topics: [Deep RL]
sources:
  - book: Grokking Deep RL
    ref: "Ch. 12 — Advanced actor-critic methods; DDPG, TD3, SAC"
  - book: Sutton & Barto
    ref: "Ch. 13 — Policy Gradient Methods; §13.7 Policy Parameterization for Continuous Actions"
---

[Deep Q-networks](/reinforcement-learning/deep-rl/deep-q-networks) act by computing
$\arg\max_a Q_\phi(s,a)$ — they enumerate the action values in a state and take the
largest. That works when the action set is a handful of discrete choices; it fails
when actions are **continuous**. A robot arm commands a torque on each of
seven joints; a quadruped commands a real vector; a car commands a steering angle and
a throttle. The maximization $\max_a Q_\phi(s,a)$ over $\mathbb{R}^k$ is then an inner
optimization problem, solved from scratch on every environment step — intractable in
any practical loop.

The [policy-gradient methods](/reinforcement-learning/approximation/policy-gradient-methods)
lesson already gave one escape: parameterize a policy directly and it can output a
Gaussian over continuous actions, no max required. The
[actor-critic and PPO](/reinforcement-learning/deep-rl/actor-critic-and-ppo) lesson
made that policy a deep network and toured this continuous-control family in a single
closing section. This lesson and its
[companion](/reinforcement-learning/modern-deep-rl/continuous-control-part-2) treat it
properly. The three methods in this family — **DDPG**, **TD3**, and **SAC** — share
one structural idea: keep a Q-critic $Q_\phi(s,a)$, but replace its intractable inner
max with a **learned actor** that produces the maximizing action directly. Because the
critic is a Q-function, all three learn **off-policy** from a replay buffer, which is
what makes them far more sample-efficient than on-policy PPO on the control benchmarks
they dominate. Here we build the first two — DDPG and the overestimation-fixing TD3 —
and leave SAC and the methods layered on this template to the companion.

$$
% caption: The obstacle. With discrete actions (left) $\max_a Q(s,a)$ is a lookup over
% a few values; with continuous actions (right) it is an optimization over a continuum,
% run on every step. The continuous-control family replaces that max with a learned actor.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % --- left: discrete, a few bars, pick the max ---
  \draw[black, ->] (-0.2,0) -- (3.3,0) node[anchor=north east, text=black] {action};
  \draw[black, ->] (-0.2,0) -- (-0.2,3.0) node[anchor=south east, text=black] {Q(s,a)};
  \node[anchor=south, text=black] at (1.5,3.0) {discrete};
  \foreach \x/\h in {0.3/1.1, 0.9/1.7, 1.5/2.4, 2.1/1.3, 2.7/0.8}
    \draw[acc, thick] (\x,0) -- (\x,\h);
  \fill[red] (1.5,2.4) circle (2.4pt);
  \node[red, anchor=south, font=\scriptsize] at (1.5,2.48) {argmax};
  % --- right: continuous, smooth curve, hard max ---
  \begin{scope}[xshift=5.6cm]
    \draw[black, ->] (-0.2,0) -- (3.3,0) node[anchor=north east, text=black] {action};
    \draw[black, ->] (-0.2,0) -- (-0.2,3.0) node[anchor=south east, text=black] {Q(s,a)};
    \node[anchor=south, text=black] at (1.5,3.0) {continuous};
    \draw[acc, very thick] plot[domain=0.05:3.05, samples=60]
      (\x, {2.4*exp(-((\x-1.7)^2)/0.6) + 0.7*exp(-((\x-0.5)^2)/0.3)});
    \fill[red] (1.7,2.42) circle (2.4pt);
    \node[red, anchor=west, font=\scriptsize] at (1.85,2.55) {argmax = ?};
  \end{scope}
\end{tikzpicture}
$$

## The deterministic policy gradient

Policy-gradient methods so far learned a **stochastic** policy $\pi(a \mid s)$ and
climbed the expectation $\nabla J = \mathbb{E}_\pi[\Psi_t\,\nabla \ln \pi]$. The
score-function estimator behind it averages over sampled actions, and its variance
grows with the dimension of the action space — a nuisance in high-dimensional control.
Silver et al. (2014) showed that if the policy is **deterministic**, written
$\mu_\theta : \mathcal{S} \to \mathcal{A}$, the policy gradient takes a simpler and
lower-variance form.[^dpg] A deterministic actor commits to one action per state,
$a = \mu_\theta(s)$, and the performance objective is the expected value of that action
under the critic,

$$
J(\theta) \;=\; \mathbb{E}_{s \sim \rho}\big[\,Q^{\mu}(s, \mu_\theta(s))\,\big],
$$

where $\rho$ is the state distribution induced by whatever policy fills the replay
buffer. Because the action is a deterministic function of $\theta$, the gradient flows
through it by the chain rule — no score-function trick, no integral over actions:

> **Theorem (Deterministic Policy Gradient).** For a deterministic policy $\mu_\theta$
> with a differentiable critic $Q^\mu(s,a)$,
> $$
> \nabla_\theta J(\theta) \;=\; \mathbb{E}_{s \sim \rho}\!\left[\,\nabla_\theta \mu_\theta(s)\;\nabla_a Q^\mu(s,a)\big|_{a=\mu_\theta(s)}\,\right].
> $$
> The actor is nudged in the direction that most increases the critic's value, mapped
> back through the actor's own Jacobian $\nabla_\theta \mu_\theta(s)$.

Read the two factors. $\nabla_a Q_\phi(s,a)$ is the direction in action space that
raises the critic's value; $\nabla_\theta \mu_\theta(s)$ translates that
change in action into a change in the actor's weights. The composition is one
backpropagation: differentiate the critic with respect to its action input, then
continue backward through the actor. This composition escapes the
$\arg\max_a Q$ — instead of searching the action space, the actor is trained by gradient
ascent to **output** the maximizer, and the critic's action-gradient is the training
signal.

$$
% caption: The deterministic policy gradient as one backward pass. The actor
% $\mu_\theta(s)$ feeds an action into the critic $Q_\phi(s,a)$; the critic's
% action-gradient $\nabla_a Q$ flows back through the actor's Jacobian to update
% $\theta$, so the actor learns to output the value-maximizing action.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=26mm, minimum height=13mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[box, draw=acc, text=acc, thick] (actor)  at (0,0)   {actor\\mu(s;theta)};
  \node[box] (critic) at (5.4,0) {critic\\Q(s,a;phi)};
  \node[anchor=east, text=black] at (-1.8,0) {state s};
  \draw[->, black] (-1.75,0) -- (actor.west);
  \draw[->, acc, thick] (actor) -- (critic) node[midway, above, font=\scriptsize] {action a};
  \node[anchor=west, text=black] at (7.4,0) {value Q};
  \draw[->, black] (critic.east) -- (7.35,0);
  % gradient backflow
  \draw[->, red, thick] (critic.south) to[out=-90,in=-90] node[midway, below, font=\scriptsize, text=red] {grad-a Q f\/lows back} (actor.south);
\end{tikzpicture}
$$

## DDPG: a DQN for continuous actions

**Deep deterministic policy gradient** (DDPG) is the deterministic policy gradient made
deep and off-policy, assembled from the same parts that stabilized DQN.[^ddpg] It runs
four networks: an actor $\mu_\theta(s)$, a critic $Q_\phi(s,a)$, and a slow-moving copy
of each — the **target actor** $\mu_{\theta'}$ and **target critic** $Q_{\phi'}$ — used
only to form the learning target. Transitions $(s, a, r, s')$ collected by the agent are
stored in a **replay buffer** $\mathcal{D}$ and sampled in minibatches, exactly as in
DQN, which decorrelates the updates and lets each transition train the networks many
times.

The critic is trained by regression toward a bootstrapped target, the continuous-action
analog of the DQN target. Where DQN uses $\max_{a'} Q_{\phi'}(s', a')$, DDPG replaces
the max with the target actor's action:

$$
y \;=\; r + \gamma\, Q_{\phi'}\!\big(s',\, \mu_{\theta'}(s')\big),
\qquad
\mathcal{L}(\phi) \;=\; \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}}\big[(Q_\phi(s,a) - y)^2\big].
$$

The actor is trained by the deterministic policy gradient — ascend the critic through
the actor — sampling states from the same buffer:

$$
\nabla_\theta J \;\approx\; \mathbb{E}_{s \sim \mathcal{D}}\!\left[\nabla_\theta \mu_\theta(s)\,\nabla_a Q_\phi(s,a)\big|_{a=\mu_\theta(s)}\right].
$$

### Target networks by soft update

DQN copies the online weights into the target network in a hard jump every few thousand
steps. DDPG instead updates the targets **softly** on every step, blending a small
fraction $\tau \ll 1$ of the online weights in:

$$
\phi' \gets \tau\,\phi + (1-\tau)\,\phi',
\qquad
\theta' \gets \tau\,\theta + (1-\tau)\,\theta',
\qquad \tau \approx 0.005.
$$

The targets therefore track the online networks slowly and smoothly. This keeps the
bootstrapped target $y$ stable: the critic regresses toward a value produced by
weights that change far more slowly than its own, damping the feedback loop that
would otherwise diverge.

### Exploration for a deterministic actor

A deterministic policy has no built-in exploration — asked for an action in a state, it
returns the same one every time. DDPG explores by adding **noise** to the action it
actually sends to the environment, while keeping the clean $\mu_\theta(s)$ for learning:

$$
a_t \;=\; \mu_\theta(s_t) + \mathcal{N}_t,
$$

with $\mathcal{N}_t$ a zero-mean noise process (the original paper used temporally
correlated Ornstein–Uhlenbeck noise; uncorrelated Gaussian noise works about as well in
practice). Exploration is thus **bolted on** externally, a design choice SAC will later
reject.

$$
% caption: The DDPG architecture. An actor and critic each have a slow target copy;
% transitions go into a replay buffer; sampled minibatches train the critic toward
% $r + \gamma Q_{\phi'}(s',\mu_{\theta'}(s'))$ and the actor up $\nabla_a Q_\phi$;
% exploration noise is added to the environment action.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=23mm, minimum height=11mm, align=center, font=\scriptsize},
  tgt/.style={draw, dashed, minimum width=23mm, minimum height=11mm, align=center, font=\scriptsize},
  env/.style={draw, minimum width=22mm, minimum height=11mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % environment + buffer top
  \node[env] (env) at (0,2.6) {environment};
  \node[box] (buf) at (4.3,2.6) {replay buf\/fer D};
  % online nets
  \node[box, draw=acc, text=acc, thick] (actor)  at (0,0)   {actor mu(s)};
  \node[box, draw=acc, text=acc, thick] (critic) at (4.3,0) {critic Q(s,a)};
  % target nets
  \node[tgt] (tactor)  at (0,-2.4)   {target mu-prime};
  \node[tgt] (tcritic) at (4.3,-2.4) {target Q-prime};
  % acting loop
  \draw[->, acc, thick] (actor.north) -- (env.south) node[midway, left, font=\scriptsize] {a + noise};
  \draw[->, thick] (env) -- (buf) node[midway, above, font=\scriptsize] {s,a,r,s-prime};
  \draw[->, thick] (buf) -- (critic) node[midway, right, font=\scriptsize] {minibatch};
  % critic trains actor
  \draw[->, red, thick] (critic) -- (actor) node[midway, above, font=\scriptsize, text=red] {grad-a Q};
  % targets form y
  \draw[->, black, dashed] (tactor) -- (tcritic) node[midway, below, font=\scriptsize] {mu-prime(s-prime)};
  \draw[->, black, dashed] (tcritic) -- (critic) node[midway, right, font=\scriptsize] {target y};
  % soft updates
  \draw[->, black, dashed] (actor.west) to[out=180,in=180] node[midway, left, font=\scriptsize] {soft} (tactor.west);
  \draw[->, black, dashed] (critic.east) to[out=0,in=0] node[midway, right, font=\scriptsize] {soft} (tcritic.east);
\end{tikzpicture}
$$

```algorithm
caption: $\textsc{DDPG}$ — off-policy deterministic actor-critic for continuous control
input: actor $\mu_\theta$, critic $Q_\phi$, soft-update rate $\tau$, discount $\gamma$
initialize targets $\theta' \gets \theta$, $\phi' \gets \phi$; empty replay buffer $\mathcal{D}$
for each step do
  observe $s$, select $a \gets \mu_\theta(s) + \mathcal{N}$ // exploration noise
  execute $a$, observe $r$, $s'$; store $(s,a,r,s')$ in $\mathcal{D}$
  sample a minibatch of $(s,a,r,s')$ from $\mathcal{D}$
  $y \gets r + \gamma\, Q_{\phi'}(s', \mu_{\theta'}(s'))$ // $Q_{\phi'} \gets 0$ if $s'$ terminal
  $\phi \gets \phi - \eta\,\nabla_\phi\, (Q_\phi(s,a) - y)^2$ // critic regression
  $\theta \gets \theta + \eta\,\nabla_\theta\, Q_\phi(s, \mu_\theta(s))$ // ascend critic
  $\phi' \gets \tau\phi + (1-\tau)\phi'$
  $\theta' \gets \tau\theta + (1-\tau)\theta'$
```

DDPG solved control problems DQN could not touch, but it earned a reputation for being
**brittle**: sensitive to hyperparameters, prone to sudden collapse, and — the central
diagnosis of the next method — systematically **overestimating** its own values.

## TD3: three fixes for overestimation

The failure mode DDPG inherits from Q-learning is **overestimation bias**. The critic is
noisy, and the actor is trained to maximize it. Any state where the critic happens to
overshoot the true value becomes a target the actor climbs toward, and the bootstrap
then propagates that inflated value backward through the Bellman update. Errors that
should average out instead compound, because the maximization systematically selects for
them. **Twin-delayed DDPG** (TD3) is DDPG plus three targeted fixes for this.[^td3]

### Clipped double-Q learning

The main fix is the same idea as double Q-learning: do not let a single noisy critic
evaluate its own maximizing action. TD3 learns **twin critics** $Q_{\phi_1}$ and
$Q_{\phi_2}$ with independent initializations, and forms the target from the
**minimum** of the two:

$$
y \;=\; r + \gamma\,\min_{i=1,2} Q_{\phi_i'}\!\big(s',\, \tilde a\big).
$$

Both critics are then regressed toward this same $y$. Taking the minimum is deliberately
pessimistic: where one critic has spiked high on noise, the other is unlikely to have
spiked in the same place, so the min discards the outlier. The bias does not vanish — it
is pushed to the *under*-estimating side — but a slight, consistent underestimate does
not compound the way an overestimate does, because the min never supplies an inflated
value for the actor to climb.

$$
% caption: Single vs twin critic under noise. A lone critic (left) is maximized at its
% noisy peak, so the target inherits the overshoot; twin critics (right) rarely spike
% together, and taking the elementwise minimum (lower envelope) clips the peak back
% toward the true value.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{grn}{HTML}{1F9D4D}
  % --- left: single critic, spiky, overestimates ---
  \draw[black, ->] (0,0) -- (3.5,0) node[anchor=north east, text=black] {action};
  \draw[black, ->] (0,0) -- (0,3.0) node[anchor=south east, text=black] {Q};
  \node[anchor=south, text=black] at (1.7,3.0) {single critic};
  \draw[grn, thick, dashed] plot[domain=0.15:3.2, samples=50] (\x, {1.4 + 0.35*\x});
  \node[grn, anchor=west, font=\scriptsize] at (2.5,2.0) {true};
  \draw[acc, very thick] plot[domain=0.15:3.2, samples=70]
    (\x, {1.4 + 0.35*\x + 0.75*exp(-((\x-2.0)^2)/0.12)});
  \fill[red] (2.0,2.85) circle (2pt);
  \node[red, anchor=west, font=\scriptsize] at (2.2,2.75) {overshoot};
  % --- right: twin critics, min clips ---
  \begin{scope}[xshift=5.6cm]
    \draw[black, ->] (0,0) -- (3.5,0) node[anchor=north east, text=black] {action};
    \draw[black, ->] (0,0) -- (0,3.0) node[anchor=south east, text=black] {Q};
    \node[anchor=south, text=black] at (1.7,3.0) {twin critics: min};
    \draw[grn, thick, dashed] plot[domain=0.15:3.2, samples=50] (\x, {1.4 + 0.35*\x});
    \draw[acc, thick] plot[domain=0.15:3.2, samples=70]
      (\x, {1.4 + 0.35*\x + 0.7*exp(-((\x-1.6)^2)/0.12)});
    \draw[red, thick] plot[domain=0.15:3.2, samples=70]
      (\x, {1.4 + 0.35*\x + 0.7*exp(-((\x-2.4)^2)/0.12)});
    % lower envelope (min) as thick black
    \draw[black, very thick] plot[domain=0.15:3.2, samples=90]
      (\x, {1.4 + 0.35*\x + 0.7*min(exp(-((\x-1.6)^2)/0.12), exp(-((\x-2.4)^2)/0.12))});
    \node[black, anchor=west, font=\scriptsize] at (1.4,1.35) {min};
  \end{scope}
\end{tikzpicture}
$$

For example, suppose the true value of the greedy target
action is $Q^\ast(s', \tilde a) = 10$, and each critic estimates it with independent
zero-mean noise. Say the two critics happen to read $Q_{\phi_1'} = 12$ (a $+2$
overshoot) and $Q_{\phi_2'} = 9$ (a $-1$ undershoot). A single-critic DDPG target
would use whichever critic the actor maximized — and since the actor was _trained_ to
maximize the critic, it is biased toward the one reading high, so the effective target
inflates toward $12$. TD3's target takes $\min(12, 9) = 9$, a slight underestimate of
$10$. Now iterate the bootstrap. An overestimate feeds forward: next step the target
built on $12$ pushes the critic to fit $\approx 12$, and the actor climbs the inflated
surface, so the error grows. An underestimate does not: the target built on $9$ pulls
the critic _down_, and the actor cannot exploit a value the min never inflates, so
the error stays bounded and self-corrects as more data arrives. Averaged over many
noisy reads, $\mathbb{E}[\min(X_1, X_2)] < \min(\mathbb{E}[X_1], \mathbb{E}[X_2])$ for
correlated-but-not-identical estimates — the min is a deliberately conservative
estimator, and conservative is safe here in a way optimistic is not.

### Delayed policy updates

The second fix addresses a coupling problem. The actor is trained on the critic, and the
critic is a moving target; if both change at the same rate, the actor chases a value
estimate that has not yet settled, and their errors feed each other. TD3 updates the
**policy less often than the critics** — typically one actor (and target) update for
every two critic updates. Letting the value estimates converge before they steer the
policy reduces the variance the actor accumulates from a still-thrashing critic.

$$
% caption: Delayed policy updates with delay d=2. The critics update every step; the
% actor and all target networks update once every two steps, after the critics have
% partly settled. The policy chases a steadier value estimate, so the coupled
% actor-critic loop accumulates less variance.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[black, ->] (0,0) -- (9.2,0) node[anchor=north, text=black] {step};
  \foreach \x/\n in {0.8/1, 2.0/2, 3.2/3, 4.4/4, 5.6/5, 6.8/6, 8.0/7} {
    \draw[black] (\x,-0.06) -- (\x,0.06);
    \node[anchor=north, font=\scriptsize, black] at (\x,-0.1) {\n};
    \draw[acc, line width=2.5pt] (\x,0.1) -- (\x,0.9);
  }
  \node[acc, anchor=west, font=\scriptsize] at (8.2,0.5) {critic};
  % actor updates every 2 steps: steps 2,4,6
  \foreach \x in {2.0, 4.4, 6.8} {
    \draw[red, line width=2.5pt] (\x,0.95) -- (\x,1.8);
  }
  \node[red, anchor=west, font=\scriptsize] at (8.2,1.4) {actor + targets};
\end{tikzpicture}
$$

### Target-policy smoothing

The third fix hardens the target against the actor exploiting sharp, spurious peaks in
the critic. A deterministic target action $\mu_{\theta'}(s')$ can land exactly on a noise
spike, and nothing prevents that spike from entering the target. TD3 **smooths** the target by
adding clipped noise to the target action, so the value is averaged over a small
neighborhood of actions rather than read off a single point:

$$
\tilde a \;=\; \mu_{\theta'}(s') + \epsilon,
\qquad
\epsilon \sim \clip\!\big(\mathcal{N}(0, \sigma),\, -c,\, c\big).
$$

This encodes a **regularizing** prior: similar actions should have similar values, so a
peak one clipped-noise step wide is treated as an artifact, not a real maximum. The clip
$[-c, c]$ keeps the perturbation local. Note this noise is on the *target* action inside
the Bellman update — distinct from the exploration noise added to the *behavior* action.

```algorithm
caption: $\textsc{TD3}$ — twin-delayed DDPG, three fixes over DDPG
input: actor $\mu_\theta$, twin critics $Q_{\phi_1}, Q_{\phi_2}$, delay $d$, noise $\sigma, c$
initialize targets $\theta', \phi_1', \phi_2'$; empty buffer $\mathcal{D}$; step counter $t \gets 0$
for each step do
  select $a \gets \mu_\theta(s) + \mathcal{N}$, execute, store $(s,a,r,s')$ in $\mathcal{D}$
  sample a minibatch from $\mathcal{D}$
  $\tilde a \gets \mu_{\theta'}(s') + \clip(\mathcal{N}(0,\sigma), -c, c)$ // target smoothing
  $y \gets r + \gamma\,\min_{i=1,2} Q_{\phi_i'}(s', \tilde a)$ // clipped double-Q
  for $i = 1, 2$ do
    $\phi_i \gets \phi_i - \eta\,\nabla_{\phi_i}\,(Q_{\phi_i}(s,a) - y)^2$
  if $t \bmod d = 0$ then // delayed policy update
    $\theta \gets \theta + \eta\,\nabla_\theta\, Q_{\phi_1}(s, \mu_\theta(s))$
    $\phi_i' \gets \tau\phi_i + (1-\tau)\phi_i'$ for $i = 1, 2$
    $\theta' \gets \tau\theta + (1-\tau)\theta'$
  $t \gets t + 1$
```

TD3 keeps DDPG's deterministic actor and off-policy replay, adds no new objective, and
turns a brittle method into a reliable one. On the MuJoCo continuous-control benchmarks
it substantially outperforms DDPG, and the three fixes are cheap enough that TD3 is often
the first thing to try when DDPG misbehaves.

## Where this leaves us

DDPG and TD3 share one skeleton: an off-policy Q-critic trained from a replay buffer,
a deterministic actor trained to output the critic-maximizing action (the escape from
$\arg\max_a Q$), and slow target networks to keep the bootstrap from diverging. DDPG
proves the idea works; TD3 makes it reliable by refusing to trust a single noisy
critic — twin critics and a minimum target, delayed policy updates, and target
smoothing.

Both actors are **deterministic**, so exploration has to be bolted on as external
action noise. The next method rejects that: SAC learns a genuinely stochastic policy,
folds exploration into the objective through an entropy bonus, and turns
out to be the more robust choice. That maximum-entropy objective, the reparameterized
squashed-Gaussian actor, automatic temperature tuning, and the distributional-critic /
ensemble / pixel extensions built on this template continue in
[Continuous Control: SAC and Beyond](/reinforcement-learning/modern-deep-rl/continuous-control-part-2).

[^dpg]: **Silver et al. (2014)**, "Deterministic Policy Gradient Algorithms," _ICML_ — the deterministic policy gradient theorem, showing that for a deterministic policy $\mu_\theta$ the gradient of performance reduces to $\mathbb{E}[\nabla_\theta \mu_\theta(s)\,\nabla_a Q^\mu(s,a)|_{a=\mu_\theta(s)}]$, a lower-variance estimator than the stochastic score-function form; also **Morales**, _Grokking Deep Reinforcement Learning_, Ch. 12.
[^ddpg]: **Lillicrap et al. (2016)**, "Continuous Control with Deep Reinforcement Learning," _ICLR_ — DDPG: the deterministic policy gradient made deep and off-policy with a replay buffer, actor and critic target networks updated by soft (Polyak) averaging $\theta' \gets \tau\theta + (1-\tau)\theta'$, and Ornstein–Uhlenbeck exploration noise; **Morales**, Ch. 12, "DDPG: Approximating a deterministic policy."
[^td3]: **Fujimoto et al. (2018)**, "Addressing Function Approximation Error in Actor-Critic Methods," _ICML_ — TD3: clipped double-Q learning (twin critics, minimum target), delayed policy and target updates, and target-policy smoothing via clipped noise on the target action, together correcting DDPG's overestimation bias on the MuJoCo benchmarks; **Morales**, Ch. 12, "TD3: State-of-the-art improvements over DDPG."