Continuous Control: DDPG and TD3

Deep Q-networks act by computing $\arg {\max }_a{Q}_{\varphi }(s,a)$ — they enumerate the action values in a state and take the largest. That works when the action set is a handful of discrete choices; it fails when actions are continuous. A robot arm commands a torque on each of seven joints; a quadruped commands a real vector; a car commands a steering angle and a throttle. The maximization ${\max }_a{Q}_{\varphi }(s,a)$ over ${\mathbb{R}}^k$ is then an inner optimization problem, solved from scratch on every environment step — intractable in any practical loop.

The policy-gradient methods lesson already gave one escape: parameterize a policy directly and it can output a Gaussian over continuous actions, no max required. The actor-critic and PPO lesson made that policy a deep network and toured this continuous-control family in a single closing section. This lesson and its companion treat it properly. The three methods in this family — DDPG, TD3, and SAC — share one structural idea: keep a Q-critic $Q_{\varphi }(s,a)$, but replace its intractable inner max with a learned actor that produces the maximizing action directly. Because the critic is a Q-function, all three learn off-policy from a replay buffer, which is what makes them far more sample-efficient than on-policy PPO on the control benchmarks they dominate. Here we build the first two — DDPG and the overestimation-fixing TD3 — and leave SAC and the methods layered on this template to the companion.

The deterministic policy gradient

Policy-gradient methods so far learned a stochastic policy $\pi (a\mid s)$ and climbed the expectation $\nabla J={\mathbb{E}}_{\pi }[{\Psi }_t\nabla \ln \pi ]$. The score-function estimator behind it averages over sampled actions, and its variance grows with the dimension of the action space — a nuisance in high-dimensional control. Silver et al. (2014) showed that if the policy is deterministic, written ${\mu }_{\theta }:\mathcal{S}\to \mathcal{A}$, the policy gradient takes a simpler and lower-variance form.¹ A deterministic actor commits to one action per state, $a={\mu }_{\theta }(s)$, and the performance objective is the expected value of that action under the critic,

where $\rho$ is the state distribution induced by whatever policy fills the replay buffer. Because the action is a deterministic function of $\theta$, the gradient flows through it by the chain rule — no score-function trick, no integral over actions:

Read the two factors. ${\nabla }_a{Q}_{\varphi }(s,a)$ is the direction in action space that raises the critic's value; ${\nabla }_{\theta }{\mu }_{\theta }(s)$ translates that change in action into a change in the actor's weights. The composition is one backpropagation: differentiate the critic with respect to its action input, then continue backward through the actor. This composition escapes the $\arg {\max }_{a}Q$ — instead of searching the action space, the actor is trained by gradient ascent to output the maximizer, and the critic's action-gradient is the training signal.

DDPG: a DQN for continuous actions

Deep deterministic policy gradient (DDPG) is the deterministic policy gradient made deep and off-policy, assembled from the same parts that stabilized DQN.² It runs four networks: an actor ${\mu }_{\theta }(s)$, a critic $Q_{\varphi }(s,a)$, and a slow-moving copy of each — the target actor ${\mu }_{{\theta }^{\prime }}$ and target critic $Q_{{\varphi }^{\prime }}$ — used only to form the learning target. Transitions $(s,a,r,s^{\prime })$ collected by the agent are stored in a replay buffer $\mathcal{D}$ and sampled in minibatches, exactly as in DQN, which decorrelates the updates and lets each transition train the networks many times.

The critic is trained by regression toward a bootstrapped target, the continuous-action analog of the DQN target. Where DQN uses ${\max }_{a^{\prime }}{Q}_{{\varphi }^{\prime }}(s^{\prime },a^{\prime })$, DDPG replaces the max with the target actor's action:

The actor is trained by the deterministic policy gradient — ascend the critic through the actor — sampling states from the same buffer:

Target networks by soft update

DQN copies the online weights into the target network in a hard jump every few thousand steps. DDPG instead updates the targets softly on every step, blending a small fraction $\tau \ll 1$ of the online weights in:

The targets therefore track the online networks slowly and smoothly. This keeps the bootstrapped target $y$ stable: the critic regresses toward a value produced by weights that change far more slowly than its own, damping the feedback loop that would otherwise diverge.

Exploration for a deterministic actor

A deterministic policy has no built-in exploration — asked for an action in a state, it returns the same one every time. DDPG explores by adding noise to the action it actually sends to the environment, while keeping the clean ${\mu }_{\theta }(s)$ for learning:

with ${\mathcal{N}}_t$ a zero-mean noise process (the original paper used temporally correlated Ornstein–Uhlenbeck noise; uncorrelated Gaussian noise works about as well in practice). Exploration is thus bolted on externally, a design choice SAC will later reject.

DDPG solved control problems DQN could not touch, but it earned a reputation for being brittle: sensitive to hyperparameters, prone to sudden collapse, and — the central diagnosis of the next method — systematically overestimating its own values.

TD3: three fixes for overestimation

The failure mode DDPG inherits from Q-learning is overestimation bias. The critic is noisy, and the actor is trained to maximize it. Any state where the critic happens to overshoot the true value becomes a target the actor climbs toward, and the bootstrap then propagates that inflated value backward through the Bellman update. Errors that should average out instead compound, because the maximization systematically selects for them. Twin-delayed DDPG (TD3) is DDPG plus three targeted fixes for this.³

Clipped double-Q learning

The main fix is the same idea as double Q-learning: do not let a single noisy critic evaluate its own maximizing action. TD3 learns twin critics $Q_{{\varphi }_1}$ and $Q_{{\varphi }_2}$ with independent initializations, and forms the target from the minimum of the two:

Both critics are then regressed toward this same $y$. Taking the minimum is deliberately pessimistic: where one critic has spiked high on noise, the other is unlikely to have spiked in the same place, so the min discards the outlier. The bias does not vanish — it is pushed to the under-estimating side — but a slight, consistent underestimate does not compound the way an overestimate does, because the min never supplies an inflated value for the actor to climb.

For example, suppose the true value of the greedy target action is $Q^{\ast }(s^{\prime },\tilde{a})=10$, and each critic estimates it with independent zero-mean noise. Say the two critics happen to read $Q_{{\varphi }_1^{\prime }}=12$ (a $+2$ overshoot) and $Q_{{\varphi }_2^{\prime }}=9$ (a $-1$ undershoot). A single-critic DDPG target would use whichever critic the actor maximized — and since the actor was trained to maximize the critic, it is biased toward the one reading high, so the effective target inflates toward $12$. TD3's target takes $\min (12,9)=9$, a slight underestimate of $10$. Now iterate the bootstrap. An overestimate feeds forward: next step the target built on $12$ pushes the critic to fit $\approx 12$, and the actor climbs the inflated surface, so the error grows. An underestimate does not: the target built on $9$ pulls the critic down, and the actor cannot exploit a value the min never inflates, so the error stays bounded and self-corrects as more data arrives. Averaged over many noisy reads, $\mathbb{E}[\min (X_1,X_2)]<\min (\mathbb{E}[X_1],\mathbb{E}[X_2])$ for correlated-but-not-identical estimates — the min is a deliberately conservative estimator, and conservative is safe here in a way optimistic is not.

Delayed policy updates

The second fix addresses a coupling problem. The actor is trained on the critic, and the critic is a moving target; if both change at the same rate, the actor chases a value estimate that has not yet settled, and their errors feed each other. TD3 updates the policy less often than the critics — typically one actor (and target) update for every two critic updates. Letting the value estimates converge before they steer the policy reduces the variance the actor accumulates from a still-thrashing critic.

Target-policy smoothing

The third fix hardens the target against the actor exploiting sharp, spurious peaks in the critic. A deterministic target action ${\mu }_{{\theta }^{\prime }}(s^{\prime })$ can land exactly on a noise spike, and nothing prevents that spike from entering the target. TD3 smooths the target by adding clipped noise to the target action, so the value is averaged over a small neighborhood of actions rather than read off a single point:

This encodes a regularizing prior: similar actions should have similar values, so a peak one clipped-noise step wide is treated as an artifact, not a real maximum. The clip $[-c,c]$ keeps the perturbation local. Note this noise is on the target action inside the Bellman update — distinct from the exploration noise added to the behavior action.

TD3 keeps DDPG's deterministic actor and off-policy replay, adds no new objective, and turns a brittle method into a reliable one. On the MuJoCo continuous-control benchmarks it substantially outperforms DDPG, and the three fixes are cheap enough that TD3 is often the first thing to try when DDPG misbehaves.

Where this leaves us

DDPG and TD3 share one skeleton: an off-policy Q-critic trained from a replay buffer, a deterministic actor trained to output the critic-maximizing action (the escape from $\arg {\max }_{a}Q$), and slow target networks to keep the bootstrap from diverging. DDPG proves the idea works; TD3 makes it reliable by refusing to trust a single noisy critic — twin critics and a minimum target, delayed policy updates, and target smoothing.

Both actors are deterministic, so exploration has to be bolted on as external action noise. The next method rejects that: SAC learns a genuinely stochastic policy, folds exploration into the objective through an entropy bonus, and turns out to be the more robust choice. That maximum-entropy objective, the reparameterized squashed-Gaussian actor, automatic temperature tuning, and the distributional-critic / ensemble / pixel extensions built on this template continue in Continuous Control: SAC and Beyond.