Continuous Control: SAC and Beyond

This builds on Continuous Control: DDPG and TD3, which built the off-policy actor-critic template — a Q-critic trained from a replay buffer and a deterministic actor trained to output the critic-maximizing action, sidestepping the intractable $\arg {\max }_{a}Q$ of continuous-action Q-learning. DDPG established the idea; TD3 made it reliable with twin critics, delayed policy updates, and target smoothing.

Both of those actors are deterministic, so exploration had to be added from outside as action noise. This lesson takes up the third member of the family, which rejects that design, and then the methods that build on the shared template.

SAC: the maximum-entropy objective

TD3 fights instability by tightening the value estimate. Soft actor-critic (SAC) takes a different route: it changes the objective itself, adding a term that rewards the policy for staying random.¹ Standard RL maximizes expected return; SAC maximizes expected return plus the entropy of the policy at every visited state,

The entropy $\mathcal{H}(\pi )$ is largest when the policy spreads its probability mass widely and smallest when it collapses onto one action. The temperature $\alpha \ge 0$ sets the trade-off between the two terms: at $\alpha \to 0$ the objective reverts to ordinary reward maximization, and as $\alpha$ grows the entropy term weighs more heavily and the policy stays more random. Unlike A2C's entropy bonus — a term tacked onto the loss to discourage premature collapse — this entropy lives inside the value function, so it shapes not just the immediate action but the long-run bootstrap.

SAC learns a stochastic actor — typically a squashed Gaussian, $a=\tanh ({\mu }_{\theta }(s)+{\sigma }_{\theta }(s)\odot \xi )$ with $\xi \sim \mathcal{N}(0,I)$ — and, borrowing from TD3, twin critics whose minimum forms the target. Exploration is now intrinsic: because the policy is genuinely stochastic and is rewarded for staying so, it explores on its own, and there is no external noise process to design. The actor is improved not by the deterministic policy gradient but by the reparameterized gradient of the soft objective, minimizing the KL between the policy and the exponentiated soft-Q:

where $a_{\theta }(s,\xi )$ is the reparameterized sample, differentiable in $\theta$.

The reparameterization trick and the squashed Gaussian

The actor gradient above hides a subtlety: it is what keeps SAC's stochastic actor trainable. The naive way to differentiate ${\mathbb{E}}_{a\sim {\pi }_{\theta }}[\cdot ]$ with respect to $\theta$ is the score-function estimator — the same ${\nabla }_{\theta }\ln {\pi }_{\theta }$ trick behind REINFORCE — but its variance is high, exactly the problem the deterministic policy gradient was introduced to escape. SAC keeps a stochastic policy and a low-variance gradient by reparameterizing the sample: instead of drawing $a\sim \mathcal{N}({\mu }_{\theta },{\sigma }_{\theta }^2)$ directly, draw a fixed-distribution noise $\xi \sim \mathcal{N}(0,I)$ and compute

Now the randomness lives in $\xi$, which does not depend on $\theta$, so the whole expression is a deterministic differentiable function of $\theta$ for each fixed $\xi$. The gradient flows straight through ${\mu }_{\theta }$ and ${\sigma }_{\theta }$ by the chain rule — the same one backward pass that trained the DDPG actor — and the expectation over $\xi$ is estimated by sampling. This is why the SAC actor update reads like a deterministic gradient of $\alpha \ln \pi -Q$ despite the policy being genuinely random.

The outer $\tanh$ is the squash: it maps the unbounded Gaussian sample into $(-1,1)$ so the action respects the environment's bounded action range (torque limits, steering stops). It is not free — squashing changes the density, so the log-probability needs a change-of-variables correction. For the pre-squash sample $u=\mu +\sigma \xi$ with density ${\mu }_u(u)$ and $a=\tanh (u)$, the corrected log-density is

where the subtracted term is $\ln |\det (\partial a/\partial u)|$, the Jacobian of the $\tanh$. Without it the entropy term $\alpha \ln \pi$ in every update would be wrong, and the temperature tuning below would chase a mismeasured entropy. The correction is a few lines of code, but it is essential.

Automatic temperature tuning

The one awkward hyperparameter is $\alpha$: too high and the agent acts near-randomly and never exploits; too low and it collapses to a deterministic policy and stops exploring. Worse, the right value drifts during training — early on more entropy helps, later less does. The refinement is to tune $\alpha$ automatically by making it satisfy an entropy constraint. Fix a target entropy $\bar{\mathcal{H}}$ (a common default is $-\dim (\mathcal{A})$, one nat of negative entropy per action dimension) and adjust $\alpha$ by gradient descent on

When the policy's entropy sits below the target, this pushes $\alpha$ up (more exploration); when it sits above, it pushes $\alpha$ down (more weight on reward). The temperature becomes a learned control that holds the policy at a chosen level of randomness throughout training, removing the single most finicky knob.

For example, take a $2$-dimensional action space, so the default target entropy is $\bar{\mathcal{H}}=-\dim (\mathcal{A})=-2$ nats. Suppose at the current step the sampled action has log-probability $\ln {\pi }_{\theta }(\hat{a}\mid s)=-0.5$, so the policy's per-sample entropy estimate is $-\ln \pi =+0.5$ nats. The objective for $\alpha$ is $J(\alpha )=-\alpha (\ln \pi +\bar{\mathcal{H}})$, and its gradient is

The gradient is positive, so a descent step $\alpha \leftarrow \alpha -\eta {\nabla }_{\alpha }J$ decreases $\alpha$. That is the correct direction: the policy's entropy ($+0.5$ per sample) is above the target ($-2$ would correspond to a much more concentrated policy), so the constraint is satisfied with slack and the update shifts weight from entropy toward reward. With $\alpha =0.20$ and learning rate $\eta =0.01$, the new temperature is $0.20-0.01\cdot 2.5=0.175$. Flip the scenario: if a mastered state gave $\ln \pi =-3.0$ (entropy $+3.0$, still above target) the gradient would be $-(-3.0-2)=5.0$ and $\alpha$ would fall faster; and if the policy collapsed so that $\ln \pi =+1.5$ (entropy $-1.5$, now below the target $-2$ in the sense that $-1.5+(-2)=-3.5<0$), the gradient would be $+3.5$, pushing $\alpha$ up to restore exploration. The temperature tracks the entropy constraint automatically, tightening when the policy over-explores and loosening when it collapses.

The entropy term does more than sustain exploration. Because the agent is rewarded for keeping several actions viable, it does not commit to a single narrow solution that a small change in dynamics would break, which gives SAC its robustness — it transfers better across perturbations and tolerates imperfect models. Together with strong sample efficiency from the off-policy buffer, that is why SAC, alongside TD3, is a default choice for continuous control.

Beyond the three: distributional critics, ensembles, and pixels

DDPG, TD3, and SAC set the template, and the work after them mostly pushes on two levers the template exposes: make the critic better, and make the whole loop work from images.

Distributional critics. The critic in all three methods predicts a scalar $Q(s,a)$. Replacing it with a distributional critic — the same return-distribution idea that powered Rainbow — sharpens the value signal. D4PG (Barth-Maron et al., 2018) is TD3-era DDPG with a C51-style categorical critic, $n$-step returns, and distributed data collection; the distributional target measurably improved stability and final performance on hard continuous-control tasks.² This carries the distributional lesson's point into continuous control: an agent that models the spread of returns, not just the mean, both learns a better mean and can act risk-sensitively when the spread matters.

Ensembled critics and high replay ratios. TD3's twin critics are the smallest possible ensemble. REDQ (Chen et al., 2021) scales that idea: keep an ensemble of $N$ critics (say $10$), form each target from a random subset of them (a random pair), and — the decisive move — perform many gradient updates per environment step (a high update-to-data ratio, e.g. $20$).³ The ensemble controls the overestimation that a high replay ratio would otherwise amplify, and the result matches the sample efficiency of model-based methods like PETS while staying fully model-free. It is a direct answer to the sample-efficiency case made in the model-based lesson: much of model-based sample efficiency can be had with a bigger critic ensemble and more updates, without ever learning a dynamics model.

Control from pixels. DDPG/TD3/SAC as stated assume a compact state vector. Running them from raw images was long thought to require a separate representation-learning stage, but two 2020–2021 results showed that plain image augmentation is enough. RAD (Laskin et al., 2020) and DrQ (Kostrikov et al., 2021) apply small random shifts and crops to the input frames and average the $Q$-target over a few augmented copies, and with nothing more than that, SAC from pixels reaches the sample efficiency of SAC from states on the DeepMind Control suite.⁴ The augmentation regularizes the critic — it encodes the prior that a two-pixel shift should not change the value — and that single inductive bias closed most of the pixels-to-states gap that had made image-based continuous control seem to need elaborate machinery.

Comparing the three

All three share the off-policy, replay-buffer, twin-or-single-critic skeleton and differ in the policy and the exploration mechanism.

The Q-critic gives off-policy sample efficiency; the learned actor replaces the intractable $\arg {\max }_{a}Q$; and the three methods disagree only on how to keep the coupled learning stable and how to explore. DDPG proves the idea works, TD3 makes it reliable by refusing to trust a single critic, and SAC makes it robust by rewarding entropy. When a modern robotics or MuJoCo agent learns continuous control off-policy, one of these three — most often TD3 or SAC — is almost certainly the algorithm underneath.