--- title: "Continuous Control: SAC and Beyond" module: Modern Deep Reinforcement Learning moduleNumber: 5 lessonNumber: 4 order: 504 summary: > A companion to the DDPG and TD3 lesson. Where those actors are deterministic and explore with bolted-on noise, soft actor-critic (SAC) changes the objective itself: maximize return plus the entropy of the policy, so exploration becomes intrinsic and the agent stays robust. We develop the maximum-entropy objective, the reparameterized squashed-Gaussian actor, and automatic temperature tuning, then survey the methods built on this off-policy template — distributional critics (D4PG), critic ensembles (REDQ), and control from pixels (DrQ, RAD). topics: [Deep RL] sources: - book: Grokking Deep RL ref: "Ch. 12 — Advanced actor-critic methods; DDPG, TD3, SAC" - book: Sutton & Barto ref: "Ch. 13 — Policy Gradient Methods; §13.7 Policy Parameterization for Continuous Actions" --- This builds on [Continuous Control: DDPG and TD3](/reinforcement-learning/modern-deep-rl/continuous-control), which built the off-policy actor-critic template — a Q-critic trained from a replay buffer and a **deterministic** actor trained to output the critic-maximizing action, sidestepping the intractable $\arg\max_a Q$ of continuous-action Q-learning. DDPG established the idea; TD3 made it reliable with twin critics, delayed policy updates, and target smoothing. Both of those actors are deterministic, so exploration had to be added from outside as action noise. This lesson takes up the third member of the family, which rejects that design, and then the methods that build on the shared template. ## SAC: the maximum-entropy objective TD3 fights instability by tightening the value estimate. **Soft actor-critic** (SAC) takes a different route: it changes the objective itself, adding a term that rewards the policy for staying **random**.[^sac] Standard RL maximizes expected return; SAC maximizes expected return **plus** the entropy of the policy at every visited state, $$ J(\pi) \;=\; \mathbb{E}_{\pi}\!\left[\sum_{t} \gamma^t \Big(r(s_t, a_t) + \alpha\,\mathcal{H}\big(\pi(\cdot \mid s_t)\big)\Big)\right], \qquad \mathcal{H}(\pi(\cdot \mid s)) = \mathbb{E}_{a \sim \pi}[-\ln \pi(a \mid s)]. $$ The entropy $\mathcal{H}(\pi)$ is largest when the policy spreads its probability mass widely and smallest when it collapses onto one action. The **temperature** $\alpha \ge 0$ sets the trade-off between the two terms: at $\alpha \to 0$ the objective reverts to ordinary reward maximization, and as $\alpha$ grows the entropy term weighs more heavily and the policy stays more random. Unlike A2C's entropy _bonus_ — a term tacked onto the loss to discourage premature collapse — this entropy lives **inside** the value function, so it shapes not just the immediate action but the long-run bootstrap. > **Definition (Soft value functions).** Under the maximum-entropy objective the state > value absorbs the future entropy stream, and the soft Q-target adds the entropy of the > next action: > $$ > y \;=\; r + \gamma\,\mathbb{E}_{a' \sim \pi}\!\big[\,Q_{\phi'}(s', a') - \alpha \ln \pi(a' \mid s')\,\big]. > $$ > The $-\alpha \ln \pi$ term is the per-step entropy reward: actions the policy is > confident about (high $\ln\pi$) are penalized, pushing the policy to keep alternatives > alive. SAC learns a **stochastic** actor — typically a squashed Gaussian, $a = \tanh(\mu_\theta(s) + \sigma_\theta(s)\odot\xi)$ with $\xi \sim \mathcal{N}(0, I)$ — and, borrowing from TD3, twin critics whose minimum forms the target. Exploration is now **intrinsic**: because the policy is genuinely stochastic and is rewarded for staying so, it explores on its own, and there is no external noise process to design. The actor is improved not by the deterministic policy gradient but by the **reparameterized** gradient of the soft objective, minimizing the KL between the policy and the exponentiated soft-Q: $$ \nabla_\theta J_\pi \;=\; \nabla_\theta\,\mathbb{E}_{s \sim \mathcal{D},\,\xi}\big[\,\alpha \ln \pi_\theta(a_\theta(s,\xi) \mid s) - Q_\phi(s, a_\theta(s,\xi))\,\big], $$ where $a_\theta(s,\xi)$ is the reparameterized sample, differentiable in $\theta$. $$ % caption: The maximum-entropy objective. SAC maximizes reward plus $\alpha$ times the % policy entropy at every state; a high-entropy (spread-out) policy earns a bonus, so % the actor keeps several actions plausible instead of collapsing onto one. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % --- left: peaked, low entropy --- \draw[black, ->] (-1.6,0) -- (1.9,0) node[anchor=north east, text=black] {a}; \draw[acc, very thick] plot[domain=-1.55:1.85, samples=50] (\x, {2.7*exp(-((\x-0.15)^2)/0.09)}); \node[anchor=south, text=black, font=\scriptsize] at (0.15,2.55) {low H (collapsed)}; \node[anchor=north, text=red, font=\scriptsize] at (0.15,-0.15) {small bonus}; % arrow between \draw[->, black, very thick] (2.5,1.2) -- (3.9,1.2); \node[anchor=south, text=black, font=\scriptsize] at (3.2,1.25) {+ alpha H}; % --- right: spread, high entropy --- \begin{scope}[xshift=6.2cm] \draw[black, ->] (-1.9,0) -- (2.1,0) node[anchor=north east, text=black] {a}; \draw[acc, very thick] plot[domain=-1.85:2.05, samples=60] (\x, {1.35*exp(-((\x-0.1)^2)/0.9)}); \node[anchor=south, text=black, font=\scriptsize] at (0.1,1.5) {high H (spread)}; \node[anchor=north, text=red, font=\scriptsize] at (0.1,-0.15) {large bonus}; \end{scope} \end{tikzpicture} $$ ### The reparameterization trick and the squashed Gaussian The actor gradient above hides a subtlety: it is what keeps SAC's stochastic actor trainable. The naive way to differentiate $\mathbb{E}_{a \sim \pi_\theta}[\,\cdot\,]$ with respect to $\theta$ is the score-function estimator — the same $\nabla_\theta \ln\pi_\theta$ trick behind REINFORCE — but its variance is high, exactly the problem the deterministic policy gradient was introduced to escape. SAC keeps a stochastic policy _and_ a low-variance gradient by **reparameterizing** the sample: instead of drawing $a \sim \mathcal{N}(\mu_\theta, \sigma_\theta^2)$ directly, draw a fixed-distribution noise $\xi \sim \mathcal{N}(0, I)$ and _compute_ $$ a_\theta(s, \xi) = \tanh\!\bigl(\mu_\theta(s) + \sigma_\theta(s) \odot \xi\bigr). $$ Now the randomness lives in $\xi$, which does not depend on $\theta$, so the whole expression is a deterministic differentiable function of $\theta$ for each fixed $\xi$. The gradient flows straight through $\mu_\theta$ and $\sigma_\theta$ by the chain rule — the same one backward pass that trained the DDPG actor — and the expectation over $\xi$ is estimated by sampling. This is why the SAC actor update reads like a deterministic gradient of $\alpha\ln\pi - Q$ despite the policy being genuinely random. The outer $\tanh$ is the **squash**: it maps the unbounded Gaussian sample into $(-1, 1)$ so the action respects the environment's bounded action range (torque limits, steering stops). It is not free — squashing changes the density, so the log-probability needs a change-of-variables correction. For the pre-squash sample $u = \mu + \sigma\xi$ with density $\mu_u(u)$ and $a = \tanh(u)$, the corrected log-density is $$ \ln\pi(a \mid s) = \ln\mu_u(u \mid s) - \sum_{i} \ln\!\bigl(1 - \tanh^2(u_i)\bigr), $$ where the subtracted term is $\ln|\det(\partial a / \partial u)|$, the Jacobian of the $\tanh$. Without it the entropy term $\alpha\ln\pi$ in every update would be wrong, and the temperature tuning below would chase a mismeasured entropy. The correction is a few lines of code, but it is essential. $$ % caption: The reparameterized squashed-Gaussian actor. Fixed noise xi enters, the % network produces a mean and spread, their combination is squashed by tanh into the % bounded action range; because the randomness (xi) is independent of theta, gradients % pass straight through mu and sigma in one backward pass. A tanh Jacobian term % corrects the log-probability. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=20mm, minimum height=10mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[box] (xi) at (0,-1.3) {noise xi\\N(0, I)}; \node[box, draw=acc, text=acc] (net) at (0,0.6) {net: mu(s), sig(s)}; \node[box] (comb) at (3.8,-0.35) {u = mu + sig xi}; \node[box, draw=acc, text=acc, thick] (sq) at (7.6,-0.35) {a = tanh(u)}; \node[box, draw=red, text=red] (jac) at (7.6,-2.4) {log-prob\\+ tanh correction}; \draw[->, black] (xi) -- (comb); \draw[->, acc, thick] (net) -- (comb); \draw[->, acc, thick] (comb) -- (sq) node[midway, above, font=\scriptsize, text=black] {squash}; \draw[->, red, thick] (sq) -- (jac); \node[anchor=west, font=\scriptsize, text=black] at (8.7,-0.35) {bounded action}; \end{tikzpicture} $$ ### Automatic temperature tuning The one awkward hyperparameter is $\alpha$: too high and the agent acts near-randomly and never exploits; too low and it collapses to a deterministic policy and stops exploring. Worse, the right value drifts during training — early on more entropy helps, later less does. The refinement is to **tune $\alpha$ automatically** by making it satisfy an entropy constraint. Fix a target entropy $\bar{\mathcal{H}}$ (a common default is $-\dim(\mathcal{A})$, one nat of negative entropy per action dimension) and adjust $\alpha$ by gradient descent on $$ J(\alpha) \;=\; \mathbb{E}_{a \sim \pi}\big[-\alpha\,\ln \pi(a \mid s) - \alpha\,\bar{\mathcal{H}}\big]. $$ When the policy's entropy sits below the target, this pushes $\alpha$ up (more exploration); when it sits above, it pushes $\alpha$ down (more weight on reward). The temperature becomes a learned control that holds the policy at a chosen level of randomness throughout training, removing the single most finicky knob. For example, take a $2$-dimensional action space, so the default target entropy is $\bar{\mathcal{H}} = -\dim(\mathcal{A}) = -2$ nats. Suppose at the current step the sampled action has log-probability $\ln \pi_\theta(\hat a \mid s) = -0.5$, so the policy's per-sample entropy estimate is $-\ln\pi = +0.5$ nats. The objective for $\alpha$ is $J(\alpha) = -\alpha(\ln\pi + \bar{\mathcal H})$, and its gradient is $$ \nabla_\alpha J(\alpha) = -\bigl(\ln\pi_\theta(\hat a \mid s) + \bar{\mathcal H}\bigr) = -(-0.5 + (-2)) = 2.5. $$ The gradient is positive, so a descent step $\alpha \gets \alpha - \eta\,\nabla_\alpha J$ _decreases_ $\alpha$. That is the correct direction: the policy's entropy ($+0.5$ per sample) is _above_ the target ($-2$ would correspond to a much more concentrated policy), so the constraint is satisfied with slack and the update shifts weight from entropy toward reward. With $\alpha = 0.20$ and learning rate $\eta = 0.01$, the new temperature is $0.20 - 0.01 \cdot 2.5 = 0.175$. Flip the scenario: if a mastered state gave $\ln\pi = -3.0$ (entropy $+3.0$, still above target) the gradient would be $-(-3.0 - 2) = 5.0$ and $\alpha$ would fall faster; and if the policy collapsed so that $\ln\pi = +1.5$ (entropy $-1.5$, now _below_ the target $-2$ in the sense that $-1.5 + (-2) = -3.5 < 0$), the gradient would be $+3.5$, pushing $\alpha$ _up_ to restore exploration. The temperature tracks the entropy constraint automatically, tightening when the policy over-explores and loosening when it collapses. $$ % caption: Automatic temperature control as a feedback loop. The policy entropy is % measured against the target H-bar; if entropy is too high the update lowers alpha % (spend slack on reward), if too low it raises alpha (buy exploration). The % temperature settles where the policy holds the chosen entropy. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=26mm, minimum height=11mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[box] (ent) at (0,0) {measure entropy H}; \node[box, draw=acc, text=acc] (cmp) at (5.0,0) {compare to H-bar}; \node[box, draw=red, text=red, thick] (upd) at (10.0,0) {adjust alpha}; \draw[->, acc, thick] (ent) -- (cmp); \draw[->, red, thick] (cmp) -- (upd) node[midway, above, font=\scriptsize, text=acc] {gap}; % feedback: alpha changes policy, changes entropy \draw[->, black, thick] (upd.south) .. controls (10.0,-1.9) and (0,-1.9) .. (ent.south) node[midway, below, font=\scriptsize, text=black] {new alpha reshapes the policy}; \node[anchor=south, font=\scriptsize, text=black] at (2.5,0.65) {H too high: lower alpha}; \node[anchor=north, font=\scriptsize, text=black] at (2.5,-0.65) {H too low: raise alpha}; \end{tikzpicture} $$ ```algorithm caption: $\textsc{SAC}$ — soft (maximum-entropy) actor-critic input: stochastic actor $\pi_\theta$, twin critics $Q_{\phi_1}, Q_{\phi_2}$, target entropy $\bar{\mathcal{H}}$ initialize targets $\phi_1', \phi_2'$; empty buffer $\mathcal{D}$ for each step do sample $a \sim \pi_\theta(\cdot \mid s)$, execute, store $(s,a,r,s')$ in $\mathcal{D}$ sample a minibatch from $\mathcal{D}$ $a' \sim \pi_\theta(\cdot \mid s')$ $y \gets r + \gamma\,[\min_{i} Q_{\phi_i'}(s', a') - \alpha \ln \pi_\theta(a' \mid s')]$ // soft target for $i = 1, 2$ do $\phi_i \gets \phi_i - \eta\,\nabla_{\phi_i}\,(Q_{\phi_i}(s,a) - y)^2$ $\theta \gets \theta - \eta\,\nabla_\theta\,[\alpha \ln \pi_\theta(\hat a \mid s) - \min_i Q_{\phi_i}(s, \hat a)]$ // reparam $\hat a$ $\alpha \gets \alpha - \eta\,\nabla_\alpha\,[-\alpha (\ln \pi_\theta(\hat a \mid s) + \bar{\mathcal{H}})]$ // tune temperature $\phi_i' \gets \tau\phi_i + (1-\tau)\phi_i'$ for $i = 1, 2$ ``` The entropy term does more than sustain exploration. Because the agent is rewarded for keeping several actions viable, it does not commit to a single narrow solution that a small change in dynamics would break, which gives SAC its **robustness** — it transfers better across perturbations and tolerates imperfect models. Together with strong sample efficiency from the off-policy buffer, that is why SAC, alongside TD3, is a default choice for continuous control. ## Beyond the three: distributional critics, ensembles, and pixels DDPG, TD3, and SAC set the template, and the work after them mostly pushes on two levers the template exposes: make the critic better, and make the whole loop work from images. **Distributional critics.** The critic in all three methods predicts a scalar $Q(s,a)$. Replacing it with a _distributional_ critic — the same [return-distribution idea](/reinforcement-learning/modern-deep-rl/distributional-and-rainbow) that powered Rainbow — sharpens the value signal. **D4PG** (Barth-Maron et al., 2018) is TD3-era DDPG with a C51-style categorical critic, $n$-step returns, and distributed data collection; the distributional target measurably improved stability and final performance on hard continuous-control tasks.[^d4pg] This carries the distributional lesson's point into continuous control: an agent that models the _spread_ of returns, not just the mean, both learns a better mean and can act risk-sensitively when the spread matters. **Ensembled critics and high replay ratios.** TD3's twin critics are the smallest possible ensemble. **REDQ** (Chen et al., 2021) scales that idea: keep an ensemble of $N$ critics (say $10$), form each target from a random subset of them (a random pair), and — the decisive move — perform _many_ gradient updates per environment step (a high update-to-data ratio, e.g. $20$).[^redq] The ensemble controls the overestimation that a high replay ratio would otherwise amplify, and the result matches the _sample efficiency of model-based methods_ like PETS while staying fully model-free. It is a direct answer to the sample-efficiency case made in the [model-based lesson](/reinforcement-learning/modern-deep-rl/model-based-rl): much of model-based sample efficiency can be had with a bigger critic ensemble and more updates, without ever learning a dynamics model. **Control from pixels.** DDPG/TD3/SAC as stated assume a compact state vector. Running them from raw images was long thought to require a separate representation-learning stage, but two 2020–2021 results showed that plain **image augmentation** is enough. **RAD** (Laskin et al., 2020) and **DrQ** (Kostrikov et al., 2021) apply small random shifts and crops to the input frames and average the $Q$-target over a few augmented copies, and with nothing more than that, SAC from pixels reaches the sample efficiency of SAC from states on the DeepMind Control suite.[^drq] The augmentation regularizes the critic — it encodes the prior that a two-pixel shift should not change the value — and that single inductive bias closed most of the pixels-to-states gap that had made image-based continuous control seem to need elaborate machinery. $$ % caption: Three post-SAC directions, each pushing one lever of the shared template. % A distributional critic (D4PG) enriches what the critic predicts; a critic ensemble % with many updates (REDQ) buys model-based-level sample efficiency; image % augmentation (DrQ, RAD) makes the same agents learn from pixels. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=28mm, minimum height=16mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[box, draw=acc, text=acc, thick] (core) at (0,0) {SAC / TD3 core\\of\/f-policy actor-critic}; \node[box] (dist) at (5.2,1.9) {D4PG:\\distributional critic}; \node[box] (redq) at (5.2,0) {REDQ:\\critic ensemble\\+ many updates}; \node[box] (drq) at (5.2,-1.9) {DrQ / RAD:\\image augmentation}; \draw[->, acc, thick] (core) -- (dist); \draw[->, acc, thick] (core) -- (redq); \draw[->, acc, thick] (core) -- (drq); \end{tikzpicture} $$ ## Comparing the three All three share the off-policy, replay-buffer, twin-or-single-critic skeleton and differ in the policy and the exploration mechanism. | Method | Policy | Exploration | Key trick | Overestimation cure | | --- | --- | --- | --- | --- | | DDPG | deterministic $\mu_\theta(s)$ | external action noise | deterministic policy gradient, target nets | none | | TD3 | deterministic $\mu_\theta(s)$ | external action noise | delayed updates, target smoothing | clipped double-Q (twin min) | | SAC | stochastic $\pi_\theta(a\mid s)$ | intrinsic (entropy reward) | max-entropy objective, tuned $\alpha$ | clipped double-Q (twin min) | $$ % caption: The lineage. DDPG establishes the off-policy deterministic actor-critic; % TD3 adds twin critics, delayed updates, and target smoothing to fix overestimation; % SAC swaps the deterministic actor for a stochastic one under a maximum-entropy objective. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=30mm, minimum height=17mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[box, draw=acc, text=acc, thick] (ddpg) at (0,0) {DDPG\\deterministic actor\\replay + target nets}; \node[box] (td3) at (4.6,0) {TD3\\twin min-critic\\delay + smoothing}; \node[box, draw=red, text=red] (sac) at (9.2,0) {SAC\\stochastic actor\\max-entropy + tuned alpha}; \draw[->, black, thick] (ddpg) -- (td3) node[midway, above, font=\scriptsize] {f\/ix bias}; \draw[->, black, thick] (td3) -- (sac) node[midway, above, font=\scriptsize] {add entropy}; \node[anchor=north, text=black, font=\scriptsize] at (0,-1.25) {2016}; \node[anchor=north, text=black, font=\scriptsize] at (4.6,-1.25) {2018}; \node[anchor=north, text=black, font=\scriptsize] at (9.2,-1.25) {2018}; \end{tikzpicture} $$ The Q-critic gives off-policy sample efficiency; the learned actor replaces the intractable $\arg\max_a Q$; and the three methods disagree only on how to keep the coupled learning stable and how to explore. DDPG proves the idea works, TD3 makes it reliable by refusing to trust a single critic, and SAC makes it robust by rewarding entropy. When a modern robotics or MuJoCo agent learns continuous control off-policy, one of these three — most often TD3 or SAC — is almost certainly the algorithm underneath. [^sac]: **Haarnoja et al. (2018)**, "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor," _ICML_, and the follow-up "Soft Actor-Critic Algorithms and Applications" (2018) — the maximum-entropy objective $J = \mathbb{E}[\sum_t \gamma^t (r + \alpha\mathcal{H}(\pi))]$, the soft value functions and squashed-Gaussian stochastic actor, twin critics from TD3, and automatic temperature tuning against a target entropy $\bar{\mathcal{H}}$; **Morales**, Ch. 12, "SAC: Maximizing the expected return and entropy." [^d4pg]: **Barth-Maron et al. (2018)**, "Distributed Distributional Deterministic Policy Gradients," _ICLR_ — D4PG: DDPG with a C51-style categorical distributional critic, $n$-step returns, prioritized replay, and distributed actors, improving stability and final performance on continuous-control tasks over scalar-critic DDPG. [^redq]: **Chen, Wang, Zhou, Ross (2021)**, "Randomized Ensembled Double Q-Learning: Learning Fast Without a Model," _ICLR_ — REDQ: an ensemble of $N$ critics with targets formed from a random in-target subset, combined with a high update-to-data ratio, matching the sample efficiency of model-based methods (e.g. PETS) while remaining model-free. [^drq]: **Laskin, Lee, Stooke, Pinto, Abbeel, Srinivas (2020)**, "Reinforcement Learning with Augmented Data" (RAD), _NeurIPS_; **Kostrikov, Yarats, Fergus (2021)**, "Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels" (DrQ), _ICLR_ — random shift/crop augmentation of image observations, with the $Q$-target averaged over augmented copies, lets SAC learn from pixels at the sample efficiency of learning from states on the DeepMind Control suite.