--- title: "PPO and Continuous Control" module: Deep Reinforcement Learning moduleNumber: 4 lessonNumber: 4 order: 404 summary: > Keeping the policy-gradient step from destroying the policy, and the algorithms that result. Trust-region optimization bounds each update by a KL constraint; PPO keeps that goal but replaces the second-order machinery with a first-order clip on the probability ratio, which is why it is the modern default and the optimizer inside RLHF. We then tour the off-policy continuous-control family — DDPG, TD3, and SAC — and where actor-critic went at scale, from OpenAI Five to language-model alignment. topics: [Deep RL] sources: - book: Grokking Deep RL ref: "Ch. 12 — Advanced actor-critic methods; DDPG, TD3, SAC, PPO" - book: Sutton & Barto ref: "Ch. 13 — Policy Gradient Methods; §13.5 Actor–Critic Methods" --- This builds on [Actor–Critic and GAE](/reinforcement-learning/deep-rl/actor-critic-and-ppo), which built the neural actor-critic, the advantage that replaces the raw return, generalized advantage estimation, and the parallel-worker methods A3C and A2C. Those give a working on-policy actor-critic; the hazard they all share is that a single gradient step can destroy the policy. This lesson bounds that step. ## The problem of destructively large steps Every method so far shares a hazard. The gradient gives a direction in _parameter_ space, but performance depends on _policy_ space, and the map between the two is highly nonlinear. A modest step in $\boldsymbol{\theta}$ can swing the action distribution enough to send the agent into a region of the state space it has never seen, where the critic's estimates are meaningless and the next gradient points nowhere useful. The policy collapses, and because the data is on-policy, there is no old good experience to recover from.[^grok-ppo] This is why plain policy gradients demand tiny learning rates and still wobble. The principled response is a **trust region**: at each step, maximize the expected improvement _subject to a hard constraint_ that the new policy stay close to the old one, measured by KL divergence. **Trust Region Policy Optimization** (TRPO) solves exactly this constrained problem, $$ \max_{\boldsymbol{\theta}} \; \mathbb{E}\!\left[\frac{\pi(A_t \mid S_t; \boldsymbol{\theta})}{\pi(A_t \mid S_t; \boldsymbol{\theta}_{\text{old}})}\,\hat A_t\right] \quad\text{subject to}\quad \mathbb{E}\big[D_{\mathrm{KL}}(\pi_{\boldsymbol{\theta}_{\text{old}}} \,\|\, \pi_{\boldsymbol{\theta}})\big] \le \varepsilon, $$ and it works, but the constrained optimization needs second-order information (a conjugate-gradient solve against the Fisher matrix) that is awkward to implement and does not share code with a plain SGD loop. PPO keeps the trust-region _intent_ and throws away the machinery. Why the Fisher matrix appears at all is worth a sentence, because it connects to the [natural policy gradient](/reinforcement-learning/approximation/policy-gradient-methods). KL divergence between two nearby policies is, to second order, a quadratic form in the parameter change with the Fisher information matrix $F$ as its metric: $D_{\mathrm{KL}} \approx \tfrac{1}{2}\,\Delta\boldsymbol{\theta}^\top F\, \Delta\boldsymbol{\theta}$. So "stay within an $\varepsilon$-ball of KL" is "stay within an ellipsoid shaped by $F$," and the exact solution steps along the _natural_ gradient $F^{-1}\nabla J$ rather than the raw gradient $\nabla J$. TRPO computes that step; its cost is the $F^{-1}$ solve. Schulman, Levine, Abbeel, Jordan, and Moritz (2015) proved a monotonic-improvement guarantee for a penalized version of this objective, then relaxed it to the hard KL constraint for practicality. PPO's contribution (Schulman, Wolski, Dhariwal, Radford, and Klimov, 2017) is to observe that the entire second-order apparatus can be replaced by a first-order clip that _approximates_ the same "don't move too far" effect, at a fraction of the code and compute.[^trpo-paper] $$ % caption: Three ways to bound the step. Gradient ascent (left) takes an unbounded % step and can overshoot into a bad policy region. TRPO (center) constrains the step % to a KL trust region, an ellipsoid in policy space. PPO (right) approximates the % same bound cheaply by clipping the probability ratio to $[1-\varepsilon,\, % 1+\varepsilon]$. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % left: unbounded gradient \node[font=\scriptsize\bfseries] at (1.1,2.3) {gradient ascent}; \fill[black] (0.3,0.3) circle (1.8pt); \node[font=\scriptsize, anchor=north] at (0.3,0.2) {old}; \draw[->, acc, very thick] (0.3,0.3) -- (2.2,1.8); \node[red, font=\scriptsize, anchor=west] at (1.7,1.85) {overshoot}; % center: TRPO region \begin{scope}[xshift=4.2cm] \node[font=\scriptsize\bfseries] at (1.1,2.3) {TRPO region}; \draw[acc, dashed] (0.9,0.7) ellipse (1.05 and 0.6); \fill[black] (0.9,0.7) circle (1.8pt); \draw[->, acc, very thick] (0.9,0.7) -- (1.7,1.0); \node[acc, font=\scriptsize, anchor=west] at (0.2,1.55) {KL ball}; \end{scope} % right: PPO clip \begin{scope}[xshift=8.4cm] \node[font=\scriptsize\bfseries] at (1.1,2.3) {PPO clip}; \draw[acc, dashed] (0.3,0.2) rectangle (1.9,1.3); \fill[black] (1.1,0.75) circle (1.8pt); \draw[->, acc, very thick] (1.1,0.75) -- (1.75,1.05); \node[acc, font=\scriptsize, anchor=north] at (1.1,0.15) {ratio box}; \end{scope} \end{tikzpicture} $$ PPO ships in two forms. The **clipped** objective above is the common one. An alternative **adaptive-KL-penalty** form adds $-\beta\,D_{\mathrm{KL}}$ to the objective and raises or lowers $\beta$ between updates to hold the realized KL near a target — closer in spirit to the Lagrangian of TRPO, but still first-order. The clip usually wins in practice and is what "PPO" now means by default.[^ppo-paper] ## Proximal policy optimization PPO lets the update improve an action's probability, but stops rewarding it once the new policy has drifted more than a set fraction away from the old one — no trust-region solver, just a clamp on how far the probabilities may move per update. **Proximal policy optimization** (PPO) enforces the trust region with nothing more than a clip inside the objective. Let $r(\boldsymbol{\theta})$ be the **probability ratio** between the new and old policies on the sampled action, $$ r(\boldsymbol{\theta}) \;=\; \frac{\pi(A_t \mid S_t; \boldsymbol{\theta})}{\pi(A_t \mid S_t; \boldsymbol{\theta}_{\text{old}})}, $$ which is $1$ when the policy is unchanged, above $1$ where the update has made the action more likely, and below $1$ where less. The unconstrained surrogate objective is $r(\boldsymbol{\theta})\,\hat A_t$ — push the ratio up for good actions, down for bad ones. PPO clips it.[^grok-ppo] > **Definition (PPO clipped surrogate objective).** > $$L^{\text{CLIP}}(\boldsymbol{\theta}) \;=\; \mathbb{E}\!\Big[\min\!\big(r(\boldsymbol{\theta})\,\hat A_t,\; \clip(r(\boldsymbol{\theta}),\, 1-\varepsilon,\, 1+\varepsilon)\,\hat A_t\big)\Big],$$ > where $\clip$ confines the ratio to $[1-\varepsilon,\, 1+\varepsilon]$ > (typically $\varepsilon = 0.2$) and the outer $\min$ takes the more pessimistic of > the clipped and unclipped terms. Read the two cases. When the advantage is **positive**, the objective grows with $r$ until $r$ hits $1+\varepsilon$, then flattens — beyond that point the gradient is zero, so there is no incentive to push the good action's probability any higher in a single update. When the advantage is **negative**, the objective is bounded below at $r = 1-\varepsilon$, again flattening so the update cannot slam a bad action's probability toward zero all at once. The $\min$ is what makes the clip pessimistic: it only ever removes incentive to move, never adds it, so a ratio that overshoots the trust region contributes no misleading gradient. $$ % caption: The PPO clipped objective $L^{\text{CLIP}}$ as a function of the ratio % $r(\boldsymbol{\theta})$. For a positive advantage the objective rises then % flattens past $1+\varepsilon$; for a negative advantage it falls then flattens % below $1-\varepsilon$. Outside the band the gradient is zero. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % axes \draw[black, ->] (0,0) -- (7.4,0) node[anchor=north east, text=black] {ratio r}; \draw[black, ->] (0,-2.2) -- (0,2.6) node[anchor=south east, text=black] {objective}; \draw[black, dashed] (0,0) -- (7.2,0); % r = 1 marker \draw[black, dashed] (3.4,-2.1) -- (3.4,2.5); \node[black, anchor=south] at (3.4,2.45) {r = 1}; % clip band boundaries \draw[black, dotted] (2.5,-2.1) -- (2.5,2.5); \draw[black, dotted] (4.3,-2.1) -- (4.3,2.5); \node[black, anchor=north east, font=\scriptsize] at (2.42,-0.08) {1-eps}; \node[black, anchor=north west, font=\scriptsize] at (4.38,-0.08) {1+eps}; % positive advantage: rises with slope, flat after 1+eps \draw[acc, very thick] (0.5,-1.55) -- (4.3,1.75) -- (7.0,1.75); \node[acc, anchor=west] at (7.0,1.75) {A $>$ 0}; % negative advantage: falls with slope, flat before 1-eps \draw[red, very thick] (2.5,-1.55) -- (2.5,-1.55) -- (6.9,-1.55); \draw[red, very thick] (2.5,-1.55) -- (0.5,-0.45); \node[red, anchor=west] at (6.9,-1.55) {A $<$ 0}; \end{tikzpicture} $$ ### The clip, evaluated on numbers Fix $\varepsilon = 0.2$, so the band is $[0.8,\ 1.2]$, and walk through four sampled transitions. For each, $r$ is the probability ratio the current update has produced, $\hat A$ is the transition's advantage, and $L^{\text{CLIP}}$ is the per-sample objective $\min\!\big(r\hat A,\ \clip(r, 0.8, 1.2)\,\hat A\big)$: | case | $r$ | $\hat A$ | $r\hat A$ | $\clip(r)\hat A$ | $L^{\text{CLIP}}$ | gradient? | | --- | --- | --- | --- | --- | --- | --- | | A | $1.10$ | $+2.0$ | $2.20$ | $2.20$ | $2.20$ | yes (inside band) | | B | $1.40$ | $+2.0$ | $2.80$ | $2.40$ | $2.40$ | no (clipped, $A>0$) | | C | $0.70$ | $-3.0$ | $-2.10$ | $-2.40$ | $-2.40$ | no ($\min$ picks clipped) | | D | $1.35$ | $-3.0$ | $-4.05$ | $-3.60$ | $-4.05$ | yes ($\min$ picks unclipped) | Case A is inside the trust region, so clipping does nothing and the objective is the ordinary surrogate; the update proceeds normally. Case B has a _good_ action ($\hat A > 0$) whose probability the update has already pushed up to $r = 1.4$, past the ceiling $1.2$; the clip caps the objective at $1.2 \times 2.0 = 2.4$, and because that value no longer depends on $\boldsymbol{\theta}$, its gradient is zero — the update stops making an already-favored action even more likely. Cases C and D are the subtle ones, and they show why the $\min$ is essential. Both are a _bad_ action ($\hat A < 0$), but the update has moved them in opposite directions. In case C the update has correctly made the action _less_ likely, pushing the ratio below the floor ($r = 0.7 < 0.8$). Both branches are negative; the clipped term ($-2.40$) is _more_ negative than the unclipped one ($-2.10$), so $\min$ takes the clipped $-2.40$. Once $r$ leaves the band the clipped branch is flat in $\boldsymbol{\theta}$, so its gradient is zero — the clip stops pushing an action that has already been suppressed enough. Case C is the mirror image of case B on the other side of the band: an already-corrected action, clipped to no gradient. Case D is the guardrail: a bad action whose probability the update has instead pushed _up_ to $r = 1.35$, the _wrong_ direction. Now the unclipped $r\hat A = -4.05$ is more negative than the clipped $-3.60$, and $\min$ takes the unclipped $-4.05$. Because this is a maximization, the more-negative value keeps a strong gradient _pulling the ratio back down_ toward the band. The asymmetry is the purpose of the $\min$: once a ratio has moved the _right_ way past the band, the clip removes any further incentive (cases B and C); but a ratio moving the _wrong_ way (case D) keeps a live corrective gradient through the unclipped branch. A plain clip without the $\min$ would zero out case D's gradient too, and the policy could drift out of the region with nothing to pull it back. Because a step outside the band earns no gradient, PPO can afford to do something on-policy methods normally cannot: run **several epochs of minibatch updates on the same batch** of experience. The clip is what keeps those repeated steps from walking the policy far from the data that produced it, so PPO reuses each rollout many times and is markedly more sample-efficient than A2C, while remaining a first-order method that drops into an ordinary Adam loop. Operationally, PPO is A2C with GAE advantages, a value head trained by the same regression, an entropy bonus, and this clipped policy term run for a handful of epochs per batch. ```algorithm caption: $\textsc{PPO}$ — clipped-surrogate actor-critic input: epochs $K$, environments $n$, actor $\boldsymbol{\theta}$, critic $\mathbf{w}$ loop $\boldsymbol{\theta}_{\text{old}} \gets \boldsymbol{\theta}$ run $\pi_{\boldsymbol{\theta}_{\text{old}}}$ in $n$ parallel environments to collect a batch of transitions compute GAE advantages $\hat A_t$ and returns $G_t$ from the critic $\hat v(\cdot; \mathbf{w})$ for $k = 1$ to $K$ do for each minibatch of the batch do ascend $L^{\text{CLIP}}(\boldsymbol{\theta})$ plus an entropy bonus w.r.t. $\boldsymbol{\theta}$ // actor descend $\tfrac{1}{2}\,[\hat v(s; \mathbf{w}) - G_t]^2$ w.r.t. $\mathbf{w}$ // critic ``` $$ % caption: PPO's batch reuse. One rollout collects a batch of transitions under the % frozen $\pi_{\text{old}}$; the clip lets the actor take $K$ epochs of minibatch % updates over that same batch before the next rollout. A2C would use each batch once. \begin{tikzpicture}[>=stealth, font=\footnotesize, b/.style={draw, minimum width=22mm, minimum height=9mm, align=center, font=\scriptsize}, ep/.style={draw, minimum width=11mm, minimum height=7mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[b, draw=acc, text=acc] (roll) at (0,0) {rollout batch\\(pi-old)}; \node[ep] (e1) at (3.3,1.0) {epoch 1}; \node[ep] (e2) at (3.3,0.0) {epoch 2}; \node[font=\scriptsize] (dots) at (3.3,-0.9) {...}; \node[ep] (eK) at (3.3,-1.7) {epoch K}; \node[b] (next) at (6.6,0) {next rollout}; \draw[->, acc] (roll.east) -- (e1.west); \draw[->, acc] (roll.east) -- (e2.west); \draw[->, acc] (roll.east) -- (eK.west); \draw[->, red] (e1.east) -- (next.west); \draw[->, red] (e2.east) -- (next.west); \draw[->, red] (eK.east) -- (next.west); \node[font=\scriptsize, black, anchor=north] at (3.3,-2.3) {clip keeps K steps near the data}; \end{tikzpicture} $$ PPO's combination of stability, sample efficiency, and implementation simplicity is why it is the **modern default** for policy-gradient RL. It is also the optimizer inside **reinforcement learning from human feedback**: a language model is the actor, a learned reward model plus a KL penalty against the base model supplies the per-token reward, GAE turns that into advantages, and the clipped PPO step nudges the model without letting it drift into gibberish. The [RLHF lesson](/deep-learning/reinforcement-learning/rl-from-human-feedback) in the deep-learning notes develops that pipeline in full; the clip here is the same clip that keeps the aligned model close to its starting point. ## Continuous control: deterministic and maximum-entropy policies A2C and PPO are on-policy and handle any action space through a stochastic policy — a softmax over discrete actions or a Gaussian over continuous ones. A parallel family attacks continuous control from the value-based side, reusing a replay buffer for **off-policy** sample efficiency. **Deep deterministic policy gradient** (DDPG) is a DQN for continuous actions.[^grok-ddpg] DQN cannot act in a continuous space because $\arg\max_a \hat q(s,a)$ is an optimization over a continuum on every step; DDPG replaces that maximization with a learned **deterministic** actor $\mu(s; \boldsymbol{\theta})$ trained to output the action that maximizes the critic, $\max_{\boldsymbol{\theta}} \mathbb{E}[\hat q(s, \mu(s; \boldsymbol{\theta}); \mathbf{w})]$. The critic is a Q-network trained with the same replay buffer and target networks as DQN. Since a deterministic policy does not explore on its own, DDPG injects Gaussian noise into the actions it sends to the environment. **Twin-delayed DDPG** (TD3) fixes DDPG's tendency to overestimate values with three adjustments.[^grok-td3] It learns **twin** critics and takes the minimum of the two when forming the target — the same overestimation fix as double Q-learning. It adds clipped noise to the _target_ action, smoothing the value estimate so the policy cannot exploit a sharp spurious peak. And it **delays** policy updates, stepping the critics more often than the actor so the value estimates settle before they steer the policy. **Soft actor-critic** (SAC) keeps the off-policy replay buffer and twin critics but learns a _stochastic_ policy under a **maximum-entropy** objective.[^grok-sac] Instead of adding an entropy bonus to the loss as A2C does, SAC folds entropy directly into the value function: the agent maximizes reward _plus_ the discounted long-run entropy of the policy, $$ \mathbb{E}\!\left[\sum_t \gamma^t \big(R_{t+1} + \alpha\,H[\pi(\cdot \mid S_t)]\big)\right], $$ where the temperature $\alpha$ trades reward against randomness and can be tuned automatically toward a target entropy. Building exploration into the objective this way makes SAC one of the most robust off-policy methods, and because the policy stays stochastic, exploration is on-policy and embedded in the learned actor rather than bolted on as external noise. | Method | Policy | Actions | On/off-policy | Idea | | --- | --- | --- | --- | --- | | A3C / A2C | stochastic | any | on-policy | parallel actors, advantage critic | | PPO | stochastic | any | on-policy | clipped surrogate, reuse batches | | DDPG | deterministic | continuous | off-policy | DQN with a learned actor | | TD3 | deterministic | continuous | off-policy | twin critics, target smoothing, delay | | SAC | stochastic | continuous | off-policy | maximum-entropy actor-critic | $$ % caption: The deep actor-critic family, arranged by whether the policy is % stochastic or deterministic and whether learning is on-policy or off-policy; % PPO is the on-policy default, SAC and TD3 the off-policy continuous-control state % of the art. \begin{tikzpicture}[>=stealth, font=\footnotesize, q/.style={draw, minimum width=30mm, minimum height=16mm, align=center}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % axes labels \node[anchor=south, text=black] at (0,2.3) {on-policy}; \node[anchor=north, text=black] at (0,-2.3) {of\/f-policy}; \node[anchor=east, text=black, align=right] at (-3.1,0) {stochastic\\policy}; \node[anchor=west, text=black, align=left] at (3.1,0) {deterministic\\policy}; % quadrants \node[q, draw=acc, text=acc, thick] (a) at (-1.6,1.1) {A2C / A3C\\PPO}; \node[q] (b) at (1.6,1.1) {(rare)}; \node[q, draw=red, text=red] (c) at (-1.6,-1.1) {SAC}; \node[q, draw=red, text=red] (d) at (1.6,-1.1) {DDPG / TD3}; \draw[black] (-3.0,0) -- (3.0,0); \draw[black] (0,-2.0) -- (0,2.0); \end{tikzpicture} $$ ## Where actor-critic went Sutton and Barto's Chapter 13 stops at the actor-critic template and the policy gradient theorem; A3C, PPO, and the continuous-control family are the deep-RL papers that carried it forward, and the systems those papers enabled define the reach of the method today. Three lines are worth naming precisely. **Large-scale game play.** PPO is the algorithm behind **OpenAI Five** (OpenAI et al., 2019), which defeated the reigning Dota 2 world champions using self-play PPO scaled across thousands of GPUs and hundreds of thousands of CPU cores, on games lasting tens of thousands of time steps. DeepMind's **AlphaStar** (Vinyals et al., 2019, _Nature_) reached Grandmaster level at StarCraft II with an actor-critic policy trained by a population-based self-play league — a different algorithm, but the same actor-critic-with-advantage backbone. These results matter for the theory in two ways: they show the clipped surrogate remains stable at a scale the original paper never tested, and they show that the on-policy sample cost, which looks prohibitive on a laptop, is a throughput problem that parallelism solves. **Continuous control and robotics.** SAC and TD3 are the default off-policy methods for simulated locomotion and manipulation, and PPO is the default on-policy one. **OpenAI's dexterous in-hand manipulation** (OpenAI et al., 2019, "Solving Rubik's Cube with a Robot Hand") trained a control policy with PPO in simulation and transferred it to a physical robot hand, using **domain randomization** — randomizing physics parameters during training so the learned policy is robust to the simulation-to-reality gap. The policy-gradient machinery is unchanged; the engineering is in the environment. **Language-model alignment.** The single most widely-deployed use of PPO today is **reinforcement learning from human feedback**. Christiano et al. (2017) introduced learning a reward model from human preference comparisons and optimizing it with an RL agent; Stiennon et al. (2020) and Ouyang et al. (2022, the InstructGPT paper) applied exactly this with PPO to large language models. The setup maps cleanly onto this lesson: the language model is the actor, a learned reward model plus a per-token KL penalty against the frozen base model supplies the reward, GAE turns per-token rewards into advantages, and the clipped PPO step updates the model. The KL penalty plays the role the clip does — keeping the tuned policy near a trusted starting point — and the [RLHF lesson](/reinforcement-learning/modern-deep-rl/rlhf-and-language-models) develops the full pipeline. That a 2017 continuous-control trick became the training loop for aligned chat models is the clearest evidence that the actor-critic template is a general one, not a game-playing curiosity.[^apps] $$ % caption: RLHF as an actor-critic instance. The language model is the actor; a % learned reward model plus a per-token KL penalty against the frozen base model is % the reward; GAE turns per-token rewards into advantages; and the clipped PPO step % updates the actor while the KL penalty keeps it near the base model. \begin{tikzpicture}[>=stealth, font=\footnotesize, b/.style={draw, minimum width=24mm, minimum height=10mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[b, draw=acc, text=acc] (lm) at (0,0) {language model\\(actor)}; \node[b] (rm) at (4.2,1.3) {reward model}; \node[b] (kl) at (4.2,-1.3) {KL vs base model}; \node[b, draw=red, text=red] (ppo) at (8.4,0) {GAE + clipped\\PPO step}; \draw[->, acc] (lm.east) -- node[above, font=\scriptsize] {tokens} (rm.west); \draw[->, acc] (lm.east) -- (kl.west); \draw[->, black] (rm.east) -- node[above, font=\scriptsize] {reward} (ppo.north west); \draw[->, black] (kl.east) -- node[below, font=\scriptsize] {penalty} (ppo.south west); \draw[->, red, thick] (ppo.south) to[bend left=34] node[pos=0.5, below, font=\scriptsize] {update actor} (lm.south); \end{tikzpicture} $$ ## The shape of modern deep RL The progression from the policy-gradient lesson to here is a single architecture at increasing resolution. An actor proposes actions; a critic scores states; the score, sharpened from raw return to baselined return to advantage to GAE-blended advantage, tells the actor which way to move. Deep networks make the actor and critic expressive; parallel workers make the on-policy data uncorrelated; and a clip or a trust region makes the step safe. PPO is where those three ideas meet in the simplest form that works, which is why it is the algorithm most likely to be running when a modern agent is being trained — including language models tuned with RLHF.[^grok-ppo] [^grok-ppo]: **Morales**, Ch. 12 — "PPO: Restricting optimization steps": the trust-region motivation (small parameter changes can cause large performance swings), the probability ratio $r(\boldsymbol{\theta})$, the clipped surrogate objective with the pessimistic $\min$, and the reuse of minibatches over several epochs; PPO as an on-policy improvement to A2C (Schulman et al., 2017; TRPO from Schulman et al., 2015). [^grok-ddpg]: **Morales**, Ch. 12 — "DDPG: Approximating a deterministic policy": deep deterministic policy gradient as a continuous-action DQN, replacing $\arg\max_a \hat q$ with a learned deterministic actor $\mu(s;\boldsymbol{\theta})$, trained off-policy with a replay buffer, target networks, and injected Gaussian exploration noise (Lillicrap et al., 2015). [^grok-td3]: **Morales**, Ch. 12 — "TD3: State-of-the-art improvements over DDPG": twin critics with a minimum target (double learning), clipped noise added to the target action (target smoothing), and delayed policy and target updates (Fujimoto et al., 2018). [^grok-sac]: **Morales**, Ch. 12 — "SAC: Maximizing the expected return and entropy": soft actor-critic folds the policy entropy into the value function to maximize reward plus long-run entropy, learns a stochastic policy off-policy with twin critics, and can tune the entropy temperature $\alpha$ automatically toward a target entropy (Haarnoja et al., 2018). [^trpo-paper]: **Schulman, Levine, Abbeel, Jordan, and Moritz** (2015), "Trust Region Policy Optimization", _ICML_ — the surrogate objective with a KL trust-region constraint, its monotonic-improvement bound for the penalized form, and the Fisher-matrix / natural-gradient solve. The second-order-KL approximation $D_{\mathrm{KL}}\approx\tfrac12\Delta\theta^\top F\Delta\theta$ underlies the natural policy gradient (Kakade, 2002). [^ppo-paper]: **Schulman, Wolski, Dhariwal, Radford, and Klimov** (2017), "Proximal Policy Optimization Algorithms", arXiv — the clipped surrogate $L^{\text{CLIP}}$ with the pessimistic $\min$, the adaptive-KL-penalty variant, and multi-epoch minibatch reuse of each on-policy batch; PPO as a first-order approximation to TRPO's trust region. [^apps]: Applications of these methods to systems past Sutton & Barto. **OpenAI et al.** (2019), "Dota 2 with Large Scale Deep Reinforcement Learning" (OpenAI Five) — self-play PPO at scale defeating the Dota 2 world champions. **Vinyals et al.** (2019), "Grandmaster level in StarCraft II using multi-agent reinforcement learning", _Nature_ (AlphaStar) — actor-critic with a self-play league. **OpenAI et al.** (2019), "Solving Rubik's Cube with a Robot Hand" — PPO with domain randomization for sim-to-real manipulation. **Christiano et al.** (2017), "Deep Reinforcement Learning from Human Preferences", _NeurIPS_; **Stiennon et al.** (2020), "Learning to summarize from human feedback", _NeurIPS_; **Ouyang et al.** (2022), "Training language models to follow instructions with human feedback" (InstructGPT) — reward-model-plus-PPO alignment of language models.