PPO and Continuous Control

This builds on Actor–Critic and GAE, which built the neural actor-critic, the advantage that replaces the raw return, generalized advantage estimation, and the parallel-worker methods A3C and A2C. Those give a working on-policy actor-critic; the hazard they all share is that a single gradient step can destroy the policy. This lesson bounds that step.

The problem of destructively large steps

Every method so far shares a hazard. The gradient gives a direction in parameter space, but performance depends on policy space, and the map between the two is highly nonlinear. A modest step in $θ$ can swing the action distribution enough to send the agent into a region of the state space it has never seen, where the critic's estimates are meaningless and the next gradient points nowhere useful. The policy collapses, and because the data is on-policy, there is no old good experience to recover from.¹ This is why plain policy gradients demand tiny learning rates and still wobble.

The principled response is a trust region: at each step, maximize the expected improvement subject to a hard constraint that the new policy stay close to the old one, measured by KL divergence. Trust Region Policy Optimization (TRPO) solves exactly this constrained problem,

θ max E [\frac{π ( A _{t} ∣ S _{t} ; θ )}{π ( A _{t} ∣ S _{t} ; θ _{old} )} \hat{A}_{t}] subject to E [D_{KL} (π_{θ_{old}} ∥ π_{θ})] \leq ε,

and it works, but the constrained optimization needs second-order information (a conjugate-gradient solve against the Fisher matrix) that is awkward to implement and does not share code with a plain SGD loop. PPO keeps the trust-region intent and throws away the machinery.

Why the Fisher matrix appears at all is worth a sentence, because it connects to the natural policy gradient. KL divergence between two nearby policies is, to second order, a quadratic form in the parameter change with the Fisher information matrix $F$ as its metric: $D_{KL} \approx \frac{1}{2} Δ θ^{⊤} F Δ θ$ . So stay within an $ε$ -ball of KL is stay within an ellipsoid shaped by $F$ , and the exact solution steps along the natural gradient $F^{- 1} \nabla J$ rather than the raw gradient $\nabla J$ . TRPO computes that step; its cost is the $F^{- 1}$ solve. Schulman, Levine, Abbeel, Jordan, and Moritz (2015) proved a monotonic-improvement guarantee for a penalized version of this objective, then relaxed it to the hard KL constraint for practicality. PPO's contribution (Schulman, Wolski, Dhariwal, Radford, and Klimov, 2017) is to observe that the entire second-order apparatus can be replaced by a first-order clip that approximates the same don't move too far effect, at a fraction of the code and compute.²

Three ways to bound the step. Gradient ascent (left) takes an unbounded step and can overshoot into a bad policy region. TRPO (center) constrains the step to a KL trust region, an ellipsoid in policy space. PPO (right) approximates the same bound cheaply by clipping the probability ratio to

[1 - ε, 1 + ε]

PPO ships in two forms. The clipped objective above is the common one. An alternative adaptive-KL-penalty form adds $- β D_{KL}$ to the objective and raises or lowers $β$ between updates to hold the realized KL near a target — closer in spirit to the Lagrangian of TRPO, but still first-order. The clip usually wins in practice and is what PPO now means by default.³

Proximal policy optimization

PPO lets the update improve an action's probability, but stops rewarding it once the new policy has drifted more than a set fraction away from the old one — no trust-region solver, just a clamp on how far the probabilities may move per update.

Proximal policy optimization (PPO) enforces the trust region with nothing more than a clip inside the objective. Let $r (θ)$ be the probability ratio between the new and old policies on the sampled action,

r (θ) = \frac{π ( A _{t} ∣ S _{t} ; θ )}{π ( A _{t} ∣ S _{t} ; θ _{old} )},

which is $1$ when the policy is unchanged, above $1$ where the update has made the action more likely, and below $1$ where less. The unconstrained surrogate objective is $r (θ) \hat{A}_{t}$ — push the ratio up for good actions, down for bad ones. PPO clips it.¹

Read the two cases. When the advantage is positive, the objective grows with $r$ until $r$ hits $1 + ε$ , then flattens — beyond that point the gradient is zero, so there is no incentive to push the good action's probability any higher in a single update. When the advantage is negative, the objective is bounded below at $r = 1 - ε$ , again flattening so the update cannot slam a bad action's probability toward zero all at once. The $min$ is what makes the clip pessimistic: it only ever removes incentive to move, never adds it, so a ratio that overshoots the trust region contributes no misleading gradient.

The PPO clipped objective

L^{CLIP}

as a function of the ratio

r (θ)

. For a positive advantage the objective rises then flattens past

1 + ε

; for a negative advantage it falls then flattens below

1 - ε

. Outside the band the gradient is zero.

The clip, evaluated on numbers

Fix $ε = 0.2$ , so the band is $[0.8, 1.2]$ , and walk through four sampled transitions. For each, $r$ is the probability ratio the current update has produced, $\hat{A}$ is the transition's advantage, and $L^{CLIP}$ is the per-sample objective $min (r \hat{A}, clip (r, 0.8, 1.2) \hat{A})$ :

case	$r$	$\hat{A}$	$r \hat{A}$	$clip (r) \hat{A}$	$L^{CLIP}$	gradient?
A	$1.10$	$+ 2.0$	$2.20$	$2.20$	$2.20$	yes (inside band)
B	$1.40$	$+ 2.0$	$2.80$	$2.40$	$2.40$	no (clipped, $A > 0$ )
C	$0.70$	$- 3.0$	$- 2.10$	$- 2.40$	$- 2.40$	no ( $min$ picks clipped)
D	$1.35$	$- 3.0$	$- 4.05$	$- 3.60$	$- 4.05$	yes ( $min$ picks unclipped)

Case A is inside the trust region, so clipping does nothing and the objective is the ordinary surrogate; the update proceeds normally. Case B has a good action ( $\hat{A} > 0$ ) whose probability the update has already pushed up to $r = 1.4$ , past the ceiling $1.2$ ; the clip caps the objective at $1.2 \times 2.0 = 2.4$ , and because that value no longer depends on $θ$ , its gradient is zero — the update stops making an already-favored action even more likely.

Cases C and D are the subtle ones, and they show why the $min$ is essential. Both are a bad action ( $\hat{A} < 0$ ), but the update has moved them in opposite directions. In case C the update has correctly made the action less likely, pushing the ratio below the floor ( $r = 0.7 < 0.8$ ). Both branches are negative; the clipped term ( $- 2.40$ ) is more negative than the unclipped one ( $- 2.10$ ), so $min$ takes the clipped $- 2.40$ . Once $r$ leaves the band the clipped branch is flat in $θ$ , so its gradient is zero — the clip stops pushing an action that has already been suppressed enough. Case C is the mirror image of case B on the other side of the band: an already-corrected action, clipped to no gradient. Case D is the guardrail: a bad action whose probability the update has instead pushed up to $r = 1.35$ , the wrong direction. Now the unclipped $r \hat{A} = - 4.05$ is more negative than the clipped $- 3.60$ , and $min$ takes the unclipped $- 4.05$ . Because this is a maximization, the more-negative value keeps a strong gradient pulling the ratio back down toward the band. The asymmetry is the purpose of the $min$ : once a ratio has moved the right way past the band, the clip removes any further incentive (cases B and C); but a ratio moving the wrong way (case D) keeps a live corrective gradient through the unclipped branch. A plain clip without the $min$ would zero out case D's gradient too, and the policy could drift out of the region with nothing to pull it back.

Because a step outside the band earns no gradient, PPO can afford to do something on-policy methods normally cannot: run several epochs of minibatch updates on the same batch of experience. The clip is what keeps those repeated steps from walking the policy far from the data that produced it, so PPO reuses each rollout many times and is markedly more sample-efficient than A2C, while remaining a first-order method that drops into an ordinary Adam loop. Operationally, PPO is A2C with GAE advantages, a value head trained by the same regression, an entropy bonus, and this clipped policy term run for a handful of epochs per batch.

Algorithm:

\textsc{PPO}

— clipped-surrogate actor-critic

1
input: epochs $K$ , environments $n$ , actor $\boldsymbol{\theta}$ , critic $\mathbf{w}$
2
loop
3
$\boldsymbol{\theta}_{\text{old}} \gets \boldsymbol{\theta}$
4
run $\pi_{\boldsymbol{\theta}_{\text{old}}}$ in $n$ parallel environments to collect a batch of transitions
5
compute GAE advantages $\hat A_t$ and returns $G_t$ from the critic $\hat v(\cdot; \mathbf{w})$
6
for $k = 1$ to $K$ do
7
for each minibatch of the batch do
8
ascend $L^{\text{CLIP}}(\boldsymbol{\theta})$ plus an entropy bonus w.r.t. $\boldsymbol{\theta}$
actor
9
descend $\tfrac{1}{2}\,[\hat v(s; \mathbf{w}) - G_t]^2$ w.r.t. $\mathbf{w}$
critic

PPO's batch reuse. One rollout collects a batch of transitions under the frozen

π_{old}

; the clip lets the actor take

K

epochs of minibatch updates over that same batch before the next rollout. A2C would use each batch once.

PPO's combination of stability, sample efficiency, and implementation simplicity is why it is the modern default for policy-gradient RL. It is also the optimizer inside reinforcement learning from human feedback: a language model is the actor, a learned reward model plus a KL penalty against the base model supplies the per-token reward, GAE turns that into advantages, and the clipped PPO step nudges the model without letting it drift into gibberish. The RLHF lesson in the deep-learning notes develops that pipeline in full; the clip here is the same clip that keeps the aligned model close to its starting point.

Continuous control: deterministic and maximum-entropy policies

A2C and PPO are on-policy and handle any action space through a stochastic policy — a softmax over discrete actions or a Gaussian over continuous ones. A parallel family attacks continuous control from the value-based side, reusing a replay buffer for off-policy sample efficiency.

Deep deterministic policy gradient (DDPG) is a DQN for continuous actions.⁴ DQN cannot act in a continuous space because $ar g max_{a} \overset{q}{^} (s, a)$ is an optimization over a continuum on every step; DDPG replaces that maximization with a learned deterministic actor $μ (s; θ)$ trained to output the action that maximizes the critic, $max_{θ} E [\overset{q}{^} (s, μ (s; θ); w)]$ . The critic is a Q-network trained with the same replay buffer and target networks as DQN. Since a deterministic policy does not explore on its own, DDPG injects Gaussian noise into the actions it sends to the environment.

Twin-delayed DDPG (TD3) fixes DDPG's tendency to overestimate values with three adjustments.⁵ It learns twin critics and takes the minimum of the two when forming the target — the same overestimation fix as double Q-learning. It adds clipped noise to the target action, smoothing the value estimate so the policy cannot exploit a sharp spurious peak. And it delays policy updates, stepping the critics more often than the actor so the value estimates settle before they steer the policy.

Soft actor-critic (SAC) keeps the off-policy replay buffer and twin critics but learns a stochastic policy under a maximum-entropy objective.⁶ Instead of adding an entropy bonus to the loss as A2C does, SAC folds entropy directly into the value function: the agent maximizes reward plus the discounted long-run entropy of the policy,

E [t \sum γ^{t} (R_{t + 1} + α H [π (\cdot ∣ S_{t})])],

where the temperature $α$ trades reward against randomness and can be tuned automatically toward a target entropy. Building exploration into the objective this way makes SAC one of the most robust off-policy methods, and because the policy stays stochastic, exploration is on-policy and embedded in the learned actor rather than bolted on as external noise.

Method	Policy	Actions	On/off-policy	Idea
A3C / A2C	stochastic	any	on-policy	parallel actors, advantage critic
PPO	stochastic	any	on-policy	clipped surrogate, reuse batches
DDPG	deterministic	continuous	off-policy	DQN with a learned actor
TD3	deterministic	continuous	off-policy	twin critics, target smoothing, delay
SAC	stochastic	continuous	off-policy	maximum-entropy actor-critic

The deep actor-critic family, arranged by whether the policy is stochastic or deterministic and whether learning is on-policy or off-policy; PPO is the on-policy default, SAC and TD3 the off-policy continuous-control state of the art.

Where actor-critic went

Sutton and Barto's Chapter 13 stops at the actor-critic template and the policy gradient theorem; A3C, PPO, and the continuous-control family are the deep-RL papers that carried it forward, and the systems those papers enabled define the reach of the method today. Three lines are worth naming precisely.

Large-scale game play. PPO is the algorithm behind OpenAI Five (OpenAI et al., 2019), which defeated the reigning Dota 2 world champions using self-play PPO scaled across thousands of GPUs and hundreds of thousands of CPU cores, on games lasting tens of thousands of time steps. DeepMind's AlphaStar (Vinyals et al., 2019, Nature) reached Grandmaster level at StarCraft II with an actor-critic policy trained by a population-based self-play league — a different algorithm, but the same actor-critic-with-advantage backbone. These results matter for the theory in two ways: they show the clipped surrogate remains stable at a scale the original paper never tested, and they show that the on-policy sample cost, which looks prohibitive on a laptop, is a throughput problem that parallelism solves.

Continuous control and robotics. SAC and TD3 are the default off-policy methods for simulated locomotion and manipulation, and PPO is the default on-policy one. OpenAI's dexterous in-hand manipulation (OpenAI et al., 2019, Solving Rubik's Cube with a Robot Hand) trained a control policy with PPO in simulation and transferred it to a physical robot hand, using domain randomization — randomizing physics parameters during training so the learned policy is robust to the simulation-to-reality gap. The policy-gradient machinery is unchanged; the engineering is in the environment.

Language-model alignment. The single most widely-deployed use of PPO today is reinforcement learning from human feedback. Christiano et al. (2017) introduced learning a reward model from human preference comparisons and optimizing it with an RL agent; Stiennon et al. (2020) and Ouyang et al. (2022, the InstructGPT paper) applied exactly this with PPO to large language models. The setup maps cleanly onto this lesson: the language model is the actor, a learned reward model plus a per-token KL penalty against the frozen base model supplies the reward, GAE turns per-token rewards into advantages, and the clipped PPO step updates the model. The KL penalty plays the role the clip does — keeping the tuned policy near a trusted starting point — and the RLHF lesson develops the full pipeline. That a 2017 continuous-control trick became the training loop for aligned chat models is the clearest evidence that the actor-critic template is a general one, not a game-playing curiosity.⁷

RLHF as an actor-critic instance. The language model is the actor; a learned reward model plus a per-token KL penalty against the frozen base model is the reward; GAE turns per-token rewards into advantages; and the clipped PPO step updates the actor while the KL penalty keeps it near the base model.

The shape of modern deep RL

The progression from the policy-gradient lesson to here is a single architecture at increasing resolution. An actor proposes actions; a critic scores states; the score, sharpened from raw return to baselined return to advantage to GAE-blended advantage, tells the actor which way to move. Deep networks make the actor and critic expressive; parallel workers make the on-policy data uncorrelated; and a clip or a trust region makes the step safe. PPO is where those three ideas meet in the simplest form that works, which is why it is the algorithm most likely to be running when a modern agent is being trained — including language models tuned with RLHF.¹

Morales, Ch. 12 — PPO: Restricting optimization steps: the trust-region motivation (small parameter changes can cause large performance swings), the probability ratio $r (θ)$ , the clipped surrogate objective with the pessimistic $min$ , and the reuse of minibatches over several epochs; PPO as an on-policy improvement to A2C (Schulman et al., 2017; TRPO from Schulman et al., 2015). ↩ ↩² ↩³
Schulman, Levine, Abbeel, Jordan, and Moritz (2015), Trust Region Policy Optimization, ICML — the surrogate objective with a KL trust-region constraint, its monotonic-improvement bound for the penalized form, and the Fisher-matrix / natural-gradient solve. The second-order-KL approximation $D_{KL} \approx \frac{1}{2} Δ θ^{⊤} F Δ θ$ underlies the natural policy gradient (Kakade, 2002). ↩
Schulman, Wolski, Dhariwal, Radford, and Klimov (2017), Proximal Policy Optimization Algorithms, arXiv — the clipped surrogate $L^{CLIP}$ with the pessimistic $min$ , the adaptive-KL-penalty variant, and multi-epoch minibatch reuse of each on-policy batch; PPO as a first-order approximation to TRPO's trust region. ↩
Morales, Ch. 12 — DDPG: Approximating a deterministic policy: deep deterministic policy gradient as a continuous-action DQN, replacing $ar g max_{a} \overset{q}{^}$ with a learned deterministic actor $μ (s; θ)$ , trained off-policy with a replay buffer, target networks, and injected Gaussian exploration noise (Lillicrap et al., 2015). ↩
Morales, Ch. 12 — TD3: State-of-the-art improvements over DDPG: twin critics with a minimum target (double learning), clipped noise added to the target action (target smoothing), and delayed policy and target updates (Fujimoto et al., 2018). ↩
Morales, Ch. 12 — SAC: Maximizing the expected return and entropy: soft actor-critic folds the policy entropy into the value function to maximize reward plus long-run entropy, learns a stochastic policy off-policy with twin critics, and can tune the entropy temperature $α$ automatically toward a target entropy (Haarnoja et al., 2018). ↩
Applications of these methods to systems past Sutton & Barto. OpenAI et al. (2019), Dota 2 with Large Scale Deep Reinforcement Learning (OpenAI Five) — self-play PPO at scale defeating the Dota 2 world champions. Vinyals et al. (2019), Grandmaster level in StarCraft II using multi-agent reinforcement learning, Nature (AlphaStar) — actor-critic with a self-play league. OpenAI et al. (2019), Solving Rubik's Cube with a Robot Hand — PPO with domain randomization for sim-to-real manipulation. Christiano et al. (2017), Deep Reinforcement Learning from Human Preferences, NeurIPS; Stiennon et al. (2020), Learning to summarize from human feedback, NeurIPS; Ouyang et al. (2022), Training language models to follow instructions with human feedback (InstructGPT) — reward-model-plus-PPO alignment of language models. ↩