Policy Gradients and Actor-Critic Methods

Value-based methods learn a value function and read a policy off it: estimate $Q(s,a)$, then act greedily, $\pi (s)=\arg {\max }_{a}Q(s,a)$. The policy is implicit, a byproduct of the value estimate, and the $\arg \max$ ties the approach to discrete action sets and to a deterministic, sometimes brittle, greedy rule. Policy-gradient methods invert the dependency: parameterize the policy itself as ${\pi }_{\theta }(a\mid s)$ and adjust $\theta$ to make good trajectories more probable. There is no $\arg \max$, the policy is stochastic by construction, and continuous action spaces come for free.

The parameterization is a design choice with concrete tensor shapes. A shared torso maps the state to a feature vector $h=f_{\theta }(s)\in {\mathbb{R}}^d$; a policy head then reads a distribution off $h$. The two standard heads differ only in what the head produces.

• Discrete actions. A linear head $Wh+b$ with $W\in {\mathbb{R}}^{|\mathcal{A}|\times d}$ emits one logit per action, $z\in {\mathbb{R}}^{|\mathcal{A}|}$, and a softmax turns the logits into probabilities, ${\pi }_{\theta }(a\mid s)=\mathrm{softmax}(z{)}_a=e^{z_a}/{\sum }_{a^{\prime }}{e}^{z_{a^{\prime }}}$. The output is a length-$|\mathcal{A}|$ probability vector; sampling picks an index. The score has the softmax form ${\nabla }_{\theta }\log {\pi }_{\theta }(a\mid s)={\nabla }_{\theta }{z}_a-{\sum }_{a^{\prime }}{\pi }_{\theta }(a^{\prime }\mid s){\nabla }_{\theta }{z}_{a^{\prime }}$, the gradient of the chosen logit minus the policy-weighted average logit gradient.
• Continuous actions. For an $m$-dimensional action the head emits a mean ${\mu }_{\theta }(s)\in {\mathbb{R}}^m$ and a (log) standard deviation, and the action is drawn from a diagonal Gaussian $a\sim \mathcal{N}({\mu }_{\theta }(s),\mathrm{diag}({\sigma }^2))$. The log-density is $\log {\pi }_{\theta }(a\mid s)=-\frac{1}{2}{\sum }_i\left(\frac{(a_i-{\mu }_i{)}^2}{{\sigma }_i^2}+\log (2\pi {\sigma }_i^2)\right)$, and its gradient in the mean is ${\nabla }_{\mu }\log {\pi }_{\theta }=(a-\mu )/{\sigma }^2$ — the score pushes the mean toward actions that scored well. Parameterizing $\log \sigma$ rather than $\sigma$ keeps the standard deviation positive without a constraint.

The two paradigms differ in what they represent and how they improve it. The table fixes the contrast that the rest of the lesson refines.

The objective

Fix a parameterization. The agent interacts with a Markov decision process, drawing a trajectory $\tau =(s_0,a_0,r_1,s_1,a_1,\dots )$ by sampling $a_t\sim {\pi }_{\theta }(\cdot \mid s_t)$ and $s_{t+1}\sim P(\cdot \mid s_t,a_t)$. The return is the discounted sum of rewards $G(\tau )={\sum }_{t\ge 0}{\gamma }^t{r}_{t+1}$, and the quantity to maximize is its expectation under the trajectory distribution the policy induces.

The obstacle is that the expectation is taken over a distribution that itself depends on $\theta$. We cannot push ${\nabla }_{\theta }$ inside an expectation whose measure moves with $\theta$, and the transition kernel $P$ is unknown, so we cannot differentiate the trajectory probability analytically. The policy gradient theorem removes both problems at once.

The policy gradient theorem

Write the trajectory probability as a product of the initial-state distribution, the policy, and the transitions:

The key device is the log-derivative trick (the score function identity): for any differentiable density, ${\nabla }_{\theta }{p}_{\theta }=p_{\theta }{\nabla }_{\theta }\log p_{\theta }$, which converts the gradient of a probability into a probability times the gradient of its log. That lets the gradient of the integral become an expectation, recoverable from sampled trajectories.

Every method in this lesson builds on this result. The gradient is an expectation, so a Monte Carlo average over sampled trajectories estimates it without a model; each term is a score ${\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)$ weighted by how good the action turned out to be.

REINFORCE

The simplest realization replaces the unknown $Q^{\pi }(s_t,a_t)$ with the actual sampled return-to-go $G_t={\sum }_{k\ge t}{\gamma }^{k-t}{r}_{k+1}$ from a complete episode. This is unbiased: the sampled return is an unbiased estimate of its own expectation $Q^{\pi }$. The result is a Monte Carlo policy gradient.

The estimator is correct in expectation but its variance is enormous. $G_t$ is a sum of many random rewards across a stochastic policy and stochastic dynamics, so a single episode's return can swing wildly, and the gradient estimate swings with it. Learning is slow and noisy, and the noise grows with the episode horizon.

Baselines and the advantage

Variance reduction is the central engineering problem of policy gradients. The first and most important tool is a baseline: subtract a state-dependent quantity $b(s)$ from the return before weighting the score. This leaves the gradient unbiased for any $b(s)$ that does not depend on the action.

The variance-minimizing choice of baseline is close to the state-value $V^{\pi }(s)$, the expected return from $s$. Using $b(s)=V^{\pi }(s)$ replaces the raw return $Q^{\pi }(s,a)$ with the advantage, which measures how much better action $a$ is than the policy's average behavior in that state.

Centering the weight is what shrinks variance. Without a baseline every action in a high-reward state gets reinforced, including the mediocre ones; with the advantage, above-average actions gain probability and below-average actions lose it. The policy gradient takes its canonical, low-variance form.

Actor-critic

REINFORCE waits for a full episode to compute $G_t$. Actor-critic methods estimate the value online with a second learned function, the critic, and let the actor (the policy) step every transition using a bootstrapped advantage rather than a Monte Carlo return. The critic supplies a low-variance, biased estimate where REINFORCE used a high-variance, unbiased one, trading a little bias for a large variance reduction.

The cheapest critic-based advantage is the one-step temporal-difference error, which estimates $A^{\pi }(s_t,a_t)$ from a single transition and the critic:

The same ${\delta }_t$ drives both updates: the actor ascends ${\delta }_t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)$, and the critic descends the squared TD error ${\delta }_t^2$. The loop is symmetric and runs every step.

The synchronous batched form is A2C (advantage actor-critic): run several environment copies in parallel, collect a batch of transitions, and update both networks on the averaged gradient. A3C is the asynchronous predecessor, where many workers compute gradients on their own copies and apply them to shared parameters without locking, decorrelating the data the way a replay buffer does for value methods.

Generalized advantage estimation

The TD error is one extreme of a spectrum. A one-step advantage ${\delta }_t$ has low variance but high bias from the imperfect critic; the full Monte Carlo return has zero bias but high variance. The $n$-step advantage interpolates by bootstrapping after $n$ real rewards. Generalized advantage estimation (GAE) takes an exponentially weighted average over all $n$, governed by a single parameter $\lambda$.

The parameter $\lambda$ controls the bias-variance trade-off, mirroring the role of $\lambda$ in $\mathrm{TD}(\lambda )$ for value estimation. Typical values sit near $\lambda =0.95$, keeping most of the variance reduction while leaving the bias small. The estimator is a geometric blend: each TD residual ${\delta }_{t+l}$ enters the advantage with weight $(\gamma \lambda {)}^l$, so nearby residuals dominate and distant ones fade, which is what keeps variance in check.

Trust-region methods

Vanilla policy gradients are fragile in step size. A learning rate that is fine in one region of policy space causes a catastrophic, irreversible collapse in another, because a small change in parameters $\theta$ can be a large change in the policy distribution ${\pi }_{\theta }$. The remedy is to bound how far the policy may move per update, measuring distance in distribution space rather than parameter space.

TRPO (trust region policy optimization) makes this a hard constraint: maximize a surrogate advantage objective subject to a cap on the average KL divergence between the new and old policies. Write the probability ratio $r_t(\theta )={\pi }_{\theta }(a_t\mid s_t)/{\pi }_{{\theta }_{\text{old}}}(a_t\mid s_t)$.

TRPO works but is heavy: it solves a constrained optimization with conjugate gradients and a line search every update. PPO (proximal policy optimization) achieves nearly the same effect with a first-order objective by clipping the ratio, removing the incentive to move it outside a small interval $[1-ϵ,1+ϵ]$.

The clip is asymmetric in the sign of the advantage. For ${\hat{A}}_t>0$ the objective flattens once $r_t$ exceeds $1+ϵ$, capping how much a good action's probability can rise in one step; for ${\hat{A}}_t<0$ it flattens below $1-ϵ$. Either way the gradient is zeroed outside the interval, so a single batch cannot overshoot.

Deterministic and continuous control

The methods so far carry a stochastic policy. For continuous control an alternative learns a deterministic policy ${\mu }_{\theta }(s)$ and pushes its gradient through a learned $Q$-critic via the chain rule, the deterministic policy gradient. DDPG combines this with a replay buffer and target networks, an off-policy actor-critic that is to continuous actions what DQN is to discrete ones.

DDPG is brittle because the critic overestimates. TD3 adds three fixes: two critics with a $\min$ to curb overestimation, delayed actor updates, and target-policy smoothing. SAC instead makes stochasticity a first-class objective by adding the policy's entropy to the reward, the maximum-entropy formulation, which keeps the policy exploratory and the optimization well-conditioned.

The family spans on-policy and off-policy, discrete and continuous, stochastic and deterministic. The master table places each method on those axes.

Why PPO won practice

Goodfellow's text predates most of this family, so the citations are the papers themselves: the policy-gradient theorem,¹ TRPO,² GAE,³ PPO,⁴ DDPG,⁵ TD3,⁶ and SAC.⁷ Two observations connect this lesson to the full reinforcement-learning subject's policy-gradient module and to modern practice.

PPO became the default for a mundane reason: it is robust and simple. TRPO's KL constraint is provably better-behaved, but its conjugate-gradient inner loop is fiddly to implement and tune. PPO throws away the hard constraint for a clipped objective that any autodiff framework optimizes with plain SGD, and in exchange for a small loss in principle it gains enormous robustness across tasks and hyperparameters. That practicality is why PPO, not TRPO, is the algorithm behind the RLHF of the large-language-model alignment lesson and the next lesson: when the environment is a reward model over text, PPO's stability is worth more than TRPO's tighter guarantee.

The stochastic/deterministic split maps onto exploration. On-policy stochastic methods (PPO, A2C) explore through the policy's own randomness and are simple but sample-hungry; off-policy deterministic methods (DDPG, TD3) reuse a replay buffer for sample efficiency but need injected exploration noise and are brittle. SAC's maximum-entropy objective is the reconciliation: off-policy sample efficiency and built-in stochastic exploration, which is why it became a default for continuous control. The choice among them is, at bottom, a choice about how the agent explores.

The reading is that the policy-gradient theorem is one clean idea — reweight action log-probabilities by how good the action was — and everything after is variance reduction (baselines, advantages, GAE) and step-size control (TRPO, PPO), with the continuous-control branch (DDPG, TD3, SAC) adding a critic-driven gradient for actions the $\arg \max$ of value methods cannot enumerate.

Takeaways

• Policy-gradient methods parameterize ${\pi }_{\theta }(a\mid s)$ and maximize expected return $J(\theta )$ by ascent, skipping the value-function detour and the $\arg \max$; they handle continuous actions and stochastic policies natively.
• The policy gradient theorem uses the log-derivative trick to write ${\nabla }_{\theta }J=\mathbb{E}\left[{\nabla }_{\theta }\log {\pi }_{\theta }(a\mid s)Q^{\pi }(s,a)\right]$, an expectation estimable by sampling, with the unknown dynamics differentiated away.
• REINFORCE plugs in the Monte Carlo return $G_t$; it is unbiased but high-variance. A state-dependent baseline $b(s)$ leaves the gradient unbiased and, taken as $V^{\pi }(s)$, yields the advantage $A^{\pi }=Q^{\pi }-V^{\pi }$ that sharply reduces variance.
• Actor-critic learns the value online with a critic and steps on the TD error ${\delta }_t$; A2C/A3C parallelize it, and GAE dials bias against variance with $\lambda$.
• Trust-region methods bound the per-step policy change: TRPO by a hard KL constraint, PPO by clipping the probability ratio to $[1-ϵ,1+ϵ]$.
• Continuous control specializes further: DDPG and TD3 push a deterministic gradient through a $Q$-critic off-policy, while SAC maximizes reward plus entropy.
• In practice: PPO displaced TRPO because a clipped first-order objective is far simpler and more robust, which is why it is the optimizer behind RLHF; the stochastic/deterministic split is at bottom a choice about how to explore.