---
title: "Policy Gradients and Actor-Critic Methods"
module: Reinforcement Learning
moduleNumber: 11
lessonNumber: 4
order: 1104
summary: >
  Value-based reinforcement learning learns what each state is worth and acts
  greedily; policy-gradient methods skip the detour and optimize a parameterized
  policy directly by ascending the gradient of expected return. The policy
  gradient theorem makes this tractable through the log-derivative trick, turning
  an intractable gradient of an expectation into an expectation of a gradient.
  REINFORCE realizes the idea but suffers high variance; baselines, the
  advantage function, and actor-critic learning reduce it, and trust-region methods
  (TRPO, PPO) keep each update from destroying the policy it just learned.
topics: [Reinforcement Learning]
sources:
  - book: Goodfellow
    ref: "Ch. 1 — directly optimizing a parameterized policy by gradient ascent"
---

[Value-based methods](/deep-learning/reinforcement-learning/deep-q-networks) learn a
value function and read a policy off it: estimate $Q(s,a)$, then act greedily,
$\pi(s) = \arg\max_a Q(s,a)$. The policy is implicit, a byproduct of the value
estimate, and the $\arg\max$ ties the approach to discrete action sets and to a
deterministic, sometimes brittle, greedy rule. **Policy-gradient methods** invert
the dependency: parameterize the policy itself as $\pi_\theta(a \mid s)$ and adjust
$\theta$ to make good trajectories more probable. There is no $\arg\max$, the policy
is stochastic by construction, and continuous action spaces come for free.

> **Definition (Parameterized stochastic policy).** A conditional distribution
> $\pi_\theta(a \mid s)$ over actions given states, differentiable in $\theta$. For
> discrete actions a network outputs logits fed to a softmax,
> $\pi_\theta(a\mid s) = \mathrm{softmax}\parens{f_\theta(s)}_a$; for continuous
> actions it outputs the mean (and possibly variance) of a Gaussian,
> $\pi_\theta(a\mid s) = \mathcal N\parens{\mu_\theta(s),\, \Sigma_\theta(s)}$.

The parameterization is a design choice with concrete tensor shapes. A shared torso
maps the state to a feature vector $h = f_\theta(s) \in \mathbb R^{d}$; a policy head
then reads a distribution off $h$. The two standard heads differ only in what the head
produces.

- **Discrete actions.** A linear head $W h + b$ with $W \in \mathbb R^{|\mathcal A|\times d}$
  emits one logit per action, $z \in \mathbb R^{|\mathcal A|}$, and a softmax turns the
  logits into probabilities, $\pi_\theta(a\mid s) = \mathrm{softmax}(z)_a = e^{z_a}/\sum_{a'} e^{z_{a'}}$.
  The output is a length-$|\mathcal A|$ probability vector; sampling picks an index. The
  score has the softmax form
  $\nabla_\theta \log\pi_\theta(a\mid s) = \nabla_\theta z_a - \sum_{a'}\pi_\theta(a'\mid s)\,\nabla_\theta z_{a'}$,
  the gradient of the chosen logit minus the policy-weighted average logit gradient.
- **Continuous actions.** For an $m$-dimensional action the head emits a mean
  $\mu_\theta(s) \in \mathbb R^{m}$ and a (log) standard deviation, and the action is
  drawn from a diagonal Gaussian $a \sim \mathcal N(\mu_\theta(s), \diag(\sigma^2))$.
  The log-density is $\log\pi_\theta(a\mid s) = -\tfrac12\sum_i\parens{\tfrac{(a_i-\mu_i)^2}{\sigma_i^2} + \log(2\pi\sigma_i^2)}$,
  and its gradient in the mean is $\nabla_\mu \log\pi_\theta = (a-\mu)/\sigma^2$ — the
  score pushes the mean toward actions that scored well. Parameterizing $\log\sigma$
  rather than $\sigma$ keeps the standard deviation positive without a constraint.

$$
% caption: One shared torso feeds two head types: a softmax over the $|\mathcal A|$ discrete actions, or the mean $\mu$ and (log) standard deviation $\sigma$ of a Gaussian for continuous actions.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=20mm, minimum height=10mm, align=center},
  small/.style={draw, minimum width=15mm, minimum height=8mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \node[box] (s) at (0,0) {\texttt{state s}};
  \node[box, draw=acc, text=acc, thick] (torso) at (3.2,0) {\texttt{shared torso}};
  % discrete head (top)
  \node[small, draw=green, text=green, thick] (disc) at (7.0,1.5) {\texttt{softmax head}};
  \node[align=left, font=\scriptsize, anchor=west] at (9.0,1.5)
    {\texttt{probs over $|A|$ actions}};
  % continuous head (bottom)
  \node[small, draw=red, text=red, thick] (cont) at (7.0,-1.5) {\texttt{Gaussian head}};
  \node[align=left, font=\scriptsize, anchor=west] at (9.0,-1.5)
    {\texttt{mean mu, log-std sigma}};
  \draw[->, black, thick] (s) -- (torso);
  \draw[->, acc, thick] (torso) -- (disc)
    node[pos=0.5, above=2.5pt, sloped, font=\footnotesize, text=acc] {\texttt{discrete}};
  \draw[->, acc, thick] (torso) -- (cont)
    node[pos=0.5, below=2.5pt, sloped, font=\footnotesize, text=acc] {\texttt{continuous}};
  \draw[->, green, thick] (disc) -- (9.0,1.5);
  \draw[->, red, thick] (cont) -- (9.0,-1.5);
\end{tikzpicture}
$$

The two paradigms differ in what they represent and how they improve it. The table
fixes the contrast that the rest of the lesson refines.

| | Value-based | Policy-based |
| --- | --- | --- |
| Learns | $Q(s,a)$ or $V(s)$ | $\pi_\theta(a\mid s)$ directly |
| Policy | implicit ($\arg\max_a Q$) | explicit, parameterized |
| Action space | discrete (needs the $\arg\max$) | discrete _or_ continuous |
| Output | deterministic greedy | stochastic |
| Optimizes | Bellman error (indirect) | expected return (direct) |
| Convergence | can oscillate / diverge with approximation | smooth gradient ascent, local optimum |

$$
% caption: Value-based RL routes through an $\arg\max$ over learned $Q$-values; policy-based RL parameterizes $\pi_\theta$ and ascends the return gradient directly.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=25mm, minimum height=11mm, align=center},
  lbl/.style={font=\scriptsize, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % --- value-based (top row) ---
  \node[box] (s1) at (0,1.6) {\texttt{state s}};
  \node[box, draw=red, text=red, thick] (q) at (3.8,1.6) {\texttt{value Q(s,a)}};
  \node[box] (am) at (7.6,1.6) {\texttt{arg max}};
  \node[box] (a1) at (11.2,1.6) {\texttt{action a}};
  \draw[->, black, thick] (s1) -- (q);
  \draw[->, red, thick] (q) -- (am);
  \draw[->, black, thick] (am) -- (a1);
  \node[lbl, anchor=south] at (3.8,2.3) {\texttt{value-based}};
  % --- policy-based (bottom row) ---
  \node[box] (s2) at (0,-1.0) {\texttt{state s}};
  \node[box, draw=acc, text=acc, thick] (p) at (3.8,-1.0) {\texttt{policy net}};
  \node[box] (a2) at (7.6,-1.0) {\texttt{sample a}};
  \draw[->, black, thick] (s2) -- (p);
  \draw[->, acc, thick] (p) -- (a2);
  \node[lbl, anchor=north] at (3.8,-1.7) {\texttt{policy-based}};
  % return gradient feedback to the policy net
  \draw[->, acc, thick, dashed] (a2.south) .. controls (7.6,-2.7) and (3.8,-2.7) .. (p.south)
    node[pos=0.5, below, lbl, text=acc] {\texttt{ascend return gradient}};
\end{tikzpicture}
$$

## The objective

Fix a parameterization. The agent interacts with a Markov decision process, drawing
a trajectory $\tau = (s_0, a_0, r_1, s_1, a_1, \dots)$ by sampling
$a_t \sim \pi_\theta(\cdot \mid s_t)$ and $s_{t+1} \sim P(\cdot \mid s_t, a_t)$. The
return is the discounted sum of rewards $G(\tau) = \sum_{t\ge 0} \gamma^t r_{t+1}$,
and the quantity to maximize is its expectation under the trajectory distribution the
policy induces.

> **Definition (Policy objective).** The expected return as a function of the policy
> parameters,
> $$
> J(\theta) = \mathbb E_{\tau \sim \pi_\theta}\!\brackets{G(\tau)}
> = \mathbb E_{\tau \sim \pi_\theta}\!\brackets{\textstyle\sum_{t\ge 0} \gamma^t r_{t+1}}.
> $$
> The training problem is $\max_\theta J(\theta)$, solved by gradient _ascent_
> $\theta \gets \theta + \alpha\,\nabla_\theta J(\theta)$.

The obstacle is that the expectation is taken over a distribution that itself depends
on $\theta$. We cannot push $\nabla_\theta$ inside an expectation whose measure moves
with $\theta$, and the transition kernel $P$ is unknown, so we cannot differentiate
the trajectory probability analytically. The policy gradient theorem removes both
problems at once.

## The policy gradient theorem

Write the trajectory probability as a product of the initial-state distribution, the
policy, and the transitions:

$$
p_\theta(\tau) = \rho_0(s_0)\prod_{t\ge 0} \pi_\theta(a_t\mid s_t)\,P(s_{t+1}\mid s_t,a_t).
$$

The key device is the **log-derivative trick** (the _score function_ identity):
for any differentiable density, $\nabla_\theta p_\theta = p_\theta\,\nabla_\theta \log p_\theta$,
which converts the gradient of a probability into a probability times the gradient of
its log. That lets the gradient of the integral become an expectation, recoverable
from sampled trajectories.

> **Theorem (Policy gradient theorem).** The gradient of the expected return is
> $$
> \nabla_\theta J(\theta)
> = \mathbb E_{\tau\sim\pi_\theta}\!\brackets{\textstyle\sum_{t\ge 0}
> \nabla_\theta \log \pi_\theta(a_t\mid s_t)\;Q^{\pi_\theta}(s_t,a_t)},
> $$
> where $Q^{\pi_\theta}(s,a) = \mathbb E_{\pi_\theta}\!\brackets{G \mid s_0{=}s,\, a_0{=}a}$
> is the action-value of the current policy. The unknown transition dynamics
> $P$ and the initial distribution $\rho_0$ do not appear.

> **Proof.** Start from $J(\theta) = \int p_\theta(\tau)\,G(\tau)\,d\tau$ and
> differentiate, applying the log-derivative trick $\nabla_\theta p_\theta = p_\theta \nabla_\theta \log p_\theta$:
> $$
> \nabla_\theta J(\theta) = \int \nabla_\theta p_\theta(\tau)\,G(\tau)\,d\tau
> = \int p_\theta(\tau)\,\nabla_\theta \log p_\theta(\tau)\,G(\tau)\,d\tau
> = \mathbb E_{\tau}\!\brackets{\nabla_\theta \log p_\theta(\tau)\,G(\tau)}.
> $$
> Take the log of the trajectory factorization. The terms $\log\rho_0(s_0)$ and
> $\log P(s_{t+1}\mid s_t,a_t)$ do not depend on $\theta$, so they vanish under
> $\nabla_\theta$, leaving only the policy terms:
> $$
> \nabla_\theta \log p_\theta(\tau) = \sum_{t\ge 0}\nabla_\theta \log \pi_\theta(a_t\mid s_t).
> $$
> Substituting gives $\nabla_\theta J = \mathbb E_\tau\!\brackets{\big(\sum_t \nabla_\theta \log\pi_\theta(a_t\mid s_t)\big)G(\tau)}$.
> A causality argument removes rewards earned _before_ action $a_t$: action $a_t$
> cannot influence past reward, so its expected contribution is uncorrelated with
> those terms and drops out. The remaining future return, conditioned on
> $(s_t,a_t)$, is exactly $Q^{\pi_\theta}(s_t,a_t)$, yielding the stated form. $\qed$

Every method in this lesson builds on this result. The gradient is an expectation, so a
Monte Carlo average over sampled trajectories estimates it without a model; each term
is a score $\nabla_\theta \log\pi_\theta(a_t\mid s_t)$ weighted by how good the action
turned out to be.

> **Remark (What the update does).** A step along
> $\nabla_\theta \log\pi_\theta(a\mid s)\,Q^{\pi}(s,a)$ raises the log-probability of
> action $a$ in state $s$ in proportion to its value $Q^\pi(s,a)$. Good actions
> ($Q>0$) become more likely, bad ones ($Q<0$) less likely. The policy is reinforced
> by outcome, which is the origin of the name REINFORCE.

## REINFORCE

The simplest realization replaces the unknown $Q^{\pi}(s_t,a_t)$ with the actual
sampled return-to-go $G_t = \sum_{k\ge t}\gamma^{k-t} r_{k+1}$ from a complete
episode. This is unbiased: the sampled return is an unbiased estimate of its own
expectation $Q^\pi$. The result is a Monte Carlo policy gradient.

> **Definition (REINFORCE estimator).** The single-trajectory estimate
> $$
> \widehat{\nabla_\theta J} = \sum_{t\ge 0}\nabla_\theta \log\pi_\theta(a_t\mid s_t)\,G_t,
> \qquad G_t = \sum_{k\ge t}\gamma^{\,k-t} r_{k+1},
> $$
> averaged over one or more sampled episodes. It requires only the ability to sample
> the policy and observe returns, with no model and no value function.

```algorithm
caption: $\textsc{Reinforce}(\pi_\theta, \alpha, \gamma)$ — Monte Carlo policy gradient
initialize policy parameters $\theta$
repeat
  sample an episode $s_0, a_0, r_1, \dots, s_{T-1}, a_{T-1}, r_T \sim \pi_\theta$
  for $t = 0$ to $T-1$ do
    $G_t \gets \sum_{k=t}^{T-1} \gamma^{\,k-t}\, r_{k+1}$ // return-to-go
    $\theta \gets \theta + \alpha\,\gamma^{\,t}\, G_t\,\nabla_\theta \log \pi_\theta(a_t \mid s_t)$ // ascend
until converged
return $\theta$
```

> **Worked example (one REINFORCE step).** Take a two-action softmax policy in a state
> $s$ with logits $z = (z_{\text{left}}, z_{\text{right}}) = (0.4,\, 0.0)$, so
> $\pi(\text{left}\mid s) = e^{0.4}/(e^{0.4}+e^{0.0}) = 1.492/2.492 = 0.599$ and
> $\pi(\text{right}\mid s) = 0.401$. The agent samples $a = \text{right}$ and the episode
> from there earns return-to-go $G_t = +2$. The softmax score in logit space is
> $\nabla_{z}\log\pi(\text{right}\mid s) = e_{\text{right}} - \pi(\cdot\mid s) = (0,1) - (0.599, 0.401) = (-0.599,\, 0.599)$,
> where $e_{\text{right}}$ is the indicator of the chosen action. Weighting by $G_t=2$
> gives the logit-space gradient $G_t\,\nabla_z\log\pi = (-1.198,\, 1.199)$. With learning
> rate $\alpha = 0.1$ the logits move to $z \gets z + \alpha\,G_t\,\nabla_z\log\pi = (0.4 - 0.120,\; 0.0 + 0.120) = (0.280,\, 0.120)$.
> Recomputing, $\pi(\text{right}\mid s)$ rises from $0.401$ to $0.460$: the sampled action
> earned positive return, so its probability went up, exactly as the score-times-return
> update prescribes. Had $G_t$ been negative, the same arithmetic would have _lowered_ it.

The estimator is correct in expectation but its variance is enormous. $G_t$ is a sum
of many random rewards across a stochastic policy and stochastic dynamics, so a single
episode's return can swing wildly, and the gradient estimate swings with it. Learning
is slow and noisy, and the noise grows with the episode horizon.

$$
% caption: REINFORCE's gradient estimate (blue) has high variance around the true gradient (black); each dot is one episode's noisy single-sample estimate.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{black}{HTML}{000000}
  \draw[->, thick] (0,0) -- (8.0,0) node[right, font=\footnotesize] {\texttt{episode}};
  \draw[->, thick] (0,-1.9) -- (0,1.9) node[above, font=\footnotesize] {\texttt{gradient estimate}};
  % true gradient (constant, black)
  \draw[black, very thick] (0,0.35) -- (7.6,0.35);
  \node[black, anchor=south west, font=\footnotesize] at (0.05,0.42) {\texttt{true gradient}};
  % scattered noisy single-sample estimates (acc dots) around the true line
  \foreach \x/\y in {0.5/1.55, 1.0/-1.2, 1.5/1.0, 2.0/-0.55, 2.5/1.7, 3.0/-1.55,
    3.5/0.2, 4.0/1.35, 4.5/-1.0, 5.0/1.6, 5.5/-1.35, 6.0/0.85, 6.5/-0.7, 7.0/1.2}
    \fill[acc] (\x,\y) circle (2.4pt);
  \node[acc, anchor=south, font=\footnotesize] at (2.6,1.78) {\texttt{single-sample estimates}};
\end{tikzpicture}
$$

## Baselines and the advantage

Variance reduction is the central engineering problem of policy gradients. The first
and most important tool is a **baseline**: subtract a state-dependent quantity $b(s)$
from the return before weighting the score. This leaves
the gradient unbiased for _any_ $b(s)$ that does not depend on the action.

> **Theorem (Baseline invariance).** For any function $b(s)$ of state alone,
> $$
> \mathbb E_{\tau}\!\brackets{\textstyle\sum_t \nabla_\theta\log\pi_\theta(a_t\mid s_t)\,b(s_t)} = 0,
> $$
> so subtracting $b(s_t)$ from the weight changes the variance of the estimator but
> not its expectation.

> **Proof.** Condition on a fixed state $s$ and take the expectation over the action
> drawn from $\pi_\theta(\cdot\mid s)$. Since $b(s)$ is constant in $a$, pull it out
> and apply the score identity in reverse:
> $$
> \mathbb E_{a\sim\pi_\theta}\!\brackets{\nabla_\theta\log\pi_\theta(a\mid s)\,b(s)}
> = b(s)\sum_a \pi_\theta(a\mid s)\,\nabla_\theta\log\pi_\theta(a\mid s)
> = b(s)\sum_a \nabla_\theta \pi_\theta(a\mid s).
> $$
> The sum and gradient commute, and $\sum_a \pi_\theta(a\mid s) = 1$ is constant, so
> $\sum_a \nabla_\theta\pi_\theta(a\mid s) = \nabla_\theta 1 = 0$. The conditional
> expectation is zero at every $s$, hence the full expectation is zero. $\qed$

The variance-minimizing choice of baseline is close to the state-value $V^\pi(s)$, the
expected return from $s$. Using $b(s) = V^\pi(s)$ replaces the raw return $Q^\pi(s,a)$
with the **advantage**, which measures how much better action $a$ is than the policy's
average behavior in that state.

> **Definition (Advantage function).** The action-value relative to the state-value,
> $$
> A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s),
> \qquad V^\pi(s) = \mathbb E_{a\sim\pi_\theta}\!\brackets{Q^\pi(s,a)}.
> $$
> By construction $\mathbb E_{a\sim\pi}\!\brackets{A^\pi(s,a)} = 0$: it is positive for
> better-than-average actions, negative for worse, and centered at zero.

Centering the weight is what shrinks variance. Without a baseline every action
in a high-reward state gets reinforced, including the mediocre ones; with the advantage,
above-average actions gain probability and below-average actions lose it. The
policy gradient takes its canonical, low-variance form.

$$
\nabla_\theta J(\theta) = \mathbb E_{\pi_\theta}\!\brackets{\nabla_\theta \log\pi_\theta(a\mid s)\;A^\pi(s,a)}.
$$

$$
% caption: Subtracting the baseline $V(s)$ recenters returns: raw returns (blue) all reinforce; advantages (split green/red) push above-average actions up and below-average down.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % --- left: raw returns, all positive ---
  \begin{scope}
    \draw[->, thick] (0,0) -- (4.4,0) node[right, font=\footnotesize] {\texttt{action}};
    \draw[->, thick] (0,-0.2) -- (0,3.2) node[above, font=\footnotesize] {\texttt{weight}};
    \foreach \x/\h in {0.7/1.6, 1.5/2.7, 2.3/1.1, 3.1/2.2} {
      \draw[acc, very thick, fill=acc!15] (\x-0.22,0) rectangle (\x+0.22,\h);
    }
    \node[anchor=north, font=\footnotesize] at (2.0,-0.4) {\texttt{raw return G}};
  \end{scope}
  % --- right: advantages centered at zero ---
  \begin{scope}[xshift=6.2cm]
    \draw[->, thick] (0,-1.6) -- (4.4,-1.6) node[right, font=\footnotesize] {\texttt{action}};
    \draw[->, thick] (0,-1.8) -- (0,1.8) node[above, font=\footnotesize] {\texttt{advantage}};
    \draw[black, thick] (0,0) -- (4.2,0);
    \node[black, anchor=west, font=\footnotesize] at (3.35,0.24) {\texttt{baseline}};
    \draw[red, very thick, fill=red!15] (0.7-0.22,0) rectangle (0.7+0.22,-0.5);
    \draw[green, very thick, fill=green!15] (1.5-0.22,0) rectangle (1.5+0.22,1.1);
    \draw[red, very thick, fill=red!15] (2.3-0.22,0) rectangle (2.3+0.22,-1.0);
    \draw[green, very thick, fill=green!15] (3.1-0.22,0) rectangle (3.1+0.22,0.6);
    \node[anchor=north, font=\footnotesize] at (2.0,-1.8) {\texttt{advantage (centered)}};
  \end{scope}
\end{tikzpicture}
$$

## Actor-critic

REINFORCE waits for a full episode to compute $G_t$. **Actor-critic** methods estimate
the value online with a second learned function, the **critic**, and let the **actor**
(the policy) step every transition using a bootstrapped advantage rather than a Monte
Carlo return. The critic supplies a low-variance, biased estimate where REINFORCE used
a high-variance, unbiased one, trading a little bias for a large variance reduction.

> **Definition (Actor-critic).** Two parameterized functions trained jointly: an
> **actor** $\pi_\theta(a\mid s)$ updated by the policy gradient, and a **critic**
> $V_w(s)$ (or $Q_w$) updated to fit the value of the current policy. The actor's
> weight is the critic-estimated advantage; the critic's target is the bootstrapped
> Bellman return.

The cheapest critic-based advantage is the one-step **temporal-difference error**,
which estimates $A^\pi(s_t,a_t)$ from a single transition and the critic:

$$
\delta_t = r_{t+1} + \gamma\,V_w(s_{t+1}) - V_w(s_t) \;\approx\; A^\pi(s_t,a_t).
$$

The same $\delta_t$ drives both updates: the actor ascends $\delta_t\,\nabla_\theta\log\pi_\theta(a_t\mid s_t)$,
and the critic descends the squared TD error $\delta_t^2$. The loop is symmetric and
runs every step.

$$
% caption: The actor-critic loop: the actor picks actions, the environment returns reward and next state, and the critic's TD error $\delta$ trains both the actor and itself.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=24mm, minimum height=13mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \node[box, draw=acc, text=acc, thick] (actor) at (0,1.7)
    {\texttt{actor}\\\texttt{(policy net)}};
  \node[box] (env) at (5.8,1.7) {\texttt{environment}};
  \node[box, draw=red, text=red, thick] (critic) at (5.8,-1.4)
    {\texttt{critic}\\\texttt{(value net)}};
  % actor -> env: action
  \draw[->, acc, thick] (actor) -- (env)
    node[pos=0.5, above, font=\footnotesize, text=acc] {\texttt{action a}};
  % env -> critic: reward, state
  \draw[->, black, thick] (env) -- (critic)
    node[pos=0.5, right, font=\footnotesize] {\texttt{reward, next state}};
  % critic -> actor: TD error trains actor
  \draw[->, green, thick] (critic) -- (actor)
    node[pos=0.5, above, sloped, yshift=3pt, font=\footnotesize, text=green] {\texttt{TD error}};
  % critic self-update
  \draw[->, red, thick] (critic.south) .. controls (8.2,-2.9) and (8.2,-0.1) .. (critic.east)
    node[pos=0.5, right, font=\footnotesize, text=red] {\texttt{fit value}};
\end{tikzpicture}
$$

The synchronous batched form is **A2C** (advantage actor-critic): run several
environment copies in parallel, collect a batch of transitions, and update both
networks on the averaged gradient. **A3C** is the asynchronous predecessor, where many
workers compute gradients on their own copies and apply them to shared parameters
without locking, decorrelating the data the way a replay buffer does for value methods.

```algorithm
caption: $\textsc{A2C}(\pi_\theta, V_w, \alpha, \beta, \gamma)$ — one synchronous advantage actor-critic update
initialize actor params $\theta$ and critic params $w$
repeat
  collect a batch of transitions $(s_t, a_t, r_{t+1}, s_{t+1})$ from parallel actors
  for each transition do
    $\delta_t \gets r_{t+1} + \gamma\, V_w(s_{t+1}) - V_w(s_t)$ // TD error = advantage estimate
  $\theta \gets \theta + \alpha\, \overline{\delta_t\, \nabla_\theta \log \pi_\theta(a_t \mid s_t)}$ // actor ascends
  $w \gets w + \beta\, \overline{\delta_t\, \nabla_w V_w(s_t)}$ // critic fits the return
until converged
return $\theta,\, w$
```

## Generalized advantage estimation

The TD error is one extreme of a spectrum. A one-step advantage $\delta_t$ has low
variance but high bias from the imperfect critic; the full Monte Carlo return has zero
bias but high variance. The $n$-step advantage interpolates by bootstrapping after $n$
real rewards. **Generalized advantage estimation** (GAE) takes an exponentially
weighted average over all $n$, governed by a single parameter $\lambda$.

> **Definition (Generalized advantage estimation).** With TD residuals
> $\delta_t = r_{t+1} + \gamma V(s_{t+1}) - V(s_t)$, the GAE advantage is
> $$
> \hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l\ge 0}(\gamma\lambda)^l\,\delta_{t+l}.
> $$
> $\lambda = 0$ recovers the one-step TD advantage $\delta_t$ (low variance, biased);
> $\lambda = 1$ recovers the Monte Carlo advantage $\sum_l \gamma^l \delta_{t+l} = G_t - V(s_t)$
> (unbiased, high variance). Intermediate $\lambda$ trades the two.

The parameter $\lambda$ controls the bias-variance trade-off, mirroring the role of
$\lambda$ in $\mathrm{TD}(\lambda)$ for value estimation. Typical values sit near
$\lambda = 0.95$, keeping most of the variance reduction while leaving the bias small.
The estimator is a geometric blend: each TD residual $\delta_{t+l}$ enters the advantage
with weight $(\gamma\lambda)^l$, so nearby residuals dominate and distant ones fade,
which is what keeps variance in check.

$$
% caption: GAE weights the TD residual $\delta_{t+l}$ by $(\gamma\lambda)^l$: near residuals count most, far ones decay geometrically. Larger $\lambda$ flattens the decay toward the Monte Carlo return.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, thick] (0,0) -- (8.6,0) node[right, font=\footnotesize] {\texttt{offset l}};
  \draw[->, thick] (0,0) -- (0,3.6) node[above, font=\footnotesize] {\texttt{weight}};
  % lambda = 0.95 gamma = 0.99 => decay ~0.94 per step (acc, slow)
  \foreach \l in {0,1,2,3,4,5,6,7} {
    \pgfmathsetmacro\hh{2.2*pow(0.94,\l)}
    \draw[acc, very thick, fill=acc!15]
      ({0.55+\l*0.82-0.15},0) rectangle ({0.55+\l*0.82+0.15},\hh);
  }
  % lambda = 0.5 => faster decay (red, offset bars)
  \foreach \l in {0,1,2,3,4,5,6,7} {
    \pgfmathsetmacro\hh{2.2*pow(0.5,\l)}
    \draw[red, very thick, fill=red!12]
      ({0.55+\l*0.82-0.15+0.02},0) rectangle ({0.55+\l*0.82+0.15+0.02},\hh);
  }
  % legend swatches + labels in the clear top-right band
  \draw[acc, very thick, fill=acc!15] (4.9,3.3) rectangle (5.2,3.5);
  \node[acc, anchor=west, font=\footnotesize] at (5.3,3.4) {\texttt{lambda = 0.95}};
  \draw[red, very thick, fill=red!12] (4.9,2.75) rectangle (5.2,2.95);
  \node[red, anchor=west, font=\footnotesize] at (5.3,2.85) {\texttt{lambda = 0.5}};
\end{tikzpicture}
$$

| | $n$-step / $\lambda$ | Bias | Variance | Bootstraps after |
| --- | --- | --- | --- | --- |
| One-step TD | $\lambda = 0$ | high | low | $1$ reward |
| GAE | $0 < \lambda < 1$ | tunable | tunable | weighted mix |
| Monte Carlo | $\lambda = 1$ | none | high | end of episode |

$$
% caption: The bias-variance trade in advantage estimation: short horizons ($\lambda \to 0$) are low-variance but biased; long horizons ($\lambda \to 1$) are unbiased but high-variance.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, thick] (0,0) -- (8.0,0) node[right, font=\footnotesize] {\texttt{horizon (lambda)}};
  \draw[->, thick] (0,0) -- (0,3.6) node[above, font=\footnotesize] {\texttt{magnitude}};
  % bias decreasing (acc)
  \draw[acc, very thick] plot[domain=0.3:7.4, samples=80]
    (\x, {2.9*exp(-0.55*\x) + 0.15});
  \node[acc, anchor=south west, font=\footnotesize] at (0.6,2.5) {\texttt{bias}};
  % variance increasing (red)
  \draw[red, very thick] plot[domain=0.3:7.4, samples=80]
    (\x, {0.2 + 0.32*\x*\x*0.16 + 0.18*\x});
  \node[red, anchor=north east, font=\footnotesize] at (7.3,3.0) {\texttt{variance}};
  % sweet spot marker
  \draw[black, dashed] (3.7,0) -- (3.7,2.4);
  \node[black, anchor=south, font=\footnotesize] at (3.7,2.45) {\texttt{sweet spot}};
  \node[anchor=north east, font=\footnotesize] at (-0.05,-0.05) {\texttt{0}};
\end{tikzpicture}
$$

## Trust-region methods

Vanilla policy gradients are fragile in step size. A learning rate that is fine in one
region of policy space causes a catastrophic, irreversible collapse in another, because
a small change in parameters $\theta$ can be a large change in the policy distribution
$\pi_\theta$. The remedy is to bound how far the policy may move per update, measuring
distance in distribution space rather than parameter space.

> **Definition (Natural policy gradient).** Precondition the gradient by the inverse
> Fisher information matrix $F(\theta) = \mathbb E\!\brackets{\nabla_\theta\log\pi_\theta\,\nabla_\theta\log\pi_\theta^{\mathsf T}}$,
> $$
> \tilde\nabla_\theta J = F(\theta)^{-1}\,\nabla_\theta J(\theta).
> $$
> The natural gradient is the steepest-ascent direction under the KL metric on the
> policy distribution, invariant to how the policy is parameterized.

**TRPO** (trust region policy optimization) makes this a hard constraint: maximize a
surrogate advantage objective subject to a cap on the average KL divergence between the
new and old policies. Write the probability ratio $r_t(\theta) = \pi_\theta(a_t\mid s_t)/\pi_{\theta_{\text{old}}}(a_t\mid s_t)$.

$$
\max_\theta\; \mathbb E_t\!\brackets{r_t(\theta)\,\hat A_t}
\quad\text{subject to}\quad
\mathbb E_t\!\brackets{\mathrm{KL}\parens{\pi_{\theta_{\text{old}}}(\cdot\mid s_t)\,\|\,\pi_\theta(\cdot\mid s_t)}} \le \delta.
$$

TRPO works but is heavy: it solves a constrained optimization with conjugate gradients
and a line search every update. **PPO** (proximal policy optimization) achieves nearly
the same effect with a first-order objective by _clipping_ the ratio, removing the
incentive to move it outside a small interval $[1-\epsilon,\,1+\epsilon]$.

> **Definition (PPO clipped surrogate).** With ratio $r_t(\theta)$ and advantage
> estimate $\hat A_t$,
> $$
> L^{\mathrm{CLIP}}(\theta) = \mathbb E_t\!\brackets{\min\parens{r_t(\theta)\,\hat A_t,\;\
> \mathrm{clip}\parens{r_t(\theta),\, 1-\epsilon,\, 1+\epsilon}\,\hat A_t}}.
> $$
> The $\min$ takes the more pessimistic of the clipped and unclipped objectives, so the
> update gains nothing from pushing $r_t$ beyond the trust interval; typical $\epsilon = 0.2$.

The clip is asymmetric in the sign of the advantage. For $\hat A_t > 0$ the objective
flattens once $r_t$ exceeds $1+\epsilon$, capping how much a good action's probability
can rise in one step; for $\hat A_t < 0$ it flattens below $1-\epsilon$. Either way the
gradient is zeroed outside the interval, so a single batch cannot overshoot.

$$
% caption: PPO's clipped objective in the ratio $r$. For positive advantage (blue) the gain saturates above $1+\epsilon$; for negative advantage (red) it saturates below $1-\epsilon$.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, thick] (0,0) -- (7.6,0) node[right, font=\footnotesize] {\texttt{ratio r}};
  \draw[->, thick] (0,-2.1) -- (0,2.1) node[above, font=\footnotesize] {\texttt{objective}};
  % center r = 1 at x = 3.5; 1-eps at 2.4, 1+eps at 4.6
  \draw[black, dashed] (3.5,-0.1) -- (3.5,1.9);
  \node[black, anchor=south, font=\footnotesize] at (3.5,1.9) {\texttt{r = 1}};
  \draw[black, dashed] (2.4,-1.9) -- (2.4,1.9);
  \node[black, anchor=north, font=\footnotesize] at (2.4,-1.95) {\texttt{1 - eps}};
  \draw[black, dashed] (4.6,-1.9) -- (4.6,1.9);
  \node[black, anchor=north, font=\footnotesize] at (4.6,-1.95) {\texttt{1 + eps}};
  % positive advantage (acc): rises with r, then flat above 1+eps
  \draw[acc, very thick] (0.6,-1.45) -- (4.6,1.5) -- (7.2,1.5);
  \node[acc, anchor=south east, font=\footnotesize] at (7.1,1.55) {\texttt{A $>$ 0}};
  % negative advantage (red): falls with r, then flat below 1-eps
  \draw[red, very thick] (0.6,1.1) -- (2.4,-1.1) -- (7.2,-1.1);
  \node[red, anchor=north east, font=\footnotesize] at (7.1,-1.15) {\texttt{A $<$ 0}};
\end{tikzpicture}
$$

```algorithm
caption: $\textsc{PPO}(\pi_\theta, V_w, \epsilon, K)$ — clipped proximal policy optimization
initialize actor $\theta$ and critic $w$
repeat
  run policy $\pi_{\theta_{\text{old}}}$, collect trajectories, store $\pi_{\theta_{\text{old}}}(a_t\mid s_t)$
  compute advantages $\hat A_t$ by GAE and value targets $\hat R_t$
  for $K$ epochs over the batch do
    $r_t \gets \pi_\theta(a_t\mid s_t) / \pi_{\theta_{\text{old}}}(a_t\mid s_t)$ // probability ratio
    $L^{\mathrm{CLIP}} \gets \overline{\min\parens{r_t \hat A_t,\ \mathrm{clip}(r_t, 1-\epsilon, 1+\epsilon)\,\hat A_t}}$
    $\theta \gets \theta + \alpha\,\nabla_\theta L^{\mathrm{CLIP}}$ // clipped actor step
    $w \gets w - \beta\,\nabla_w\, \overline{\parens{V_w(s_t) - \hat R_t}^2}$ // critic regression
  $\theta_{\text{old}} \gets \theta$
until converged
return $\theta$
```

## Deterministic and continuous control

The methods so far carry a stochastic policy. For continuous control an alternative
learns a **deterministic** policy $\mu_\theta(s)$ and pushes its gradient through a
learned $Q$-critic via the chain rule, the deterministic policy gradient. **DDPG**
combines this with a replay buffer and target networks, an off-policy actor-critic that
is to continuous actions what DQN is to discrete ones.

> **Definition (Deterministic policy gradient).** For a deterministic actor
> $\mu_\theta(s)$ and critic $Q_w(s,a)$, the actor gradient flows through the critic,
> $$
> \nabla_\theta J = \mathbb E_s\!\brackets{\nabla_\theta \mu_\theta(s)\;\nabla_a Q_w(s,a)\big|_{a=\mu_\theta(s)}}.
> $$
> There is no score function: the gradient is the critic's sensitivity to the action,
> backpropagated into the actor.

DDPG is brittle because the critic overestimates. **TD3** adds three fixes: two critics
with a $\min$ to curb overestimation, delayed actor updates, and target-policy
smoothing. **SAC** instead makes stochasticity a first-class objective by adding
the policy's entropy to the reward, the **maximum-entropy** formulation, which keeps the
policy exploratory and the optimization well-conditioned.

> **Definition (Maximum-entropy objective).** Augment the return with the policy
> entropy, weighted by a temperature $\alpha$,
> $$
> J(\theta) = \mathbb E_{\pi_\theta}\!\brackets{\textstyle\sum_t \gamma^t\parens{r_{t+1} + \alpha\,\mathcal H\parens{\pi_\theta(\cdot\mid s_t)}}}.
> $$
> Maximizing entropy alongside reward favors policies that succeed _and_ stay random,
> improving exploration and robustness; $\alpha$ trades reward against entropy.

The family spans on-policy and off-policy, discrete and continuous, stochastic and
deterministic. The master table places each method on those axes.

| Method | Policy | Action space | On/off-policy | Key idea |
| --- | --- | --- | --- | --- |
| REINFORCE | stochastic | discrete or continuous | on-policy | Monte Carlo policy gradient |
| A2C / A3C | stochastic | discrete or continuous | on-policy | learned critic gives the advantage |
| TRPO | stochastic | discrete or continuous | on-policy | KL-constrained trust region |
| PPO | stochastic | discrete or continuous | on-policy | clipped surrogate, first-order |
| DDPG | deterministic | continuous | off-policy | deterministic gradient + replay |
| TD3 | deterministic | continuous | off-policy | twin critics, delayed updates |
| SAC | stochastic | continuous | off-policy | maximum-entropy actor-critic |

## Why PPO won practice

Goodfellow's text predates most of this family, so the citations are the papers themselves: the policy-gradient theorem,[^sutton-pg] TRPO,[^schulman-trpo] GAE,[^schulman-gae] PPO,[^schulman-ppo] DDPG,[^lillicrap-ddpg] TD3,[^fujimoto-td3] and SAC.[^haarnoja-sac] Two observations connect this lesson to the full [reinforcement-learning subject's policy-gradient module](/reinforcement-learning/approximation/policy-gradient-methods) and to modern practice.

**PPO became the default for a mundane reason: it is robust and simple.** TRPO's KL constraint is provably better-behaved, but its conjugate-gradient inner loop is fiddly to implement and tune. PPO throws away the hard constraint for a clipped objective that any autodiff framework optimizes with plain SGD, and in exchange for a small loss in principle it gains enormous robustness across tasks and hyperparameters. That practicality is why PPO, not TRPO, is the algorithm behind the RLHF of the [large-language-model alignment](/deep-learning/large-models-and-agents/scaling-inference-and-alignment) lesson and the [next lesson](/deep-learning/reinforcement-learning/rl-from-human-feedback): when the "environment" is a reward model over text, PPO's stability is worth more than TRPO's tighter guarantee.

**The stochastic/deterministic split maps onto exploration.** On-policy stochastic methods (PPO, A2C) explore through the policy's own randomness and are simple but sample-hungry; off-policy deterministic methods (DDPG, TD3) reuse a replay buffer for sample efficiency but need injected exploration noise and are brittle. SAC's maximum-entropy objective is the reconciliation: off-policy sample efficiency _and_ built-in stochastic exploration, which is why it became a default for continuous control. The choice among them is, at bottom, a choice about how the agent explores.

The reading is that the policy-gradient theorem is one clean idea — reweight action log-probabilities by how good the action was — and everything after is variance reduction (baselines, advantages, GAE) and step-size control (TRPO, PPO), with the continuous-control branch (DDPG, TD3, SAC) adding a critic-driven gradient for actions the $\arg\max$ of value methods cannot enumerate.

## Takeaways

- **Policy-gradient methods** parameterize $\pi_\theta(a\mid s)$ and maximize expected
  return $J(\theta)$ by ascent, skipping the value-function detour and the $\arg\max$;
  they handle continuous actions and stochastic policies natively.
- The **policy gradient theorem** uses the log-derivative trick to write
  $\nabla_\theta J = \mathbb E\!\brackets{\nabla_\theta\log\pi_\theta(a\mid s)\,Q^\pi(s,a)}$,
  an expectation estimable by sampling, with the unknown dynamics differentiated away.
- **REINFORCE** plugs in the Monte Carlo return $G_t$; it is unbiased but high-variance.
  A state-dependent **baseline** $b(s)$ leaves the gradient unbiased and, taken as
  $V^\pi(s)$, yields the **advantage** $A^\pi = Q^\pi - V^\pi$ that sharply reduces variance.
- **Actor-critic** learns the value online with a critic and steps on the TD error
  $\delta_t$; **A2C/A3C** parallelize it, and **GAE** dials bias against variance with
  $\lambda$.
- **Trust-region** methods bound the per-step policy change: **TRPO** by a hard KL
  constraint, **PPO** by clipping the probability ratio to $[1-\epsilon,1+\epsilon]$.
- **Continuous control** specializes further: **DDPG** and **TD3** push a deterministic
  gradient through a $Q$-critic off-policy, while **SAC** maximizes reward plus entropy.
- **In practice:** **PPO** displaced TRPO because a clipped first-order
  objective is far simpler and more robust, which is why it is the optimizer behind
  RLHF; the stochastic/deterministic split is at bottom a choice about how to explore.

[^sutton-pg]: **Sutton et al.**, _Policy Gradient Methods for Reinforcement Learning with Function Approximation_, NeurIPS 2000 — the policy gradient theorem and its actor-critic realization.
[^schulman-trpo]: **Schulman et al.**, _Trust Region Policy Optimization_, ICML 2015 — maximizes a surrogate advantage subject to a hard KL-divergence trust region on the policy update.
[^schulman-gae]: **Schulman et al.**, _High-Dimensional Continuous Control Using Generalized Advantage Estimation_, ICLR 2016 — the $\gamma\lambda$-weighted advantage estimator that dials bias against variance.
[^schulman-ppo]: **Schulman et al.**, _Proximal Policy Optimization Algorithms_, 2017 — replaces TRPO's hard constraint with a clipped first-order surrogate, the default deep-RL policy optimizer.
[^lillicrap-ddpg]: **Lillicrap et al.**, _Continuous Control with Deep Reinforcement Learning_ (DDPG), ICLR 2016 — an off-policy deterministic actor-critic with replay and target networks for continuous actions.
[^fujimoto-td3]: **Fujimoto et al.**, _Addressing Function Approximation Error in Actor-Critic Methods_ (TD3), ICML 2018 — twin critics with a min, delayed actor updates, and target-policy smoothing to curb overestimation.
[^haarnoja-sac]: **Haarnoja et al.**, _Soft Actor-Critic_, ICML 2018 — a maximum-entropy off-policy actor-critic that augments reward with policy entropy for exploration and stability.