---
title: Actor–Critic and GAE
module: Deep Reinforcement Learning
moduleNumber: 4
lessonNumber: 3
order: 403
summary: >
  Make the actor and the critic deep networks and the policy-gradient
  architecture becomes modern deep RL. We build the neural actor-critic, the
  advantage estimate that replaces the raw return, and Generalized Advantage
  Estimation as a λ-blend of n-step advantages, then the parallel-worker methods
  A3C and A2C that decorrelate on-policy data. The step-size constraints — trust
  regions, PPO, and the continuous-control family — follow in the next lesson.
topics: [Deep RL]
sources:
  - book: Grokking Deep RL
    ref: "Ch. 11 — Policy-gradient and actor-critic methods; REINFORCE, VPG, A3C, GAE, A2C"
  - book: Sutton & Barto
    ref: "Ch. 13 — Policy Gradient Methods; §13.5 Actor–Critic Methods"
---

The [policy-gradient lesson](/reinforcement-learning/approximation/policy-gradient-methods)
left every function approximator abstract: the action preferences $h$, the value
$\hat v$, the Gaussian's mean and spread were "some differentiable function of
$\boldsymbol{\theta}$." Make each of them a deep neural network and the same three
equations underlie modern reinforcement learning. The **actor** is a
network $\pi(a \mid s; \boldsymbol{\theta})$ that maps a state to a distribution
over actions; the **critic** is a network $\hat v(s; \mathbf{w})$ that scores how
good the state is. They train together, driven by the same error signal, and the
whole apparatus is what plays Atari from pixels, controls simulated robots, and
aligns language models.[^grok-intro]

We do not re-derive the policy gradient theorem here — recall it from the
[previous lesson](/reinforcement-learning/approximation/policy-gradient-methods):
the gradient of performance is an expectation, under the policy's own state
visitation, of the log-probability eligibility vector weighted by how good the
action was,

$$
\nabla J(\boldsymbol{\theta}) \;=\; \mathbb{E}_\pi\!\big[\,\Psi_t\,\nabla \ln \pi(A_t \mid S_t; \boldsymbol{\theta})\,\big].
$$

Everything in this lesson is a choice of the score $\Psi_t$ — the number that
multiplies the eligibility vector — and a rule for keeping the resulting update
from destroying the policy. The full Monte Carlo return $G_t$ (REINFORCE) is one
choice; it is unbiased but very noisy. The rest of the progression trades a
little bias for far less variance, and then guards the step size.

$$
% caption: The neural actor-critic. The actor $\pi(a\mid s;\boldsymbol{\theta})$
% samples an action; the environment returns a reward and next state; the critic
% $\hat v(s;\mathbf{w})$ turns the transition into a score that trains both heads.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=26mm, minimum height=12mm, align=center},
  env/.style={draw, minimum width=26mm, minimum height=12mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[box, draw=acc, text=acc, thick] (actor)  at (0,1.7)  {actor net\\pi(a:s;theta)};
  \node[box] (critic) at (0,-1.7) {critic net\\v-hat(s;w)};
  \node[env] (env)    at (5.6,0)  {environment};
  \node[red] (sc)     at (2.8,0)  {score};
  \draw[->, acc, thick] (actor) to[out=0,in=120] node[midway, above, font=\scriptsize] {action} (env);
  \draw[->, thick] (env) to[out=-120,in=0] node[midway, below, font=\scriptsize] {reward, next state} (critic);
  \draw[->, red, thick] (critic) -- (sc);
  \draw[->, red, thick] (sc) -- (actor) node[midway, left, font=\scriptsize] {train actor};
  \draw[->, red, thick] (sc) to[out=-90,in=90] node[midway, right, font=\scriptsize] {train critic} (critic.east);
\end{tikzpicture}
$$

## The advantage replaces the return

Instead of scoring an action by the total reward that followed it (the return),
score it by how much better it was than the state's average. That difference is
the advantage, and it is far less noisy to learn from.

REINFORCE weights each eligibility vector by the return $G_t$, and even a good
baseline only recenters it. The cleaner object to weight by is the **advantage**,
which measures how much better an action is than the state's average.[^grok-vpg]

> **Definition (Advantage function).** For a policy $\pi$, the advantage of taking
> action $a$ in state $s$ is
> $$A_\pi(s,a) \;\doteq\; q_\pi(s,a) - v_\pi(s),$$
> the action value minus the state value. It is positive when $a$ beats what the
> policy would do on average from $s$, negative when it does worse, and exactly
> zero in expectation over actions drawn from $\pi$.

The advantage is the natural score because it removes the part of the return that
depends only on the state — the part every action in that state shares, and that
therefore carries no information about which action to prefer. In an environment
like cart-pole, where every reward is positive and every return is large, the raw
return pushes the probability of _every_ action up; only the differences between
returns tell good actions from mediocre ones, and the advantage is precisely those
differences. Centering the score around zero means better-than-average actions get
a positive push and worse-than-average actions a negative one.

$$
% caption: Why the advantage centers the score. Left: raw returns in a
% positive-reward task are all large and positive, so weighting by the return pushes
% every action's probability up — only the differences matter. Right: subtracting the
% baseline $\hat v(s)$ recenters the scores on zero, so above-average actions get a
% positive push and below-average ones a negative push.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % left: raw returns
  \draw[black, ->] (0,0) -- (0,2.7) node[above, black, font=\scriptsize] {return G};
  \draw[black] (0,0) -- (3.0,0);
  \foreach \x/\h in {0.5/1.9, 1.1/2.3, 1.7/2.0, 2.3/2.4}
    \draw[acc, very thick] (\x,0) -- (\x,\h);
  \node[font=\scriptsize, black, anchor=north] at (1.4,-0.15) {all push up};
  % right: advantages, centered
  \begin{scope}[xshift=5.0cm]
    \draw[black, ->] (0,-1.4) -- (0,1.6) node[above, black, font=\scriptsize] {advantage A};
    \draw[black, dashed] (0,0) -- (3.0,0);
    \node[black, font=\scriptsize, anchor=east] at (-0.05,0) {0};
    \foreach \x/\h in {0.5/-0.5, 1.1/0.9, 1.7/-0.9, 2.3/1.0}
      \draw[red, very thick] (\x,0) -- (\x,\h);
    \node[font=\scriptsize, black, anchor=north] at (1.4,-1.5) {up or down};
  \end{scope}
\end{tikzpicture}
$$

We rarely know $q_\pi$ and $v_\pi$ exactly, so we estimate the advantage. The
critic supplies $\hat v(s; \mathbf{w})$; the simplest advantage estimate is the
one-step **TD residual**, which is an unbiased sample of $q_\pi(s,a)$ minus the
baseline:

$$
\hat A_t \;=\; \delta_t \;=\; R_{t+1} + \gamma\,\hat v(S_{t+1}; \mathbf{w}) - \hat v(S_t; \mathbf{w}).
$$

A one-step residual has low variance but is biased by whatever error the critic
carries. Using the full Monte Carlo return in place of the bootstrap gives the
opposite: unbiased, but as noisy as REINFORCE. Between them sits the whole family
of **$n$-step advantages**, which bootstrap only after $n$ real rewards,

$$
\hat A_t^{(n)} \;=\; \underbrace{R_{t+1} + \gamma R_{t+2} + \cdots + \gamma^{n-1} R_{t+n}}_{n\text{ real rewards}} + \gamma^{n}\,\hat v(S_{t+n}; \mathbf{w}) - \hat v(S_t; \mathbf{w}).
$$

Small $n$ leans on the critic (low variance, higher bias); large $n$ leans on
real experience (low bias, higher variance). The integer $n$ turns the same
bias-variance dial seen in
[n-step bootstrapping](/reinforcement-learning/tabular-methods/n-step-bootstrapping),
now applied to advantages instead of value targets.

## Generalized advantage estimation

Rather than pick how many real steps to use before bootstrapping on the critic,
GAE averages over _all_ those choices, weighting shorter horizons by a single
parameter $\lambda$ — one estimator that slides from noisy and unbiased to
low-variance and slightly biased.

Choosing a single $n$ forces a hard commitment to one point on the bias-variance
line. **Generalized advantage estimation** (GAE) avoids the choice: it takes an
exponentially weighted average of _every_ $n$-step advantage, exactly as the
[λ-return](/reinforcement-learning/approximation/eligibility-traces) takes an
exponentially weighted average of every $n$-step value target.[^grok-gae] The same
$\lambda$ that set the fade rate of an eligibility trace here sets the blend of
$n$-step advantages.

> **Definition (Generalized advantage estimation).** With TD residuals
> $\delta_t = R_{t+1} + \gamma\,\hat v(S_{t+1};\mathbf{w}) - \hat v(S_t;\mathbf{w})$,
> the GAE($\gamma,\lambda$) advantage is the geometric sum
> $$\hat A_t^{\text{GAE}} \;\doteq\; \sum_{l=0}^{\infty} (\gamma\lambda)^{l}\,\delta_{t+l}.$$
> At $\lambda = 0$ it collapses to the one-step residual $\delta_t$ (maximum
> bootstrapping, minimum variance); at $\lambda = 1$ it becomes the full
> Monte Carlo advantage $\sum_l \gamma^l \delta_{t+l}$ (no bootstrapping, minimum
> bias). Interior $\lambda$ interpolates.

The identity to $\lambda=1$ is worth seeing: telescoping the discounted sum of TD
residuals collapses the successive $\hat v$ terms and leaves $\sum_l \gamma^l
R_{t+1+l} - \hat v(S_t)$, the Monte Carlo return minus the baseline. So GAE is
genuinely a single parameter that slides from the noisy unbiased estimate to the
biased low-variance one, and in practice a value like $\lambda = 0.95$ sits close to the
low-variance end while paying only slight bias. GAE is not an algorithm on its own;
it is the advantage estimator that A2C and PPO plug in.[^gae-paper]

$$
% caption: GAE blends every n-step advantage under weight $(\gamma\lambda)^l$ on
% the residual $\delta_{t+l}$: $\lambda=0$ keeps only the one-step term, $\lambda=1$
% recovers the full Monte Carlo advantage, and interior $\lambda$ tapers between them.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[black, ->] (0,0) -- (7.6,0) node[anchor=north east, text=black] {horizon l};
  \draw[black, ->] (0,0) -- (0,3.3) node[anchor=south east, text=black] {weight};
  % lambda near 1: slow geometric decay
  \foreach \i/\h in {0/2.9, 1/2.55, 2/2.25, 3/1.98, 4/1.74, 5/1.53, 6/1.35}
    \draw[acc, thick] ({0.55+\i*0.95},0) -- ({0.55+\i*0.95},\h);
  \node[acc, anchor=west] at (5.9,2.55) {lambda near 1};
  % lambda near 0: fast decay
  \foreach \i/\h in {0/2.9, 1/0.9, 2/0.28, 3/0.09}
    \draw[red, thick] ({0.75+\i*0.95},0) -- ({0.75+\i*0.95},\h);
  \node[red, anchor=west] at (2.65,1.85) {lambda near 0};
\end{tikzpicture}
$$

### GAE by hand

Run the estimator on a real short rollout. Take $\gamma = 0.99$, $\lambda = 0.95$,
and a five-step segment that ends in a terminal state. Suppose the rewards and the
critic's state values are

| $t$ | $R_{t+1}$ | $\hat v(S_t)$ | $\hat v(S_{t+1})$ |
| --- | --- | --- | --- |
| 0 | $0$ | $5.0$ | $5.2$ |
| 1 | $0$ | $5.2$ | $6.0$ |
| 2 | $+1$ | $6.0$ | $4.0$ |
| 3 | $0$ | $4.0$ | $2.0$ |
| 4 | $+1$ | $2.0$ | $0$ (terminal) |

First form each one-step TD residual $\delta_t = R_{t+1} + \gamma\,\hat v(S_{t+1}) -
\hat v(S_t)$, using $\hat v = 0$ past the terminal step:

$$
\begin{aligned}
\delta_0 &= 0 + 0.99(5.2) - 5.0 = 0.148, \\
\delta_1 &= 0 + 0.99(6.0) - 5.2 = 0.740, \\
\delta_2 &= 1 + 0.99(4.0) - 6.0 = -1.040, \\
\delta_3 &= 0 + 0.99(2.0) - 4.0 = -2.020, \\
\delta_4 &= 1 + 0.99(0) - 2.0 = -1.000.
\end{aligned}
$$

The GAE advantage at each step is the discounted-by-$\gamma\lambda$ sum of the
_remaining_ residuals, with weight $(\gamma\lambda)^l = 0.9405^l$. The clean way to
compute it is the backward recursion $\hat A_t = \delta_t + \gamma\lambda\,\hat
A_{t+1}$, starting from the last step:

$$
\begin{aligned}
\hat A_4 &= \delta_4 = -1.000, \\
\hat A_3 &= \delta_3 + 0.9405\,\hat A_4 = -2.020 + 0.9405(-1.000) = -2.961, \\
\hat A_2 &= \delta_2 + 0.9405\,\hat A_3 = -1.040 + 0.9405(-2.961) = -3.824, \\
\hat A_1 &= \delta_1 + 0.9405\,\hat A_2 = 0.740 + 0.9405(-3.824) = -2.856, \\
\hat A_0 &= \delta_0 + 0.9405\,\hat A_1 = 0.148 + 0.9405(-2.856) = -2.538.
\end{aligned}
$$

Every advantage came out negative: this trajectory did worse than the critic
expected, because the two rewards it collected did not justify the state values the
critic had assigned. The agent will _decrease_ the probability of the actions it took
here. Contrast the two extremes on the same data. At $\lambda = 0$ only the local
residual survives, so $\hat A_0 = \delta_0 = 0.148$ — a small _positive_ advantage,
because step 0's own one-step lookahead looked fine and the recursion never sees the
disappointing future. At $\lambda = 1$ the advantage is the full discounted return
minus the baseline, which folds in every later residual at the largest weight and
gives the noisiest, least-biased estimate. GAE's $\lambda = 0.95$ sits close to that
Monte Carlo end but tempers the variance, which is why the negative future does reach
$\hat A_0$ here but slightly damped. PPO and A2C compute advantages with this
backward recursion in practice: one pass over the collected batch, $O(1)$ work per
step.

$$
% caption: GAE by backward recursion. Each residual $\delta_t$ (top row) is folded
% into the advantage right-to-left as $\hat A_t = \delta_t + \gamma\lambda\,\hat
% A_{t+1}$; the arrows carry the discounted future advantage back one step at a time.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  d/.style={draw, minimum width=13mm, minimum height=7mm, align=center, font=\scriptsize},
  a/.style={draw, minimum width=13mm, minimum height=7mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \foreach \i/\dv in {0/0.15, 1/0.74, 2/{-1.04}, 3/{-2.02}, 4/{-1.00}} {
    \node[d] (d\i) at ({\i*2.4},1.4) {\dv};
  }
  \foreach \i/\av in {0/{-2.54}, 1/{-2.86}, 2/{-3.82}, 3/{-2.96}, 4/{-1.00}} {
    \node[a, draw=red, text=red] (a\i) at ({\i*2.4},0) {\av};
  }
  \foreach \i in {0,1,2,3,4} \draw[->, black] (d\i) -- (a\i);
  \foreach \i/\j in {4/3, 3/2, 2/1, 1/0} \draw[->, red, thick] (a\i) to[bend left=22] node[above, font=\scriptsize] {x g-lam} (a\j);
  \node[font=\scriptsize, black, anchor=east] at (-0.9,1.4) {residual d};
  \node[font=\scriptsize, text=red, anchor=east] at (-0.9,0) {advantage A};
\end{tikzpicture}
$$

## A3C and A2C: many actors at once

REINFORCE-with-baseline is unbiased but sample-inefficient, and its
[TD-actor-critic](/reinforcement-learning/approximation/policy-gradient-methods)
descendant learns online but from a stream of highly correlated samples. Value
methods break that correlation with a replay buffer, but a buffer holds experience
from _old_ policies, and an on-policy gradient cannot reuse it — every update needs
fresh samples from the current policy. The fix is parallelism: run many copies of
the environment at once and the batch of experiences gathered across them is
decorrelated by construction.[^grok-a3c]

**Asynchronous advantage actor-critic** (A3C) spawns several worker processes, each
with its own copy of the environment and the networks. A worker rolls out a short
$n$-step segment, computes the advantage and the policy and value gradients on it,
and pushes those gradients to a shared global model — without any lock, overwriting
freely. It then reloads the global weights and continues. The lack of coordination
is deliberate; the lock-free scheme converges as fast as a locked one and runs an
order of magnitude quicker. Each worker's independent trajectory supplies the
diversity a replay buffer would have.

$$
% caption: A3C runs several worker-learners, each with an environment copy and a
% local actor-critic; each pushes its gradients to a shared global model
% asynchronously and reloads the updated weights.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  wk/.style={draw, minimum width=24mm, minimum height=15mm, align=center, font=\scriptsize},
  gl/.style={draw, thick, minimum width=30mm, minimum height=13mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[wk] (w1) at (0,0)    {worker 1\\env + actor-critic};
  \node[wk] (w2) at (3.4,0)  {worker 2\\env + actor-critic};
  \node[font=\large] (dots) at (6.0,0) {...};
  \node[wk] (wn) at (8.5,0)  {worker n\\env + actor-critic};
  \node[gl, draw=acc, text=acc] (g) at (4.25,-2.7) {global model\\policy + value};
  \draw[->, acc, thick] (w1) to[out=-90,in=160] node[near end, left, font=\scriptsize] {grads} (g.west);
  \draw[->, acc, thick] (w2) -- (g);
  \draw[->, acc, thick] (wn) to[out=-90,in=20]  node[near end, right, font=\scriptsize] {grads} (g.east);
  \draw[->, black, dashed] (g.north) to[out=110,in=-90] node[midway, above, font=\scriptsize] {reload} (w1.south);
  \draw[->, black, dashed] (g.north) to[out=70,in=-90]  node[midway, above, font=\scriptsize] {reload} (wn.south);
\end{tikzpicture}
$$

The surprise, found soon after, was that the _asynchrony_ was not what made A3C
work — the multiple rollout workers were. **Advantage actor-critic** (A2C) makes
the scheme synchronous: a single learner drives a vectorized environment that steps
$n$ parallel copies at once, collects the batch, computes one advantage-weighted
gradient over all of it, and takes a single update.[^grok-a2c] With one batched
update instead of many racing ones, A2C matches A3C's performance, is simpler, and
puts the whole batch on a GPU. Its loss ties the two heads together with an
**entropy** bonus $H$ that keeps the policy from collapsing onto one action too
early:

$$
\mathcal{L}(\boldsymbol{\theta}, \mathbf{w}) \;=\; \underbrace{-\,\hat A_t\,\ln \pi(A_t \mid S_t; \boldsymbol{\theta})}_{\text{policy}} \;+\; c_v\,\underbrace{\big(\hat v(S_t; \mathbf{w}) - G_t\big)^2}_{\text{value}} \;-\; c_H\,\underbrace{H\!\big[\pi(\cdot \mid S_t; \boldsymbol{\theta})\big]}_{\text{entropy}}.
$$

The policy term is the negated advantage-weighted log-probability (negated because
optimizers descend and we want to ascend $J$); the value term is the critic's mean
squared error against the return; the entropy term, weighted by a small $c_H$,
rewards a spread-out action distribution and so sustains exploration. A2C and A3C
are the standard synchronous and asynchronous forms of deep advantage actor-critic,
and PPO is built directly on A2C's skeleton.
This lesson built the actor-critic and stabilized its advantage estimate. The
complementary problem — keeping a single gradient step from destroying the policy,
which gives PPO and the continuous-control family (DDPG, TD3, SAC) — continues in
[PPO and Continuous Control](/reinforcement-learning/deep-rl/ppo-and-continuous-control).

[^grok-intro]: **Morales**, _Grokking Deep Reinforcement Learning_, Ch. 11 — policy-gradient and actor-critic methods: the actor as the policy that selects actions and the critic as the value function that evaluates them, learned together, with the critic earning its name only when it bootstraps (matching Sutton & Barto §13.5).
[^grok-vpg]: **Morales**, Ch. 11 — VPG / "Using estimated advantages": replacing the raw return with an advantage estimate $A \approx G_t - \hat v(S_t)$ centers the policy-gradient score around zero, separating better- from worse-than-average actions and cutting variance; the entropy bonus added to the loss to sustain exploration.
[^grok-gae]: **Morales**, Ch. 11 — "GAE: Robust advantage estimation": generalized advantage estimation as an exponentially weighted mixture of $n$-step advantage estimates, analogous to the λ-return of TD(λ) but for advantages, with $\lambda = 0$ giving the one-step residual and $\lambda = 1$ the infinite-step (Monte Carlo) advantage (Schulman et al., 2015).
[^grok-a3c]: **Morales**, Ch. 11 — "A3C: Parallel policy updates": asynchronous advantage actor-critic uses $n$-step bootstrapped returns and multiple lock-free worker-learners (the Hogwild! scheme) to decorrelate on-policy experience without a replay buffer (Mnih et al., 2016).
[^grok-a2c]: **Morales**, Ch. 11 — "A2C: Synchronous policy updates": the synchronous form drives a multi-process vectorized environment from a single learner, matches A3C, enables GPU batching, and combines the policy, value, and entropy terms in one weighted loss.
[^gae-paper]: **Schulman, Moritz, Levine, Jordan, and Abbeel** (2016), "High-Dimensional Continuous Control Using Generalized Advantage Estimation", _ICLR_ — the exponentially-weighted advantage estimator $\hat A_t = \sum_{l\ge 0}(\gamma\lambda)^l\delta_{t+l}$, its $\lambda=0$ (one-step) and $\lambda=1$ (Monte Carlo) limits, and the backward recursion $\hat A_t = \delta_t + \gamma\lambda\,\hat A_{t+1}$.