Actor–Critic and GAE

The policy-gradient lesson left every function approximator abstract: the action preferences $h$, the value $\hat{v}$, the Gaussian's mean and spread were some differentiable function of $\mathbf{\theta }$. Make each of them a deep neural network and the same three equations underlie modern reinforcement learning. The actor is a network $\pi (a\mid s;\mathbf{\theta })$ that maps a state to a distribution over actions; the critic is a network $\hat{v}(s;\mathbf{w})$ that scores how good the state is. They train together, driven by the same error signal, and the whole apparatus is what plays Atari from pixels, controls simulated robots, and aligns language models.¹

We do not re-derive the policy gradient theorem here — recall it from the previous lesson: the gradient of performance is an expectation, under the policy's own state visitation, of the log-probability eligibility vector weighted by how good the action was,

Everything in this lesson is a choice of the score ${\Psi }_t$ — the number that multiplies the eligibility vector — and a rule for keeping the resulting update from destroying the policy. The full Monte Carlo return $G_t$ (REINFORCE) is one choice; it is unbiased but very noisy. The rest of the progression trades a little bias for far less variance, and then guards the step size.

The advantage replaces the return

Instead of scoring an action by the total reward that followed it (the return), score it by how much better it was than the state's average. That difference is the advantage, and it is far less noisy to learn from.

REINFORCE weights each eligibility vector by the return $G_t$, and even a good baseline only recenters it. The cleaner object to weight by is the advantage, which measures how much better an action is than the state's average.²

The advantage is the natural score because it removes the part of the return that depends only on the state — the part every action in that state shares, and that therefore carries no information about which action to prefer. In an environment like cart-pole, where every reward is positive and every return is large, the raw return pushes the probability of every action up; only the differences between returns tell good actions from mediocre ones, and the advantage is precisely those differences. Centering the score around zero means better-than-average actions get a positive push and worse-than-average actions a negative one.

We rarely know $q_{\pi }$ and $v_{\pi }$ exactly, so we estimate the advantage. The critic supplies $\hat{v}(s;\mathbf{w})$; the simplest advantage estimate is the one-step TD residual, which is an unbiased sample of $q_{\pi }(s,a)$ minus the baseline:

A one-step residual has low variance but is biased by whatever error the critic carries. Using the full Monte Carlo return in place of the bootstrap gives the opposite: unbiased, but as noisy as REINFORCE. Between them sits the whole family of $n$-step advantages, which bootstrap only after $n$ real rewards,

Small $n$ leans on the critic (low variance, higher bias); large $n$ leans on real experience (low bias, higher variance). The integer $n$ turns the same bias-variance dial seen in n-step bootstrapping, now applied to advantages instead of value targets.

Generalized advantage estimation

Rather than pick how many real steps to use before bootstrapping on the critic, GAE averages over all those choices, weighting shorter horizons by a single parameter $\lambda$ — one estimator that slides from noisy and unbiased to low-variance and slightly biased.

Choosing a single $n$ forces a hard commitment to one point on the bias-variance line. Generalized advantage estimation (GAE) avoids the choice: it takes an exponentially weighted average of every $n$-step advantage, exactly as the λ-return takes an exponentially weighted average of every $n$-step value target.³ The same $\lambda$ that set the fade rate of an eligibility trace here sets the blend of $n$-step advantages.

The identity to $\lambda =1$ is worth seeing: telescoping the discounted sum of TD residuals collapses the successive $\hat{v}$ terms and leaves ${\sum }_l{\gamma }^l{R}_{t+1+l}-\hat{v}(S_t)$, the Monte Carlo return minus the baseline. So GAE is genuinely a single parameter that slides from the noisy unbiased estimate to the biased low-variance one, and in practice a value like $\lambda =0.95$ sits close to the low-variance end while paying only slight bias. GAE is not an algorithm on its own; it is the advantage estimator that A2C and PPO plug in.⁴

GAE by hand

Run the estimator on a real short rollout. Take $\gamma =0.99$, $\lambda =0.95$, and a five-step segment that ends in a terminal state. Suppose the rewards and the critic's state values are

First form each one-step TD residual ${\delta }_t=R_{t+1}+\gamma \hat{v}(S_{t+1})-\hat{v}(S_t)$, using $\hat{v}=0$ past the terminal step:

The GAE advantage at each step is the discounted-by-$\gamma \lambda$ sum of the remaining residuals, with weight $(\gamma \lambda {)}^l=0.9405^l$. The clean way to compute it is the backward recursion ${\hat{A}}_t={\delta }_t+\gamma \lambda {\hat{A}}_{t+1}$, starting from the last step:

Every advantage came out negative: this trajectory did worse than the critic expected, because the two rewards it collected did not justify the state values the critic had assigned. The agent will decrease the probability of the actions it took here. Contrast the two extremes on the same data. At $\lambda =0$ only the local residual survives, so ${\hat{A}}_0={\delta }_0=0.148$ — a small positive advantage, because step 0's own one-step lookahead looked fine and the recursion never sees the disappointing future. At $\lambda =1$ the advantage is the full discounted return minus the baseline, which folds in every later residual at the largest weight and gives the noisiest, least-biased estimate. GAE's $\lambda =0.95$ sits close to that Monte Carlo end but tempers the variance, which is why the negative future does reach ${\hat{A}}_0$ here but slightly damped. PPO and A2C compute advantages with this backward recursion in practice: one pass over the collected batch, $O(1)$ work per step.

A3C and A2C: many actors at once

REINFORCE-with-baseline is unbiased but sample-inefficient, and its TD-actor-critic descendant learns online but from a stream of highly correlated samples. Value methods break that correlation with a replay buffer, but a buffer holds experience from old policies, and an on-policy gradient cannot reuse it — every update needs fresh samples from the current policy. The fix is parallelism: run many copies of the environment at once and the batch of experiences gathered across them is decorrelated by construction.⁵

Asynchronous advantage actor-critic (A3C) spawns several worker processes, each with its own copy of the environment and the networks. A worker rolls out a short $n$-step segment, computes the advantage and the policy and value gradients on it, and pushes those gradients to a shared global model — without any lock, overwriting freely. It then reloads the global weights and continues. The lack of coordination is deliberate; the lock-free scheme converges as fast as a locked one and runs an order of magnitude quicker. Each worker's independent trajectory supplies the diversity a replay buffer would have.

The surprise, found soon after, was that the asynchrony was not what made A3C work — the multiple rollout workers were. Advantage actor-critic (A2C) makes the scheme synchronous: a single learner drives a vectorized environment that steps $n$ parallel copies at once, collects the batch, computes one advantage-weighted gradient over all of it, and takes a single update.⁶ With one batched update instead of many racing ones, A2C matches A3C's performance, is simpler, and puts the whole batch on a GPU. Its loss ties the two heads together with an entropy bonus $H$ that keeps the policy from collapsing onto one action too early:

The policy term is the negated advantage-weighted log-probability (negated because optimizers descend and we want to ascend $J$); the value term is the critic's mean squared error against the return; the entropy term, weighted by a small $c_H$, rewards a spread-out action distribution and so sustains exploration. A2C and A3C are the standard synchronous and asynchronous forms of deep advantage actor-critic, and PPO is built directly on A2C's skeleton. This lesson built the actor-critic and stabilized its advantage estimate. The complementary problem — keeping a single gradient step from destroying the policy, which gives PPO and the continuous-control family (DDPG, TD3, SAC) — continues in PPO and Continuous Control.