Actor-Critic Methods and Continuous Actions

This builds on policy gradient methods, which parameterized the policy directly, proved the policy gradient theorem, and derived REINFORCE and its variance-cutting baseline. REINFORCE with a baseline already learns both a policy and a value function — but the value function only recenters the Monte Carlo return; it never bootstraps. This lesson adds the bootstrapping step that turns the baseline into a true critic, then extends the architecture to continuing tasks and continuous actions.

Actor-critic methods

REINFORCE with baseline learns both a policy and a value function, yet it is not an actor-critic method, because its value function never bootstraps. It supplies a baseline for one state, but the target for the policy update is still the full Monte Carlo return $G_t$. An actor-critic method uses the learned value estimate to bootstrap — to replace the full return with a one-step estimate — and that is what earns the name critic.¹

The move repeats the one that took us from Monte Carlo to temporal-difference learning. Replace the full return $G_t$ in the baselined update with the one-step return $R_{t+1}+\gamma \hat{v}(S_{t+1},\mathbf{w})$, and the difference between that target and the current estimate is precisely the TD error:

Bootstrapping introduces bias and an asymptotic dependence on the quality of the value approximation, a cost that REINFORCE-with-baseline avoids. The tradeoff is usually favorable: bootstrapping substantially reduces variance and lets the method run online and incrementally, updating every step with no wait for the episode to end. It works for continuing tasks, where Monte Carlo returns do not even exist.

The architecture splits cleanly into two learners driven by the same TD error.

The actor is the policy $\pi (a\mid s,\mathbf{\theta })$ — it selects actions. The critic is the value function $\hat{v}(s,\mathbf{w})$ — it evaluates them, producing ${\delta }_t$. A positive TD error means the action led somewhere better than the critic expected, so the actor pushes to make that action more likely and the critic revises its estimate upward; a negative error does the reverse. Both learners consume the single scalar ${\delta }_t$.

This one-step method is the analog of TD(0). Generalizing it to $n$-step returns or to a $\lambda$-return with separate eligibility traces for actor and critic is mechanical — swap the one-step target for the longer one — and gives a family that trades bias against variance by the degree of bootstrapping, exactly as in the value-based case. The critic's own weights $\mathbf{w}$ are learned by semi-gradient TD, paired to the actor's update through their shared ${\delta }_t$.

Policy gradient for continuing problems

Episodic performance is the value of a start state, but continuing problems have no episode boundary and no start state to anchor $J$. For them, performance is the average rate of reward per step under the policy:²

where $\mu$ is the steady-state distribution under $\pi$, defined by $\mu (s)\doteq {\lim }_{t\to \infty }\mathrm{Pr}\{S_t=s\mid A_{0:t}\sim \pi \}$ and assumed to exist independently of the start state (an ergodicity assumption). It is the special distribution in which staying with $\pi$ keeps you: ${\sum }_s\mu (s){\sum }_a\pi (a\mid s,\mathbf{\theta })p(s^{\prime }\mid s,a)=\mu (s^{\prime })$ for every $s^{\prime }$.

Values are then defined against the differential return, rewards measured relative to the average rate,

and with these definitions the policy gradient theorem holds unchanged — the same $\nabla J\propto {\sum }_s\mu (s){\sum }_a{q}_{\pi }(s,a)\nabla \pi (a\mid s,\mathbf{\theta })$, now with the continuing $\mu$ and $q_{\pi }$. The proof runs parallel to the episodic one. In the actor-critic algorithm the only practical change is that the TD error subtracts a running estimate $\bar{R}$ of the average reward: $\delta \leftarrow R-\bar{R}+\hat{v}(S^{\prime },\mathbf{w})-\hat{v}(S,\mathbf{w})$, with $\bar{R}$ itself updated toward $R$ by the same error. No discounting appears; the average-reward setting replaces $\gamma$ with the differential value.

Continuous actions: the Gaussian policy

The softmax parameterization enumerates a probability per action, which is hopeless when there are infinitely many. For continuous actions the policy instead learns the statistics of a distribution and samples from it. The standard choice on a real-valued scalar action is a Gaussian policy: output a state-dependent mean and standard deviation, then draw the action from the normal density.³

where $\mu :\mathcal{S}\times {\mathbb{R}}^{d^{\prime }}\to \mathbb{R}$ and $\sigma :\mathcal{S}\times {\mathbb{R}}^{d^{\prime }}\to {\mathbb{R}}^{+}$ are two parameterized function approximators. The mean sets where the policy aims; the standard deviation sets how much it explores — a large $\sigma$ spreads mass widely and tries varied actions, a small $\sigma$ concentrates near the mean and commits. Exploration is now a learned quantity, not a fixed $\epsilon$.

To finish the construction, split the parameter vector into two blocks, $\mathbf{\theta }=[{\mathbf{\theta }}_{\mu },{\mathbf{\theta }}_{\sigma }{]}^{\top }$, one for each statistic. The mean is a linear function of state features; the standard deviation, which must stay positive, is the exponential of a linear function:

Both are differentiable in $\mathbf{\theta }$, so the eligibility vector $\nabla \ln \pi (a\mid s,\mathbf{\theta })$ is available in closed form and every algorithm in this lesson — REINFORCE, baselined REINFORCE, actor-critic — applies verbatim to real-valued control.

Differentiating $\ln \pi$ under the normal density (13.19) splits the eligibility vector into a mean part and a spread part:

Each has a plain meaning. The mean part pushes $\mu$ toward the sampled action when the return was good — the size of the push is the standardized surprise $(a-\mu )/\sigma$, scaled again by $1/\sigma$ so that a confident (small-$\sigma$) policy reacts more sharply to the same deviation. The spread part widens $\sigma$ when the action landed far from the mean (the squared standardized deviation exceeds $1$) and narrows it when actions cluster near the mean (below $1$). Exploration tunes itself: consistently rewarded actions near the mean shrink $\sigma$ and sharpen the policy; surprising high-return outliers grow $\sigma$ and keep it searching.

A worked step. Suppose $\mu (s)=2.0$, $\sigma (s)=0.5$, and the agent samples $a=2.4$ with a positive advantage $G-b=+3$. The standardized deviation is $(2.4-2.0)/0.5=0.8$. The mean eligibility is $(0.4/0.25){\mathbf{x}}_{\mu }=1.6{\mathbf{x}}_{\mu }$, so the update moves $\mu$ toward $2.4$. The spread eligibility is $(0.8^2-1){\mathbf{x}}_{\sigma }=-0.36{\mathbf{x}}_{\sigma }$; because the sampled action sat inside one standard deviation, the positive advantage times a negative eligibility shrinks $\sigma$, tightening the policy around a region with high return.

Natural policy gradients and the modern lineage

The plain gradient $\nabla J(\mathbf{\theta })$ ascends performance in the geometry of the raw parameter space, treating every coordinate of $\mathbf{\theta }$ as equally scaled. That geometry is arbitrary: rescaling one weight, or reparameterizing the policy, changes the gradient direction even though the policy — the actual distribution over actions — is unchanged. As a result, vanilla policy gradient can be slow. A small step in $\mathbf{\theta }$ can produce a large, destabilizing change in the action distribution in one region and a negligible change in another.

The natural policy gradient fixes this by measuring distance between policies in the space of distributions rather than of parameters, using the Fisher information matrix $\mathbf{F}(\mathbf{\theta })={\mathbb{E}}_{\pi }[\nabla \ln \pi \nabla \ln {\pi }^{\top }]$ as the local metric. The natural gradient premultiplies the ordinary gradient by its inverse,

which points in the direction of steepest ascent per unit of change in the policy's output distribution, invariant to how the policy is parameterized. Amari (1998, Neural Computation) established that this natural gradient is the correct steepest descent direction on a statistical manifold;⁴ Kakade (2002, NeurIPS, A Natural Policy Gradient) brought it to reinforcement learning and showed it moves toward the greedy policy and often converges far faster than the vanilla gradient.⁵ Sutton and Barto flag natural-gradient methods (Amari 1998; Kakade 2002; Peters and Schaal 2008) as the main further development past their own treatment.

This line of work leads to the algorithms that train today's agents. Trust region policy optimization (Schulman et al. 2015, ICML) constrains each update to a small KL-divergence step — a practical approximation of the natural-gradient idea, capping how far the action distribution may move per update.⁶Proximal policy optimization (Schulman et al. 2017, Proximal Policy Optimization Algorithms) replaces the hard constraint with a clipped surrogate objective that is cheaper and more stable, and is the actor-critic loop of this lesson with the raw ${\delta }_t$ swapped for a generalized advantage estimate (Schulman et al. 2016, ICLR) and the policy step clipped.⁷Deterministic policy gradients (Silver et al. 2014, ICML) take the opposite turn from the Gaussian policy: they learn a deterministic $\mu (s)$ and differentiate through a critic, the basis for DDPG in continuous control. And off-policy policy-gradient methods (Degris, White, and Sutton 2012, ICML) reweight the gradient by importance ratios so the policy can be improved from data a different behavior policy generated⁸ — the same off-policy correction that runs through the eligibility-trace lessons. All rest on the policy gradient theorem; they differ in the metric of the step and the estimator of the advantage.

The bridge to deep RL

Every method here left the function approximator abstract: the preferences $h$, the value $\hat{v}$, the Gaussian's mean and spread are some differentiable function of $\mathbf{\theta }$. Make that function a deep neural network and the same three equations underlie modern reinforcement learning. The eligibility vector $\nabla \ln \pi$ is computed by backpropagation; the actor and critic become two networks (or two heads of one); the TD error ${\delta }_t$ is the training signal for both. Advantage actor-critic and PPO — the algorithms that train game-playing agents and align language models — are the actor-critic loop of this lesson with a neural policy, a clipped or trust-region step to keep the update stable, and an advantage estimate in place of the raw ${\delta }_t$. We follow that thread into actor-critic and PPO.

The larger shift is conceptual. Before this chapter, learning meant estimating values and reading a policy off them. Policy gradient methods invert that: the policy is the object of learning, performance is a differentiable function of its parameters, and everything reduces to gradient ascent on $J(\mathbf{\theta })$. The policy gradient theorem makes this possible without a model of the environment, and the actor-critic architecture makes it fast.