Imitation as Adversarial Matching: GAIL and AIRL

This builds on Imitation Learning: Cloning, DAgger, and Inverse RL, which built behavioral cloning (and its $T^2ϵ$ compounding error), DAgger (which queries an interactive expert on the learner's own states), and inverse RL (which recovers the reward the expert optimizes, resolving the ill-posedness with the maximum-entropy principle at the cost of a forward-RL solve in the inner loop).

That inner solve is the opening this lesson exploits. If recovering a reward only ever serves to re-run RL and reproduce the expert, the reward can be dissolved entirely — match the behavior directly, as an adversarial game — and then, if a portable reward is still wanted, read one back out of that game.

GAIL: imitation as a GAN

Maximum-entropy IRL suggests a suspicion that Ho and Ermon made precise: if the whole point of recovering a reward is to run RL and reproduce the expert's behavior, maybe you can skip the reward and match the behavior directly. Their reframing starts from the object that actually characterizes a policy's long-run behavior, the occupancy measure.

Ho and Ermon show that running RL on the reward IRL recovers is equivalent to finding a policy whose occupancy measure ${\rho }_{\pi }$ is close to the expert's ${\rho }_{{\pi }_E}$ under a particular divergence. The reward is an intermediary that can be dissolved: directly minimize a divergence between ${\rho }_{\pi }$ and ${\rho }_{{\pi }_E}$ and you imitate without ever naming a reward. And when the divergence is the Jensen-Shannon divergence, minimizing it becomes the objective a generative adversarial network optimizes.¹

This yields Generative Adversarial Imitation Learning (GAIL). A discriminator$D_{\psi }(s,a)\to (0,1)$ is trained to tell expert state-action pairs from the learner's, outputting the probability that a pair came from the expert. The policy ${\pi }_{\theta }$ plays the role of the GAN's generator: it is trained to produce state-action pairs the discriminator cannot distinguish from the expert's. The two optimize the adversarial objective

with an entropy bonus $H({\pi }_{\theta })$ inherited from the maximum-entropy view. The discriminator maximizes its ability to separate the two distributions; the policy minimizes it, driving ${\rho }_{{\pi }_{\theta }}$ toward ${\rho }_{{\pi }_E}$. The policy cannot be trained by supervised backprop through $D$ — the state-action pairs come from running the policy in the environment, so the discriminator's verdict becomes a reward signal the policy improves against by an ordinary policy-gradient update. Concretely, $-\log (1-D_{\psi }(s,a))$ (high when $D$ is fooled) plays the role of $r(s,a)$, and TRPO or PPO does the value-function-based credit assignment.

GAIL matches the theoretical power of maximum-entropy IRL — it recovers the same occupancy-matching behavior — but pays none of IRL's double-loop cost, because the discriminator replaces the nested forward-RL solve with a cheap classification step. It scales to high-dimensional continuous control where classical IRL is intractable, and it learns from a fixed demonstration set (no interactive expert, unlike DAgger). The price is GAN training: adversarial optimization is delicate, sample-hungry in environment steps (each policy-gradient update needs fresh rollouts), and prone to the instabilities familiar from the generative-adversarial-network lesson — mode collapse, oscillation, and a discriminator that outpaces the generator. GAIL is the conceptual bridge: the same min-max game that generates photorealistic images, pointed at trajectories, produces expert-like behavior.

From GAIL back to a reward: adversarial IRL

GAIL buys efficiency by dissolving the reward — it matches occupancy measures and never names $r(s,a)$. That is the right trade when you only want the behavior, but it forfeits IRL's original benefit: a reward that transfers to new dynamics. The resolution is to keep GAIL's adversarial machinery yet read a reward back out of the discriminator, and two lines of work do exactly that.

Guided Cost Learning (Finn, Levine, Abbeel 2016) makes maximum-entropy IRL practical for deep continuous control by replacing the intractable partition function $Z$ with a sampling-based estimate — importance-sampled from the current policy — so the reward network can be trained by gradient descent without an exact inner solve.² Finn et al. then showed this sampling-based IRL is formally the same algorithm as a GAN with a particular discriminator form, making explicit the GAIL connection: the discriminator and the reward are two views of one object.

Adversarial Inverse Reinforcement Learning (Fu, Luo, Levine 2018) exploits that view to recover a reward that generalizes. It constrains the discriminator to the form

so that at convergence $g_{\psi }(s)$ recovers the reward and $h_{\psi }$ absorbs the potential-shaping term the reward is ambiguous up to (the same $\gamma \varphi (s^{\prime })-\varphi (s)$ shaping that made IRL ill-posed).³ Because AIRL isolates the state-only reward $g_{\psi }$ from the shaping, the recovered reward transfers across changes in dynamics: under new physics it still specifies the same goal, where GAIL's occupancy-matched policy would have to relearn from scratch. AIRL is the deep, adversarial descendant of MaxEnt IRL, closing the loop back to a portable reward without paying the classical double-loop cost.

Choosing an approach

The four methods sort along one axis — how much work they do to handle the covariate shift that sinks naive cloning.

Behavioral cloning is the honest baseline: reach for it first, especially as a warm start, and accept that its policy drifts once it leaves the demonstrated states. If you have a queryable expert, DAgger fixes the drift directly and cheaply. If you need a reward that transfers to new tasks or want the possibility of beating the demonstrator, IRL recovers one, at the cost of a nested RL solve. If you only need the behavior and have environment access but no interactive expert, GAIL matches the expert's occupancy measure through the adversarial game, trading IRL's inner loop for GAN instability.

Reward models, diffusion policies, and scaled cloning

Imitation learning has moved well past the four classical methods, in directions Sutton & Barto do not cover.

IRL is the reward model in RLHF. The single largest deployment of inverse reward learning is aligning language models. A reward model trained on human preference comparisons is an IRL step in disguise: it recovers a scalar reward that explains which of two responses a human preferred, then ordinary RL (PPO) optimizes a policy against it (Christiano et al. 2017; Ouyang et al. 2022, InstructGPT).⁴ The connection to this lesson is exact — preference-based reward learning is IRL with pairwise-comparison supervision instead of full demonstrations, and the KL penalty that keeps the tuned model near the reference is the same stay near the demonstrator pressure that stabilizes imitation. Direct Preference Optimization (Rafailov et al. 2023) later showed the reward model can be skipped entirely, folding the IRL and RL steps into one supervised loss.

Behavioral cloning, scaled, is competitive again. The $T^2ϵ$ pessimism is a worst case, and modern practice has learned to blunt it without an interactive expert. Two levers matter. First, sheer scale: robotics foundation models such as RT-1 (Brohan et al. 2022) and RT-2 are behavioral clones trained on enormous, diverse teleoperation datasets, where broad coverage means the policy rarely leaves the demonstrated distribution in the first place. Second, expressive action distributions: Diffusion Policy (Chi et al. 2023) clones by fitting a diffusion model over action sequences rather than a unimodal Gaussian, which captures the multi-modality of human demonstrations (a human might swerve left or right around an obstacle) that a mean-squared clone averages into an unusable in-between. Modeling the full action distribution, not its mean, turns out to fix much of what made naive cloning brittle.

Offline imitation and observation-only learning. GAIL needs environment rollouts; ValueDICE and related DICE methods (Kostrikov et al. 2019) match occupancy measures entirely offline, from a fixed dataset, closing the gap to the offline-RL lesson. And when only expert states are available with no action labels — video of an expert, say — imitation-from-observation methods recover behavior by matching state occupancy alone, extending the occupancy-matching view of GAIL to the label-free setting.

The through-line is the covariate-shift diagnosis. Behavioral cloning fails because it trains on one distribution and is tested on another that its own actions create; every method that beats it does so by confronting the distribution the learner actually visits — DAgger by labelling those states, IRL and GAIL by matching the long-run occupancy the learner induces rather than the individual actions the expert took. Demonstrations can stand in for a reward, but only once you stop treating control as if it were supervised learning.