---
title: "Imitation as Adversarial Matching: GAIL and AIRL"
module: Modern Deep Reinforcement Learning
moduleNumber: 5
lessonNumber: 12
order: 512
summary: >
  A companion to the imitation-learning lesson. If the point of recovering a reward is
  only to re-run RL and match the expert, you can skip the reward and match the
  behavior directly. GAIL casts imitation as a GAN — a discriminator separating expert
  from learner state-action pairs supplies the reward a policy-gradient method
  optimizes — matching occupancy measures without ever naming a reward. AIRL reads a
  transferable reward back out of the discriminator. We compare all four methods and
  close with reward models in RLHF, scaled cloning, and diffusion policies.
topics: [Deep RL]
sources:
  - book: Sutton & Barto
    ref: "Ch. 3 — Finite Markov Decision Processes; Ch. 13 — Policy Gradient Methods"
  - book: Grokking Deep RL
    ref: "Ch. 12 — advanced actor-critic; learning from demonstrations"
---

This builds on
[Imitation Learning: Cloning, DAgger, and Inverse RL](/reinforcement-learning/modern-deep-rl/imitation-and-inverse-rl),
which built behavioral cloning (and its $T^2\epsilon$ compounding error), DAgger (which
queries an interactive expert on the learner's own states), and inverse RL (which
recovers the reward the expert optimizes, resolving the ill-posedness with the
maximum-entropy principle at the cost of a forward-RL solve in the inner loop).

That inner solve is the opening this lesson exploits. If recovering a reward only ever
serves to re-run RL and reproduce the expert, the reward can be dissolved entirely —
match the behavior directly, as an adversarial game — and then, if a portable reward is
still wanted, read one back out of that game.

## GAIL: imitation as a GAN

Maximum-entropy IRL suggests a suspicion that Ho and Ermon made precise: if the
whole point of recovering a reward is to run RL and reproduce the expert's behavior,
maybe you can **skip the reward** and match the behavior directly. Their reframing
starts from the object that actually characterizes a policy's long-run behavior, the
**occupancy measure**.

> **Definition (Occupancy measure).** For a policy $\pi$, the discounted
> distribution of state-action pairs it visits,
> $\rho_\pi(s,a) = \pi(a \mid s)\sum_{t} \gamma^t\, \Pr(s_t = s \mid \pi)$. Two
> policies are identical if and only if their occupancy measures are identical, so
> matching occupancy measures amounts to imitation. Expected reward is linear in
> the occupancy measure, $\sum_{s,a}\rho_\pi(s,a)\,r(s,a)$.

$$
% caption: Occupancy-measure matching. The expert's discounted state-action visits
% (blue) form a distribution over the space; the learner's current occupancy (red)
% differs. GAIL drives the learner's occupancy toward the expert's until the two
% distributions coincide, which is exactly imitation, since two policies are equal
% iff their occupancy measures are equal.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (-0.2,0) -- (9.2,0) node[anchor=north, black, font=\footnotesize] {state-action space};
  \draw[->, black] (0,-0.2) -- (0,3.4) node[anchor=east, black, font=\footnotesize] {visitation};
  % expert occupancy: peak around 3
  \draw[acc, very thick, smooth] plot coordinates {(0.4,0.3)(1.4,0.7)(2.4,1.9)(3.2,2.6)(4.0,2.0)(5.0,0.9)(6.2,0.4)(7.6,0.25)(8.7,0.2)};
  \node[acc, anchor=west, font=\scriptsize] at (3.3,2.7) {expert occupancy};
  % learner occupancy: peak shifted right
  \draw[red, thick, dashed, smooth] plot coordinates {(0.4,0.2)(1.4,0.3)(2.4,0.5)(3.6,0.9)(4.8,1.7)(5.8,2.1)(6.6,1.5)(7.6,0.7)(8.7,0.35)};
  \node[red, anchor=west, font=\scriptsize] at (5.8,2.15) {learner occupancy};
  \draw[->, black, thick] (5.6,1.0) -- (4.2,1.0) node[midway, above, font=\scriptsize] {push toward expert};
\end{tikzpicture}
$$

Ho and Ermon show that running RL on the reward IRL recovers is
equivalent to finding a policy whose occupancy measure $\rho_\pi$ is close
to the expert's $\rho_{\pi_E}$ under a particular divergence. The reward is an
intermediary that can be dissolved: **directly minimize a divergence between
$\rho_\pi$ and $\rho_{\pi_E}$** and you imitate without ever naming a reward. And
when the divergence is the Jensen-Shannon divergence, minimizing it becomes the
objective a **generative adversarial network** optimizes.[^gail]

This yields **Generative Adversarial Imitation Learning** (GAIL). A **discriminator**
$D_\psi(s,a) \to (0,1)$ is trained to tell expert state-action pairs from the
learner's, outputting the probability that a pair came from the expert. The
**policy** $\pi_\theta$ plays the role of the GAN's generator: it is trained to
produce state-action pairs the discriminator _cannot_ distinguish from the expert's.
The two optimize the adversarial objective

$$
\min_{\theta}\ \max_{\psi}\;
\mathbb{E}_{(s,a)\sim \rho_{\pi_E}}\!\bigl[\log D_\psi(s,a)\bigr]
\;+\;
\mathbb{E}_{(s,a)\sim \rho_{\pi_\theta}}\!\bigl[\log\bigl(1 - D_\psi(s,a)\bigr)\bigr]
\;-\; \lambda\, H(\pi_\theta),
$$

with an entropy bonus $H(\pi_\theta)$ inherited from the maximum-entropy view. The
discriminator maximizes its ability to separate the two distributions; the policy
minimizes it, driving $\rho_{\pi_\theta}$ toward $\rho_{\pi_E}$. The
policy cannot be trained by supervised backprop through $D$ — the state-action pairs
come from _running the policy in the environment_, so the discriminator's verdict
becomes a **reward signal** the policy improves against by an ordinary
[policy-gradient](/reinforcement-learning/approximation/policy-gradient-methods)
update. Concretely, $-\log\bigl(1 - D_\psi(s,a)\bigr)$ (high when $D$ is fooled)
plays the role of $r(s,a)$, and TRPO or PPO does the [value-function](
/reinforcement-learning/foundations/value-functions-and-optimality)-based credit
assignment.

$$
% caption: The GAIL adversarial loop. The discriminator learns to separate expert
% state-action pairs from the policy's rollouts; its output is fed back as a reward
% the policy maximizes by policy gradient, pushing the learner's occupancy measure
% toward the expert's until the discriminator can no longer tell them apart.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=28mm, minimum height=12mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[box, draw=acc, text=acc, thick] (pol) at (0,0) {policy (generator)};
  \node[box] (env) at (0,-2.6) {environment:\\rollout (s, a) pairs};
  \node[box, draw=red, text=red, thick] (disc) at (6.4,-1.3) {discriminator D};
  \node[box] (exp) at (6.4,1.5) {expert pairs};
  % policy -> env
  \draw[->, acc, thick] (pol) -- (env) node[midway, left, font=\scriptsize] {act};
  % env rollouts -> discriminator
  \draw[->, black, thick] (env) -- (disc) node[midway, below, font=\scriptsize] {learner (s, a)};
  % expert -> discriminator
  \draw[->, black, thick] (exp) -- (disc) node[midway, right, font=\scriptsize] {expert (s, a)};
  % discriminator -> policy as reward
  \draw[->, red, thick] (disc) to[bend right=22] node[midway, above, font=\scriptsize] {reward = -log(1 - D)} (pol);
\end{tikzpicture}
$$

GAIL matches the theoretical power of maximum-entropy IRL — it recovers the same
occupancy-matching behavior — but pays none of IRL's double-loop cost, because the
discriminator replaces the nested forward-RL solve with a cheap classification step.
It scales to high-dimensional continuous control where classical IRL is intractable,
and it learns from a fixed demonstration set (no interactive expert, unlike DAgger).
The price is GAN training: adversarial optimization is delicate, sample-hungry in
environment steps (each policy-gradient update needs fresh rollouts), and prone to
the instabilities familiar from the
[generative-adversarial-network](/deep-learning/generative-models/generative-adversarial-networks)
lesson — mode collapse, oscillation, and a discriminator that outpaces the
generator. GAIL is the conceptual bridge: the same min-max game that generates
photorealistic images, pointed at trajectories, produces expert-like behavior.

## From GAIL back to a reward: adversarial IRL

GAIL buys efficiency by _dissolving_ the reward — it matches occupancy measures and
never names $r(s,a)$. That is the right trade when you only want the behavior, but
it forfeits IRL's original benefit: a reward that transfers to new dynamics. The
resolution is to keep GAIL's adversarial machinery yet read a reward back out of the
discriminator, and two lines of work do exactly that.

**Guided Cost Learning** (Finn, Levine, Abbeel 2016) makes maximum-entropy IRL
practical for deep continuous control by replacing the intractable partition
function $Z$ with a sampling-based estimate — importance-sampled from the current
policy — so the reward network can be trained by gradient descent without an exact
inner solve.[^gcl] Finn et al. then showed this sampling-based IRL is _formally the
same algorithm_ as a GAN with a particular discriminator form, making explicit the
GAIL connection: the discriminator and the reward are two views of one object.

**Adversarial Inverse Reinforcement Learning** (Fu, Luo, Levine 2018) exploits that
view to recover a reward that generalizes. It constrains the discriminator to the
form

$$
D_\psi(s, a, s') = \frac{\exp\bigl(f_\psi(s, a, s')\bigr)}{\exp\bigl(f_\psi(s,a,s')\bigr) + \pi(a \mid s)},
\qquad
f_\psi = g_\psi(s) + \gamma\, h_\psi(s') - h_\psi(s),
$$

so that at convergence $g_\psi(s)$ recovers the reward and $h_\psi$ absorbs the
potential-shaping term the reward is ambiguous up to (the same $\gamma\phi(s') -
\phi(s)$ shaping that made IRL ill-posed).[^airl] Because AIRL isolates the
_state-only_ reward $g_\psi$ from the shaping, the recovered reward transfers across
changes in dynamics: under new physics it still specifies the same goal, where
GAIL's occupancy-matched policy would have to relearn from scratch. AIRL is the deep,
adversarial descendant of MaxEnt IRL, closing the loop back to a portable reward
without paying the classical double-loop cost.

$$
% caption: The imitation-learning family as a single spectrum. Behavioral cloning
% copies actions; DAgger copies actions on the learner's own states; GAIL matches
% occupancy without a reward; MaxEnt IRL, Guided Cost Learning, and AIRL recover an
% explicit reward, trading more computation for a transferable object.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  node distance=0mm]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (-0.3,0) -- (11.2,0);
  \node[black, anchor=north west, font=\scriptsize] at (-0.3,-0.15) {copy behavior};
  \node[black, anchor=north east, font=\scriptsize] at (11.2,-0.15) {recover reward (transfers)};
  \foreach \x/\lbl/\c in {0.3/BC/acc, 2.6/DAgger/acc, 5.4/GAIL/red, 7.7/{MaxEnt IRL}/red, 10.4/AIRL/red} {
    \fill[\c] (\x,0) circle (2pt);
    \node[\c, anchor=south, font=\scriptsize] at (\x,0.15) {\lbl};
  }
\end{tikzpicture}
$$

## Choosing an approach

The four methods sort along one axis — how much work they do to handle the
covariate shift that sinks naive cloning.

| Method | What it produces | Handles covariate shift? | Cost | Needs |
| --- | --- | --- | --- | --- |
| Behavioral cloning | a policy, by supervised learning | no — $T^2\epsilon$ compounding | very cheap (one fit) | a fixed demo set |
| DAgger | a policy, over aggregation rounds | yes — trains on learner's states | moderate | an **interactive** expert |
| Max-Ent IRL | a reward function | yes — reward generalizes | expensive (RL in the inner loop) | a fixed demo set + forward solver |
| GAIL | a policy, adversarially | yes — matches occupancy measure | expensive (GAN + env rollouts) | a fixed demo set + environment |

Behavioral cloning is the honest baseline: reach for it first, especially as a warm
start, and accept that its policy drifts once it leaves the demonstrated states. If
you have a queryable expert, DAgger fixes the drift directly and cheaply. If you
need a reward that transfers to new tasks or want the possibility of beating the
demonstrator, IRL recovers one, at the cost of a nested RL solve. If you only need
the behavior and have environment access but no interactive expert, GAIL matches the
expert's occupancy measure through the adversarial game, trading IRL's inner loop for
GAN instability.

## Reward models, diffusion policies, and scaled cloning

Imitation learning has moved well past the four classical methods, in directions
Sutton & Barto do not cover.

**IRL is the reward model in RLHF.** The single largest deployment of inverse
reward learning is aligning language models. A reward model trained on human
preference comparisons is an IRL step in disguise: it recovers a scalar reward that
"explains" which of two responses a human preferred, then ordinary RL (PPO)
optimizes a policy against it (Christiano et al. 2017; Ouyang et al. 2022,
InstructGPT).[^rlhf] The connection to this lesson is exact — preference-based
reward learning is IRL with pairwise-comparison supervision instead of full
demonstrations, and the KL penalty that keeps the tuned model near the reference is
the same "stay near the demonstrator" pressure that stabilizes imitation. Direct
Preference Optimization (Rafailov et al. 2023) later showed the reward model can be
skipped entirely, folding the IRL and RL steps into one supervised loss.

**Behavioral cloning, scaled, is competitive again.** The $T^2\epsilon$ pessimism
is a worst case, and modern practice has learned to blunt it without an interactive
expert. Two levers matter. First, _sheer scale_: robotics foundation models such as
**RT-1** (Brohan et al. 2022) and **RT-2** are behavioral clones trained on
enormous, diverse teleoperation datasets, where broad coverage means the policy
rarely leaves the demonstrated distribution in the first place. Second, _expressive
action distributions_: **Diffusion Policy** (Chi et al. 2023) clones by fitting a
diffusion model over action sequences rather than a unimodal Gaussian, which
captures the multi-modality of human demonstrations (a human might swerve left _or_
right around an obstacle) that a mean-squared clone averages into an unusable
in-between. Modeling the full action distribution, not its mean, turns out to fix
much of what made naive cloning brittle.

$$
% caption: Why expressive action distributions matter for cloning. Human
% demonstrations are multi-modal: at an obstacle, experts go left OR right. A
% unimodal (Gaussian) clone fits the mean of the two modes and drives straight into
% the obstacle; a diffusion policy captures both modes and commits to one.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (-0.2,0) -- (9.2,0) node[anchor=north, black, font=\footnotesize] {steering action};
  \draw[->, black] (0,-0.2) -- (0,3.2) node[anchor=east, black, font=\footnotesize] {density};
  % two expert modes
  \draw[acc, very thick, smooth] plot coordinates {(0.5,0.1)(1.6,0.5)(2.4,2.4)(3.0,0.6)(3.8,0.15)(4.6,0.6)(5.6,2.4)(6.4,0.5)(7.4,0.1)};
  \node[acc, anchor=south, font=\scriptsize] at (2.4,2.4) {left};
  \node[acc, anchor=south, font=\scriptsize] at (5.6,2.4) {right};
  \node[acc, anchor=west, font=\scriptsize] at (6.6,1.4) {di\/f\/fusion: both modes};
  % unimodal clone: mean between them -> straight into obstacle
  \draw[red, thick, dashed, smooth] plot coordinates {(2.6,0.1)(3.4,0.8)(4.0,1.5)(4.6,0.8)(5.4,0.1)};
  \fill[red] (4.0,1.5) circle (2pt);
  \node[red, anchor=west, font=\scriptsize] at (0.3,0.75) {clone averages: hits obstacle};
\end{tikzpicture}
$$

**Offline imitation and observation-only learning.** GAIL needs environment
rollouts; **ValueDICE** and related "DICE" methods (Kostrikov et al. 2019) match
occupancy measures entirely offline, from a fixed dataset, closing the gap to the
offline-RL lesson. And when only expert _states_ are available with no action
labels — video of an expert, say — imitation-from-observation methods recover
behavior by matching state occupancy alone, extending the occupancy-matching view of
GAIL to the label-free setting.

The through-line is the covariate-shift diagnosis. Behavioral cloning fails because
it trains on one distribution and is tested on another that its own actions create;
every method that beats it does so by confronting the distribution the learner
actually visits — DAgger by labelling those states, IRL and GAIL by matching the
long-run occupancy the learner induces rather than the individual actions the expert
took. Demonstrations can stand in for a reward, but only once you stop treating
control as if it were supervised learning.


[^gail]: **Ho & Ermon (2016)**, "Generative Adversarial Imitation Learning", NeurIPS — shows that RL on the IRL-recovered reward reduces to occupancy-measure matching, and casts it as a GAN: a discriminator separating expert from learner state-action pairs supplies the reward a policy-gradient method optimizes, imitating without an explicit reward.
[^gcl]: **Finn, Levine, Abbeel (2016)**, "Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization", ICML — sampling-based maximum-entropy IRL for deep continuous control, estimating the partition function from policy samples; the companion note (Finn, Christiano, Abbeel, Levine 2016) shows the algorithm is equivalent to a GAN with a specific discriminator, tying IRL to GAIL.
[^airl]: **Fu, Luo, Levine (2018)**, "Learning Robust Rewards with Adversarial Inverse Reinforcement Learning", ICLR — AIRL constrains the discriminator to a reward-plus-shaping form $f = g(s) + \gamma h(s') - h(s)$, recovering a state-only reward $g$ that is disentangled from potential shaping and therefore transfers across changes in the dynamics.
[^rlhf]: **Christiano, Leike, Brown, Martic, Legg, Amodei (2017)**, "Deep Reinforcement Learning from Human Preferences", NeurIPS — learns a reward model from pairwise human preferences and optimizes a policy against it; **Ouyang et al. (2022)**, "Training language models to follow instructions with human feedback" (InstructGPT), applies the same preference-based reward-learning-then-PPO pipeline to align a large language model.