---
title: "Imitation Learning: Cloning, DAgger, and Inverse RL"
module: Modern Deep Reinforcement Learning
moduleNumber: 5
lessonNumber: 11
order: 511
summary: >
  When a reward is hard to specify but an expert is easy to watch, learn from
  demonstrations instead. Behavioral cloning treats control as supervised
  learning of the expert's state-to-action map, and fails through compounding
  error: small mistakes carry the agent off the expert's distribution, where it
  was never trained. DAgger fixes the mismatch by querying the expert on the
  learner's own states. Inverse RL instead recovers the reward the expert seems
  to optimize — an ill-posed problem that maximum-entropy IRL disambiguates. A
  companion lesson casts imitation as adversarial occupancy matching (GAIL, AIRL).
topics: [Deep RL]
sources:
  - book: Sutton & Barto
    ref: "Ch. 3 — Finite Markov Decision Processes; Ch. 13 — Policy Gradient Methods"
  - book: Grokking Deep RL
    ref: "Ch. 12 — advanced actor-critic; learning from demonstrations"
---

Every method so far assumed a reward. The environment hands the agent a scalar
$r_t$, and all the machinery — value functions, Bellman backups, policy gradients —
exists to turn that signal into behavior. But in many of the tasks people most
want to automate, the reward is the hard part. What is the numeric reward for
_driving well_, for a surgical suture, for a natural backflip? Write one down by
hand and the agent will exploit its every gap: reward forward speed and it drives
into walls sideways, reward staying on the road and it circles forever going
nowhere. The signal a designer can articulate is rarely the signal the task
actually needs.

What is often available instead is an **expert**. A human driver, a teleoperated
robot arm, a champion game log — a policy $\pi_E$ we cannot write down but can
watch, producing demonstrations

$$
\mathcal{D} = \{\tau_1, \tau_2, \ldots, \tau_m\},
\qquad
\tau = (s_0, a_0, s_1, a_1, \ldots, s_T),
$$

trajectories of states paired with the actions the expert chose. **Imitation
learning** asks: given $\mathcal{D}$, recover a policy that behaves like $\pi_E$,
using the demonstrations _in place of_ a reward. **Inverse reinforcement
learning** asks the deeper question: what reward was the expert optimizing, so
that we could re-run ordinary RL and perhaps _surpass_ the demonstrator? Both
sidestep reward specification; they differ in whether they copy the behavior or
reverse-engineer its cause.

This lesson builds the classical trio: behavioral cloning and its compounding-error
flaw, DAgger's fix by querying the expert, and inverse RL's recovery of the reward
itself. A [companion lesson](/reinforcement-learning/modern-deep-rl/imitation-and-inverse-rl-part-2)
then dissolves the reward entirely, casting imitation as an adversarial game (GAIL)
and reading a transferable reward back out of it (AIRL).

## Behavioral cloning

The simplest idea is also the most tempting: forget that this is a control problem
at all. Each demonstrated transition is a labelled example $(s, a)$ — input state,
target action — so a supervised learner can fit a policy $\pi_\theta(a \mid s)$ by
maximum likelihood, exactly as it would fit any classifier or regressor.

$$
\theta^\star \;=\; \argmax_\theta \;
\sum_{(s,a) \in \mathcal{D}} \log \pi_\theta(a \mid s).
$$

This is **behavioral cloning** (BC): imitation reduced to a single pass of
supervised learning over the demonstration set.[^bc] No environment interaction,
no value function, no reward — just a labelled dataset and a cross-entropy (for
discrete actions) or mean-squared (for continuous actions) loss. When it works it
is unbeatably cheap, and it is genuinely useful as a **warm start**: pretraining a
policy by BC before switching to
[policy-gradient](/reinforcement-learning/approximation/policy-gradient-methods)
fine-tuning is standard practice, because a randomly initialized policy wastes
enormous effort just discovering the plausible actions BC already knows.

$$
% caption: Behavioral cloning as supervised learning. Each demonstrated transition
% is treated as a labeled example: the state is the input, the expert's action is
% the target. A single supervised fit maps states to actions, ignoring that the
% policy will later choose the states it sees.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=20mm, minimum height=10mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[box] (s) at (0,0) {state s\\(input)};
  \node[box, draw=acc, text=acc, thick] (pi) at (4.0,0) {policy\\pi(a given s)};
  \node[box, draw=red, text=red] (a) at (8.0,0) {expert action a\\(label)};
  \draw[->, black, thick] (s) -- (pi) node[midway, above, font=\scriptsize] {feed};
  \draw[->, red, thick, dashed] (a.south) to[out=270, in=270] node[pos=0.5, below, font=\scriptsize, text=red] {supervised loss: match the label} (pi.south);
  \node[black, anchor=north, font=\scriptsize] at (4.0,-1.9) {one pass, no environment, no reward};
\end{tikzpicture}
$$

> **Definition (Behavioral cloning).** Learning a policy $\pi_\theta(a \mid s)$ by
> supervised maximum likelihood on the expert's state-action pairs
> $\{(s,a)\} \subset \mathcal{D}$, treating each demonstrated action as the
> ground-truth label for its state and ignoring the sequential structure of the
> task.

### The fatal flaw: compounding error

Behavioral cloning makes an assumption supervised learning always makes and control
never permits: that the test inputs are drawn from the same distribution as the
training inputs. In classification, the images you evaluate on look like the images
you trained on. In control, **the policy chooses its own inputs**. The states the
cloned policy visits at test time are generated by _its own_ actions, not the
expert's, and the moment it makes one imperfect prediction it lands in a state the
expert would never have entered — a state absent from $\mathcal{D}$, where the
policy's behavior is pure extrapolation.

This is **covariate shift**, and control turns it into a feedback loop. Suppose the
cloned policy matches the expert with per-step error probability $\epsilon$. A
single mistake nudges the trajectory slightly off the demonstrated path. Off-path
states were undertrained, so the next prediction is _more_ likely to err, nudging
the trajectory further still. Errors do not average out; they accumulate, because
each one changes the distribution of states the next decision faces.

$$
% caption: Compounding error in behavioral cloning. The expert path (blue) stays
% in the demonstrated region (shaded); the cloned policy (red) makes a small error,
% enters an off-distribution state it was never trained on, errs more there, and
% drifts away — quadratically in the horizon rather than linearly.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % demonstrated tube
  \fill[acc!7] plot[smooth] coordinates {(0,0.35)(2,0.45)(4,0.5)(6,0.55)(8,0.6)}
    -- plot[smooth] coordinates {(8,-0.6)(6,-0.55)(4,-0.5)(2,-0.45)(0,-0.35)} -- cycle;
  \node[acc, anchor=south, font=\scriptsize] at (1.5,0.5) {expert distribution};
  % expert trajectory
  \draw[acc, very thick] plot[smooth] coordinates {(0,0)(2,0.05)(4,0)(6,0.05)(8,0)};
  \fill[acc] (0,0) circle (2pt);
  \node[acc, anchor=east, font=\scriptsize] at (-0.1,0) {start};
  \node[acc, anchor=west, font=\scriptsize] at (8.1,0) {expert};
  % cloned trajectory: diverges quadratically
  \draw[red, very thick] plot[smooth] coordinates {(0,0)(2,-0.2)(3.5,-0.55)(5,-1.25)(6.5,-2.3)(8,-3.7)};
  \node[red, anchor=west, font=\scriptsize] at (8.1,-3.7) {cloned policy};
  % annotate first slip
  \fill[red] (2,-0.2) circle (1.8pt);
  \draw[red!70, ->] (3.3,-0.15) to[bend right=12] (2.15,-0.22);
  \node[red, anchor=west, font=\scriptsize] at (3.35,-0.12) {f\/irst small error};
  % annotate off-distribution
  \fill[red] (5,-1.25) circle (1.8pt);
  \draw[red!70, ->] (6.1,-1.0) to[bend left=10] (5.15,-1.22);
  \node[red, anchor=west, font=\scriptsize] at (6.15,-0.95) {o\/f\/f-distribution: untrained};
\end{tikzpicture}
$$

Ross and Bagnell made the scaling precise: a policy
with per-step error $\epsilon$ under the expert's distribution can suffer total
cost that grows as $T^2\epsilon$ over a horizon of $T$ steps, rather than the
$T\epsilon$ a naive per-step bound would suggest.[^bagnell] The extra factor of $T$
is precisely the compounding: the policy is not just wrong at $\epsilon T$ steps,
it is wrong in _states it drove itself into_, which are worse than the states the
bound was measured on. Double the demonstrations and $\epsilon$ shrinks, but the
$T^2$ structure remains — behavioral cloning is brittle in a way no amount of data
of the same kind fully removes.

**Where the extra $T$ comes from, concretely.** Picture a corridor task with an
absorbing "off-track" trap: at each step the cloned policy matches the expert with
probability $1 - \epsilon$ and, with probability $\epsilon$, takes a wrong action
that dumps it into the trap, where it stays and accrues cost $1$ per remaining step.
A mistake at step $t$ therefore costs about $T - t$ — the entire remaining horizon —
not a single unit. Summing over when the first mistake lands, the expected total
cost is on the order of

$$
\sum_{t=1}^{T} \Pr(\text{first slip at } t)\cdot(T - t)
\;\approx\; \epsilon \sum_{t=1}^{T}(T - t)
\;=\; \epsilon \cdot \frac{T(T-1)}{2}
\;=\; O(T^2\epsilon).
$$

Put numbers on it: with $T = 100$ steps and a per-step error $\epsilon = 0.01$
(the clone is right $99\%$ of the time), the naive bound $T\epsilon = 1$ suggests
about one unit of cost, but the compounding bound $\tfrac{1}{2}T^2\epsilon =
\tfrac12\cdot 10000\cdot 0.01 = 50$ gives fifty — a fiftyfold gap, and it widens
with the horizon. To halve the realized cost you must halve $\epsilon$, which under
supervised learning means roughly quadrupling the demonstration set; the horizon
term $T^2$ never goes away. This is why a behavioral-cloning driving policy that
looks flawless on held-out frames still wanders off the road after a few seconds of
control: the held-out frames measure $\epsilon$, the road measures $T^2\epsilon$.

$$
% caption: Cost growth with the horizon. A method trained on its own induced
% distribution (DAgger, blue) accrues cost linearly in T; behavioral cloning (red)
% accrues cost quadratically, because each early mistake pushes the policy into
% off-distribution states that stay costly for the rest of the episode. The gap
% widens without bound as the horizon grows.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (-0.2,0) -- (8.6,0) node[anchor=north, black, font=\footnotesize] {horizon T};
  \draw[->, black] (0,-0.2) -- (0,4.4) node[anchor=east, black, font=\footnotesize] {expected cost};
  % linear (DAgger)
  \draw[acc, very thick] (0,0) -- (8.0,1.6);
  \node[acc, anchor=north west, font=\scriptsize] at (5.6,0.95) {DAgger: T epsilon};
  % quadratic (BC)
  \draw[red, very thick] plot coordinates {(0,0)(2,0.28)(4,1.1)(6,2.5)(7.6,4.0)};
  \node[red, anchor=east, font=\scriptsize] at (6.6,3.6) {cloning: T-squared epsilon};
\end{tikzpicture}
$$

> **Definition (Covariate shift, in imitation).** The mismatch between the
> state distribution $d^{\pi_E}$ on which a cloned policy is trained and the
> distribution $d^{\pi_\theta}$ it induces at test time. Because the policy
> selects its own future states, any error moves it toward states with no
> training support, where errors are larger — a self-reinforcing drift absent
> from ordinary supervised learning.

The root cause is the i.i.d. assumption. Supervised learning is sound only when
train and test draw from one fixed distribution; a policy _breaks_ that assumption
by construction, since $d^{\pi_\theta}$ depends on $\pi_\theta$ itself. Any honest
fix must confront the distribution the learner actually visits — not the one the
expert did.

## DAgger: fixing the distribution mismatch

If the problem is that the learner is trained on the expert's states but tested on
its own, the fix writes itself: train it on its _own_ states. The catch is that we
only have labels for the expert's states — for a state the learner drifts into, we
do not know what the expert would have done. **DAgger** (Dataset Aggregation)
resolves this by keeping the expert **in the loop**: run the current learner, let
it visit its own (possibly mistaken) states, and _query the expert_ for the correct
action at each one.[^dagger]

The result is an iterative dataset-aggregation loop. At iteration $i$, roll out the
current policy $\pi_i$ to collect the states it actually visits, ask the expert
$\pi_E$ to label those states with correct actions, add the new labelled pairs to
the growing dataset, and retrain on the aggregate. Over iterations the training
distribution converges to the distribution the learner induces — the very
distribution it will be tested on.

```algorithm
caption: $\textsc{DAgger}$ — dataset aggregation with an interactive expert
input: expert $\pi_E$, number of iterations $N$, mixing schedule $\beta_i$
$\mathcal{D} \gets \varnothing$
initialize $\pi_1$ (e.g. by behavioral cloning on expert demonstrations)
for $i = 1$ to $N$ do
  $\pi \gets \beta_i\,\pi_E + (1 - \beta_i)\,\pi_i$ // mix in the expert early on
  sample trajectories by running $\pi$ in the environment
  collect the visited states $\{s\}$ into a dataset $\mathcal{D}_i$
  for each $s \in \mathcal{D}_i$ do
    query the expert for $a = \pi_E(s)$ // label the LEARNER's states
  $\mathcal{D} \gets \mathcal{D} \cup \mathcal{D}_i$ // aggregate, do not discard
  train $\pi_{i+1}$ on all of $\mathcal{D}$ by supervised learning
return the best $\pi_i$ on validation
```

The mixing coefficient $\beta_i$ is the one subtlety. Early on the learner is bad,
so rolling it out unguided produces useless (or dangerous) states; DAgger blends in
the expert with weight $\beta_i$, starting near $\beta_1 = 1$ (roll out the expert)
and decaying to $0$ (roll out the pure learner). By the final iterations the states
collected are the learner's own, and the aggregated dataset covers exactly the
region the learner drifts into.

$$
% caption: The DAgger loop. Run the current policy to collect the states it
% actually visits, have the expert label those states with correct actions,
% aggregate them into the dataset, and retrain — closing the gap between the
% training distribution and the distribution the learner induces.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=27mm, minimum height=12mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box, draw=acc, text=acc, thick] (pol) at (0,0) {current policy};
  \node[box] (roll) at (4.6,0) {roll out:\\collect visited states};
  \node[box] (exp) at (4.6,-2.6) {expert labels\\each state};
  \node[box] (agg) at (0,-2.6) {aggregate into\\dataset, retrain};
  \draw[->, acc, thick] (pol) -- (roll) node[midway, above, font=\scriptsize] {run};
  \draw[->, acc, thick] (roll) -- (exp) node[midway, right, font=\scriptsize] {query};
  \draw[->, acc, thick] (exp) -- (agg) node[midway, below, font=\scriptsize] {label};
  \draw[->, acc, thick] (agg) -- (pol) node[midway, left, font=\scriptsize] {train};
\end{tikzpicture}
$$

DAgger converts imitation into a **no-regret online learning** problem, and Ross et
al. prove that under this scheme the total error grows linearly, $T\epsilon$,
recovering the benign scaling that behavioral cloning loses. The price:
DAgger needs an **interactive expert**, one that can be queried on
arbitrary states the learner invents — not a fixed log of past demonstrations. A
recorded dataset of expert driving cannot tell you what the expert would do in the
weird half-off-road state your bad policy produced; only a live expert can. Where
that expert is available (a human labeller, a slow-but-correct planner, a
teleoperator), DAgger is the standard fix for compounding error. Where it is not,
you are back to either plain cloning or the reward-recovery methods below.

## Inverse reinforcement learning

Cloning copies _what_ the expert did. **Inverse reinforcement learning** (IRL)
recovers the _why_: the reward function $r(s,a)$ under which the observed
behavior is optimal. The payoff is generalization: a reward is a far more portable
object than a policy. If you can recover the reward, you can re-run ordinary RL from
it in a new environment, with different dynamics, or for longer than the expert
ever demonstrated — and potentially exceed the demonstrator, since a reward
describes the _goal_ rather than one particular way of reaching it.[^ng-russell]

> **Definition (Inverse reinforcement learning).** Given demonstrations from an
> expert assumed (near-)optimal, recover a reward function $r$ such that the
> expert's policy $\pi_E$ is optimal (or high-value) under $r$. The recovered
> reward can then be handed to a forward RL algorithm to reproduce or improve on
> the demonstrated behavior.

### Why it is ill-posed

IRL is fundamentally **underdetermined**, and Ng and Russell made the point exactly
in the paper that framed the problem: for any expert policy, _many_ reward functions
make it optimal. The trivial witness is $r \equiv 0$: under a constant zero reward
every policy is optimal, so the expert's policy is optimal too, and $r=0$ "explains"
any demonstration whatsoever. Less trivially, reward shaping shows that adding a
potential-based term $\gamma\,\phi(s') - \phi(s)$ to any reward leaves the optimal
policy unchanged, so an entire family of rewards induces identical behavior. The
mapping from behavior back to reward is many-to-one; observing a policy determines
only an equivalence class of rewards, never a single one.

$$
% caption: Inverse RL is ill-posed. Many distinct reward functions (left) all make
% the same expert policy optimal (right), so behavior alone cannot single one out —
% the trivial $r = 0$ among them. A criterion beyond consistency, such as maximum
% entropy, is needed to choose.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  rw/.style={draw, minimum width=22mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[rw] (r1) at (0,1.6)  {reward A};
  \node[rw] (r2) at (0,0)    {reward B};
  \node[rw] (r3) at (0,-1.6) {reward C (r = 0)};
  \node[rw, draw=acc, text=acc, thick] (pi) at (5.4,0) {one expert policy};
  \draw[->, black] (r1) -- (pi);
  \draw[->, black] (r2) -- (pi);
  \draw[->, black] (r3) -- (pi);
  \node[anchor=south, font=\scriptsize] at (0,2.15) {many rewards};
  \node[acc, anchor=west, font=\scriptsize] at (5.55,-1.3) {same behavior};
\end{tikzpicture}
$$

Early IRL work resolved the ambiguity with heuristics — prefer a reward under which
the expert beats alternatives by the largest margin, or match the expected feature
counts of the demonstrations. **Feature matching** is the key structural idea:
assume the reward is linear in known features, $r(s) = w^\top \phi(s)$, and require
that the learner's policy match the expert's _expected discounted feature counts_,
$\mathbb{E}[\sum_t \gamma^t \phi(s_t)]$. Matching feature counts guarantees the
learner's value equals the expert's under _that_ linear reward. But feature matching
alone is still degenerate: many policies (many stochastic mixtures) match the same
feature counts, and nothing selects among them.

### Maximum-entropy IRL

The clean resolution is a principled tie-breaker: among all policies (or all
trajectory distributions) that match the expert's feature counts, pick the one that
is **maximally random** otherwise. Committing to nothing beyond what the data forces
is the maximum-entropy principle, and applying it to IRL gives a distribution over
trajectories that is exponential in reward.[^maxent] Maximum-entropy IRL models the
expert as producing trajectories with probability proportional to the exponential of
their total reward,

$$
p(\tau) \;=\; \frac{1}{Z}\, \exp\!\Bigl(\textstyle\sum_{t} r_w(s_t, a_t)\Bigr),
\qquad
Z \;=\; \sum_{\tau} \exp\!\Bigl(\textstyle\sum_{t} r_w(s_t, a_t)\Bigr),
$$

where higher-reward trajectories are exponentially more likely but no trajectory is
ever ruled out. This is a soft, probabilistic model of a _near_-optimal expert: it
tolerates the occasional suboptimal action rather than demanding strict optimality,
which is both more realistic and mathematically better-behaved. Fitting $w$ by
maximum likelihood on the demonstrations has a clean gradient — the difference
between the expert's expected features and the model's expected features — and its
maximum-entropy form uniquely resolves the ambiguity that made plain feature
matching degenerate. Ziebart et al. introduced it for modeling real driver-route
choices, and it became the template for essentially all subsequent probabilistic
IRL, including the deep variants that replace $w^\top\phi$ with a neural reward.

> **Definition (Maximum-entropy IRL).** An IRL method that models expert
> trajectories as $p(\tau) \propto \exp\bigl(\sum_t r_w(s_t, a_t)\bigr)$ and fits
> the reward parameters $w$ by maximum likelihood. Among all reward-consistent
> trajectory distributions it selects the highest-entropy one, uniquely resolving
> the ill-posedness while modeling the expert as noisily rather than perfectly
> optimal.

**A worked feature-matching step.** Take a gridworld with
two features per state: $\phi_1(s) = 1$ if $s$ is on a highway, and $\phi_2(s) = 1$
if $s$ is a scenic back road, both $0$ otherwise, with reward $r_w(s) = w_1\phi_1 +
w_2\phi_2$. Suppose the expert's demonstrated trajectories spend, in expected
discounted feature counts, $\bar\phi_E = (8, 2)$ — mostly highway. Start the reward
at $w = (0, 0)$; then every action is equally good, the model's induced policy
wanders uniformly, and its feature counts come out $\bar\phi_{\text{model}} =
(5, 5)$. The maximum-entropy gradient equals the feature-count mismatch,

$$
\nabla_w \mathcal{L} = \bar\phi_E - \bar\phi_{\text{model}} = (8, 2) - (5, 5) = (3, -3),
$$

so a gradient step _raises_ $w_1$ (reward highway more) and _lowers_ $w_2$ (reward
back roads less). Re-solve the forward problem under the new reward, the induced
policy now favors the highway, its feature counts move toward $(8, 2)$, the
gradient shrinks, and at convergence the model matches the expert's counts exactly —
$\bar\phi_{\text{model}} = \bar\phi_E$ — which under the linear reward means the
model's value equals the expert's. The recovered $w$ encodes that the expert values
highway over scenery — an explanation of the behavior that transfers to a new map
with different roads, where a cloned policy transfers nothing.

The recurring cost of IRL is that it is a **double loop**. Every gradient step on
the reward $w$ requires computing the model's expected feature counts (the
$\bar\phi_{\text{model}}$ above), which means solving the _forward_ RL problem under
the current reward — a full planning or RL run nested inside each reward update. In
the worked example, each of the arrows "re-solve the forward problem" is an entire
value iteration. IRL recovers a portable reward, but pays for it with an expensive
inner optimization that BC never needs.

$$
% caption: The IRL double loop. An outer loop adjusts the reward weights w by the
% feature-count mismatch (expert counts minus model counts); each such update
% requires an inner loop that fully re-solves the forward RL problem under the
% current reward to recompute the model's feature counts. The nested forward solve
% is what makes IRL expensive.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=32mm, minimum height=12mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[box, draw=acc, text=acc, thick] (rew) at (0,0) {update reward w\\by count mismatch};
  \node[box, draw=red, text=red] (fwd) at (6.4,0) {inner loop:\\solve forward RL};
  \draw[->, acc, thick] (rew) to[bend left=22] node[midway, above, font=\scriptsize] {new reward} (fwd);
  \draw[->, red, thick] (fwd) to[bend left=22] node[midway, below, font=\scriptsize] {model feature counts} (rew);
  \node[black, anchor=north, font=\scriptsize] at (3.2,-1.5) {outer loop repeats until counts match};
\end{tikzpicture}
$$

## Where this leaves us

Three ways to learn from demonstrations are now in hand, sorted by how they confront
the covariate shift that sinks naive cloning:

- **Behavioral cloning** treats control as supervised learning and pays for the
  category error with $T^2\epsilon$ compounding — it trains on the expert's states but
  is tested on the ones its own mistakes create.
- **DAgger** fixes the mismatch directly by querying an interactive expert on the
  learner's own states, recovering the benign $T\epsilon$ scaling.
- **Inverse RL** recovers the reward the expert seems to optimize — a portable object
  that transfers to new dynamics — resolving the ill-posedness with the
  maximum-entropy principle, at the cost of a forward-RL solve in the inner loop.

That IRL double loop is the opening the next lesson exploits. If the point of
recovering a reward is only to re-run RL and match the expert, maybe you can skip the
reward and match the behavior directly — an occupancy-matching game that turns out to
be a GAN. That reframing (GAIL), and the way to read a transferable reward back out of
it (AIRL), plus a comparison of all the methods and the modern lines that grew from
them, continue in
[Imitation as Adversarial Matching: GAIL and AIRL](/reinforcement-learning/modern-deep-rl/imitation-and-inverse-rl-part-2).

[^bc]: The supervised-imitation baseline predates deep RL; it appears as **ALVINN** (Pomerleau, 1989), "ALVINN: An Autonomous Land Vehicle in a Neural Network", NeurIPS — a neural net cloning a human driver's steering, and an early demonstration of both the appeal and the brittleness of behavioral cloning.
[^bagnell]: **Ross & Bagnell (2010)**, "Efficient Reductions for Imitation Learning", AISTATS — the analysis showing behavioral cloning incurs cost growing as $T^2\epsilon$ in the horizon, versus $T\epsilon$ for methods that train on the learner's own induced distribution.
[^dagger]: **Ross, Gordon, Bagnell (2011)**, "A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning", AISTATS — introduces DAgger, aggregating expert labels on the learner's visited states and proving the resulting linear-in-horizon error bound.
[^ng-russell]: **Ng & Russell (2000)**, "Algorithms for Inverse Reinforcement Learning", ICML — the paper that framed IRL, characterized the set of rewards consistent with an optimal policy, and noted the degeneracy (including $r \equiv 0$) that makes the problem ill-posed.
[^maxent]: **Ziebart, Maas, Bagnell, Dey (2008)**, "Maximum Entropy Inverse Reinforcement Learning", AAAI — resolves IRL's ambiguity with the maximum-entropy principle, modeling trajectories as $p(\tau) \propto \exp(\sum_t r_w(s_t,a_t))$ and fitting the reward by maximum likelihood on demonstrated feature counts.