Imitation Learning: Cloning, DAgger, and Inverse RL

Every method so far assumed a reward. The environment hands the agent a scalar $r_t$, and all the machinery — value functions, Bellman backups, policy gradients — exists to turn that signal into behavior. But in many of the tasks people most want to automate, the reward is the hard part. What is the numeric reward for driving well, for a surgical suture, for a natural backflip? Write one down by hand and the agent will exploit its every gap: reward forward speed and it drives into walls sideways, reward staying on the road and it circles forever going nowhere. The signal a designer can articulate is rarely the signal the task actually needs.

What is often available instead is an expert. A human driver, a teleoperated robot arm, a champion game log — a policy ${\pi }_E$ we cannot write down but can watch, producing demonstrations

trajectories of states paired with the actions the expert chose. Imitation learning asks: given $\mathcal{D}$, recover a policy that behaves like ${\pi }_E$, using the demonstrations in place of a reward. Inverse reinforcement learning asks the deeper question: what reward was the expert optimizing, so that we could re-run ordinary RL and perhaps surpass the demonstrator? Both sidestep reward specification; they differ in whether they copy the behavior or reverse-engineer its cause.

This lesson builds the classical trio: behavioral cloning and its compounding-error flaw, DAgger's fix by querying the expert, and inverse RL's recovery of the reward itself. A companion lesson then dissolves the reward entirely, casting imitation as an adversarial game (GAIL) and reading a transferable reward back out of it (AIRL).

Behavioral cloning

The simplest idea is also the most tempting: forget that this is a control problem at all. Each demonstrated transition is a labelled example $(s,a)$ — input state, target action — so a supervised learner can fit a policy ${\pi }_{\theta }(a\mid s)$ by maximum likelihood, exactly as it would fit any classifier or regressor.

This is behavioral cloning (BC): imitation reduced to a single pass of supervised learning over the demonstration set.¹ No environment interaction, no value function, no reward — just a labelled dataset and a cross-entropy (for discrete actions) or mean-squared (for continuous actions) loss. When it works it is unbeatably cheap, and it is genuinely useful as a warm start: pretraining a policy by BC before switching to policy-gradient fine-tuning is standard practice, because a randomly initialized policy wastes enormous effort just discovering the plausible actions BC already knows.

The fatal flaw: compounding error

Behavioral cloning makes an assumption supervised learning always makes and control never permits: that the test inputs are drawn from the same distribution as the training inputs. In classification, the images you evaluate on look like the images you trained on. In control, the policy chooses its own inputs. The states the cloned policy visits at test time are generated by its own actions, not the expert's, and the moment it makes one imperfect prediction it lands in a state the expert would never have entered — a state absent from $\mathcal{D}$, where the policy's behavior is pure extrapolation.

This is covariate shift, and control turns it into a feedback loop. Suppose the cloned policy matches the expert with per-step error probability $ϵ$. A single mistake nudges the trajectory slightly off the demonstrated path. Off-path states were undertrained, so the next prediction is more likely to err, nudging the trajectory further still. Errors do not average out; they accumulate, because each one changes the distribution of states the next decision faces.

Ross and Bagnell made the scaling precise: a policy with per-step error $ϵ$ under the expert's distribution can suffer total cost that grows as $T^2ϵ$ over a horizon of $T$ steps, rather than the $Tϵ$ a naive per-step bound would suggest.² The extra factor of $T$ is precisely the compounding: the policy is not just wrong at $ϵT$ steps, it is wrong in states it drove itself into, which are worse than the states the bound was measured on. Double the demonstrations and $ϵ$ shrinks, but the $T^2$ structure remains — behavioral cloning is brittle in a way no amount of data of the same kind fully removes.

Where the extra $T$ comes from, concretely. Picture a corridor task with an absorbing off-track trap: at each step the cloned policy matches the expert with probability $1-ϵ$ and, with probability $ϵ$, takes a wrong action that dumps it into the trap, where it stays and accrues cost $1$ per remaining step. A mistake at step $t$ therefore costs about $T-t$ — the entire remaining horizon — not a single unit. Summing over when the first mistake lands, the expected total cost is on the order of

Put numbers on it: with $T=100$ steps and a per-step error $ϵ=0.01$ (the clone is right $99\%$ of the time), the naive bound $Tϵ=1$ suggests about one unit of cost, but the compounding bound $\frac{1}{2}{T}^2ϵ=\frac{1}{2}\cdot 10000\cdot 0.01=50$ gives fifty — a fiftyfold gap, and it widens with the horizon. To halve the realized cost you must halve $ϵ$, which under supervised learning means roughly quadrupling the demonstration set; the horizon term $T^2$ never goes away. This is why a behavioral-cloning driving policy that looks flawless on held-out frames still wanders off the road after a few seconds of control: the held-out frames measure $ϵ$, the road measures $T^2ϵ$.

The root cause is the i.i.d. assumption. Supervised learning is sound only when train and test draw from one fixed distribution; a policy breaks that assumption by construction, since $d^{{\pi }_{\theta }}$ depends on ${\pi }_{\theta }$ itself. Any honest fix must confront the distribution the learner actually visits — not the one the expert did.

DAgger: fixing the distribution mismatch

If the problem is that the learner is trained on the expert's states but tested on its own, the fix writes itself: train it on its own states. The catch is that we only have labels for the expert's states — for a state the learner drifts into, we do not know what the expert would have done. DAgger (Dataset Aggregation) resolves this by keeping the expert in the loop: run the current learner, let it visit its own (possibly mistaken) states, and query the expert for the correct action at each one.³

The result is an iterative dataset-aggregation loop. At iteration $i$, roll out the current policy ${\pi }_i$ to collect the states it actually visits, ask the expert ${\pi }_E$ to label those states with correct actions, add the new labelled pairs to the growing dataset, and retrain on the aggregate. Over iterations the training distribution converges to the distribution the learner induces — the very distribution it will be tested on.

The mixing coefficient ${\beta }_i$ is the one subtlety. Early on the learner is bad, so rolling it out unguided produces useless (or dangerous) states; DAgger blends in the expert with weight ${\beta }_i$, starting near ${\beta }_1=1$ (roll out the expert) and decaying to $0$ (roll out the pure learner). By the final iterations the states collected are the learner's own, and the aggregated dataset covers exactly the region the learner drifts into.

DAgger converts imitation into a no-regret online learning problem, and Ross et al. prove that under this scheme the total error grows linearly, $Tϵ$, recovering the benign scaling that behavioral cloning loses. The price: DAgger needs an interactive expert, one that can be queried on arbitrary states the learner invents — not a fixed log of past demonstrations. A recorded dataset of expert driving cannot tell you what the expert would do in the weird half-off-road state your bad policy produced; only a live expert can. Where that expert is available (a human labeller, a slow-but-correct planner, a teleoperator), DAgger is the standard fix for compounding error. Where it is not, you are back to either plain cloning or the reward-recovery methods below.

Inverse reinforcement learning

Cloning copies what the expert did. Inverse reinforcement learning (IRL) recovers the why: the reward function $r(s,a)$ under which the observed behavior is optimal. The payoff is generalization: a reward is a far more portable object than a policy. If you can recover the reward, you can re-run ordinary RL from it in a new environment, with different dynamics, or for longer than the expert ever demonstrated — and potentially exceed the demonstrator, since a reward describes the goal rather than one particular way of reaching it.⁴

Why it is ill-posed

IRL is fundamentally underdetermined, and Ng and Russell made the point exactly in the paper that framed the problem: for any expert policy, many reward functions make it optimal. The trivial witness is $r\equiv 0$: under a constant zero reward every policy is optimal, so the expert's policy is optimal too, and $r=0$ explains any demonstration whatsoever. Less trivially, reward shaping shows that adding a potential-based term $\gamma \varphi (s^{\prime })-\varphi (s)$ to any reward leaves the optimal policy unchanged, so an entire family of rewards induces identical behavior. The mapping from behavior back to reward is many-to-one; observing a policy determines only an equivalence class of rewards, never a single one.

Early IRL work resolved the ambiguity with heuristics — prefer a reward under which the expert beats alternatives by the largest margin, or match the expected feature counts of the demonstrations. Feature matching is the key structural idea: assume the reward is linear in known features, $r(s)=w^{\top }\varphi (s)$, and require that the learner's policy match the expert's expected discounted feature counts, $\mathbb{E}[{\sum }_t{\gamma }^t\varphi (s_t)]$. Matching feature counts guarantees the learner's value equals the expert's under that linear reward. But feature matching alone is still degenerate: many policies (many stochastic mixtures) match the same feature counts, and nothing selects among them.

Maximum-entropy IRL

The clean resolution is a principled tie-breaker: among all policies (or all trajectory distributions) that match the expert's feature counts, pick the one that is maximally random otherwise. Committing to nothing beyond what the data forces is the maximum-entropy principle, and applying it to IRL gives a distribution over trajectories that is exponential in reward.⁵ Maximum-entropy IRL models the expert as producing trajectories with probability proportional to the exponential of their total reward,

where higher-reward trajectories are exponentially more likely but no trajectory is ever ruled out. This is a soft, probabilistic model of a near-optimal expert: it tolerates the occasional suboptimal action rather than demanding strict optimality, which is both more realistic and mathematically better-behaved. Fitting $w$ by maximum likelihood on the demonstrations has a clean gradient — the difference between the expert's expected features and the model's expected features — and its maximum-entropy form uniquely resolves the ambiguity that made plain feature matching degenerate. Ziebart et al. introduced it for modeling real driver-route choices, and it became the template for essentially all subsequent probabilistic IRL, including the deep variants that replace $w^{\top }\varphi$ with a neural reward.

A worked feature-matching step. Take a gridworld with two features per state: ${\varphi }_1(s)=1$ if $s$ is on a highway, and ${\varphi }_2(s)=1$ if $s$ is a scenic back road, both $0$ otherwise, with reward $r_w(s)=w_1{\varphi }_1+w_2{\varphi }_2$. Suppose the expert's demonstrated trajectories spend, in expected discounted feature counts, ${\bar{\varphi }}_E=(8,2)$ — mostly highway. Start the reward at $w=(0,0)$; then every action is equally good, the model's induced policy wanders uniformly, and its feature counts come out ${\bar{\varphi }}_{\text{model}}=(5,5)$. The maximum-entropy gradient equals the feature-count mismatch,

so a gradient step raises $w_1$ (reward highway more) and lowers $w_2$ (reward back roads less). Re-solve the forward problem under the new reward, the induced policy now favors the highway, its feature counts move toward $(8,2)$, the gradient shrinks, and at convergence the model matches the expert's counts exactly — ${\bar{\varphi }}_{\text{model}}={\bar{\varphi }}_E$ — which under the linear reward means the model's value equals the expert's. The recovered $w$ encodes that the expert values highway over scenery — an explanation of the behavior that transfers to a new map with different roads, where a cloned policy transfers nothing.

The recurring cost of IRL is that it is a double loop. Every gradient step on the reward $w$ requires computing the model's expected feature counts (the ${\bar{\varphi }}_{\text{model}}$ above), which means solving the forward RL problem under the current reward — a full planning or RL run nested inside each reward update. In the worked example, each of the arrows re-solve the forward problem is an entire value iteration. IRL recovers a portable reward, but pays for it with an expensive inner optimization that BC never needs.

Where this leaves us

Three ways to learn from demonstrations are now in hand, sorted by how they confront the covariate shift that sinks naive cloning:

• Behavioral cloning treats control as supervised learning and pays for the category error with $T^2ϵ$ compounding — it trains on the expert's states but is tested on the ones its own mistakes create.
• DAgger fixes the mismatch directly by querying an interactive expert on the learner's own states, recovering the benign $Tϵ$ scaling.
• Inverse RL recovers the reward the expert seems to optimize — a portable object that transfers to new dynamics — resolving the ill-posedness with the maximum-entropy principle, at the cost of a forward-RL solve in the inner loop.

That IRL double loop is the opening the next lesson exploits. If the point of recovering a reward is only to re-run RL and match the expert, maybe you can skip the reward and match the behavior directly — an occupancy-matching game that turns out to be a GAN. That reframing (GAIL), and the way to read a transferable reward back out of it (AIRL), plus a comparison of all the methods and the modern lines that grew from them, continue in Imitation as Adversarial Matching: GAIL and AIRL.