---
title: Meta-RL and Generalization
module: Modern Deep Reinforcement Learning
moduleNumber: 5
lessonNumber: 22
order: 522
summary: >
  An agent that masters one task often fails on the next; it has overfit to a
  single environment. This lesson treats fast adaptation as a meta-problem over a
  distribution of tasks: meta-train so that a few episodes at meta-test time
  suffice. We cover the two families — optimization-based (MAML learns an
  initialization) and context-based (RL-squared and PEARL infer a latent task) —
  the exploration cost of adaptation, and the parallel problem of generalization:
  why deep RL memorizes environments and what fixes it (domain randomization,
  procedural generation, augmentation, regularization). It closes on foundation
  models and sequence-model agents as the generalist endpoint.
topics: [Deep RL]
sources:
  - book: Grokking Deep RL
    ref: "Ch. 4 — Balancing information gathering and use; adaptation as a bandit-like problem"
  - book: Grokking Deep RL
    ref: "Ch. 12 — Generalization and the road to general agents"
---

Every method in this course so far assumes a fixed environment. You point the
[deep-Q learner](/reinforcement-learning/deep-rl/deep-q-networks) or
[PPO](/reinforcement-learning/deep-rl/actor-critic-and-ppo) at one MDP, spend
millions of frames, and out comes a policy that plays that MDP well. Change the
maze, shift the physics, recolor the sprites, and the same policy often falls to
chance. It did not learn to solve mazes; it learned _this_ maze. Closing that gap
is the frontier of the field.

Two problems live in that gap, and they are duals of each other. **Meta-learning**
trains a _learner_ rather than a policy: something that adapts to a new
task in a handful of episodes. **Generalization** seeks a single fixed policy that
works on tasks it never saw, with no adaptation at all. The first allows a little
test-time experience; the second allows none. Both start from the same
admission — that the object worth producing is not a policy for one task but
something that transfers across a _family_ of them.

## The overfitting problem

Standard RL optimizes return on the environment it trains in, and delivers just
that: a policy tuned to the specific transitions, layouts, and reward
placements it saw. Nothing in the objective rewards transfer. When the training
and test environments are drawn from a distribution and the agent only ever sees a
few draws, it memorizes those draws.

$$
% caption: Train on a handful of levels and the agent memorizes them: training
% return climbs while return on held-out levels stalls, the RL analogue of the
% train/test generalization gap.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (0,0) -- (6.4,0) node[anchor=north west] {training frames};
  \draw[->, black] (0,0) -- (0,3.6) node[anchor=south east] {return};
  % training curve: rises high
  \draw[acc, very thick] (0,0.25)
    .. controls (1.6,1.9) and (3.0,3.0) .. (6.0,3.25);
  \node[acc, anchor=west] at (6.05,3.25) {train levels};
  % test curve: rises then plateaus low
  \draw[red, very thick] (0,0.2)
    .. controls (1.6,1.2) and (3.0,1.55) .. (6.0,1.5);
  \node[red, anchor=west] at (6.05,1.5) {test levels};
  % the gap
  \draw[black, <->] (5.4,1.55) -- (5.4,3.2);
  \node[black, anchor=east, align=center] at (5.35,2.4) {generalization\\gap};
\end{tikzpicture}
$$

The clearest evidence came from **ProcGen**, a suite of sixteen procedurally
generated arcade games.[^procgen] When agents were trained on a few hundred levels
of a game and tested on unseen levels of the _same_ game, they needed on the order
of thousands of training levels before test performance approached training
performance. With few levels the agent scored near-perfectly on what it had seen
and poorly on what it had not — memorization, plain and measurable. The same effect
appears in **CoinRun**, an earlier single-game benchmark built to expose it: a
network trained on a fixed set of levels overfits their specific geometry.[^coinrun]

> **Definition (Generalization gap in RL).** The difference between an agent's
> expected return on the tasks (levels, MDPs) it was trained on and its expected
> return on unseen tasks drawn from the same distribution. A large gap means the
> policy has fit task-specific detail rather than the shared structure the
> distribution defines.

This reframes the goal. If tasks come from a distribution $p(\mathcal{T})$, the
thing worth optimizing is not return on one task $\mathcal{T}_i$ but expected
return over fresh draws — either after a short adaptation phase (meta-learning) or
with none (zero-shot generalization).

## The meta-learning setup

Fast adaptation is a familiar problem at a new scale. The
[multi-armed bandit](/reinforcement-learning/foundations/multi-armed-bandits) was
already a meta-problem in miniature: you do not know which arm pays, so you spend
early pulls _learning_ the task (which arm is best) and later pulls _exploiting_
what you learned. Meta-RL scales that exact shape up. Each "arm" becomes a whole
MDP, each "pull" a whole episode, and the agent must, within a small budget of
episodes, figure out which MDP it is in and act well.

Formally, we assume a distribution over tasks $p(\mathcal{T})$, where each task
$\mathcal{T}_i = (\mathcal{S}, \mathcal{A}, P_i, R_i)$ is an MDP that shares the
state and action spaces but has its own dynamics $P_i$ and reward $R_i$. Meta-RL
runs in two nested loops.

$$
% caption: The nested loops of meta-RL. The outer loop draws tasks from $p(\mathcal{T})$
% and updates the meta-parameters; the inner loop adapts to one drawn task using a
% small budget of experience.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=30mm, minimum height=11mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box, draw=acc, text=acc, thick] (meta) at (0,0) {outer loop\\(meta-train)};
  \node[box] (draw) at (5,1.4) {draw task T from p(T)};
  \node[box] (adapt) at (5,-1.4) {inner loop:\\adapt on T};
  \draw[->, acc, thick] (meta.east) .. controls (2.6,0.9) .. (draw.west);
  \draw[->, black, thick] (draw.south) -- (adapt.north) node[midway, right, font=\scriptsize] {few episodes};
  \draw[->, black, thick] (adapt.west) .. controls (2.6,-0.9) .. (meta.east)
    node[midway, below, font=\scriptsize] {update theta};
\end{tikzpicture}
$$

- The **inner loop** (adaptation) takes one task and a small amount of experience
  and produces a task-specialized behavior. This is what happens at meta-test time,
  fast.
- The **outer loop** (meta-training) draws many tasks, measures how well the inner
  loop does on each, and updates the shared **meta-parameters** $\theta$ so that
  the inner loop adapts _better_ next time. This is slow and done once, offline.

> **Definition (Meta-reinforcement learning).** Learning, from a distribution of
> tasks $p(\mathcal{T})$, a procedure that adapts to a new task drawn from
> $p(\mathcal{T})$ using only a small amount of task-specific experience. The
> outer loop optimizes the meta-parameters $\theta$; the inner loop produces the
> adapted behavior for one task.

The families differ in _what_ the inner loop is. In **optimization-based** meta-RL
the inner loop is a few steps of gradient descent, and $\theta$ is a good starting
point for them. In **context-based** meta-RL the inner loop is a forward pass of a
network that has _read_ the recent experience, and $\theta$ are that network's
weights. We take them in turn.

## Optimization-based: MAML

**Model-Agnostic Meta-Learning** seeks an initialization $\theta$ from which a
single gradient step on any task lands on a good task-specific policy.[^maml] It is
"model-agnostic" because it makes no assumption about the architecture — anything
trained by gradient descent qualifies.

The picture in plain terms: there are two nested loops. An **inner loop** takes the
shared starting weights and does a quick task-specific update, as if adapting to that
one task. An **outer loop** then evaluates the starting weights not on their own
performance but on their performance _after_ that quick adaptation, and nudges the
starting point so that the after-adaptation policy is good. MAML optimizes for
_adaptability_, not for any single task.

The inner loop is one (or a few) policy-gradient steps. Starting from the shared
$\theta$, for task $\mathcal{T}_i$ it computes adapted parameters

$$
\theta_i' \;=\; \theta + \alpha\,\nabla_\theta J_{\mathcal{T}_i}(\theta),
$$

where $J_{\mathcal{T}_i}$ is the expected return on task $i$ (we ascend, so the sign
is $+$) and $\alpha$ is the inner step size. The outer loop then evaluates
$\theta_i'$ — the point reached _after_ adapting — across all tasks, and updates
$\theta$ to make that post-adaptation return high:

$$
\theta \;\gets\; \theta + \beta\,\nabla_\theta \sum_i J_{\mathcal{T}_i}(\theta_i').
$$

The subtlety is that $\theta_i'$ is itself a function of $\theta$, so the outer
gradient differentiates _through_ the inner update — a gradient of a gradient, the
**meta-gradient**. This is what makes MAML learn an initialization tuned for
adaptability rather than for any one task's performance.

$$
% caption: The MAML meta-gradient. From the shared initialization one inner step
% lands at an adapted point per task; the outer update moves the initialization so
% that those post-adaptation points score high, differentiating through the inner step.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % shared init
  \fill[acc] (0,0) circle (2.6pt);
  \node[acc, anchor=east] at (-0.12,0) {theta};
  % three task-adapted points
  \coordinate (t1) at (2.6,1.7);
  \coordinate (t2) at (3.0,0.1);
  \coordinate (t3) at (2.4,-1.6);
  \draw[->, black, thick] (0.12,0.06) -- (t1);
  \draw[->, black, thick] (0.15,0.0) -- (t2);
  \draw[->, black, thick] (0.12,-0.06) -- (t3);
  \fill[black] (t1) circle (2.2pt); \node[anchor=west] at ($(t1)+(0.12,0)$) {theta-1 (task 1)};
  \fill[black] (t2) circle (2.2pt); \node[anchor=west] at ($(t2)+(0.12,0)$) {theta-2 (task 2)};
  \fill[black] (t3) circle (2.2pt); \node[anchor=west] at ($(t3)+(0.12,0)$) {theta-3 (task 3)};
  % inner-step label
  \node[black, anchor=south, font=\scriptsize] at (1.3,0.75) {inner step};
  % meta update
  \draw[->, red, very thick] (0,0) -- (0.95,-0.55);
  \node[red, anchor=north east, font=\scriptsize] at (0.95,-0.6) {meta-step};
\end{tikzpicture}
$$

For RL the return $J$ is estimated from sampled trajectories, so both the inner and
outer gradients are policy-gradient estimates; the outer estimate must account for
the sampling done during adaptation. The full procedure:

```algorithm
caption: $\textsc{MAML-RL}$ — learn an initialization that adapts in a few gradient steps
input: task distribution $p(\mathcal{T})$, inner step $\alpha$, outer step $\beta$
initialize meta-parameters $\theta$
repeat
  sample a batch of tasks $\mathcal{T}_i \sim p(\mathcal{T})$
  for each task $\mathcal{T}_i$ do
    collect trajectories with $\pi_\theta$ on $\mathcal{T}_i$
    $\theta_i' \gets \theta + \alpha\,\nabla_\theta J_{\mathcal{T}_i}(\theta)$ // inner adaptation
    collect fresh trajectories with $\pi_{\theta_i'}$ on $\mathcal{T}_i$
  $\theta \gets \theta + \beta\,\nabla_\theta \sum_i J_{\mathcal{T}_i}(\theta_i')$ // meta-update through the inner step
until converged
return $\theta$
```

At meta-test time you drop the outer loop: draw a new task, run one or a few inner
steps from $\theta$, and act. On locomotion tasks where the reward is a different
target velocity or direction per task, a MAML-initialized policy reaches good
behavior after a single gradient step, where a network trained from a random or
jointly-pretrained start needs many more. The cost is second-order gradients (a
first-order approximation, FOMAML, drops them with little loss) and the expense of
the nested sampling.

#### Why the meta-gradient differs from joint training

It is easy to mistake MAML for "just train on all tasks at once" (joint training,
sometimes called multi-task pretraining). The two optimize different things, and a
one-dimensional picture makes the difference sharp. Imagine two tasks whose
per-task optima sit at parameters $\theta_1^\ast$ and $\theta_2^\ast$, some
distance apart. Joint training moves $\theta$ toward the point that is best _on
average right now_, which is the midpoint — a compromise that is mediocre on both
and adapts slowly to either, because the loss surface at the midpoint may point
nowhere useful. MAML instead moves $\theta$ toward the point from which _one
gradient step_ reaches a good task solution, which can be a completely different
location: a spot on the slope where the task-1 gradient points at $\theta_1^\ast$
and the task-2 gradient points at $\theta_2^\ast$. MAML optimizes for _post-step_
performance, so it prefers a good starting point for adaptation over a compromise.

$$
% caption: MAML initialization versus joint-training compromise. Two tasks have
% optima theta-1 and theta-2. Joint training settles at the average (mediocre on
% both, poor launch point). MAML places the initialization where a single inner
% gradient step reaches each task's optimum, trading current performance for
% post-adaptation performance.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % two task optima
  \fill[black] (0,0) circle (2.4pt);
  \node[anchor=north, font=\scriptsize] at (0,-0.15) {theta-1 (task 1 optimum)};
  \fill[black] (7.0,0) circle (2.4pt);
  \node[anchor=north, font=\scriptsize] at (7.0,-0.15) {theta-2 (task 2 optimum)};
  % joint-training compromise (midpoint)
  \fill[black] (3.5,0) circle (2.4pt);
  \node[black, anchor=south, font=\scriptsize] at (3.5,0.15) {joint training (average)};
  % MAML init, off the line, one step reaches each
  \coordinate (maml) at (3.5,2.2);
  \fill[acc] (maml) circle (2.6pt);
  \node[acc, anchor=south, font=\scriptsize] at (3.5,2.35) {MAML init};
  \draw[->, acc, thick] (maml) -- (0.2,0.12) node[midway, above left, black, font=\scriptsize] {1 step};
  \draw[->, acc, thick] (maml) -- (6.8,0.12) node[midway, above right, black, font=\scriptsize] {1 step};
\end{tikzpicture}
$$

#### Worked example: the meta-gradient on a scalar task

Reduce MAML to one dimension to see the meta-gradient's second-order term
explicitly. Let a task have loss $J(\theta)$ (here we _descend_ a loss for
clarity), inner step $\alpha$, and one inner step $\theta' = \theta - \alpha
J'(\theta)$. The outer objective is $J(\theta')$, and by the chain rule its
gradient with respect to the initialization is

$$
\frac{d}{d\theta}\, J(\theta')
=
J'(\theta')\cdot \frac{d\theta'}{d\theta}
=
J'(\theta')\,\bigl(1 - \alpha\, J''(\theta)\bigr).
$$

The factor $(1 - \alpha J''(\theta))$ is the meta-gradient's signature: it carries
the _curvature_ $J''$ of the inner loss. Where the loss is sharply curved
(large $J''$), the inner step overshoots or undershoots, and the meta-gradient
discounts that direction; where the loss is gently curved, the inner step is
reliable and passes through at nearly full weight. Plug numbers in: with $\alpha = 0.1$
and a curvature $J'' = 4$, the factor is $1 - 0.4 = 0.6$, so the outer update is
scaled to 60% of the naive first-order value in that direction. FOMAML simply sets
this factor to $1$ — it drops the $-\alpha J''$ term — which is why it is cheaper
and why it loses a little: it can no longer tell a well-conditioned adaptation
direction from a badly-conditioned one.

## Context-based: RL-squared and PEARL

The second family keeps the parameters fixed at meta-test time and instead lets the
policy _read_ its recent experience. Adaptation becomes inference, not gradient
descent.

### RL-squared: recurrence as the learning algorithm

**RL-squared** (written $\text{RL}^2$) makes the observation that a
recurrent network run across episodes already implements a learning algorithm — one
we can meta-learn.[^rl2] The key choice is to _not_ reset the hidden state between
episodes of the same task. The recurrent policy receives, at each step, not just the
observation but the previous action, previous reward, and a done flag; its hidden
state $h_t$ therefore accumulates a summary of everything seen so far on this task.

$$
% caption: In RL-squared the recurrent hidden state carries across episode
% boundaries within a task, so by the second and third episodes the policy has
% "read" the earlier ones and acts as an adapted learner.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  cell/.style={draw, minimum width=15mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[cell] (e1) at (0,0) {episode 1};
  \node[cell] (e2) at (2.6,0) {episode 2};
  \node[cell] (e3) at (5.2,0) {episode 3};
  \node[cell, draw=acc, text=acc] (out) at (8.0,0) {adapted};
  \draw[->, thick] (e1) -- (e2) node[midway, above, font=\scriptsize] {keep h};
  \draw[->, thick] (e2) -- (e3) node[midway, above, font=\scriptsize] {keep h};
  \draw[->, acc, thick] (e3) -- (out);
  \node[black, anchor=north, font=\scriptsize] at (2.6,-0.75) {hidden state accumulates task evidence};
\end{tikzpicture}
$$

The meta-loss is the total reward across the _whole_ sequence of
episodes, so the network is pushed to explore in the early episodes (to identify the
task fast) and exploit in the later ones. The recurrent weights _become_ the
adaptation algorithm; standard policy gradient trains them. Nothing about the update
rule at meta-test time is hand-designed — the RNN's forward pass is the learner,
which is why the method is "RL, squared."

> **Definition (Context-based adaptation).** Adapting to a task by conditioning a
> fixed-weight policy on a summary of recent experience — a recurrent hidden state
> or an inferred latent variable — rather than by changing the weights. The
> adaptation "algorithm" is learned into the network and executed by a forward pass.

### PEARL: a latent task variable

$\text{RL}^2$ blends task inference and control into one opaque hidden state.
**PEARL** (Probabilistic Embeddings for Actor-critic RL) separates them.[^pearl] It
posits an explicit **latent task variable** $z$ and learns an inference network
$q_\phi(z \mid c)$ that reads a context $c$ — a small set of recently collected
$(s, a, r, s')$ transitions — and outputs a posterior over $z$. The policy
$\pi_\theta(a \mid s, z)$ and critic are then conditioned on a sample of $z$. Give
the same policy a different $z$ and it behaves as if in a different task.

$$
% caption: PEARL infers a latent task variable $z$ from a context of recent
% transitions, then conditions the actor and critic on $z$; sampling $z$ from the
% posterior drives exploration for identifying the task.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=24mm, minimum height=10mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box] (ctx) at (0,0) {context c\\(recent s,a,r,s')};
  \node[box, draw=acc, text=acc] (q) at (3.6,0) {encoder q(z given c)};
  \node[box] (z) at (7.0,0) {latent z};
  \node[box] (pi) at (7.0,-1.9) {actor pi(a given s, z)};
  \draw[->, thick] (ctx) -- (q);
  \draw[->, acc, thick] (q) -- (z) node[midway, below, font=\scriptsize] {posterior};
  \draw[->, thick] (z) -- (pi) node[midway, right, font=\scriptsize] {condition};
\end{tikzpicture}
$$

Two design choices make PEARL efficient. First, the encoder treats the context as an
_unordered set_ — it factors the posterior as a product of per-transition Gaussian
factors, so it is permutation-invariant and cheap. Second,
the inference is decoupled from the control policy, which lets PEARL be **off-policy**:
the actor-critic (built on [SAC](/reinforcement-learning/deep-rl/actor-critic-and-ppo))
reuses stored transitions, while only the fresh context feeds the encoder. That
decoupling is why PEARL reached roughly one to two orders of magnitude better
sample efficiency than the on-policy meta-methods that came before it, on continuous
control task distributions.

Exploration falls out naturally: early in a new task the posterior $q_\phi(z \mid c)$
is broad, so sampling $z$ from it produces varied, hypothesis-testing behavior — a
form of **posterior sampling** for task identity, exactly the Thompson-sampling idea
from the [bandit](/reinforcement-learning/foundations/multi-armed-bandits) lifted to
whole MDPs. As context accumulates, the posterior sharpens and behavior commits.

#### Worked example: the posterior collapsing on a task

Watch the latent posterior sharpen on a toy task distribution where each task is a
2-D navigation problem with a hidden goal, and the latent $z$ encodes the goal
location. Before any experience, the posterior is the prior — say a broad Gaussian
$\mathcal{N}(0, \sigma^2 I)$ with $\sigma = 1.0$ over goal position. The agent
samples $z \sim \mathcal{N}(0, 1)$, which corresponds to a guess (say, the goal is
to the northeast), and it walks that way. It collects a handful of transitions; the reward
it sees (getting warmer, getting colder) is context $c$, and the encoder
$q_\phi(z \mid c)$ tightens the posterior to, say, $\mathcal{N}(\mu = (0.6, 0.4),
\sigma = 0.4)$ — still uncertain but now leaning toward one quadrant. A second
batch of context tightens it further to $\sigma = 0.15$, at which point sampled
$z$'s barely differ and the policy commits to the inferred goal. The variance
$\sigma$ falling from $1.0$ to $0.4$ to $0.15$ _is_ the exploration schedule: PEARL
never hand-codes an $\epsilon$ that decays; the shrinking posterior does the
decaying automatically, and it decays fast precisely when the context is
informative.

$$
% caption: PEARL's latent posterior sharpening over three context batches. The
% posterior over the latent task variable z starts broad (prior, high variance),
% so sampled z drives varied exploration; each batch of context transitions
% tightens it (variance 1.0, then 0.4, then 0.15), until sampled z's agree and the
% policy commits. The shrinking spread is an automatic exploration schedule.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (-3.6,0) -- (3.8,0) node[anchor=north, black] {latent z};
  \draw[->, black] (0,-0.1) -- (0,3.6) node[anchor=east, black] {q(z given c)};
  % broad prior (low, wide)
  \draw[black, thick, smooth] plot coordinates {
    (-3.4,0.15)(-2.4,0.45)(-1.4,0.95)(-0.4,1.35)(0,1.45)(0.6,1.35)(1.6,0.9)(2.6,0.42)(3.4,0.14)};
  \node[black, anchor=west, font=\scriptsize] at (2.0,1.05) {prior (var 1.0)};
  % after one batch (medium)
  \draw[red, thick, smooth] plot coordinates {
    (-1.6,0.1)(-0.8,0.6)(-0.2,1.7)(0.4,2.3)(0.9,1.7)(1.5,0.6)(2.2,0.12)};
  \node[red, anchor=west, font=\scriptsize] at (1.5,1.6) {after batch 1 (var 0.4)};
  % after two batches (sharp)
  \draw[acc, very thick, smooth] plot coordinates {
    (0.1,0.1)(0.4,0.9)(0.6,3.2)(0.8,0.9)(1.1,0.1)};
  \node[acc, anchor=west, font=\scriptsize] at (0.85,3.0) {after batch 2 (var 0.15)};
\end{tikzpicture}
$$

## Exploration for adaptation

Adaptation makes the exploration problem explicit. To adapt to a task you
must first _identify_ it, and identifying it may require actions that pay nothing
immediately — probing a lever, walking to a landmark — purely to reduce uncertainty
about which task you are in. This is a strictly harder problem than the exploration
in [single-task deep RL](/reinforcement-learning/modern-deep-rl/exploration), because
the informative action and the rewarding action can be entirely different.

$$
% caption: Meta-test exploration versus exploitation. The first episodes spend
% actions to identify the task (information); once identified, later episodes cash
% in on the right behavior (reward). A good meta-learner allocates this budget well.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  ep/.style={draw, minimum width=13mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[ep, draw=red, text=red] (a) at (0,0) {ep 1};
  \node[ep, draw=red, text=red] (b) at (1.7,0) {ep 2};
  \node[ep, draw=acc, text=acc] (c) at (3.4,0) {ep 3};
  \node[ep, draw=acc, text=acc] (d) at (5.1,0) {ep 4};
  \node[red, anchor=north, font=\scriptsize] at (0.85,-0.75) {identify (explore)};
  \node[acc, anchor=north, font=\scriptsize] at (4.25,-0.75) {exploit};
  \draw[->, black] (-0.8,0.85) -- (5.9,0.85) node[anchor=west, black, font=\scriptsize] {episodes within one task};
\end{tikzpicture}
$$

The meta-training objective handles this automatically when it scores the _whole_
adaptation sequence, not each episode alone. Because $\text{RL}^2$ and PEARL are both
graded on cumulative return over the episodes spent on a task, they are rewarded for
"wasting" early actions on identification if that pays off later — the meta-learner
discovers an exploration strategy specialized to the task distribution, rather than
using a generic one. This is the sense in which meta-RL learns _how_ to explore.

## Generalization: closing the train/test gap

Meta-learning achieves competence after a few adaptation episodes. Generalization
requires it with zero — a single fixed policy that works on unseen tasks. The obstacle,
seen already with ProcGen, is that deep RL overfits its training environments. The
fixes modify the training distribution or the network, and they are largely
orthogonal, so they compose.

$$
% caption: Four handles on the generalization gap, arranged by what they change:
% the training tasks (randomization, procedural generation) or the learner (data
% augmentation, network regularization).
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=34mm, minimum height=10mm, align=center, font=\scriptsize},
  hdr/.style={draw, thick, minimum width=34mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[hdr, draw=acc, text=acc] (t) at (0,0) {widen the tasks};
  \node[hdr, draw=acc, text=acc] (l) at (5.2,0) {regularize the learner};
  \node[box] (dr) at (0,-1.4) {domain randomization};
  \node[box] (pg) at (0,-2.8) {procedural generation};
  \node[box] (aug) at (5.2,-1.4) {data augmentation};
  \node[box] (reg) at (5.2,-2.8) {dropout / L2 / batchnorm};
  \draw[->, thick] (t) -- (dr);
  \draw[->, thick] (dr) -- (pg);
  \draw[->, thick] (l) -- (aug);
  \draw[->, thick] (aug) -- (reg);
\end{tikzpicture}
$$

### Domain randomization

If the test environment differs from training in physics, textures, or lighting,
_randomize those factors during training_ so the policy sees a broad range and
cannot lean on any single setting. Trained across enough randomized variants, the
real environment looks like just one more sample, and the policy transfers to it
zero-shot. This is how a robot control policy trained entirely in simulation — with
randomized friction, masses, and visual appearance — can be deployed on physical
hardware it never touched during training.[^domainrand] The bet is that if the real
world lies inside the span of the randomization, the policy already covers it.

> **Definition (Domain randomization).** Training across a wide random distribution
> of environment parameters (dynamics, visuals) so that the policy is forced to be
> invariant to them, making a specific unseen environment — including the real
> world — appear as one more draw from the training distribution.

### Procedural generation

Domain randomization perturbs a fixed environment; **procedural generation** builds
each environment from scratch by an algorithm, producing an effectively unbounded
supply of distinct-but-related tasks. Benchmarks like ProcGen and CoinRun generate a
new level layout every episode. The lesson from them is quantitative and blunt:
generalization improves smoothly with the _number of distinct training levels_. Train
on a hundred and the agent memorizes; train on tens of thousands and it must learn
the shared mechanics, because memorizing that many is not an option.[^procgen] More
diversity in the training tasks is the single most reliable lever on the gap.

### Data augmentation

Borrowed from supervised vision, **data augmentation** applies label-preserving
transforms — random crops, color jitter, cutout — to observations during training.
In RL a subtlety appears: the augmentation must not change the optimal action, and
the value estimate must stay consistent across augmented views of the same state.
Methods that respect this (RAD, which simply augments, and DrQ, which additionally
averages the Q-target over augmentations) sharply improved sample efficiency and
generalization on pixel-based control from a small pool of environments, sometimes
matching methods that used far more data.[^drq] Augmentation creates
task diversity from the observations you already have.

### Network regularization

The last handle is the network. A high-capacity policy net has ample room to
memorize levels, so the standard [regularizers](/deep-learning/regularization/regularization-overview)
apply: L2 weight decay, dropout, and batch normalization all measurably shrink the
generalization gap on ProcGen, and combining them helps more.[^procgen] The
counterintuitive part is that regularization matters _more_ in RL than in supervised
learning, because the agent generates its own training data and can drive itself into
a narrow, self-reinforcing slice of the state space — overfitting to a distribution
it created.

### Zero-shot versus few-shot transfer

These two goals sit on a spectrum.

| | Zero-shot transfer | Few-shot transfer |
| --- | --- | --- |
| Test-time task data | none | a few episodes |
| Weights at test time | frozen | may adapt (MAML) or condition (PEARL) |
| Bet | training diversity already covers the test task | test task is identifiable from little data |
| Typical tools | domain randomization, procedural generation, augmentation | MAML, RL-squared, PEARL |
| Failure mode | test task outside the training span | too little data to identify the task |

Domain randomization and procedural generation aim for the left column: no
adaptation, just enough training breadth that the test task is already inside the
distribution. Meta-RL occupies the right: a small adaptation budget the agent spends
to identify which task it faces. Real systems mix them — a broadly randomized policy
that _also_ adapts online is strictly harder to break.

## In-context RL and vision-language-action agents

The methods above were the state of the art around 2019. Three developments since
then reshaped what "meta-RL" and "generalization" mean in practice.

**In-context RL at scale (emergent adaptation).** RL² showed a recurrent network
_can_ implement a learned adaptation algorithm; the question left open was how far
this scales. **AdA** (Adaptive Agent) answered it by meta-training a large
transformer agent across a vast, procedurally generated task space and showing it
adapts to genuinely held-out tasks in a handful of trials — human-timescale
in-context adaptation — with the adaptation done entirely by the forward pass over
the context, no weight updates.[^ada] The lesson is that RL²'s mechanism does not
saturate: scale the task diversity and the model, and the in-context learner keeps
improving, the same scaling story that drove language models.

$$
% caption: In-context adaptation without weight updates. A meta-trained sequence
% model receives a growing context of trials on a new task; each trial's result is
% appended to the context, and the next action is produced by a forward pass that
% reads the whole history. Performance rises within a single deployment as the
% context fills, with no gradient step at test time.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  tr/.style={draw, minimum width=15mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[tr] (t1) at (0,0) {trial 1};
  \node[tr] (t2) at (2.0,0) {trial 2};
  \node[tr] (t3) at (4.0,0) {trial 3};
  \node[tr, draw=acc, text=acc] (t4) at (6.0,0) {trial k};
  \draw[->, black] (t1) -- (t2);
  \draw[->, black] (t2) -- (t3);
  \draw[->, black, dashed] (t3) -- (t4);
  \node[black, anchor=north, font=\scriptsize] at (3.0,-0.65) {context grows, weights frozen};
  % performance arrow rising
  \draw[->, acc, thick] (0,1.2) -- (6.0,2.2) node[anchor=west, font=\scriptsize] {rising performance};
  \node[acc, anchor=east, font=\scriptsize] at (0,1.2) {low};
\end{tikzpicture}
$$

**Meta-RL as what already happens inside language models.** DPO and RLHF aside, the
sharpest modern statement of RL² is that a large pretrained sequence model, given a
few examples in its prompt, adapts to a new task by a forward pass — which is
exactly context-based meta-learning, arrived at from the language side rather than
the control side. The "few-shot learning" of a large language model and the
in-context adaptation of RL² are the same computational object: a fixed network
whose activations carry a learned learning algorithm. Meta-RL's central bet — that
adaptation can be learned into weights and executed by inference — turned out to be
how the largest models adapt at all.

**Vision-language-action models (generalist robot policies).** The generalist bet
reached physical robots through **RT-2**, which fine-tunes a large vision-language
model on robot trajectories so that the same network that answers questions about
images also emits robot actions as output tokens.[^rt2] Because it inherits the
web-scale semantics of its vision-language backbone, RT-2 generalizes to objects
and instructions absent from the robot data — pick up "the extinct animal" and it
reaches for the toy dinosaur — a kind of zero-shot semantic transfer no
robot-data-only policy achieves. The **Open X-Embodiment** effort then pooled robot
data across dozens of labs and platforms to train cross-embodiment policies,
pushing the same generalist recipe toward a single policy that spans many
robots.[^oxe] These are the endpoint of the lesson's progression: one model
spanning many tasks rather than a policy per task.

## Toward generalist agents

Pushed far enough, generalization blurs the distinction between a task and the
world. If a single network can be trained on a wide enough distribution
of tasks, why not train it on _all_ of them at once and let the same weights serve
every task, selected by the prompt or context? This is the foundation-model bet
applied to control.

The bridge is the [sequence-model view](/reinforcement-learning/modern-deep-rl/offline-rl)
from offline RL. **Decision Transformer** already recast control as autoregressive
sequence modeling over $(\text{return}, s, a)$ tokens — no value function, no
bootstrapping, just a Transformer predicting the next action conditioned on a desired
return.[^dt] Once behavior is a sequence-modeling problem, the machinery of large
language models transfers directly: scale the model, scale the data, and condition on
a task specification rather than training a fresh policy each time.

$$
% caption: The generalist trajectory. A single sequence model, trained on many
% tasks (and modalities), selects behavior by conditioning on a prompt or desired
% return rather than by retraining — control folded into the foundation-model recipe.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=26mm, minimum height=10mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box] (t1) at (0,1.1) {task 1 data};
  \node[box] (t2) at (0,0) {task 2 data};
  \node[box] (t3) at (0,-1.1) {task 3 data};
  \node[box, draw=acc, text=acc, thick] (m) at (4.4,0) {one sequence model};
  \node[box] (p) at (9.0,0) {prompt-selected\\behavior};
  \draw[->, thick] (t1) -- (m);
  \draw[->, thick] (t2) -- (m);
  \draw[->, thick] (t3) -- (m);
  \draw[->, acc, thick] (m) -- (p) node[midway, above, font=\scriptsize] {condition};
\end{tikzpicture}
$$

**Gato** is the concrete instance: one Transformer, one set of weights, trained on
hundreds of tasks spanning Atari, robotic manipulation, image captioning, and dialog,
with the task implied by the tokens in context.[^gato] It is not the best agent at any
single task, and that is the point — it is a demonstration that a _single generalist_
policy can span radically different domains, the opposite of the one-maze specialist
we started with. The through-line of the whole lesson is that same shift in the object
of study: from a policy that masters one task to a learner (or a model) that spans a
distribution of them. Mastering the maze was never the goal; generalizing across mazes
is.

[^procgen]: **Cobbe, Hesse, Hilton, Schulman (2020)**, "Leveraging Procedural Generation to Benchmark Reinforcement Learning", ICML — the ProcGen benchmark; shows test performance improving with the number of training levels and quantifies the effect of L2, dropout, and batchnorm on the generalization gap.
[^coinrun]: **Cobbe, Klimov, Hesse, Kim, Schulman (2019)**, "Quantifying Generalization in Reinforcement Learning", ICML — the CoinRun benchmark isolating overfitting to specific level geometry in deep RL.
[^maml]: **Finn, Abbeel, Levine (2017)**, "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks", ICML — learns an initialization such that one or a few gradient steps on a new task yield good performance, via a meta-gradient through the inner update.
[^rl2]: **Duan, Schulman, Chen, Bartlett, Sutskever, Abbeel (2016)**, "RL-squared: Fast Reinforcement Learning via Slow Reinforcement Learning", arXiv:1611.02779 — a recurrent policy whose hidden state persists across episodes implements a learned, task-adapted RL algorithm.
[^pearl]: **Rakelly, Zhou, Quillen, Finn, Levine (2019)**, "Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables", ICML — PEARL; infers a latent task variable from a context set and conditions an off-policy actor-critic on it, with posterior sampling driving exploration.
[^domainrand]: **Tobin, Fong, Ray, Schneider, Zaremba, Abbeel (2017)**, "Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World", IROS — randomizing simulator appearance/dynamics so a policy transfers zero-shot to real hardware.
[^drq]: **Kostrikov, Yarats, Fergus (2020)**, "Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels", arXiv:2004.13649 (ICLR 2021) — DrQ; regularizes Q-learning by averaging targets over augmented observations. See also **Laskin, Lee, Stooke, Pinto, Abbeel, Srinivas (2020)**, "Reinforcement Learning with Augmented Data" (RAD), NeurIPS.
[^dt]: **Chen, Lu, Rajeswaran, Lee, Grover, Laskin, Abbeel, Srinivas, Mordatch (2021)**, "Decision Transformer: Reinforcement Learning via Sequence Modeling", NeurIPS — return-conditioned autoregressive sequence modeling as an alternative to value-based control.
[^gato]: **Reed et al. (2022)**, "A Generalist Agent" (Gato), Transactions on Machine Learning Research — a single Transformer with one set of weights trained across hundreds of tasks and modalities, task-selected by context.
[^ada]: **Adaptive Agent Team (DeepMind) (2023)**, "Human-Timescale Adaptation in an Open-Ended Task Space" (AdA), arXiv:2301.07608 — meta-trains a large transformer agent over a vast procedurally generated task space and demonstrates in-context adaptation to held-out tasks within a few trials, with no test-time weight updates, showing RL²-style in-context learning scales.
[^rt2]: **Brohan et al. (Google DeepMind) (2023)**, "RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control", arXiv:2307.15818 — fine-tunes a vision-language model to emit robot actions as tokens, inheriting web-scale semantics so the policy generalizes to novel objects and instructions unseen in the robot data.
[^oxe]: **Open X-Embodiment Collaboration (2023)**, "Open X-Embodiment: Robotic Learning Datasets and RT-X Models", arXiv:2310.08864 — pools robot demonstration data across many labs and robot embodiments and trains cross-embodiment (RT-X) policies, advancing a single generalist policy that transfers across different robots.