Meta-RL and Generalization

Every method in this course so far assumes a fixed environment. You point the deep-Q learner or PPO at one MDP, spend millions of frames, and out comes a policy that plays that MDP well. Change the maze, shift the physics, recolor the sprites, and the same policy often falls to chance. It did not learn to solve mazes; it learned this maze. Closing that gap is the frontier of the field.

Two problems live in that gap, and they are duals of each other. Meta-learning trains a learner rather than a policy: something that adapts to a new task in a handful of episodes. Generalization seeks a single fixed policy that works on tasks it never saw, with no adaptation at all. The first allows a little test-time experience; the second allows none. Both start from the same admission — that the object worth producing is not a policy for one task but something that transfers across a family of them.

The overfitting problem

Standard RL optimizes return on the environment it trains in, and delivers just that: a policy tuned to the specific transitions, layouts, and reward placements it saw. Nothing in the objective rewards transfer. When the training and test environments are drawn from a distribution and the agent only ever sees a few draws, it memorizes those draws.

The clearest evidence came from ProcGen, a suite of sixteen procedurally generated arcade games.¹ When agents were trained on a few hundred levels of a game and tested on unseen levels of the same game, they needed on the order of thousands of training levels before test performance approached training performance. With few levels the agent scored near-perfectly on what it had seen and poorly on what it had not — memorization, plain and measurable. The same effect appears in CoinRun, an earlier single-game benchmark built to expose it: a network trained on a fixed set of levels overfits their specific geometry.²

This reframes the goal. If tasks come from a distribution $p(\mathcal{T})$, the thing worth optimizing is not return on one task ${\mathcal{T}}_i$ but expected return over fresh draws — either after a short adaptation phase (meta-learning) or with none (zero-shot generalization).

The meta-learning setup

Fast adaptation is a familiar problem at a new scale. The multi-armed bandit was already a meta-problem in miniature: you do not know which arm pays, so you spend early pulls learning the task (which arm is best) and later pulls exploiting what you learned. Meta-RL scales that exact shape up. Each arm becomes a whole MDP, each pull a whole episode, and the agent must, within a small budget of episodes, figure out which MDP it is in and act well.

Formally, we assume a distribution over tasks $p(\mathcal{T})$, where each task ${\mathcal{T}}_i=(\mathcal{S},\mathcal{A},P_i,R_i)$ is an MDP that shares the state and action spaces but has its own dynamics $P_i$ and reward $R_i$. Meta-RL runs in two nested loops.

• The inner loop (adaptation) takes one task and a small amount of experience and produces a task-specialized behavior. This is what happens at meta-test time, fast.
• The outer loop (meta-training) draws many tasks, measures how well the inner loop does on each, and updates the shared meta-parameters $\theta$ so that the inner loop adapts better next time. This is slow and done once, offline.

The families differ in what the inner loop is. In optimization-based meta-RL the inner loop is a few steps of gradient descent, and $\theta$ is a good starting point for them. In context-based meta-RL the inner loop is a forward pass of a network that has read the recent experience, and $\theta$ are that network's weights. We take them in turn.

Optimization-based: MAML

Model-Agnostic Meta-Learning seeks an initialization $\theta$ from which a single gradient step on any task lands on a good task-specific policy.³ It is model-agnostic because it makes no assumption about the architecture — anything trained by gradient descent qualifies.

The picture in plain terms: there are two nested loops. An inner loop takes the shared starting weights and does a quick task-specific update, as if adapting to that one task. An outer loop then evaluates the starting weights not on their own performance but on their performance after that quick adaptation, and nudges the starting point so that the after-adaptation policy is good. MAML optimizes for adaptability, not for any single task.

The inner loop is one (or a few) policy-gradient steps. Starting from the shared $\theta$, for task ${\mathcal{T}}_i$ it computes adapted parameters

where $J_{{\mathcal{T}}_i}$ is the expected return on task $i$ (we ascend, so the sign is $+$) and $\alpha$ is the inner step size. The outer loop then evaluates ${\theta }_i^{\prime }$ — the point reached after adapting — across all tasks, and updates $\theta$ to make that post-adaptation return high:

The subtlety is that ${\theta }_i^{\prime }$ is itself a function of $\theta$, so the outer gradient differentiates through the inner update — a gradient of a gradient, the meta-gradient. This is what makes MAML learn an initialization tuned for adaptability rather than for any one task's performance.

For RL the return $J$ is estimated from sampled trajectories, so both the inner and outer gradients are policy-gradient estimates; the outer estimate must account for the sampling done during adaptation. The full procedure:

At meta-test time you drop the outer loop: draw a new task, run one or a few inner steps from $\theta$, and act. On locomotion tasks where the reward is a different target velocity or direction per task, a MAML-initialized policy reaches good behavior after a single gradient step, where a network trained from a random or jointly-pretrained start needs many more. The cost is second-order gradients (a first-order approximation, FOMAML, drops them with little loss) and the expense of the nested sampling.

Why the meta-gradient differs from joint training

It is easy to mistake MAML for just train on all tasks at once (joint training, sometimes called multi-task pretraining). The two optimize different things, and a one-dimensional picture makes the difference sharp. Imagine two tasks whose per-task optima sit at parameters ${\theta }_1^{\ast }$ and ${\theta }_2^{\ast }$, some distance apart. Joint training moves $\theta$ toward the point that is best on average right now, which is the midpoint — a compromise that is mediocre on both and adapts slowly to either, because the loss surface at the midpoint may point nowhere useful. MAML instead moves $\theta$ toward the point from which one gradient step reaches a good task solution, which can be a completely different location: a spot on the slope where the task-1 gradient points at ${\theta }_1^{\ast }$ and the task-2 gradient points at ${\theta }_2^{\ast }$. MAML optimizes for post-step performance, so it prefers a good starting point for adaptation over a compromise.

Worked example: the meta-gradient on a scalar task

Reduce MAML to one dimension to see the meta-gradient's second-order term explicitly. Let a task have loss $J(\theta )$ (here we descend a loss for clarity), inner step $\alpha$, and one inner step ${\theta }^{\prime }=\theta -\alpha J^{\prime }(\theta )$. The outer objective is $J({\theta }^{\prime })$, and by the chain rule its gradient with respect to the initialization is

The factor $(1-\alpha J^{\prime \prime }(\theta ))$ is the meta-gradient's signature: it carries the curvature $J^{\prime \prime }$ of the inner loss. Where the loss is sharply curved (large $J^{\prime \prime }$), the inner step overshoots or undershoots, and the meta-gradient discounts that direction; where the loss is gently curved, the inner step is reliable and passes through at nearly full weight. Plug numbers in: with $\alpha =0.1$ and a curvature $J^{\prime \prime }=4$, the factor is $1-0.4=0.6$, so the outer update is scaled to 60% of the naive first-order value in that direction. FOMAML simply sets this factor to $1$ — it drops the $-\alpha J^{\prime \prime }$ term — which is why it is cheaper and why it loses a little: it can no longer tell a well-conditioned adaptation direction from a badly-conditioned one.

Context-based: RL-squared and PEARL

The second family keeps the parameters fixed at meta-test time and instead lets the policy read its recent experience. Adaptation becomes inference, not gradient descent.

RL-squared: recurrence as the learning algorithm

RL-squared (written ${\text{RL}}^2$) makes the observation that a recurrent network run across episodes already implements a learning algorithm — one we can meta-learn.⁴ The key choice is to not reset the hidden state between episodes of the same task. The recurrent policy receives, at each step, not just the observation but the previous action, previous reward, and a done flag; its hidden state $h_t$ therefore accumulates a summary of everything seen so far on this task.

The meta-loss is the total reward across the whole sequence of episodes, so the network is pushed to explore in the early episodes (to identify the task fast) and exploit in the later ones. The recurrent weights become the adaptation algorithm; standard policy gradient trains them. Nothing about the update rule at meta-test time is hand-designed — the RNN's forward pass is the learner, which is why the method is RL, squared.

PEARL: a latent task variable

${\text{RL}}^2$ blends task inference and control into one opaque hidden state. PEARL (Probabilistic Embeddings for Actor-critic RL) separates them.⁵ It posits an explicit latent task variable $z$ and learns an inference network $q_{\varphi }(z\mid c)$ that reads a context $c$ — a small set of recently collected $(s,a,r,s^{\prime })$ transitions — and outputs a posterior over $z$. The policy ${\pi }_{\theta }(a\mid s,z)$ and critic are then conditioned on a sample of $z$. Give the same policy a different $z$ and it behaves as if in a different task.

Two design choices make PEARL efficient. First, the encoder treats the context as an unordered set — it factors the posterior as a product of per-transition Gaussian factors, so it is permutation-invariant and cheap. Second, the inference is decoupled from the control policy, which lets PEARL be off-policy: the actor-critic (built on SAC) reuses stored transitions, while only the fresh context feeds the encoder. That decoupling is why PEARL reached roughly one to two orders of magnitude better sample efficiency than the on-policy meta-methods that came before it, on continuous control task distributions.

Exploration falls out naturally: early in a new task the posterior $q_{\varphi }(z\mid c)$ is broad, so sampling $z$ from it produces varied, hypothesis-testing behavior — a form of posterior sampling for task identity, exactly the Thompson-sampling idea from the bandit lifted to whole MDPs. As context accumulates, the posterior sharpens and behavior commits.

Worked example: the posterior collapsing on a task

Watch the latent posterior sharpen on a toy task distribution where each task is a 2-D navigation problem with a hidden goal, and the latent $z$ encodes the goal location. Before any experience, the posterior is the prior — say a broad Gaussian $\mathcal{N}(0,{\sigma }^{2}I)$ with $\sigma =1.0$ over goal position. The agent samples $z\sim \mathcal{N}(0,1)$, which corresponds to a guess (say, the goal is to the northeast), and it walks that way. It collects a handful of transitions; the reward it sees (getting warmer, getting colder) is context $c$, and the encoder $q_{\varphi }(z\mid c)$ tightens the posterior to, say, $\mathcal{N}(\mu =(0.6,0.4),\sigma =0.4)$ — still uncertain but now leaning toward one quadrant. A second batch of context tightens it further to $\sigma =0.15$, at which point sampled $z$'s barely differ and the policy commits to the inferred goal. The variance $\sigma$ falling from $1.0$ to $0.4$ to $0.15$ is the exploration schedule: PEARL never hand-codes an $ϵ$ that decays; the shrinking posterior does the decaying automatically, and it decays fast precisely when the context is informative.

Exploration for adaptation

Adaptation makes the exploration problem explicit. To adapt to a task you must first identify it, and identifying it may require actions that pay nothing immediately — probing a lever, walking to a landmark — purely to reduce uncertainty about which task you are in. This is a strictly harder problem than the exploration in single-task deep RL, because the informative action and the rewarding action can be entirely different.

The meta-training objective handles this automatically when it scores the whole adaptation sequence, not each episode alone. Because ${\text{RL}}^2$ and PEARL are both graded on cumulative return over the episodes spent on a task, they are rewarded for wasting early actions on identification if that pays off later — the meta-learner discovers an exploration strategy specialized to the task distribution, rather than using a generic one. This is the sense in which meta-RL learns how to explore.

Generalization: closing the train/test gap

Meta-learning achieves competence after a few adaptation episodes. Generalization requires it with zero — a single fixed policy that works on unseen tasks. The obstacle, seen already with ProcGen, is that deep RL overfits its training environments. The fixes modify the training distribution or the network, and they are largely orthogonal, so they compose.

Domain randomization

If the test environment differs from training in physics, textures, or lighting, randomize those factors during training so the policy sees a broad range and cannot lean on any single setting. Trained across enough randomized variants, the real environment looks like just one more sample, and the policy transfers to it zero-shot. This is how a robot control policy trained entirely in simulation — with randomized friction, masses, and visual appearance — can be deployed on physical hardware it never touched during training.⁶ The bet is that if the real world lies inside the span of the randomization, the policy already covers it.

Procedural generation

Domain randomization perturbs a fixed environment; procedural generation builds each environment from scratch by an algorithm, producing an effectively unbounded supply of distinct-but-related tasks. Benchmarks like ProcGen and CoinRun generate a new level layout every episode. The lesson from them is quantitative and blunt: generalization improves smoothly with the number of distinct training levels. Train on a hundred and the agent memorizes; train on tens of thousands and it must learn the shared mechanics, because memorizing that many is not an option.¹ More diversity in the training tasks is the single most reliable lever on the gap.

Data augmentation

Borrowed from supervised vision, data augmentation applies label-preserving transforms — random crops, color jitter, cutout — to observations during training. In RL a subtlety appears: the augmentation must not change the optimal action, and the value estimate must stay consistent across augmented views of the same state. Methods that respect this (RAD, which simply augments, and DrQ, which additionally averages the Q-target over augmentations) sharply improved sample efficiency and generalization on pixel-based control from a small pool of environments, sometimes matching methods that used far more data.⁷ Augmentation creates task diversity from the observations you already have.

Network regularization

The last handle is the network. A high-capacity policy net has ample room to memorize levels, so the standard regularizers apply: L2 weight decay, dropout, and batch normalization all measurably shrink the generalization gap on ProcGen, and combining them helps more.¹ The counterintuitive part is that regularization matters more in RL than in supervised learning, because the agent generates its own training data and can drive itself into a narrow, self-reinforcing slice of the state space — overfitting to a distribution it created.

Zero-shot versus few-shot transfer

These two goals sit on a spectrum.

Domain randomization and procedural generation aim for the left column: no adaptation, just enough training breadth that the test task is already inside the distribution. Meta-RL occupies the right: a small adaptation budget the agent spends to identify which task it faces. Real systems mix them — a broadly randomized policy that also adapts online is strictly harder to break.

In-context RL and vision-language-action agents

The methods above were the state of the art around 2019. Three developments since then reshaped what meta-RL and generalization mean in practice.

In-context RL at scale (emergent adaptation). RL² showed a recurrent network can implement a learned adaptation algorithm; the question left open was how far this scales. AdA (Adaptive Agent) answered it by meta-training a large transformer agent across a vast, procedurally generated task space and showing it adapts to genuinely held-out tasks in a handful of trials — human-timescale in-context adaptation — with the adaptation done entirely by the forward pass over the context, no weight updates.⁸ The lesson is that RL²'s mechanism does not saturate: scale the task diversity and the model, and the in-context learner keeps improving, the same scaling story that drove language models.

Meta-RL as what already happens inside language models. DPO and RLHF aside, the sharpest modern statement of RL² is that a large pretrained sequence model, given a few examples in its prompt, adapts to a new task by a forward pass — which is exactly context-based meta-learning, arrived at from the language side rather than the control side. The few-shot learning of a large language model and the in-context adaptation of RL² are the same computational object: a fixed network whose activations carry a learned learning algorithm. Meta-RL's central bet — that adaptation can be learned into weights and executed by inference — turned out to be how the largest models adapt at all.

Vision-language-action models (generalist robot policies). The generalist bet reached physical robots through RT-2, which fine-tunes a large vision-language model on robot trajectories so that the same network that answers questions about images also emits robot actions as output tokens.⁹ Because it inherits the web-scale semantics of its vision-language backbone, RT-2 generalizes to objects and instructions absent from the robot data — pick up the extinct animal and it reaches for the toy dinosaur — a kind of zero-shot semantic transfer no robot-data-only policy achieves. The Open X-Embodiment effort then pooled robot data across dozens of labs and platforms to train cross-embodiment policies, pushing the same generalist recipe toward a single policy that spans many robots.¹⁰ These are the endpoint of the lesson's progression: one model spanning many tasks rather than a policy per task.

Toward generalist agents

Pushed far enough, generalization blurs the distinction between a task and the world. If a single network can be trained on a wide enough distribution of tasks, why not train it on all of them at once and let the same weights serve every task, selected by the prompt or context? This is the foundation-model bet applied to control.

The bridge is the sequence-model view from offline RL. Decision Transformer already recast control as autoregressive sequence modeling over $(\text{return},s,a)$ tokens — no value function, no bootstrapping, just a Transformer predicting the next action conditioned on a desired return.¹¹ Once behavior is a sequence-modeling problem, the machinery of large language models transfers directly: scale the model, scale the data, and condition on a task specification rather than training a fresh policy each time.

Gato is the concrete instance: one Transformer, one set of weights, trained on hundreds of tasks spanning Atari, robotic manipulation, image captioning, and dialog, with the task implied by the tokens in context.¹² It is not the best agent at any single task, and that is the point — it is a demonstration that a single generalist policy can span radically different domains, the opposite of the one-maze specialist we started with. The through-line of the whole lesson is that same shift in the object of study: from a policy that masters one task to a learner (or a model) that spans a distribution of them. Mastering the maze was never the goal; generalizing across mazes is.