---
title: "Meta-Learning and Few-Shot Learning"
module: Practical Deep Learning
moduleNumber: 9
lessonNumber: 7
order: 907
summary: >
  A deep network trained on one example per class overfits. Meta-learning
  targets this few-shot regime by training across a distribution of
  tasks so that a new task is learnable from a handful of examples. We formalize
  the $N$-way $K$-shot episode, then derive the two dominant families: metric
  methods that learn an embedding where distance classifies (Prototypical
  Networks), and optimization methods that learn an initialization a few gradient
  steps can adapt (MAML). We close on the link to transfer learning and to the
  in-context few-shot behavior of large language models.
topics: [Practical Deep Learning]
sources:
  - book: Goodfellow
    ref: "§5.6 inductive bias, §8 gradient optimization, §15 representation learning and transfer"
  - book: Chollet
    ref: "§5.3 transfer, §6.1 embeddings"
  - book: Stevens
    ref: "§5.5 the training loop"
---

A supervised learner is trained on one task and tested on held-out examples of
that _same_ task. **Meta-learning** changes the unit of experience: the learner is
trained on a distribution over _tasks_ and tested on its ability to acquire a
_new_ task from a few examples. This is "learning to learn", and it is the formal
account of why a person who has seen a thousand object categories needs only one
photograph to learn the thousand-and-first.[^gf-l2l] The right inductive bias, tuned
across many tasks, is what makes one example enough.[^gf-bias] The skill being optimized is
not a classifier but the _procedure_ that produces one.

This lesson connects directly to
[transfer learning](/deep-learning/practical/transfer-learning):
both reuse experience across tasks. Transfer reuses a _representation_; meta-learning
reuses a _learning algorithm_, and the line between them blurs at the modern end,
where a frozen [large language model](/deep-learning/large-models-and-agents/large-language-models)
performs new tasks from a few in-context examples with no gradient step at all.

## The few-shot problem

Fix a space of tasks. Each task is a small classification problem drawn from a
common distribution; the learner never sees enough examples of any one task to
solve it in isolation, but it sees _many_ tasks and must extract what they share.

> **Definition (Task and task distribution).** A task $\mathcal{T} = (\mathcal{D}, \ell)$
> is a dataset $\mathcal{D}$ with a loss $\ell$. Tasks are drawn from a distribution
> $p(\mathcal{T})$ (e.g. "classify $5$ animal species sampled from a large pool").
> Scoring adaptation on _held-out_ query examples rather than the support it fit is a
regularization choice: it targets generalization of the learned procedure, not
memorization of the shots.[^gf-reg] Meta-learning optimizes expected post-adaptation performance
> $\mathbb{E}_{\mathcal{T}\sim p(\mathcal{T})}\!\brackets{\,\ell_{\mathcal{T}}\parens{A(\mathcal{D}^{\text{spt}}_{\mathcal{T}})}\,}$,
> where $A$ is the learning procedure and $\mathcal{D}^{\text{spt}}$ is the task's
> few labeled examples.

The training signal arrives in **episodes**. Each episode samples a task, splits its
data into a tiny labeled **support set** the learner adapts on and a **query set** it
is scored on, and the meta-objective rewards low query loss _after_ adaptation. The
episode is the atom of meta-learning, and its shape is named by two numbers.

> **Definition ($N$-way $K$-shot episode).** Sample $N$ classes and, for each, $K$
> labeled support examples and $Q$ query examples. The **support set**
> $\mathcal{S} = \braces{(x_i, y_i)}_{i=1}^{NK}$ is all the supervision the learner
> gets; the **query set** $\mathcal{Q} = \braces{(x_j^\ast, y_j^\ast)}_{j=1}^{NQ}$
> measures whether it generalized. Typical regimes are $5$-way $1$-shot and
> $5$-way $5$-shot.

For example, take a $5$-way $1$-shot episode with $Q = 15$ queries
per class and a backbone that embeds each image to $d = 64$ dimensions. The support
holds $NK = 5\cdot 1 = 5$ labeled images; the query holds $NQ = 5\cdot 15 = 75$. After
embedding, the support is a matrix of shape $5\times 64$ and the query a matrix of
shape $75\times 64$. A prototypical classifier reduces the support to $N = 5$
prototypes ($5\times 64$), then computes a $75\times 5$ distance matrix, one distance
from each query to each prototype, and takes a row-wise softmax to get $75\times 5$
class probabilities. The entire episode is a handful of matrix operations; its cost
is dominated by the $80$ forward passes through the backbone.

The classes used to build meta-training episodes are disjoint from those used at
meta-test time: the model must classify _categories it never saw_, judged only on
its ability to learn them from $K$ shots.

> **Definition (Meta-train / meta-test split).** The class pool is partitioned into
> $\mathcal{C}_{\text{train}}$, $\mathcal{C}_{\text{val}}$, $\mathcal{C}_{\text{test}}$
> with $\mathcal{C}_{\text{train}} \cap \mathcal{C}_{\text{test}} = \varnothing$.
> Meta-training draws episodes only from $\mathcal{C}_{\text{train}}$; meta-testing
> draws _new_ episodes from $\mathcal{C}_{\text{test}}$. Generalization is measured
> across tasks, not across examples of one task.

$$
% caption: A 3-way 2-shot episode. The support set (left) labels two examples per
% class; the query (right) is classified by the procedure adapted on the support.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  ex/.style={draw, black, minimum size=6mm, inner sep=0pt},
  cls/.style={font=\scriptsize, anchor=east}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % ---- support set ----
  \node[font=\small] at (1.3,2.9) {support set ($K=2$)};
  \foreach \r/\col/\lbl in {0/acc/{class 1}, 1/green/{class 2}, 2/red/{class 3}} {
    \node[cls, text=\col] at (-0.35, -\r*0.95+1.9) {\lbl};
    \foreach \c in {0,1} {
      \node[ex, draw=\col, very thick, fill=\col!15] at (\c*0.8+0.2, -\r*0.95+1.9) {};
    }
  }
  % ---- adapted procedure box ----
  \node[draw=black, very thick, minimum width=15mm, minimum height=26mm, align=center]
    (proc) at (4.4,0.0) {learn\\from\\support};
  \draw[->, black, thick] (1.5,0.0) -- (proc.west);
  % ---- query set ----
  \node[font=\small] at (8.0,2.9) {query set};
  \foreach \r in {0,1,2} {
    \node[ex] (q\r) at (7.7, -\r*0.95+1.9) {?};
  }
  \draw[->, acc, thick] (proc.east) -- (6.9,0.95) node[midway,above,font=\scriptsize,text=acc]{predict};
  \draw[->, acc, thick] (proc.east) -- (6.9,0.0);
  \draw[->, acc, thick] (proc.east) -- (6.9,-0.95);
  \node[font=\scriptsize, anchor=west] at (8.2,0.0) {assign one of 3 classes};
\end{tikzpicture}
$$

Two strategies dominate, and they differ in _what_ is meta-learned. **Metric-based**
methods meta-learn an embedding so that a fixed, parameter-free rule (nearest
neighbor) classifies; **optimization-based** methods meta-learn an initialization so
that a few steps of ordinary gradient descent adapt. A third **model-based** family
folds the whole learning procedure into a network's forward pass.

| Family | What is meta-learned | Adaptation at test time | Cost |
| --- | --- | --- | --- |
| metric-based | an embedding $f_\theta$ | compute support statistics, compare distances | cheap; one forward pass |
| optimization-based | an initialization $\theta$ | run $1$–$5$ gradient steps on the support | costly; backprop-through-backprop |
| model-based | a network that ingests $(\mathcal{S}, x^\ast)$ | a single conditioned forward pass | cheap forward; hard to train |

$$
% caption: The three meta-learning families, split by what is meta-learned and
% whether a gradient step runs at adaptation time.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  root/.style={draw=black, very thick, minimum width=30mm, minimum height=8mm, align=center, font=\scriptsize},
  fam/.style={draw, thick, minimum width=26mm, minimum height=11mm, align=center, font=\scriptsize},
  leaf/.style={font=\scriptsize, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \node[root] (top) at (0,2.4) {meta-learning\\across tasks};
  \node[fam, draw=acc] (metric) at (-4.2,0.4) {metric-based};
  \node[fam, draw=green] (optim) at (0,0.4) {optimization-based};
  \node[fam, draw=red] (model) at (4.2,0.4) {\texttt{model-based}};
  \draw[->, acc, thick] (top) -- (metric);
  \draw[->, green, thick] (top) -- (optim);
  \draw[->, red, thick] (top) -- (model);
  \node[leaf, text=acc] (m1) at (-4.2,-1.4) {\texttt{Siamese, Matching,}\\\texttt{Prototypical}\\\texttt{(learn an embedding)}};
  \node[leaf, text=green] (o1) at (0,-1.4) {\texttt{MAML, FOMAML,}\\\texttt{Reptile}\\\texttt{(learn an init)}};
  \node[leaf, text=red] (b1) at (4.2,-1.4) {\texttt{memory-augmented}\\\texttt{nets}\\\texttt{(learn a forward pass)}};
  \draw[->, acc, thick] (metric) -- (m1);
  \draw[->, green, thick] (optim) -- (o1);
  \draw[->, red, thick] (model) -- (b1);
\end{tikzpicture}
$$

## Metric-based meta-learning

The idea is to learn an embedding $f_\theta : \mathcal{X} \to \mathbb{R}^d$ such
that examples of the same class land close together and different classes land far
apart. Then a new task needs no training, only its support embeddings and a distance.
This rests on distributed representations placing similar inputs near one another, so
that geometric proximity in feature space carries class information.[^gf-sim][^ch-embed]

### Siamese and matching networks

The earliest version trains a network to decide whether _two_ images share a class.
A **Siamese network** embeds both inputs with the same weights and scores their
agreement, turning classification into a learned similarity.

> **Definition (Siamese network).** A pair of weight-tied encoders $f_\theta$ maps
> two inputs to embeddings $f_\theta(x_1), f_\theta(x_2)$; a head predicts the
> probability they belong to the same class, e.g.
> $p(\text{same}) = \sigma\parens{w^\top \abs{f_\theta(x_1) - f_\theta(x_2)} + b}$.
> At test time a query is labeled by its most similar support example.

**Matching Networks** generalize the nearest-neighbor rule into a differentiable,
attention-weighted vote over the _entire_ support set, so the whole episode is
trained end-to-end against the query loss.

> **Definition (Matching Networks).** Predict the query label as an
> attention-weighted combination of support labels,
> $$
> \hat y = \sum_{i=1}^{NK} a\parens{x^\ast, x_i}\, y_i,
> \qquad
> a\parens{x^\ast, x_i} = \frac{\exp\parens{\cos\!\parens{f(x^\ast), g(x_i)}}}{\sum_{k} \exp\parens{\cos\!\parens{f(x^\ast), g(x_k)}}},
> $$
> where $a$ is a softmax over cosine similarities (optionally with a context
> encoder over $\mathcal{S}$) and $y_i$ are one-hot support labels.

### Prototypical networks

Matching Networks compare a query to every support point. **Prototypical Networks**
compress each class to a single point, its **prototype**, the mean of its support
embeddings, and classify a query by the nearest prototype. The model is a
parameter-free softmax over negative squared distances.

> **Definition (Class prototype).** For class $c$ with support examples
> $\mathcal{S}_c$, the prototype is the mean embedding
> $$
> \mathbf{c}_c = \frac{1}{\abs{\mathcal{S}_c}} \sum_{(x_i, y_i)\in \mathcal{S}_c} f_\theta(x_i)
> \;\in\; \mathbb{R}^d.
> $$

> **Definition (Prototypical classifier).** A query $x^\ast$ is assigned class
> probabilities by a softmax over negative squared Euclidean distances to the
> prototypes,
> $$
> p_\theta\parens{y = c \mid x^\ast}
> = \frac{\exp\parens{-\norm{f_\theta(x^\ast) - \mathbf{c}_c}^2}}{\sum_{c'=1}^{N}\exp\parens{-\norm{f_\theta(x^\ast) - \mathbf{c}_{c'}}^2}}.
> $$

$$
% caption: The prototypical-network pipeline for a 3-way 2-shot episode. The shared
% embedding $f_\theta$ maps every support image to $\mathbb{R}^d$; each class prototype
% is the mean of its two support embeddings; the query is embedded and assigned the
% class of the nearest prototype under squared Euclidean distance.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  img/.style={draw, black, minimum size=5mm, inner sep=0pt},
  enc/.style={draw=black, very thick, fill=black!8, minimum width=8mm, minimum height=20mm, align=center, font=\scriptsize},
  proto/.style={very thick, minimum size=4.5mm, inner sep=0pt},
  lbl/.style={font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % ---- support images, 3 classes x 2 shots ----
  \node[lbl] at (0.6,3.1) {\texttt{support 3x2}};
  \foreach \r/\col in {0/acc, 1/green, 2/red} {
    \foreach \c in {0,1} {
      \node[img, draw=\col, very thick, fill=\col!15] at (\c*0.65+0.15, -\r*0.9+2.2) {};
    }
  }
  % ---- shared encoder ----
  \node[enc] (enc) at (2.7,1.3) {\texttt{f}\\shared\\encoder};
  \draw[->, black, thick] (1.1,2.2) -- (2.3,2.2);
  \draw[->, black, thick] (1.1,1.3) -- (enc.west);
  \draw[->, black, thick] (1.1,0.4) -- (2.3,0.4);
  % ---- embeddings + prototypes column ----
  \node[lbl] at (5.6,3.1) {\texttt{prototypes (dim d)}};
  \node[proto, draw=acc, fill=acc!15] (pA) at (5.6,2.2) {};
  \node[lbl, text=acc, anchor=west] at (5.9,2.2) {mean class 1};
  \node[proto, draw=green, fill=green!15] (pB) at (5.6,1.3) {};
  \node[lbl, text=green, anchor=west] at (5.9,1.3) {mean class 2};
  \node[proto, draw=red, fill=red!15] (pC) at (5.6,0.4) {};
  \node[lbl, text=red, anchor=west] at (5.9,0.4) {mean class 3};
  \draw[->, black, thick] (3.1,2.2) -- (pA.west);
  \draw[->, black, thick] (enc.east) -- (pB.west);
  \draw[->, black, thick] (3.1,0.4) -- (pC.west);
  % ---- query path ----
  \node[img, draw=black, very thick] (qimg) at (0.5,-1.1) {?};
  \node[lbl, anchor=east] at (0.15,-1.1) {query};
  \node[enc, minimum height=8mm] (enc2) at (2.7,-1.1) {\texttt{f}};
  \draw[->, black, thick] (qimg.east) -- (enc2.west);
  \node[proto, draw=black, very thick] (qe) at (4.5,-1.1) {};
  \node[lbl, anchor=west] at (4.85,-1.1) {\texttt{query embedding}};
  \draw[->, black, thick] (enc2.east) -- (qe.west);
  % nearest prototype = class 2 (green); connectors fan out from the lower-left so
  % they clear the stacked prototype boxes instead of piercing them
  \draw[->, green, very thick] (qe) -- (pB.south) node[midway, left=1mm, font=\scriptsize, text=green] {nearest};
  \draw[black, thick, dashed] (qe) -- (pA.south);
  \draw[black, thick, dashed] (qe) -- (pC.south west);
\end{tikzpicture}
$$

For example, take $N = 3$ classes and suppose a query
embeds to $f_\theta(x^\ast)$ at squared distances $d_1 = 1$, $d_2 = 4$, $d_3 = 9$ from
the three prototypes. The logits are the negatives $-1, -4, -9$; exponentiating gives
$e^{-1} \approx 0.368$, $e^{-4} \approx 0.0183$, $e^{-9} \approx 0.000123$, which sum
to $\approx 0.386$. The probabilities are $0.953$, $0.047$, $0.0003$: the nearest
prototype dominates, and the softmax is sharp because distances enter the exponent
directly. Doubling the embedding scale (multiplying $f_\theta$ by $2$) quadruples every
distance, which sharpens the softmax further, so the embedding's overall scale acts as
an implicit temperature the network is free to learn.

Geometrically, the prototypes act as reference points, and the softmax over
negative distances partitions the embedding space into Voronoi-like cells that
are the class regions.

$$
% caption: Prototypical networks. Each class prototype (square) is the mean of its
% support embeddings (dots); a query is assigned the nearest prototype's class.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % ---- class A (acc) support cluster, lower-left ----
  \foreach \x/\y in {-2.5/-0.6, -2.0/-1.2, -2.9/-1.1, -2.3/-0.2} \fill[acc] (\x,\y) circle (2pt);
  \node[draw=acc, very thick, fill=acc!15, minimum size=4.5mm, inner sep=0pt] (pA) at (-2.42,-0.78) {};
  \node[text=acc, font=\footnotesize, anchor=north] at (-2.42,-1.6) {\texttt{prototype A}};
  % ---- class B (green) support cluster, upper-middle ----
  \foreach \x/\y in {0.2/1.6, -0.4/1.9, 0.6/2.1, 0.0/1.2} \fill[green] (\x,\y) circle (2pt);
  \node[draw=green, very thick, fill=green!15, minimum size=4.5mm, inner sep=0pt] (pB) at (0.1,1.7) {};
  \node[text=green, font=\footnotesize, anchor=south] at (0.1,2.4) {\texttt{prototype B}};
  % ---- class C (red) support cluster, lower-right ----
  \foreach \x/\y in {2.6/-0.7, 3.1/-1.2, 2.3/-1.3, 2.9/-0.3} \fill[red] (\x,\y) circle (2pt);
  \node[draw=red, very thick, fill=red!15, minimum size=4.5mm, inner sep=0pt] (pC) at (2.72,-0.87) {};
  \node[text=red, font=\footnotesize, anchor=north] at (2.72,-1.7) {\texttt{prototype C}};
  % ---- query point, near B ----
  \node[draw=black, very thick, minimum size=3.5mm, inner sep=0pt] (q) at (0.6,0.4) {};
  \node[font=\scriptsize, anchor=west] at (0.85,0.4) {query};
  % distance lines to each prototype (nearest = green)
  \draw[green, thick] (q) -- (pB) node[midway, right, font=\scriptsize, text=green] {min};
  \draw[black, thick, dashed] (q) -- (pA);
  \draw[black, thick, dashed] (q) -- (pC);
\end{tikzpicture}
$$

Training minimizes the negative log-probability of the true class on the query set,
averaged over episodes. Because every operation (embed, average, distance, softmax)
is differentiable, gradients flow back into $f_\theta$.

```algorithm
caption: $\textsc{ProtoNetEpisode}(\mathcal{S}, \mathcal{Q}, f_\theta)$ — one meta-training episode
$L \gets 0$
for $c \gets 1$ to $N$ do
  $\mathbf{c}_c \gets \frac{1}{K}\sum_{(x_i, y_i)\in \mathcal{S}_c} f_\theta(x_i)$ // class prototype = mean support embedding
for each $(x^\ast, y^\ast)$ in $\mathcal{Q}$ do
  for $c \gets 1$ to $N$ do
    $d_c \gets \norm{f_\theta(x^\ast) - \mathbf{c}_c}^2$ // squared distance to each prototype
  $L \gets L + d_{y^\ast} + \log \sum_{c} \exp\parens{-d_c}$ // negative log-softmax of true class
update $\theta$ by gradient descent on $L / \abs{\mathcal{Q}}$ // backprop into the embedding
return $\theta$
```

> **Theorem (Prototypical networks are a linear classifier in embedding space).**
> With squared Euclidean distance, the prototype logits are an affine function of
> the embedding: $-\norm{f_\theta(x^\ast) - \mathbf{c}_c}^2 = w_c^\top f_\theta(x^\ast) + b_c$
> with $w_c = 2\,\mathbf{c}_c$ and $b_c = -\norm{\mathbf{c}_c}^2$.

> **Proof.** Expand the square:
> $-\norm{f(x^\ast) - \mathbf{c}_c}^2 = -\norm{f(x^\ast)}^2 + 2\,\mathbf{c}_c^\top f(x^\ast) - \norm{\mathbf{c}_c}^2$.
> The term $-\norm{f(x^\ast)}^2$ is shared by all classes, so it cancels in the
> softmax. What remains, $2\,\mathbf{c}_c^\top f(x^\ast) - \norm{\mathbf{c}_c}^2$, is
> affine in $f(x^\ast)$ with weight $w_c = 2\,\mathbf{c}_c$ and bias
> $b_c = -\norm{\mathbf{c}_c}^2$. $\qed$

The theorem explains why squared Euclidean distance is the right choice and cosine
distance underperforms: it makes the classifier a _linear_ readout whose weights are
the prototypes, recovering a Bregman-divergence mean-classifier rather than an
arbitrary metric.[^gf-sim]

## Optimization-based meta-learning

Metric methods bake in a fixed classification rule. Optimization-based methods keep
ordinary gradient descent as the adaptation rule and instead meta-learn the _starting
point_. The adaptation itself is nothing exotic: it is the same gradient step
$\theta \gets \theta - \alpha\nabla_\theta\mathcal{L}$ that trains any network, run for
a few iterations on the support set.[^gf-opt][^st-train] The goal is an initialization
$\theta$ that sits a few gradient steps away from a good solution for _any_ task in
$p(\mathcal{T})$.

### MAML and the bi-level objective

**Model-Agnostic Meta-Learning (MAML)** is model-agnostic because it adds nothing to
the architecture: it only changes how the initialization is trained. The
**inner loop** adapts $\theta$ to a task with one (or a few) gradient steps on its
support loss; the **outer loop** updates $\theta$ so that this adaptation generalizes,
scored on the query loss _after_ adaptation.

> **Definition (MAML inner loop).** For task $\mathcal{T}_i$, take $G$ gradient steps
> on the support loss starting from $\theta$. With one step,
> $$
> \theta_i' = \theta - \alpha\,\nabla_\theta\, \mathcal{L}_{\mathcal{T}_i}^{\text{spt}}(\theta),
> $$
> producing a task-adapted parameter $\theta_i'$. The step size $\alpha$ is the
> inner learning rate (it may itself be learned).

> **Definition (MAML outer objective).** Meta-train $\theta$ to minimize the sum of
> _post-adaptation_ query losses across a batch of tasks,
> $$
> \min_\theta \; \sum_{\mathcal{T}_i \sim p(\mathcal{T})} \mathcal{L}_{\mathcal{T}_i}^{\text{qry}}\parens{\theta_i'}
> = \min_\theta \; \sum_i \mathcal{L}_{\mathcal{T}_i}^{\text{qry}}\parens{\theta - \alpha\,\nabla_\theta \mathcal{L}_{\mathcal{T}_i}^{\text{spt}}(\theta)}.
> $$

The objective is **bi-level**: the outer minimization over $\theta$ contains, inside
its loss, the inner minimization that produced $\theta_i'$. Differentiating it is the
crux. By the chain rule, the meta-gradient of one task's query loss is

$$
\nabla_\theta\, \mathcal{L}^{\text{qry}}_{\mathcal{T}_i}(\theta_i')
= \underbrace{\frac{\partial \theta_i'}{\partial \theta}}_{\text{inner Jacobian}}^{\!\!\top}
\nabla_{\theta_i'}\, \mathcal{L}^{\text{qry}}_{\mathcal{T}_i}(\theta_i').
$$

Differentiating the one-step update $\theta_i' = \theta - \alpha\,\nabla_\theta \mathcal{L}^{\text{spt}}_{\mathcal{T}_i}(\theta)$
with respect to $\theta$ gives the inner Jacobian

$$
\frac{\partial \theta_i'}{\partial \theta}
= I - \alpha\,\nabla^2_\theta\, \mathcal{L}^{\text{spt}}_{\mathcal{T}_i}(\theta),
$$

so the exact meta-gradient carries a **Hessian** of the support loss:

$$
\nabla_\theta\, \mathcal{L}^{\text{qry}}_{\mathcal{T}_i}(\theta_i')
= \parens{I - \alpha\,\nabla^2_\theta\, \mathcal{L}^{\text{spt}}_{\mathcal{T}_i}(\theta)}\,
\nabla_{\theta_i'}\, \mathcal{L}^{\text{qry}}_{\mathcal{T}_i}(\theta_i').
$$

This is the **second-order** MAML update: backpropagating through the inner gradient
step requires a Hessian-vector product, differentiating the gradient itself.

The dimensions clarify why this is expensive yet the Hessian is never materialized. Let $\theta \in \mathbb{R}^P$
with $P$ the parameter count. The query gradient $\nabla_{\theta_i'}\mathcal{L}^{\text{qry}}$ is a
vector in $\mathbb{R}^P$; the inner Jacobian $\partial\theta_i'/\partial\theta$ and the Hessian
$\nabla^2_\theta \mathcal{L}^{\text{spt}}$ are $P\times P$ matrices. Forming a $P\times P$
Hessian is impossible for a real network ($P$ is in the millions), but the meta-gradient only ever
multiplies it _by a vector_:
$$
\parens{I - \alpha\,\nabla^2_\theta \mathcal{L}^{\text{spt}}}\,v
= v - \alpha\,\nabla^2_\theta \mathcal{L}^{\text{spt}}\,v,
\qquad v = \nabla_{\theta_i'}\mathcal{L}^{\text{qry}} \in \mathbb{R}^P.
$$
The Hessian-vector product $\nabla^2_\theta \mathcal{L}^{\text{spt}}\,v$ is one extra backward pass
(reverse-mode over the inner gradient), so the exact meta-gradient costs roughly _two_ backward
passes per task, not the $P^2$ storage a full Hessian would need. For $G$ inner steps, the chain
rule nests $G$ such Jacobians, and the cost grows linearly in $G$.

$$
% caption: MAML as a computation graph for one task. Forward (top): the meta-parameter
% $\theta$ takes an inner gradient step on the support loss to the adapted $\theta_i'$,
% which is scored on the query loss. Backward (bottom): the meta-gradient flows through
% $\theta_i'$ into $\theta$, passing the inner Jacobian $I - \alpha\nabla^2\mathcal{L}^{\text{spt}}$.
% FOMAML cuts the dashed second-order edge, treating $\theta_i'$ as constant in $\theta$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  var/.style={draw=black, very thick, minimum width=14mm, minimum height=8mm, align=center, font=\scriptsize},
  loss/.style={draw, very thick, minimum width=16mm, minimum height=8mm, align=center, font=\scriptsize},
  lbl/.style={font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % forward row
  \node[var, fill=black!8] (th) at (0,1.3) {\texttt{theta}\\meta-init};
  \node[loss, draw=acc, fill=acc!12] (spt) at (3.3,1.3) {\texttt{support}\\loss};
  \node[var, draw=acc, fill=acc!12] (thp) at (6.6,1.3) {\texttt{theta'}\\adapted};
  \node[loss, draw=red, fill=red!12] (qry) at (9.6,1.3) {query\\loss};
  \draw[->, black, thick] (th) -- (spt);
  \draw[->, acc, thick] (spt) -- (thp) node[midway, above, lbl, text=acc] {inner step};
  \draw[->, red, thick] (thp) -- (qry) node[midway, above, lbl, text=red] {score};
  % meta-gradient backward row
  \draw[->, red, very thick] (qry.south) to[bend left=22] node[midway, below, lbl, text=red] {grad query} (thp.south);
  \draw[->, acc, very thick] (thp.south) to[bend left=22] node[midway, below, lbl, text=acc] {inner Jacobian} (th.south);
  \node[lbl, text=black, align=center] at (3.3,-1.15) {dashed edge = 2nd-order (Hessian);\\FOMAML drops it};
  \draw[acc, very thick, dashed] (5.1,-0.65) -- (4.3,-0.65);
\end{tikzpicture}
$$

> **Definition (First-order MAML, FOMAML).** Drop the second-order term by treating
> $\theta_i'$ as if it did not depend on $\theta$, i.e. set the inner Jacobian to
> $I$. The meta-gradient becomes the query gradient _evaluated at the adapted
> parameters_, $\nabla_\theta \mathcal{L}^{\text{qry}}_{\mathcal{T}_i}(\theta_i') \approx \nabla_{\theta_i'}\mathcal{L}^{\text{qry}}_{\mathcal{T}_i}(\theta_i')$,
> sidestepping the Hessian.

FOMAML loses little accuracy in practice while removing the cost of the Hessian
term, evidence that most of the meta-signal is in the first-order direction.[^gf-hess]
The two loops are nested: an inner trajectory per task, an outer step that moves the
shared start.

$$
% caption: MAML. From a shared initialization, each task takes inner gradient steps
% to its own optimum; the outer step moves the start to minimize post-step loss.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % meta-init
  \node[circle, draw=black, very thick, fill=black!8, minimum size=5mm, inner sep=0pt] (init) at (0,0) {};
  \node[font=\scriptsize, anchor=east] at (-0.3,0) {init};
  % task 1 optimum (acc) upper-right
  \node[circle, draw=acc, very thick, fill=acc!15, minimum size=4.5mm, inner sep=0pt] (t1) at (3.2,1.7) {};
  \node[text=acc, font=\scriptsize, anchor=west] at (3.4,1.7) {task 1};
  % task 2 optimum (green) right
  \node[circle, draw=green, very thick, fill=green!15, minimum size=4.5mm, inner sep=0pt] (t2) at (3.6,-0.1) {};
  \node[text=green, font=\scriptsize, anchor=west] at (3.8,-0.1) {task 2};
  % task 3 optimum (red) lower-right
  \node[circle, draw=red, very thick, fill=red!15, minimum size=4.5mm, inner sep=0pt] (t3) at (3.2,-1.7) {};
  \node[text=red, font=\scriptsize, anchor=west] at (3.4,-1.7) {task 3};
  % inner-loop adaptation arrows (curved, one per task)
  \draw[->, acc, thick] (init) to[bend left=18] node[midway,above,font=\scriptsize,text=acc]{inner steps} (t1);
  \draw[->, green, thick] (init) to[bend left=4] (t2);
  \draw[->, red, thick] (init) to[bend right=18] (t3);
  % outer-loop step: move the init toward the centroid of tasks
  \draw[->, black, very thick] (init) -- (1.3,0.0);
  \node[font=\scriptsize, anchor=north] at (0.7,-0.35) {outer step};
\end{tikzpicture}
$$

```algorithm
caption: $\textsc{MAML}(p(\mathcal{T}), \alpha, \beta)$ — meta-learn an initialization
initialize meta-parameters $\theta$
while not converged do
  sample a batch of tasks $\mathcal{T}_1, \dots, \mathcal{T}_B \sim p(\mathcal{T})$
  for $i \gets 1$ to $B$ do
    $\theta_i' \gets \theta - \alpha\,\nabla_\theta \mathcal{L}^{\text{spt}}_{\mathcal{T}_i}(\theta)$ // inner loop: adapt on support
  $\theta \gets \theta - \beta\,\nabla_\theta \sum_{i} \mathcal{L}^{\text{qry}}_{\mathcal{T}_i}(\theta_i')$ // outer loop: meta-update on query
return $\theta$
```

### Reptile: first-order without a query split

**Reptile** removes even the support/query distinction. It runs several SGD steps on
a task to reach $\theta_i'$, then moves the initialization _toward_ that point.[^st-train]
The meta-update is a difference of parameter vectors, no second derivative anywhere.

> **Definition (Reptile update).** For task $\mathcal{T}_i$, run $G$ SGD steps from
> $\theta$ to obtain $\theta_i'$, then nudge the initialization toward it,
> $$
> \theta \;\gets\; \theta + \beta\,\parens{\theta_i' - \theta}.
> $$
> With $G = 1$ this reduces to ordinary SGD; with $G > 1$ the expected update points
> along the same Hessian-weighted direction MAML follows, recovering MAML's behavior
> to first order without forming the Hessian.

Reptile works because $\theta_i' - \theta$, averaged over tasks, points toward a
point _near_ each task's manifold of solutions, exactly where a few more steps land
fastest. It is the cheapest of the three and a strong default when compute is tight.

| Method | Meta-gradient | Hessian needed | Query split |
| --- | --- | --- | --- |
| MAML (2nd-order) | $\parens{I - \alpha \nabla^2 \mathcal{L}^{\text{spt}}}\nabla \mathcal{L}^{\text{qry}}$ | yes | yes |
| FOMAML | $\nabla \mathcal{L}^{\text{qry}}(\theta_i')$ | no | yes |
| Reptile | $\theta - \theta_i'$ (direction) | no | no |

A related thread treats the optimizer itself as a learnable network: an LSTM
**meta-learner** that ingests the gradient $\nabla_\theta \mathcal{L}$ at each step
and emits the parameter update, learning a task-specific update rule in its hidden
state rather than a fixed $\alpha$.[^gf-opt]

## Model-based meta-learning

Metric and optimization methods both run an explicit adaptation procedure at test
time. **Model-based** (black-box) methods absorb the procedure into a single network
whose forward pass _reads_ the support set and then classifies the query, so
"learning" is one conditioned inference.

> **Definition (Model-based meta-learner).** A network with parameters $\theta$ and
> an internal state (recurrent hidden state or external memory) consumes the support
> sequence $\mathcal{S}$, updates its state to encode the task, and predicts
> $\hat y = f_\theta(x^\ast \mid \mathcal{S})$ for a query in a single forward pass.
> No gradient step occurs at test time.

The canonical instance is a **memory-augmented neural network**: a controller writes
each support embedding into an external, content-addressable memory, then reads the
slot most similar to the query to recover its label, learning a write-then-retrieve
policy that generalizes to new classes. This family is the conceptual
bridge to in-context learning: a forward pass that adapts because the support is in
its input, not in its weights.

## Connections: transfer, and in-context learning

Meta-learning, transfer learning, and the few-shot behavior of large language models
form a spectrum distinguished by _where_ task-specific information lives and _whether_
a gradient step occurs at adaptation time.

| Paradigm | Cross-task signal reused | Adaptation mechanism | Gradient at test? |
| --- | --- | --- | --- |
| transfer / fine-tuning | a pretrained representation | fine-tune a head on target data | yes |
| metric meta-learning | an embedding metric | compute support statistics | no |
| optimization meta-learning | an initialization | a few inner gradient steps | yes (inner) |
| in-context (LLM) | a sequence model's weights | condition on examples in the prompt | no |

Transfer learning meta-learns implicitly: pretraining on a broad source task yields
features that adapt to many targets, which is the few-shot objective with a single
giant "task": feature extraction reuses the frozen representation, and fine-tuning
takes a few gradient steps on it, exactly the metric and optimization mechanisms
seen above, applied to one source instead of a task distribution.[^ch-transfer] The most recent form is **in-context learning**: a frozen
[large language model](/deep-learning/large-models-and-agents/large-language-models)
shown a few input–output pairs in its prompt performs the new task with no weight
update, the support set living entirely in the context window.

> **Definition (In-context few-shot learning).** A pretrained sequence model
> $p_\theta(y \mid x, \mathcal{S})$ conditions on a prompt containing $K$
> demonstration pairs $\mathcal{S} = \braces{(x_i, y_i)}$ followed by a query $x$,
> and emits $y$ by autoregressive prediction. The weights $\theta$ are fixed; the
> task is specified _entirely_ by the in-context examples.

This is meta-learning whose inner loop is a forward pass. Pretraining on a vast
distribution of implicit tasks (every document is a different prediction problem)
plays the role of the meta-training distribution $p(\mathcal{T})$; the prompt's
demonstrations play the role of the support set; and the model's conditioning on them
is an _implicit_ adaptation algorithm learned by the same gradient descent that
trained the weights. The forward pass has learned to be a learning algorithm.[^gf-l2l]

## Evaluation

Two benchmarks anchor the literature. **Omniglot** is $1623$ handwritten characters
from $50$ alphabets with $20$ examples each, the "transpose of MNIST" (many classes,
few examples), well suited to few-shot evaluation. **miniImageNet** is $100$
ImageNet classes ($64$ train, $16$ validation, $20$ test) at $84\times84$
resolution, the harder natural-image standard. Accuracy is reported on
held-out classes, averaged over hundreds of sampled $N$-way $K$-shot test episodes
with $95\%$ confidence intervals.

> **Definition (Few-shot evaluation protocol).** Sample many test episodes from
> $\mathcal{C}_{\text{test}}$; for each, adapt on the support and score on the
> query; report mean query accuracy and a confidence interval over episodes.
> Comparisons fix $N$, $K$, the backbone, and the episode count, since all four move
> the number.

$$
% caption: Few-shot accuracy rises with shots $K$ and falls with ways $N$. More
% support per class helps; more classes to disambiguate hurts.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, black, thick] (0,0) -- (6.2,0) node[right, font=\scriptsize] {shots $K$};
  \draw[->, black, thick] (0,0) -- (0,3.6) node[above, font=\scriptsize] {accuracy};
  \foreach \k/\x in {1/0.6, 2/1.8, 5/3.6, 10/5.4} \node[font=\scriptsize, below] at (\x,0) {\k};
  \foreach \a/\y in {50/0.6, 70/1.8, 90/3.0} {
    \node[font=\scriptsize, left] at (0,\y) {\a};
    \draw[black] (0,\y) -- (6.0,\y);
  }
  % 5-way curve (acc), higher
  \draw[acc, very thick] (0.6,1.55) .. controls (2.0,2.5) and (4.0,2.95) .. (5.4,3.15);
  \foreach \x/\y in {0.6/1.55, 1.8/2.45, 3.6/2.95, 5.4/3.15} \fill[acc] (\x,\y) circle (2pt);
  \node[text=acc, font=\footnotesize, anchor=west] at (5.5,3.15) {\texttt{5-way}};
  % 20-way curve (red), lower
  \draw[red, very thick] (0.6,0.7) .. controls (2.0,1.4) and (4.0,1.85) .. (5.4,2.05);
  \foreach \x/\y in {0.6/0.7, 1.8/1.3, 3.6/1.8, 5.4/2.05} \fill[red] (\x,\y) circle (2pt);
  \node[text=red, font=\footnotesize, anchor=west] at (5.5,2.05) {\texttt{20-way}};
\end{tikzpicture}
$$

The comparison across the families, on mechanism rather than a leaderboard
number (which shifts with backbone and year):

| Method | Mechanism | What is learned | Adaptation cost |
| --- | --- | --- | --- |
| Siamese | learned pairwise similarity | an embedding + similarity head | one forward pass per pair |
| Matching Networks | attention-weighted support vote | an embedding + context encoder | one forward pass over $\mathcal{S}$ |
| Prototypical Networks | nearest class-mean prototype | an embedding (parameter-free rule) | mean + distances, one pass |
| MAML | few inner gradient steps from a meta-init | an initialization $\theta$ | $G$ steps + 2nd-order backprop |
| Reptile | move init toward post-SGD parameters | an initialization $\theta$ | $G$ SGD steps, first-order only |

## Strong baselines and the in-context turn

Two findings since the classic meta-learning papers reshaped how the field thinks about
few-shot learning — one deflationary, one transformative.

The deflationary result is that **a good representation plus a simple classifier often
beats elaborate meta-learning**. Chen et al. (2019, _ICLR_, "A Closer Look at Few-Shot
Classification") and the "Meta-Baseline" line (Tian et al., 2020, _ECCV_) showed that
plainly pretraining a backbone on all the base classes with ordinary supervised
cross-entropy, then fitting a nearest-centroid or logistic-regression head on the support
set, matches or beats MAML and Prototypical Networks on miniImageNet. The episodic
training machinery this lesson builds turned out to be less important than the quality of
the embedding it produced — an echo of the
[representation-learning](/deep-learning/practical/representation-learning) thesis that a
good code makes the downstream task easy. A related result, **ANIL** (Raghu et al., 2020,
_ICLR_, "Almost No Inner Loop"), showed MAML's inner-loop adaptation barely changes the
feature extractor at all: nearly all the benefit comes from adapting the final layer, so
MAML is mostly learning a reusable representation, not a fast learner.

The transformative result is **in-context learning**. A large
[language model](/deep-learning/large-models-and-agents/large-language-models) (Brown et
al., 2020, _NeurIPS_, "Language Models are Few-Shot Learners") performs a new task from a
handful of examples placed in its prompt, with _no gradient step at all_ — the frozen
forward pass conditioned on the support set _is_ the adaptation. This is precisely the
model-based meta-learning goal from the previous section, achieved as an emergent property
of scale rather than an explicit meta-objective: the $N$-way $K$-shot episode becomes $K$
demonstrations in the context window. The formal account is still being worked out —
evidence suggests the forward pass implements something like implicit gradient descent or
Bayesian inference over the demonstrations — but the practical upshot is settled. For text,
and increasingly for vision and multimodal tasks, few-shot learning is now done by
prompting a foundation model, and the explicit meta-learners of this lesson remain useful
mainly where episodes are cheap and a small specialized model is preferable to a giant
general one.

## Takeaways

- Meta-learning trains across a **distribution of tasks** so a new task is learnable
  from few examples; the unit of experience is the $N$-way $K$-shot **episode** with
  its support and query sets, and meta-train/meta-test classes are **disjoint**.
- **Metric methods** learn an embedding where a fixed distance classifies.
  **Prototypical Networks** use the class-mean prototype and a softmax over negative
  squared distances, which (by the expansion of the square) is a _linear_ classifier
  whose weights are the prototypes.
- **Optimization methods** learn an initialization. MAML's **bi-level** objective
  minimizes post-adaptation query loss; its exact meta-gradient carries a Hessian
  $\parens{I - \alpha\nabla^2 \mathcal{L}^{\text{spt}}}$, which **FOMAML** drops and
  **Reptile** avoids entirely by moving the init toward post-SGD parameters.
- **Model-based** methods fold adaptation into a conditioned forward pass
  (memory-augmented nets), the bridge to in-context learning.
- The paradigm is a spectrum with [transfer learning](/deep-learning/practical/transfer-learning)
  and with **in-context learning** in [large language models](/deep-learning/large-models-and-agents/large-language-models),
  where a frozen model's forward pass, conditioned on a few prompt demonstrations,
  _is_ the learning algorithm meta-learning sought to produce.

[^gf-l2l]: **Goodfellow**, _Deep Learning_, §15.2 — representation learning: features learned on many tasks transfer to new ones, the principle that reusing cross-task experience is a route to data-efficient acquisition of a new task.
[^gf-bias]: **Goodfellow**, _Deep Learning_, §5.6 — capacity, overfitting, and inductive bias: a learner biased toward the right hypothesis class generalizes from fewer examples, which is what meta-learning tunes across tasks.
[^gf-sim]: **Goodfellow**, _Deep Learning_, §15.4 — distributed representations place similar inputs near one another in feature space, the property a learned embedding exploits so that distance carries class information.
[^ch-embed]: **Chollet**, _Deep Learning with Python_, §6.1 — embeddings map inputs to a vector space where geometric proximity reflects semantic similarity; a nearest-neighbour rule in that space is then a classifier.
[^gf-opt]: **Goodfellow**, _Deep Learning_, §8 — gradient-based optimization: a single step $\theta \gets \theta - \alpha\nabla_\theta \mathcal{L}$ decreases the loss locally, the inner-loop update meta-learning differentiates through.
[^gf-hess]: **Goodfellow**, _Deep Learning_, §4.3, §8.6 — second-order information (the Hessian) describes how the gradient changes; differentiating through a gradient step therefore introduces a Hessian-vector product.
[^gf-reg]: **Goodfellow**, _Deep Learning_, §7 — regularization and generalization: controlling the effective complexity of the adaptation is what lets a query loss, not a support loss, be the true objective.
[^st-train]: **Stevens**, _Deep Learning with PyTorch_, §5.5 — a training loop is repeated gradient steps on batches; the inner loop here runs such a loop for a few steps on the support set.
[^ch-transfer]: **Chollet**, _Deep Learning with Python_, §5.3 — feature extraction and fine-tuning reuse a pretrained representation on a small target dataset, the transfer-learning limit of the few-shot objective.