Meta-Learning and Few-Shot Learning

A supervised learner is trained on one task and tested on held-out examples of that same task. Meta-learning changes the unit of experience: the learner is trained on a distribution over tasks and tested on its ability to acquire a new task from a few examples. This is learning to learn, and it is the formal account of why a person who has seen a thousand object categories needs only one photograph to learn the thousand-and-first.¹ The right inductive bias, tuned across many tasks, is what makes one example enough.² The skill being optimized is not a classifier but the procedure that produces one.

This lesson connects directly to transfer learning: both reuse experience across tasks. Transfer reuses a representation; meta-learning reuses a learning algorithm, and the line between them blurs at the modern end, where a frozen large language model performs new tasks from a few in-context examples with no gradient step at all.

The few-shot problem

Fix a space of tasks. Each task is a small classification problem drawn from a common distribution; the learner never sees enough examples of any one task to solve it in isolation, but it sees many tasks and must extract what they share.

regularization choice: it targets generalization of the learned procedure, not memorization of the shots.³ Meta-learning optimizes expected post-adaptation performance

${\mathbb{E}}_{\mathcal{T}\sim p(\mathcal{T})}\left[{\ell }_{\mathcal{T}}\left(A({\mathcal{D}}_{\mathcal{T}}^{\text{spt}})\right)\right]$, where $A$ is the learning procedure and ${\mathcal{D}}^{\text{spt}}$ is the task's few labeled examples.

The training signal arrives in episodes. Each episode samples a task, splits its data into a tiny labeled support set the learner adapts on and a query set it is scored on, and the meta-objective rewards low query loss after adaptation. The episode is the atom of meta-learning, and its shape is named by two numbers.

For example, take a $5$-way $1$-shot episode with $Q=15$ queries per class and a backbone that embeds each image to $d=64$ dimensions. The support holds $NK=5\cdot 1=5$ labeled images; the query holds $NQ=5\cdot 15=75$. After embedding, the support is a matrix of shape $5\times 64$ and the query a matrix of shape $75\times 64$. A prototypical classifier reduces the support to $N=5$ prototypes ($5\times 64$), then computes a $75\times 5$ distance matrix, one distance from each query to each prototype, and takes a row-wise softmax to get $75\times 5$ class probabilities. The entire episode is a handful of matrix operations; its cost is dominated by the $80$ forward passes through the backbone.

The classes used to build meta-training episodes are disjoint from those used at meta-test time: the model must classify categories it never saw, judged only on its ability to learn them from $K$ shots.

Two strategies dominate, and they differ in what is meta-learned. Metric-based methods meta-learn an embedding so that a fixed, parameter-free rule (nearest neighbor) classifies; optimization-based methods meta-learn an initialization so that a few steps of ordinary gradient descent adapt. A third model-based family folds the whole learning procedure into a network's forward pass.

Metric-based meta-learning

The idea is to learn an embedding $f_{\theta }:\mathcal{X}\to {\mathbb{R}}^d$ such that examples of the same class land close together and different classes land far apart. Then a new task needs no training, only its support embeddings and a distance. This rests on distributed representations placing similar inputs near one another, so that geometric proximity in feature space carries class information.⁴⁵

Siamese and matching networks

The earliest version trains a network to decide whether two images share a class. A Siamese network embeds both inputs with the same weights and scores their agreement, turning classification into a learned similarity.

Matching Networks generalize the nearest-neighbor rule into a differentiable, attention-weighted vote over the entire support set, so the whole episode is trained end-to-end against the query loss.

Prototypical networks

Matching Networks compare a query to every support point. Prototypical Networks compress each class to a single point, its prototype, the mean of its support embeddings, and classify a query by the nearest prototype. The model is a parameter-free softmax over negative squared distances.

For example, take $N=3$ classes and suppose a query embeds to $f_{\theta }(x^{\ast })$ at squared distances $d_1=1$, $d_2=4$, $d_3=9$ from the three prototypes. The logits are the negatives $-1,-4,-9$; exponentiating gives $e^{-1}\approx 0.368$, $e^{-4}\approx 0.0183$, $e^{-9}\approx 0.000123$, which sum to $\approx 0.386$. The probabilities are $0.953$, $0.047$, $0.0003$: the nearest prototype dominates, and the softmax is sharp because distances enter the exponent directly. Doubling the embedding scale (multiplying $f_{\theta }$ by $2$) quadruples every distance, which sharpens the softmax further, so the embedding's overall scale acts as an implicit temperature the network is free to learn.

Geometrically, the prototypes act as reference points, and the softmax over negative distances partitions the embedding space into Voronoi-like cells that are the class regions.

Training minimizes the negative log-probability of the true class on the query set, averaged over episodes. Because every operation (embed, average, distance, softmax) is differentiable, gradients flow back into $f_{\theta }$.

The theorem explains why squared Euclidean distance is the right choice and cosine distance underperforms: it makes the classifier a linear readout whose weights are the prototypes, recovering a Bregman-divergence mean-classifier rather than an arbitrary metric.⁴

Optimization-based meta-learning

Metric methods bake in a fixed classification rule. Optimization-based methods keep ordinary gradient descent as the adaptation rule and instead meta-learn the starting point. The adaptation itself is nothing exotic: it is the same gradient step $\theta \leftarrow \theta -\alpha {\nabla }_{\theta }\mathcal{L}$ that trains any network, run for a few iterations on the support set.⁶⁷ The goal is an initialization $\theta$ that sits a few gradient steps away from a good solution for any task in $p(\mathcal{T})$.

MAML and the bi-level objective

Model-Agnostic Meta-Learning (MAML) is model-agnostic because it adds nothing to the architecture: it only changes how the initialization is trained. The inner loop adapts $\theta$ to a task with one (or a few) gradient steps on its support loss; the outer loop updates $\theta$ so that this adaptation generalizes, scored on the query loss after adaptation.

The objective is bi-level: the outer minimization over $\theta$ contains, inside its loss, the inner minimization that produced ${\theta }_i^{\prime }$. Differentiating it is the crux. By the chain rule, the meta-gradient of one task's query loss is

Differentiating the one-step update ${\theta }_i^{\prime }=\theta -\alpha {\nabla }_{\theta }{\mathcal{L}}_{{\mathcal{T}}_i}^{\text{spt}}(\theta )$ with respect to $\theta$ gives the inner Jacobian

so the exact meta-gradient carries a Hessian of the support loss:

This is the second-order MAML update: backpropagating through the inner gradient step requires a Hessian-vector product, differentiating the gradient itself.

The dimensions clarify why this is expensive yet the Hessian is never materialized. Let $\theta \in {\mathbb{R}}^P$ with $P$ the parameter count. The query gradient ${\nabla }_{{\theta }_i^{\prime }}{\mathcal{L}}^{\text{qry}}$ is a vector in ${\mathbb{R}}^P$; the inner Jacobian $\partial {\theta }_i^{\prime }/\partial \theta$ and the Hessian ${\nabla }_{\theta }^2{\mathcal{L}}^{\text{spt}}$ are $P\times P$ matrices. Forming a $P\times P$ Hessian is impossible for a real network ($P$ is in the millions), but the meta-gradient only ever multiplies it by a vector:

The Hessian-vector product ${\nabla }_{\theta }^2{\mathcal{L}}^{\text{spt}}v$ is one extra backward pass (reverse-mode over the inner gradient), so the exact meta-gradient costs roughly two backward passes per task, not the $P^2$ storage a full Hessian would need. For $G$ inner steps, the chain rule nests $G$ such Jacobians, and the cost grows linearly in $G$.

FOMAML loses little accuracy in practice while removing the cost of the Hessian term, evidence that most of the meta-signal is in the first-order direction.⁸ The two loops are nested: an inner trajectory per task, an outer step that moves the shared start.

Reptile: first-order without a query split

Reptile removes even the support/query distinction. It runs several SGD steps on a task to reach ${\theta }_i^{\prime }$, then moves the initialization toward that point.⁷ The meta-update is a difference of parameter vectors, no second derivative anywhere.

Reptile works because ${\theta }_i^{\prime }-\theta$, averaged over tasks, points toward a point near each task's manifold of solutions, exactly where a few more steps land fastest. It is the cheapest of the three and a strong default when compute is tight.

A related thread treats the optimizer itself as a learnable network: an LSTM meta-learner that ingests the gradient ${\nabla }_{\theta }\mathcal{L}$ at each step and emits the parameter update, learning a task-specific update rule in its hidden state rather than a fixed $\alpha$.⁶

Model-based meta-learning

Metric and optimization methods both run an explicit adaptation procedure at test time. Model-based (black-box) methods absorb the procedure into a single network whose forward pass reads the support set and then classifies the query, so learning is one conditioned inference.

The canonical instance is a memory-augmented neural network: a controller writes each support embedding into an external, content-addressable memory, then reads the slot most similar to the query to recover its label, learning a write-then-retrieve policy that generalizes to new classes. This family is the conceptual bridge to in-context learning: a forward pass that adapts because the support is in its input, not in its weights.

Connections: transfer, and in-context learning

Meta-learning, transfer learning, and the few-shot behavior of large language models form a spectrum distinguished by where task-specific information lives and whether a gradient step occurs at adaptation time.

Transfer learning meta-learns implicitly: pretraining on a broad source task yields features that adapt to many targets, which is the few-shot objective with a single giant task: feature extraction reuses the frozen representation, and fine-tuning takes a few gradient steps on it, exactly the metric and optimization mechanisms seen above, applied to one source instead of a task distribution.⁹ The most recent form is in-context learning: a frozen large language model shown a few input–output pairs in its prompt performs the new task with no weight update, the support set living entirely in the context window.

This is meta-learning whose inner loop is a forward pass. Pretraining on a vast distribution of implicit tasks (every document is a different prediction problem) plays the role of the meta-training distribution $p(\mathcal{T})$; the prompt's demonstrations play the role of the support set; and the model's conditioning on them is an implicit adaptation algorithm learned by the same gradient descent that trained the weights. The forward pass has learned to be a learning algorithm.¹

Evaluation

Two benchmarks anchor the literature. Omniglot is $1623$ handwritten characters from $50$ alphabets with $20$ examples each, the transpose of MNIST (many classes, few examples), well suited to few-shot evaluation. miniImageNet is $100$ ImageNet classes ($64$ train, $16$ validation, $20$ test) at $84\times 84$ resolution, the harder natural-image standard. Accuracy is reported on held-out classes, averaged over hundreds of sampled $N$-way $K$-shot test episodes with $95\%$ confidence intervals.

The comparison across the families, on mechanism rather than a leaderboard number (which shifts with backbone and year):

Strong baselines and the in-context turn

Two findings since the classic meta-learning papers reshaped how the field thinks about few-shot learning — one deflationary, one transformative.

The deflationary result is that a good representation plus a simple classifier often beats elaborate meta-learning. Chen et al. (2019, ICLR, A Closer Look at Few-Shot Classification) and the Meta-Baseline line (Tian et al., 2020, ECCV) showed that plainly pretraining a backbone on all the base classes with ordinary supervised cross-entropy, then fitting a nearest-centroid or logistic-regression head on the support set, matches or beats MAML and Prototypical Networks on miniImageNet. The episodic training machinery this lesson builds turned out to be less important than the quality of the embedding it produced — an echo of the representation-learning thesis that a good code makes the downstream task easy. A related result, ANIL (Raghu et al., 2020, ICLR, Almost No Inner Loop), showed MAML's inner-loop adaptation barely changes the feature extractor at all: nearly all the benefit comes from adapting the final layer, so MAML is mostly learning a reusable representation, not a fast learner.

The transformative result is in-context learning. A large language model (Brown et al., 2020, NeurIPS, Language Models are Few-Shot Learners) performs a new task from a handful of examples placed in its prompt, with no gradient step at all — the frozen forward pass conditioned on the support set is the adaptation. This is precisely the model-based meta-learning goal from the previous section, achieved as an emergent property of scale rather than an explicit meta-objective: the $N$-way $K$-shot episode becomes $K$ demonstrations in the context window. The formal account is still being worked out — evidence suggests the forward pass implements something like implicit gradient descent or Bayesian inference over the demonstrations — but the practical upshot is settled. For text, and increasingly for vision and multimodal tasks, few-shot learning is now done by prompting a foundation model, and the explicit meta-learners of this lesson remain useful mainly where episodes are cheap and a small specialized model is preferable to a giant general one.

Takeaways

• Meta-learning trains across a distribution of tasks so a new task is learnable from few examples; the unit of experience is the $N$-way $K$-shot episode with its support and query sets, and meta-train/meta-test classes are disjoint.
• Metric methods learn an embedding where a fixed distance classifies. Prototypical Networks use the class-mean prototype and a softmax over negative squared distances, which (by the expansion of the square) is a linear classifier whose weights are the prototypes.
• Optimization methods learn an initialization. MAML's bi-level objective minimizes post-adaptation query loss; its exact meta-gradient carries a Hessian $\left(I-\alpha {\nabla }^2{\mathcal{L}}^{\text{spt}}\right)$, which FOMAML drops and Reptile avoids entirely by moving the init toward post-SGD parameters.
• Model-based methods fold adaptation into a conditioned forward pass (memory-augmented nets), the bridge to in-context learning.
• The paradigm is a spectrum with transfer learning and with in-context learning in large language models, where a frozen model's forward pass, conditioned on a few prompt demonstrations, is the learning algorithm meta-learning sought to produce.