Transfer Learning

This builds on Representation Learning, which established what makes a representation good — disentangled factors, a distributed code, and respect for the data manifold — and closed on greedy layer-wise pretraining, the first method to reuse a learned representation. Here we take the reuse idea seriously as engineering: how to transfer a trained encoder to a new task, when it helps and when it hurts, and how the modern self-supervised recipe turns one representation into a foundation for everything downstream.

Transfer learning

A representation learned to solve one task is rarely specific to that task. The early layers of an ImageNet classifier compute oriented edges, color blobs, and textures, features useful for any vision problem. Transfer learning reuses that learned $f_{\theta }$ on a new task, paying the large pretraining cost once and amortizing it over many downstream problems.¹

There are two regimes for the transfer, distinguished by whether the borrowed weights $\theta$ are held fixed or adapted.

The choice between the two is governed by target-set size and source–target similarity: small data favors freezing (fewer parameters to overfit), large and dissimilar data favors fine-tuning.²

A worked example: feature extraction with shapes

Take a convolutional base pretrained on ImageNet (a VGG16-style stack) and reuse it on a target set of $2000$ labeled cat/dog images. Feature extraction runs each target image through the frozen base once and caches the output.

The convolutional base holds roughly $14.7$ million weights; every one is frozen. The head is a single dense layer $g_{\varphi }:{\mathbb{R}}^{8192}\to {\mathbb{R}}^2$ with $8192\times 2+2=16,386$ trainable parameters, about $0.1\%$ of the network. Because the base never receives a gradient, its outputs $h$ are constant across epochs: precompute the $2000$ feature maps once, store the $2000\times 8192$ matrix, and train the head on that cached matrix. This is why feature extraction is cheap.³ It is also why it resists overfitting on small data: a $16,386$-parameter head fit to $2000$ examples is a favorable ratio, whereas turning the full $14.7$ million weights loose on those same $2000$ images would memorize them.

Caching only works while the base is frozen. The moment any base layer is unfrozen its outputs change every step, so fine-tuning cannot precompute $h$ and must run the full forward and backward pass over the base each iteration, which is one reason fine-tuning costs far more per epoch.

The generic-to-specific gradient of features

The mechanism behind transfer is the layer hierarchy: low layers learn generic features (edges, color blobs, phones, character n-grams) that recur in nearly every task in a modality; high layers learn task-specific features (an ImageNet dog-snout detector, a sentiment cue) that are useful only for the source task. Transferability falls monotonically as you climb the stack, and this gradient is what dictates the freeze boundary: freeze the layers whose features are generic enough to reuse verbatim, retrain the layers whose features have specialized to the source.

Two forces set the boundary. Freezing more layers means fewer trainable parameters, which helps when target labels are scarce (less to overfit) and hurts when the target differs from the source (the frozen features are wrong and cannot adapt). Unfreezing more layers adds capacity to adapt but demands enough target data to fit it. A practical default: freeze the bottom two-thirds, fine-tune the top third and the head, and move the boundary down only if the target is both large and unlike the source.

The freeze boundary has been measured directly. Yosinski et al. (2014, NeurIPS) split ImageNet into two disjoint halves, trained a network on one, and transferred the first $k$ layers to the other for each $k$. Transferring the first one or two layers cost almost nothing; transferring the top layers hurt, and the damage grew with $k$ exactly because those layers had co-adapted to the source. That experiment is the empirical curve the diagram above draws: generic at the bottom, specific at the top, with a measurable crossover in between.

The learning-rate discipline for fine-tuning

Fine-tuning is not ordinary training. The borrowed weights already sit in a good basin, and the task is to nudge them, not to relearn them. Three rules follow.

Train the head first, then unfreeze. A freshly initialized head produces large, random gradients. If those flow back into the pretrained base before the head has converged, they overwrite the features being transferred. The standard sequence is: (1) freeze the base and train the head to convergence as feature extraction, then (2) unfreeze the top blocks and continue at a small rate. Skipping step (1) lets a random head's gradient destroy the pretrained representation on the first minibatch.⁴

Use a small learning rate. Fine-tuning uses a rate roughly one to two orders of magnitude below from-scratch training, so that each update moves the weights a short distance and the borrowed features are preserved. If pretraining used $\eta =10^{-3}$, fine-tuning typically uses $\eta \approx 10^{-5}$ to $10^{-4}$. Too large a rate erases the pretrained structure and the run reverts to training from scratch on too little data.

Discriminative (layer-wise) learning rates. Because transferability decays with depth, the layers should learn at different speeds: lower layers, being more generic, should barely move; upper layers, being more source-specific, need more adjustment. Assign each layer $\ell$ its own rate that grows with depth,

so the top layer $\ell =L$ trains at the base rate ${\eta }_{\text{top}}$ and each layer below it is discounted by a factor $\gamma$ (a common choice is $\gamma \approx 1/2.6$, giving a base layer rate hundreds of times smaller than the head). This is discriminative fine-tuning: a continuous version of the freeze boundary, where frozen is the limit ${\eta }_{\ell }\to 0$ and every layer in between gets a rate matched to how much its features should change.

Domain adaptation

A special case keeps the task fixed but changes the input distribution: a sentiment classifier trained on product reviews applied to movie reviews, or a detector trained on daytime images run at night. Here labels exist for the source but not the target.

The standard approach makes the representation domain-invariant by penalizing any statistic that separates source from target codes: minimizing a divergence $d\left(p_S(h),p_T(h)\right)$ jointly with the source classification loss, so that $f_{\theta }$ keeps what predicts $y$ and discards what reveals the domain. Geometrically, the encoder is pushed to collapse the two clouds of embeddings on top of each other while the source decision boundary stays put, so the boundary now cuts the target cloud in the same place.

The divergence-minimizing approach has an adversarial form. Domain-adversarial training (Ganin et al., 2016, JMLR) attaches a small domain classifier on top of $h$ that tries to tell source from target, and trains the encoder to fool it through a gradient-reversal layer: the encoder is pushed to make $h$ domain-agnostic at the same time as the label head is pushed to make $h$ predictive. At the saddle point the domain classifier is at chance — $p_S(h)\approx p_T(h)$ — and the label head transfers. It is the same align-the-clouds picture, with a learned adversary supplying the divergence instead of a fixed statistic.

When transfer helps, and when it hurts

Transfer is not free. Reusing $f_{\theta }$ imports whatever inductive bias the source task built into the features, and that bias helps only when source and target share the relevant structure. Three regimes:

• Positive transfer. Source and target share low-level structure and the target is small. The pretrained features are close to a good target encoder, so the head (or top blocks) adapts from a warm start with far fewer labels than training from scratch would need. This is the common case within a modality: any ImageNet base transfers to most vision tasks; any large language-model encoder transfers to most text tasks.
• Negligible transfer. The target is large and the source is unrelated. The warm start neither helps nor hurts much: with enough target data the network relearns whatever it needs, and pretraining is wasted effort but not damaging.
• Negative transfer. The source features encode structure that is actively wrong for the target, and the target is too small to correct it. Freezing then locks in the wrong features; even fine-tuning can leave the network stuck in the source basin. A detector pretrained to ignore color transferred to a task where color is the signal is worse than random initialization.

The deciding variable is the mismatch between $p(y\mid x)$ on the two tasks, not merely the input shift. Domain adaptation handles the case where only $p(x)$ shifts and $p(y\mid x)$ is shared; when $p(y\mid x)$ itself differs, the borrowed representation is optimizing for the wrong invariances and transfer degrades.

The modern arc: self-supervision and foundation models

Greedy pretraining used unlabeled data through a clumsy layer-wise proxy. Transfer learning reused supervised features. The synthesis of the two is self-supervised pretraining: train one enormous encoder end-to-end on a task whose labels are manufactured from the input itself (predict a masked-out token, the next token, or whether two augmentations came from the same image), then transfer that single representation to every downstream task.

The design of the pretext task is the central decision: it must be solvable only by learning features that also serve the real downstream tasks, so that the encoder is forced to model $p(x)$ rather than exploit a shortcut. The families that work share that property.⁵

• Masked prediction. Hide part of the input and predict it from the rest, optimizing $-\log p(x_{\text{hidden}}\mid x_{\text{visible}})$. Masking a token in a sentence forces the encoder to model syntax and semantics; masking a patch in an image forces it to model shape and texture. The label is the hidden piece itself, so no annotation is needed.
• Next-element prediction. Predict the next token in a sequence, ${\prod }_{t}p(x_t\mid x_{<t})$. Modeling the conditional over a large corpus requires a representation of everything that predicts what comes next, which is most of language.
• Contrastive agreement. Produce two augmented views of the same input and train their codes to agree while disagreeing with views of other inputs. The encoder must discard the nuisance factors the augmentation varies (crop, color jitter) and keep the identity-preserving content — disentanglement pursued directly. For a similarity $s(\cdot ,\cdot )$ and temperature $\tau$, the per-pair loss over a batch is

where $h_i^{+}$ is the other view of input $i$ and the $h_j$ are the other batch members. Minimizing it pulls matched views together and pushes mismatched ones apart, carving the code space into one cluster per underlying instance.

Each of these manufactures a supervised-looking objective out of raw data, so the encoder trains end-to-end at scale with the same backprop machinery as a labeled task, and the resulting $f_{\theta }$ transfers by feature extraction, fine-tuning, or prompting.

This collapses the pretrain–transfer pipeline into its modern form: one representation, learned once on web-scale unlabeled data, serves many tasks.

The architecture that made this scale to language and beyond is the Transformer: an attention-based encoder whose self-supervised pretraining objective (predict the next token over a trillion-word corpus) yields a representation general enough that a frozen model plus a prompt solves tasks it never saw at training time. Every property from the companion lesson recurs there at scale: the hidden states are distributed, the learned embedding space disentangles syntactic and semantic factors, and tokens that play similar roles cluster on a shared manifold.

Parameter-efficient adaptation

Fine-tuning as the standard references describe it updates the borrowed weights. Once a foundation model has billions of parameters, updating and storing a full copy per downstream task is untenable, and a family of parameter-efficient fine-tuning (PEFT) methods answers it by adapting the model while touching only a tiny fraction of its weights.

The dominant method is LoRA (low-rank adaptation; Hu et al., 2021, ICLR). Freeze the pretrained weight matrix $W_0\in {\mathbb{R}}^{d\times k}$ entirely and learn a low-rank correction $\Delta W=BA$ with $B\in {\mathbb{R}}^{d\times r}$, $A\in {\mathbb{R}}^{r\times k}$, and $r\ll \min (d,k)$, so the adapted layer computes

Only $A$ and $B$ are trained — $r(d+k)$ parameters instead of $dk$. For a $d=k=4096$ layer and rank $r=8$, that is $8\times 8192=65,536$ trainable weights against $4096^2\approx 16.8$ million, a $256\times$ reduction, and the correction folds back into $W_0$ at inference so there is no added latency. The underlying assumption is the same manifold intuition from the companion lesson: the update a downstream task needs lies in a low-dimensional subspace, so a low-rank $\Delta W$ suffices.

Two neighbors round out the family. Adapters (Houlsby et al., 2019, ICML) insert small bottleneck layers between the frozen blocks and train only those; prompt / prefix tuning (Lester et al., 2021, EMNLP) freezes the entire model and learns a handful of continuous soft prompt vectors prepended to the input — the logical end of the freeze boundary, where no model weight moves and adaptation lives entirely in the input. All three share LoRA's economics: one frozen backbone, a few megabytes of task-specific parameters each, swappable at serving time.

Takeaways

• Transfer reuses a learned encoder $f_{\theta }$ on a new task, paying the pretraining cost once. Feature extraction freezes it and trains only a head (cheap, overfits little, cacheable); fine-tuning unfreezes some layers at a small rate (adapts more, costs more). Choose by target size and source–target similarity.
• Features run generic-to-specific with depth. Low layers (edges, textures) transfer; high layers (source-specific detectors) do not. The freeze boundary sits at the crossover — measured by Yosinski et al. — and discriminative learning rates are its continuous form.
• Fine-tune by moving little: converge the head first so a random gradient cannot wreck the base, keep $\eta$ one to two orders below from-scratch, and scale it down with depth.
• Domain adaptation aligns $p_S(h)\approx p_T(h)$ when the task is shared but the input distribution shifts and the target is unlabeled — via a divergence penalty or, adversarially, a domain classifier the encoder learns to fool.
• Transfer can hurt. Negative transfer occurs when source features are wrong for the target and the target is too small to correct them; the deciding variable is the mismatch in $p(y\mid x)$, not merely the input shift.
• Self-supervised pretraining manufactures labels from raw data (masked prediction, next-token, contrastive agreement), yielding foundation models: one representation, learned once on web-scale data, adapted to many tasks — now most cheaply through parameter-efficient methods like LoRA.