--- title: Transfer Learning module: Practical Deep Learning moduleNumber: 9 lessonNumber: 4 order: 904 summary: > A representation learned once can be reused everywhere. We cover the main mechanisms of reuse: feature extraction versus fine-tuning, the generic-to-specific gradient of features that sets the freeze boundary, the learning-rate discipline that keeps borrowed weights from being erased, domain adaptation when only the input distribution shifts, and the modern arc from supervised transfer to self-supervised foundation models. topics: [Practical Deep Learning] sources: - book: Goodfellow ref: "§15.2 — Transfer Learning and Domain Adaptation" - book: Chollet ref: "§5.3 — Using a Pretrained Convnet (feature extraction, fine-tuning)" - book: Goodfellow ref: "Ch. 15 / §15.1 — Representation Learning (self-supervised pretraining)" --- This builds on [Representation Learning](/deep-learning/practical/representation-learning), which established what makes a representation good — disentangled factors, a distributed code, and respect for the data manifold — and closed on greedy layer-wise pretraining, the first method to reuse a learned representation. Here we take the reuse idea seriously as engineering: how to transfer a trained encoder to a new task, when it helps and when it hurts, and how the modern self-supervised recipe turns one representation into a foundation for everything downstream. ## Transfer learning A representation learned to solve one task is rarely specific to that task. The early layers of an ImageNet classifier compute oriented edges, color blobs, and textures, features useful for _any_ vision problem. **Transfer learning** reuses that learned $f_\theta$ on a new task, paying the large pretraining cost once and amortizing it over many downstream problems.[^gf-transfer] > **Definition (Transfer learning).** Given a representation $f_\theta$ trained on > a **source** task with abundant data $\mathcal{D}_S$, reuse it on a **target** > task with limited data $\mathcal{D}_T$ by attaching a new head $g_\phi$ and > training $g_\phi$ (and optionally adjusting $\theta$) on $\mathcal{D}_T$. Transfer > helps when the two tasks share low-level structure, so $f_\theta$ is already > close to a good target encoder. $$ % caption: Transfer learning: pretrain a deep encoder on a large source dataset, % then attach a small head and train it on limited target data. \begin{tikzpicture}[>=stealth, font=\footnotesize, blk/.style={draw, minimum width=20mm, minimum height=9mm, align=center, font=\scriptsize}, enc/.style={draw=acc, thick, minimum width=20mm, minimum height=9mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} % ---- source column ---- \node[blk] (sdata) at (0,3.0) {large source\\data}; \node[enc] (senc) at (0,1.4) {encoder $f$}; \node[blk] (shead) at (0,-0.2) {source head}; \draw[->, acc, thick] (sdata) -- (senc); \draw[->, acc, thick] (senc) -- (shead); \node[font=\scriptsize, text=acc] at (0,-1.1) {pretrain}; % ---- transfer arrow ---- \draw[->, green, very thick] (1.3,1.4) -- (3.5,1.4) node[midway, above, text=green] {transfer $f$}; % ---- target column ---- \node[blk] (tdata) at (4.8,3.0) {small target\\data}; \node[enc] (tenc) at (4.8,1.4) {encoder $f$\\(reused)}; \node[blk, draw=green, thick] (thead) at (4.8,-0.2) {new head}; \draw[->, acc, thick] (tdata) -- (tenc); \draw[->, green, thick] (tenc) -- (thead); \node[font=\scriptsize, text=green] at (4.8,-1.1) {train head}; \end{tikzpicture} $$ There are two regimes for the transfer, distinguished by whether the borrowed weights $\theta$ are held fixed or adapted. > **Definition (Feature extraction).** Freeze the pretrained encoder $f_\theta$ and > train only the new head $g_\phi$ on the target data: $h = f_\theta(x)$ is computed > once and treated as a fixed feature, $\hat y = g_\phi(h)$. No gradient flows into > $\theta$. > **Definition (Fine-tuning).** Initialize from $f_\theta$ but continue to update > some or all of its weights on the target task, usually with a small learning rate > so the borrowed features are nudged rather than overwritten. Often the lowest > layers are frozen and only the top blocks are fine-tuned. $$ % caption: Feature extraction (left) freezes every borrowed layer and trains only % the head; fine-tuning (right) unfreezes the top blocks at a small learning rate. \begin{tikzpicture}[>=stealth, font=\footnotesize, L/.style={draw, minimum width=24mm, minimum height=8mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} % ---- left stack: feature extraction ---- \begin{scope} \node[font=\small] at (0,4.0) {feature extraction}; \node[L, fill=black!8] (a1) at (0,3.1) {layer 1 (frozen)}; \node[L, fill=black!8] (a2) at (0,2.2) {layer 2 (frozen)}; \node[L, fill=black!8] (a3) at (0,1.3) {layer 3 (frozen)}; \node[L, fill=black!8] (a4) at (0,0.4) {layer 4 (frozen)}; \node[L, draw=green, thick] (ah) at (0,-0.7) {head (trained)}; \draw[->, thick] (a1)--(a2); \draw[->, thick] (a2)--(a3); \draw[->, thick] (a3)--(a4); \draw[->, green, thick] (a4)--(ah); \node[text=black, font=\footnotesize, anchor=west] at (1.45,1.75) {\texttt{no gradient}}; \end{scope} % ---- right stack: fine-tuning ---- \begin{scope}[xshift=6.4cm] \node[font=\small] at (0,4.0) {f\/ine-tuning}; \node[L, fill=black!8] (b1) at (0,3.1) {layer 1 (frozen)}; \node[L, fill=black!8] (b2) at (0,2.2) {layer 2 (frozen)}; \node[L, draw=acc, thick] (b3) at (0,1.3) {layer 3 (trained)}; \node[L, draw=acc, thick] (b4) at (0,0.4) {layer 4 (trained)}; \node[L, draw=green, thick] (bh) at (0,-0.7) {head (trained)}; \draw[->, thick] (b1)--(b2); \draw[->, acc, thick] (b2)--(b3); \draw[->, acc, thick] (b3)--(b4); \draw[->, green, thick] (b4)--(bh); \node[text=acc, font=\scriptsize, anchor=west] at (1.45,0.85) {small step}; \end{scope} \end{tikzpicture} $$ The choice between the two is governed by target-set size and source–target similarity: small data favors freezing (fewer parameters to overfit), large and dissimilar data favors fine-tuning.[^chollet-finetune] | Scenario | Source–target similarity | Target data size | Recommended strategy | | --- | --- | --- | --- | | same domain, few labels | high | small | feature extraction (freeze all, train head) | | same domain, many labels | high | large | fine-tune whole network at small rate | | related domain, few labels | medium | small | freeze low layers, fine-tune top blocks | | related domain, many labels | medium | large | fine-tune all; warm-start beats random | | unrelated domain | low | any | transfer rarely helps; train from scratch | | target unlabeled, source labeled | varies | none labeled | domain adaptation (align features) | #### A worked example: feature extraction with shapes Take a convolutional base pretrained on ImageNet (a VGG16-style stack) and reuse it on a target set of $2000$ labeled cat/dog images. Feature extraction runs each target image through the frozen base once and caches the output. $$ \underbrace{x \in \mathbb{R}^{150 \times 150 \times 3}}_{\text{RGB image}} \;\xrightarrow{\;f_\theta\ (\text{frozen})\;}\; \underbrace{h \in \mathbb{R}^{4 \times 4 \times 512}}_{\text{conv feature map}} \;\xrightarrow{\;\text{flatten}\;}\; \mathbb{R}^{8192} \;\xrightarrow{\;g_\phi\;}\; \underbrace{\hat y \in \mathbb{R}^{2}}_{\text{logits}} . $$ The convolutional base holds roughly $14.7$ million weights; every one is frozen. The head is a single dense layer $g_\phi : \mathbb{R}^{8192} \to \mathbb{R}^{2}$ with $8192 \times 2 + 2 = 16{,}386$ trainable parameters, about $0.1\%$ of the network. Because the base never receives a gradient, its outputs $h$ are constant across epochs: precompute the $2000$ feature maps once, store the $2000 \times 8192$ matrix, and train the head on that cached matrix. This is why feature extraction is cheap.[^chollet-cache] It is also why it resists overfitting on small data: a $16{,}386$-parameter head fit to $2000$ examples is a favorable ratio, whereas turning the full $14.7$ million weights loose on those same $2000$ images would memorize them. Caching only works while the base is frozen. The moment any base layer is unfrozen its outputs change every step, so fine-tuning cannot precompute $h$ and must run the full forward and backward pass over the base each iteration, which is one reason fine-tuning costs far more per epoch. #### The generic-to-specific gradient of features The mechanism behind transfer is the layer hierarchy: low layers learn _generic_ features (edges, color blobs, phones, character n-grams) that recur in nearly every task in a modality; high layers learn _task-specific_ features (an ImageNet dog-snout detector, a sentiment cue) that are useful only for the source task. Transferability falls monotonically as you climb the stack, and this gradient is what dictates the freeze boundary: freeze the layers whose features are generic enough to reuse verbatim, retrain the layers whose features have specialized to the source. $$ % caption: Feature transferability decays with depth: low layers ($h^{(1)}$, % $h^{(2)}$) hold generic edges and textures reused across tasks; high layers % ($h^{(4)}$, head) hold source-specific features. The freeze boundary sits where % features stop being generic. \begin{tikzpicture}[>=stealth, font=\footnotesize, L/.style={draw, minimum width=30mm, minimum height=8mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \definecolor{warm}{HTML}{C6551E} % layer stack, bottom = input \node[L, fill=green!12, draw=green] (l1) at (0,0.0) {\texttt{layer 1: edges}}; \node[L, fill=green!10, draw=green] (l2) at (0,1.0) {\texttt{layer 2: textures}}; \node[L, fill=black!6] (l3) at (0,2.0) {\texttt{layer 3: parts}}; \node[L, fill=warm!12, draw=warm] (l4) at (0,3.0) {\texttt{layer 4: objects}}; \node[L, fill=warm!16, draw=warm] (lh) at (0,4.0) {\texttt{head: classes}}; \draw[->, thick] (l1)--(l2); \draw[->, thick] (l2)--(l3); \draw[->, thick] (l3)--(l4); \draw[->, thick] (l4)--(lh); % transferability arrow on the right, pointing down (more transferable low) \draw[->, acc, very thick] (2.2,4.3) -- (2.2,-0.3); \node[text=acc, rotate=-90, anchor=south] at (2.55,2.0) {\texttt{less generic}}; % generic vs specific brackets on the left \node[text=green, anchor=east, font=\footnotesize] at (-1.9,0.5) {\texttt{generic}}; \node[text=green, anchor=east, font=\footnotesize] at (-1.9,0.0) {\texttt{(reuse)}}; \node[text=warm, anchor=east, font=\footnotesize] at (-1.9,3.5) {\texttt{specific}}; \node[text=warm, anchor=east, font=\footnotesize] at (-1.9,3.0) {\texttt{(retrain)}}; % freeze boundary line between layer 2 and 3 \draw[dash pattern=on 2pt off 2pt, acc, thick] (-2.2,1.5) -- (1.9,1.5); \node[text=acc, anchor=south west, font=\footnotesize] at (-2.2,1.6) {\texttt{freeze below}}; \end{tikzpicture} $$ Two forces set the boundary. Freezing more layers means fewer trainable parameters, which helps when target labels are scarce (less to overfit) and hurts when the target differs from the source (the frozen features are wrong and cannot adapt). Unfreezing more layers adds capacity to adapt but demands enough target data to fit it. A practical default: freeze the bottom two-thirds, fine-tune the top third and the head, and move the boundary down only if the target is both large and unlike the source. The freeze boundary has been measured directly. Yosinski et al. (2014, _NeurIPS_) split ImageNet into two disjoint halves, trained a network on one, and transferred the first $k$ layers to the other for each $k$. Transferring the first one or two layers cost almost nothing; transferring the top layers hurt, and the damage grew with $k$ exactly because those layers had co-adapted to the source. That experiment is the empirical curve the diagram above draws: generic at the bottom, specific at the top, with a measurable crossover in between. #### The learning-rate discipline for fine-tuning Fine-tuning is not ordinary training. The borrowed weights already sit in a good basin, and the task is to nudge them, not to relearn them. Three rules follow. **Train the head first, then unfreeze.** A freshly initialized head produces large, random gradients. If those flow back into the pretrained base before the head has converged, they overwrite the features being transferred. The standard sequence is: (1) freeze the base and train the head to convergence as feature extraction, then (2) unfreeze the top blocks and continue at a small rate. Skipping step (1) lets a random head's gradient destroy the pretrained representation on the first minibatch.[^chollet-order] **Use a small learning rate.** Fine-tuning uses a rate roughly one to two orders of magnitude below from-scratch training, so that each update moves the weights a short distance and the borrowed features are preserved. If pretraining used $\eta = 10^{-3}$, fine-tuning typically uses $\eta \approx 10^{-5}$ to $10^{-4}$. Too large a rate erases the pretrained structure and the run reverts to training from scratch on too little data. **Discriminative (layer-wise) learning rates.** Because transferability decays with depth, the layers should learn at different speeds: lower layers, being more generic, should barely move; upper layers, being more source-specific, need more adjustment. Assign each layer $\ell$ its own rate that grows with depth, $$ \eta_\ell \;=\; \eta_{\text{top}} \cdot \gamma^{\,L-\ell}, \qquad \gamma \in (0,1), $$ so the top layer $\ell = L$ trains at the base rate $\eta_{\text{top}}$ and each layer below it is discounted by a factor $\gamma$ (a common choice is $\gamma \approx 1/2.6$, giving a base layer rate hundreds of times smaller than the head). This is _discriminative fine-tuning_: a continuous version of the freeze boundary, where "frozen" is the limit $\eta_\ell \to 0$ and every layer in between gets a rate matched to how much its features should change. > **Takeaway.** Fine-tuning succeeds by moving little: converge the head first, > keep the learning rate small, and scale it down with depth so generic low-level > features stay put while source-specific high-level features adapt. The freeze > boundary and discriminative rates are the same idea (protect the generic end of > the hierarchy) expressed discretely and continuously. $$ % caption: The pretrain-then-fine-tune schedule. Phase 0 pretrains $f_\theta$ on the % source at the base rate $\eta \approx 10^{-3}$. Phase 1 freezes the base and % trains only the head. Phase 2 unfreezes the top blocks at a small rate % $\eta \approx 10^{-5}$. The order (head first) protects the borrowed features. \begin{tikzpicture}[>=stealth, font=\footnotesize, ph/.style={draw, minimum width=32mm, minimum height=13mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \node[ph, draw=acc, thick, fill=acc!6] (p0) at (0,0) {\texttt{phase 0: pretrain}\\\texttt{base f on source}\\\texttt{lr about 1e-3}}; \node[ph, fill=black!5] (p1) at (4.6,0) {\texttt{phase 1: freeze base,}\\\texttt{train head}\\\texttt{lr about 1e-3}}; \node[ph, draw=green, thick, fill=green!6] (p2) at (9.2,0) {\texttt{phase 2: unfreeze top,}\\\texttt{fine-tune}\\\texttt{lr about 1e-5}}; \draw[->, acc, thick] (p0) -- (p1); \draw[->, green, thick] (p1) -- (p2); \node[font=\footnotesize, text=black, anchor=north] at (2.3,-0.9) {\texttt{transfer f}}; \node[font=\footnotesize, text=green, anchor=north] at (6.9,-0.9) {\texttt{head converged}}; \end{tikzpicture} $$ ### Domain adaptation A special case keeps the _task_ fixed but changes the input _distribution_: a sentiment classifier trained on product reviews applied to movie reviews, or a detector trained on daytime images run at night. Here labels exist for the source but not the target. > **Definition (Domain adaptation).** Transfer where the label space and task are > identical but the marginal input distributions differ, $p_S(x) \neq p_T(x)$, > while the conditional $p(y \mid x)$ is assumed (approximately) shared. The goal > is a representation $h = f_\theta(x)$ under which the two domains are > indistinguishable, $p_S(h) \approx p_T(h)$, so a source-trained classifier > transfers without target labels. The standard approach makes the representation domain-invariant by penalizing any statistic that separates source from target codes: minimizing a divergence $d\parens{p_S(h),\, p_T(h)}$ jointly with the source classification loss, so that $f_\theta$ keeps what predicts $y$ and discards what reveals the domain. Geometrically, the encoder is pushed to collapse the two clouds of embeddings on top of each other while the source decision boundary stays put, so the boundary now cuts the target cloud in the same place. $$ % caption: Domain adaptation. Before (left) source (blue) and target (orange) % embeddings occupy different regions, so the source boundary misclassifies target % points. After (right) the two clouds are aligned, $p_S(h) \approx p_T(h)$, and one % boundary serves both. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{warm}{HTML}{C6551E} % ---- before: two separated clouds ---- \begin{scope} \node[font=\small] at (1.4,3.0) {\texttt{before}}; \draw[black] (-0.4,-0.4) rectangle (3.2,2.4); % source cloud lower-left \foreach \x/\y in {0.1/0.3, 0.3/0.7, 0.5/0.2, 0.2/0.9, 0.6/0.6, 0.4/1.1} { \fill[acc] (\x,\y) circle (1.6pt); } % target cloud upper-right, shifted \foreach \x/\y in {2.1/1.4, 2.3/1.8, 2.5/1.3, 2.2/2.0, 2.6/1.7, 2.4/1.1} { \fill[warm] (\x,\y) circle (1.6pt); } % source boundary misses target \draw[black, thick, dash pattern=on 2pt off 2pt] (1.2,-0.4) -- (1.2,2.4); \node[text=warm, font=\footnotesize, anchor=west] at (1.9,2.15) {\texttt{target off}}; \end{scope} % ---- align arrow ---- \draw[->, green, very thick] (3.6,1.0) -- (5.0,1.0); \definecolor{green}{HTML}{1F9D4D} \node[text=green, font=\footnotesize] at (4.3,1.5) {\texttt{align}}; % ---- after: overlapping clouds ---- \begin{scope}[xshift=5.6cm] \node[font=\small] at (1.4,3.0) {\texttt{after}}; \draw[black] (-0.4,-0.4) rectangle (3.2,2.4); % both clouds overlap, class structure preserved left/right \foreach \x/\y in {0.4/0.5, 0.6/1.0, 0.3/0.8, 0.7/0.4, 0.5/1.3} { \fill[acc] (\x,\y) circle (1.6pt); } \foreach \x/\y in {0.5/0.7, 0.7/1.2, 0.4/1.1, 0.6/0.6, 0.3/1.0} { \fill[warm] (\x,\y) circle (1.6pt); } \foreach \x/\y in {2.3/0.6, 2.5/1.1, 2.2/0.9, 2.6/0.5, 2.4/1.3} { \fill[acc] (\x,\y) circle (1.6pt); } \foreach \x/\y in {2.4/0.8, 2.6/1.2, 2.3/1.0, 2.5/0.6, 2.2/1.1} { \fill[warm] (\x,\y) circle (1.6pt); } \draw[black, thick, dash pattern=on 2pt off 2pt] (1.5,-0.4) -- (1.5,2.4); \node[text=green, font=\footnotesize, anchor=west] at (1.65,2.15) {\texttt{both fit}}; \end{scope} \end{tikzpicture} $$ The divergence-minimizing approach has an adversarial form. **Domain-adversarial training** (Ganin et al., 2016, _JMLR_) attaches a small _domain classifier_ on top of $h$ that tries to tell source from target, and trains the encoder to _fool_ it through a gradient-reversal layer: the encoder is pushed to make $h$ domain-agnostic at the same time as the label head is pushed to make $h$ predictive. At the saddle point the domain classifier is at chance — $p_S(h) \approx p_T(h)$ — and the label head transfers. It is the same align-the-clouds picture, with a learned adversary supplying the divergence instead of a fixed statistic. ### When transfer helps, and when it hurts Transfer is not free. Reusing $f_\theta$ imports whatever inductive bias the source task built into the features, and that bias helps only when source and target share the relevant structure. Three regimes: - **Positive transfer.** Source and target share low-level structure and the target is small. The pretrained features are close to a good target encoder, so the head (or top blocks) adapts from a warm start with far fewer labels than training from scratch would need. This is the common case within a modality: any ImageNet base transfers to most vision tasks; any large language-model encoder transfers to most text tasks. - **Negligible transfer.** The target is large and the source is unrelated. The warm start neither helps nor hurts much: with enough target data the network relearns whatever it needs, and pretraining is wasted effort but not damaging. - **Negative transfer.** The source features encode structure that is actively wrong for the target, and the target is too small to correct it. Freezing then locks in the wrong features; even fine-tuning can leave the network stuck in the source basin. A detector pretrained to ignore color transferred to a task where color is the signal is worse than random initialization. The deciding variable is the mismatch between $p(y \mid x)$ on the two tasks, not merely the input shift. Domain adaptation handles the case where only $p(x)$ shifts and $p(y \mid x)$ is shared; when $p(y \mid x)$ itself differs, the borrowed representation is optimizing for the wrong invariances and transfer degrades. ## The modern arc: self-supervision and foundation models Greedy pretraining used unlabeled data through a clumsy layer-wise proxy. Transfer learning reused _supervised_ features. The synthesis of the two is **self-supervised pretraining**: train one enormous encoder end-to-end on a task whose labels are manufactured from the input itself (predict a masked-out token, the next token, or whether two augmentations came from the same image), then transfer that single representation to every downstream task. > **Definition (Self-supervised pretraining).** Pretraining on a **pretext** task > whose targets are derived automatically from unlabeled $x$ (masked prediction > $p(x_{\text{hidden}} \mid x_{\text{visible}})$, next-token prediction, or > contrastive agreement of augmentations), so that no human annotation is needed, > yet the learned $f_\theta$ captures the structure of $p(x)$ and transfers. The design of the pretext task is the central decision: it must be solvable only by learning features that also serve the real downstream tasks, so that the encoder is forced to model $p(x)$ rather than exploit a shortcut. The families that work share that property.[^gf-selfsup] - **Masked prediction.** Hide part of the input and predict it from the rest, optimizing $-\log p(x_{\text{hidden}} \mid x_{\text{visible}})$. Masking a token in a sentence forces the encoder to model syntax and semantics; masking a patch in an image forces it to model shape and texture. The label is the hidden piece itself, so no annotation is needed. - **Next-element prediction.** Predict the next token in a sequence, $\prod_t p(x_t \mid x_{ **Definition (Foundation model).** A single large model pretrained > self-supervised on broad data at scale, whose representation is then adapted > (by fine-tuning, a lightweight head, or merely a prompt) to a wide range of > downstream tasks it was not explicitly trained for. $$ % caption: The foundation-model arc: one encoder pretrained self-supervised on % web-scale data, with many downstream tasks branching off via a small head or prompt. \begin{tikzpicture}[>=stealth, font=\footnotesize, blk/.style={draw, minimum width=22mm, minimum height=9mm, align=center, font=\scriptsize}, task/.style={draw, minimum width=24mm, minimum height=8mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \node[blk] (data) at (0,0) {\texttt{web-scale}\\\texttt{unlabeled data}}; \node[blk, draw=acc, thick, minimum height=12mm] (fm) at (3.6,0) {\texttt{foundation}\\\texttt{model} $f$}; \draw[->, acc, thick] (data) -- (fm); \node[font=\footnotesize, text=acc, anchor=south] at (1.8,0.9) {\texttt{self-supervised}}; % downstream tasks branching off \node[task, draw=green, thick] (t1) at (8.0,2.0) {classify}; \node[task, draw=green, thick] (t2) at (8.0,0.6) {translate}; \node[task, draw=green, thick] (t3) at (8.0,-0.8) {summarize}; \node[task, draw=green, thick] (t4) at (8.0,-2.2) {answer}; \draw[->, green, thick] (fm.east) -- (t1); \draw[->, green, thick] (fm.east) -- (t2); \draw[->, green, thick] (fm.east) -- (t3); \draw[->, green, thick] (fm.east) -- (t4); \node[font=\scriptsize, text=green] at (6.0,1.9) {adapt}; \end{tikzpicture} $$ The architecture that made this scale to language and beyond is the [Transformer](/deep-learning/architectures/the-transformer-architecture): an attention-based encoder whose self-supervised pretraining objective (predict the next token over a trillion-word corpus) yields a representation general enough that a frozen model plus a prompt solves tasks it never saw at training time. Every property from the companion lesson recurs there at scale: the hidden states are distributed, the learned embedding space disentangles syntactic and semantic factors, and tokens that play similar roles cluster on a shared manifold. | Era | Pretraining signal | How features are reused | Limitation | | --- | --- | --- | --- | | greedy layer-wise (2006) | unsupervised, layer-by-layer | init for supervised fine-tune | clumsy; superseded by better optimization | | supervised transfer (2014) | labeled source task | feature-extract or fine-tune | needs a large _labeled_ source | | self-supervised (2018–) | pretext task from raw $x$ | fine-tune, head, or prompt | compute-hungry; data quality matters | | foundation models (now) | web-scale self-supervision | one model, many tasks | cost, opacity, alignment | ## Parameter-efficient adaptation Fine-tuning as the standard references describe it updates the borrowed weights. Once a foundation model has billions of parameters, updating and _storing_ a full copy per downstream task is untenable, and a family of **parameter-efficient fine-tuning** (PEFT) methods answers it by adapting the model while touching only a tiny fraction of its weights. The dominant method is **LoRA** (low-rank adaptation; Hu et al., 2021, _ICLR_). Freeze the pretrained weight matrix $W_0 \in \mathbb{R}^{d \times k}$ entirely and learn a low-rank correction $\Delta W = B A$ with $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$, and $r \ll \min(d,k)$, so the adapted layer computes $$ h = W_0 x + \Delta W\, x = W_0 x + B A\, x . $$ Only $A$ and $B$ are trained — $r(d+k)$ parameters instead of $dk$. For a $d = k = 4096$ layer and rank $r = 8$, that is $8 \times 8192 = 65{,}536$ trainable weights against $4096^2 \approx 16.8$ million, a $256\times$ reduction, and the correction folds back into $W_0$ at inference so there is no added latency. The underlying assumption is the same manifold intuition from the companion lesson: the update a downstream task needs lies in a low-dimensional subspace, so a low-rank $\Delta W$ suffices. Two neighbors round out the family. **Adapters** (Houlsby et al., 2019, _ICML_) insert small bottleneck layers between the frozen blocks and train only those; **prompt / prefix tuning** (Lester et al., 2021, _EMNLP_) freezes the entire model and learns a handful of continuous "soft prompt" vectors prepended to the input — the logical end of the freeze boundary, where _no_ model weight moves and adaptation lives entirely in the input. All three share LoRA's economics: one frozen backbone, a few megabytes of task-specific parameters each, swappable at serving time. ## Takeaways - **Transfer reuses a learned encoder** $f_\theta$ on a new task, paying the pretraining cost once. **Feature extraction** freezes it and trains only a head (cheap, overfits little, cacheable); **fine-tuning** unfreezes some layers at a small rate (adapts more, costs more). Choose by target size and source–target similarity. - **Features run generic-to-specific with depth.** Low layers (edges, textures) transfer; high layers (source-specific detectors) do not. The freeze boundary sits at the crossover — measured by Yosinski et al. — and discriminative learning rates are its continuous form. - **Fine-tune by moving little:** converge the head first so a random gradient cannot wreck the base, keep $\eta$ one to two orders below from-scratch, and scale it down with depth. - **Domain adaptation** aligns $p_S(h) \approx p_T(h)$ when the task is shared but the input distribution shifts and the target is unlabeled — via a divergence penalty or, adversarially, a domain classifier the encoder learns to fool. - **Transfer can hurt.** Negative transfer occurs when source features are wrong for the target and the target is too small to correct them; the deciding variable is the mismatch in $p(y\mid x)$, not merely the input shift. - **Self-supervised pretraining** manufactures labels from raw data (masked prediction, next-token, contrastive agreement), yielding **foundation models**: one representation, learned once on web-scale data, adapted to many tasks — now most cheaply through parameter-efficient methods like LoRA. [^gf-transfer]: **Goodfellow**, _Deep Learning_, §15.2 — Transfer Learning and Domain Adaptation: reusing a source-task representation on a target task, and aligning marginals when only the input distribution shifts. [^chollet-finetune]: **Chollet**, _Deep Learning with Python_, §5.3 — Using a Pretrained Convnet: feature extraction (freeze) versus fine-tuning (unfreeze top blocks at a small learning rate), chosen by target-set size and similarity. [^chollet-cache]: **Chollet**, _Deep Learning with Python_, §5.3 — Feature extraction without data augmentation: run the frozen convolutional base over the data once, cache the extracted feature maps, and train a small densely connected classifier on the cached features. [^chollet-order]: **Chollet**, _Deep Learning with Python_, §5.3 — Fine-tuning: the pretrained base is unfrozen only after the newly added head has been trained, because otherwise the large gradient from a randomly initialized head propagates through the network and destroys the learned representations; fine-tuning then proceeds at a low learning rate. [^gf-selfsup]: **Goodfellow**, _Deep Learning_, Ch. 15 / §15.1 — Representation Learning: pretraining a representation on an unsupervised objective that models $p(x)$ so the learned features transfer to supervised tasks; the value of a pretext objective is that solving it requires capturing the structure of the data distribution.