---
title: Representation Learning
module: Practical Deep Learning
moduleNumber: 9
lessonNumber: 3
order: 903
summary: >
  A good representation makes a hard task easy by changing coordinates: it
  disentangles the factors of variation, spends its bits as a distributed code,
  and respects the low-dimensional manifold the data lives on. We make those
  three properties precise, recover the manifold hypothesis, and close on the
  first method that turned them into training practice — greedy layer-wise
  unsupervised pretraining — before the sequel picks up how the field learned to
  reuse those features.
topics: [Practical Deep Learning]
sources:
  - book: Goodfellow
    ref: "Ch. 15 — Representation Learning"
  - book: Goodfellow
    ref: "§5.11.3 Manifold Learning; §15.1 Greedy Layer-Wise Unsupervised Pretraining"
  - book: Chollet
    ref: "§5.3 — Using a Pretrained Convnet (feature extraction, fine-tuning)"
---

A learning algorithm never sees the world; it sees a representation of it. A
$28\times28$ digit is a point in $\mathbb{R}^{784}$, an utterance is a waveform, a
sentence is a sequence of token ids, and the difficulty of every downstream task
is set less by the task than by the coordinates the input arrives in. The thesis of
this lesson, and of [deep learning](/deep-learning/foundations/what-is-deep-learning)
as a whole, is that the representation is itself learnable, and that a _good_ one
is characterized by three concrete properties.

> **Definition (Representation).** A representation of an input $x$ is a vector
> $h = f_\theta(x) \in \mathbb{R}^k$ produced by a learned encoder $f_\theta$,
> such that a simple readout (typically a linear map $w^\top h + b$) solves the
> downstream task. Learning $h$ is **representation learning**; learning $w$ on
> top of a fixed $h$ is the downstream task.

The entire content of the field's progress is in what makes one $f_\theta$ better
than another. We isolate three answers (_disentanglement_, _distributedness_, and
_manifold structure_) and then show how they are reused across tasks.[^gf-repr]

## What makes a good representation

### Disentangling the factors of variation

Natural data is generated by a small number of independent causes. A photograph of
a face is determined by identity, pose, expression, lighting direction, and camera
distance; these vary independently in the world but arrive entangled in
pixel space, where changing the lighting alters every coordinate at once. A good
representation undoes the entanglement.

> **Definition (Factors of variation).** The underlying, often unobserved,
> explanatory variables $\mathbf{z} = (z_1, \dots, z_m)$ whose joint setting
> determines the observation $x = g(\mathbf{z})$ through some generative process
> $g$. A representation **disentangles** them when each learned coordinate $h_j$
> recovers one factor (or a sparse group), so that varying one $z_i$ moves few
> coordinates of $h$ and leaves the rest fixed.

Formally, disentanglement is a statement about the Jacobian of the encoder
composed with the generator. Write $h = f_\theta(g(\mathbf{z}))$; the encoder
disentangles when

$$
\frac{\partial h_j}{\partial z_i} \approx 0 \quad\text{for } i \neq \pi(j),
\qquad
\frac{\partial h_{\pi(j)}}{\partial z_i} \neq 0,
$$

for some assignment $\pi$ of factors to coordinates: the Jacobian is (up to
permutation) block-diagonal. Contrast pixel space, where $\partial x / \partial
z_{\text{light}}$ is dense: every pixel responds to the light.

$$
% caption: Factors of variation disentangled: each axis of the learned grid moves
% one independent factor (pose, lighting) while the other is held fixed.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  cell/.style={draw, black, minimum size=12mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % 3x3 grid of face glyphs: columns = pose, rows = lighting
  \foreach \c in {0,1,2} \foreach \r in {0,1,2} {
    \node[cell] (g\c\r) at (\c*1.4, \r*1.4) {};
  }
  % draw a stylized face in each cell: head circle, two eyes, pose shifts eyes, lighting shifts a shading dot
  \foreach \c in {0,1,2} \foreach \r in {0,1,2} {
    \begin{scope}[shift={(\c*1.4, \r*1.4)}]
      \draw[black] (0,0) circle (4.2mm);
      % eyes shift right with pose (column c)
      \fill[black] ({-0.18+0.12*\c},0.12) circle (0.9pt);
      \fill[black] ({0.18+0.12*\c},0.12) circle (0.9pt);
      % lighting marker rises with row r
      \fill[green] (0.18,{-0.32+0.18*\r}) circle (1.1pt);
    \end{scope}
  }
  % axis arrows
  \draw[->, acc, very thick] (-0.95,-0.95) -- (3.35,-0.95)
    node[midway, below, text=acc] {\texttt{pose} ($h_1$)};
  \draw[->, acc, very thick] (-0.95,-0.95) -- (-0.95,3.35)
    node[midway, above, rotate=90, text=acc] {\texttt{lighting} ($h_2$)};
\end{tikzpicture}
$$

The practical consequence: when factors are separated, the task-relevant ones are read
off by a linear classifier and the nuisance ones are ignored — the very invariance a
deep net needs to learn. We return to disentanglement as the
explicit objective of [variational autoencoders](/deep-learning/generative-models/variational-autoencoders).

### Distributed representations and the exponential advantage

How a representation uses its dimensions matters as much as what it encodes. A
**symbolic** or **one-hot** code uses one active unit per concept: $n$ units
distinguish $n$ concepts, and nothing is shared between them. A **distributed**
code lets concepts be patterns of activity across units, so $n$ units carve the
input space into exponentially many regions.

> **Definition (Distributed representation).** A representation in which each input
> is described by a _pattern_ of values across many units, and each unit
> participates in describing many inputs, as opposed to a **one-hot** (local)
> representation, where exactly one unit fires per input. With $n$ binary units a
> one-hot code names $n$ concepts; a distributed code names up to $2^n$.

The advantage is a counting argument. Each binary feature $h_j \in \{0,1\}$ is a
hyperplane splitting the input space in two; $n$ features in general position
partition $\mathbb{R}^d$ into a number of regions that grows as a polynomial of
degree $d$ in $n$,

$$
\#\text{regions} \;=\; \sum_{i=0}^{d} \binom{n}{i}
\;=\; O\!\parens{n^{d}} \quad (n \gg d),
$$

and when $n \le d$ this reaches the full $2^n$. Each region is a distinct concept
the representation can label differently — so $n$ features distinguish
_exponentially_ more configurations than the $n$ a one-hot code would manage.

$$
% caption: One-hot versus distributed: $n$ units name $n$ concepts, but $n$ bits as
% a pattern name $2^n$ — an exponential gap.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % cell pitch 0.9, cell box 6.5mm; on-bit fill is a centered 5.5mm square
  % ---- one-hot block (left): one lit cell per row ----
  \node[font=\small] at (0.9,3.6) {one-hot: $n$ concepts};
  \foreach \row in {0,1,2,3} {
    \foreach \col in {0,1,2} {
      \node[draw, black, minimum size=6.5mm] at (\col*0.9, -\row*0.9+2.7) {};
    }
  }
  \foreach \row/\col in {0/1, 1/0, 2/2, 3/1} {
    \draw[acc, very thick, fill=acc!15] (\col*0.9-0.275, -\row*0.9+2.7-0.275) rectangle ++(0.55,0.55);
  }
  \node[text=acc, font=\small] at (0.9,-1.0) {$n$ units $=$ $n$ codes};
  % ---- distributed block (right): each row a distinct bit pattern ----
  \begin{scope}[xshift=5.6cm]
    \node[font=\small] at (0.9,3.6) {distributed: $2^n$ concepts};
    \foreach \row in {0,1,2,3} {
      \foreach \col in {0,1,2} {
        \node[draw, black, minimum size=6.5mm] at (\col*0.9, -\row*0.9+2.7) {};
      }
    }
    % patterns: row0=001, row1=110, row2=011, row3=101
    \foreach \row/\col in {0/2, 1/0, 1/1, 2/1, 2/2, 3/0, 3/2} {
      \draw[green, very thick, fill=green!15] (\col*0.9-0.275, -\row*0.9+2.7-0.275) rectangle ++(0.55,0.55);
    }
    \node[text=green, font=\small] at (0.9,-1.0) {$n$ bits $=$ $2^n$ codes};
  \end{scope}
\end{tikzpicture}
$$

| Property | One-hot / local | Distributed |
| --- | --- | --- |
| Concepts per $n$ units | $n$ | up to $2^n$ |
| Sharing between concepts | none | features reused across concepts |
| Generalization to novel combos | none (unseen $=$ off) | interpolates in feature space |
| Statistical efficiency | one example per concept | examples share statistical strength |
| Example | a lookup table, word ids | word embeddings, hidden units |

The deeper consequence is generalization. A one-hot model has no representation for
an unseen concept; a distributed model places it at
a _new pattern_ of already-trained features and so generalizes to configurations
absent from the training set: the statistical core of why neural features transfer.[^gf-distributed]

### Smoothness and manifold structure

The third property is geometric. Useful representations are **smooth**: nearby
inputs map to nearby codes, and the encoder respects the low-dimensional surface
the data actually occupies. This is the manifold hypothesis, which deserves its own
section.

## The manifold hypothesis

High-dimensional natural data does not fill its ambient space. Sample a $32\times32$
RGB image uniformly at random (choose each of the $3072$ values independently) and
you get television static with probability arbitrarily close to one; you will
_never_ stumble onto a face, a digit, or any natural image. The set of natural
images is a vanishingly thin subset of $\mathbb{R}^{3072}$, and it is _structured_:
it concentrates near a smooth surface of far lower dimension.[^gf-manifold]

> **Definition (Manifold hypothesis).** The probability mass of natural data
> $p(x)$ over $\mathbb{R}^d$ concentrates near a low-dimensional manifold
> $\mathcal{M} \subset \mathbb{R}^d$ of intrinsic dimension $k \ll d$. Locally,
> $\mathcal{M}$ looks like $\mathbb{R}^k$: every $x \in \mathcal{M}$ has a
> neighborhood diffeomorphic to a $k$-dimensional patch, with a tangent space
> $T_x\mathcal{M}$ spanned by the directions of allowed variation.

The intrinsic dimension $k$ is the number of independent factors of variation: a
fixed digit under translation, rotation, and stroke-width has an intrinsic
dimension of a handful, even though it lives in $\mathbb{R}^{784}$. Three
observations support the hypothesis: uniform samples are never natural;
natural data admits smooth deformations (slide, rotate, dim the light) that trace
_paths_ staying on the data; and nearest-neighbor interpolation between two examples
in pixel space leaves the manifold (yielding ghosts), whereas interpolation in a
learned code stays on it.

$$
% caption: The manifold hypothesis: 3D data lies on a curved 2D sheet, and
% representation learning finds coordinates that unroll it to a flat plane.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % ---- left: swiss roll embedded in 3D ----
  \begin{scope}[xshift=0cm]
    % spiral cross-section curve (the rolled sheet), drawn as a thick gray spiral
    \draw[black, very thick] plot[domain=0:6.2, samples=80, variable=\t]
      ({0.32*\t*cos(\t r)}, {0.32*\t*sin(\t r)*0.55});
    % depth offset copy to suggest a sheet with width
    \draw[black, very thick] plot[domain=0:6.2, samples=80, variable=\t]
      ({0.32*\t*cos(\t r)}, {0.32*\t*sin(\t r)*0.55 + 0.9});
    % data points sampled along the spiral, colored by position along the roll
    \foreach \t in {0.6,1.4,2.2,3.0,3.8,4.6,5.4,6.0} {
      \fill[acc] ({0.32*\t*cos(\t r)}, {0.32*\t*sin(\t r)*0.55 + 0.45}) circle (1.7pt);
    }
    \node[font=\small] at (0,-2.4) {data on a curved sheet in 3D};
  \end{scope}
  % ---- unroll arrow ----
  \draw[->, green, very thick] (2.6,0) -- (4.4,0)
    node[midway, above, text=green] {unroll};
  \node[font=\footnotesize, text=green] at (3.5,-0.55) {learn $h$};
  % ---- right: flattened plane with same points in a line ----
  \begin{scope}[xshift=6.4cm]
    \draw[black] (-0.3,-1.3) rectangle (3.3,1.3);
    \foreach \i/\x in {0/0.1,1/0.5,2/0.9,3/1.3,4/1.7,5/2.1,6/2.5,7/2.9} {
      \fill[acc] (\x,0) circle (1.7pt);
    }
    \draw[->, thick] (-0.3,-1.6) -- (3.3,-1.6) node[right] {$h_1$};
    \draw[->, thick] (-0.6,-1.3) -- (-0.6,1.3) node[above] {$h_2$};
    \node[font=\small] at (1.5,-2.4) {\texttt{flat manifold coordinates}};
  \end{scope}
\end{tikzpicture}
$$

Representation learning, geometrically, is the search for a chart on $\mathcal{M}$:
a map $f_\theta : \mathcal{M} \to \mathbb{R}^k$ assigning intrinsic coordinates so
that geodesic distance _along the sheet_ becomes ordinary Euclidean distance in code
space. The encoder's Jacobian $J = \partial f_\theta / \partial x$ should have rank
$k$ at each point, with its row space aligned to the tangent plane $T_x\mathcal{M}$
and a null space spanning the off-manifold directions the representation is free to
ignore. This is the precise version of the coordinate-change argument from the
[machine-learning refresher](/deep-learning/foundations/machine-learning-refresher):
on raw pixels a digit shifted by a few columns lands far from itself; on manifold
coordinates the shift is a short move along one tangent direction.

> **Theorem (Linear separability after flattening).** Suppose classes occupy
> disjoint connected regions of a $k$-manifold $\mathcal{M}$ and $f_\theta$ is a
> diffeomorphism that maps $\mathcal{M}$ onto a convex set in $\mathbb{R}^k$ with
> each class image convex. Then there exists a linear classifier $w^\top h + b$
> separating the classes in code space, even when no hyperplane separates them in
> the input $\mathbb{R}^d$.

> **Proof.** Convex disjoint sets in $\mathbb{R}^k$ are separated by a hyperplane
> (separating-hyperplane theorem). Let $w^\top h + b = 0$ be that hyperplane in
> code space. Pulling back through $h = f_\theta(x)$, the decision region
> $\{x : w^\top f_\theta(x) + b > 0\}$ is a curved (generally non-convex) region
> in the input that no _input-space_ hyperplane need match. The flattening done by
> $f_\theta$ converts a nonlinear input-space boundary into a linear code-space
> one. $\qed$

This is why depth helps: each layer is a partial flattening, and the composition
straightens a manifold that no single affine map could.

## Greedy layer-wise pretraining

Before end-to-end supervised training of deep nets was reliable (roughly
2006–2011), gradients through many layers vanished or exploded, and a deep stack
initialized randomly would not train. The breakthrough that briefly made depth work
was **greedy layer-wise unsupervised pretraining**: build the representation one
layer at a time, each layer trained _unsupervised_ to model the output of the layer
below, then fine-tune the whole stack with labels.[^gf-greedy]

> **Definition (Greedy layer-wise pretraining).** Train a deep encoder in stages.
> Stage $1$ fits an unsupervised model (an RBM or autoencoder) to the raw input
> $x$, producing features $h^{(1)} = f_1(x)$. Stage $\ell$ freezes the layers below
> and fits an unsupervised model to $h^{(\ell-1)}$, producing $h^{(\ell)}$. After
> all layers are initialized this way, a supervised head is added and the entire
> network is **fine-tuned** end-to-end with backpropagation.

```algorithm
caption: $\textsc{GreedyPretrain}(\mathcal{D}, L)$ — layer-wise unsupervised init, then supervised fine-tune
$h^{(0)} \gets x$ for each $x$ in $\mathcal{D}$ // raw inputs are layer-0 codes
for $\ell \gets 1$ to $L$ do
  fit unsupervised model $f_\ell$ to codes $h^{(\ell-1)}$ // RBM or autoencoder, layers below frozen
  $h^{(\ell)} \gets f_\ell(h^{(\ell-1)})$ // encode upward, freeze, move on
add supervised head $g$ on top of $h^{(L)}$
fine-tune all of $f_1, \dots, f_L, g$ jointly by gradient descent // backprop end-to-end
return the trained network
```

It mattered for two reasons, both now understood as side effects of better
optimization and regularization rather than a deep necessity.

| Why it helped (then) | Mechanism | Why it faded |
| --- | --- | --- |
| Optimization | unsupervised init lands weights in a basin from which backprop descends | ReLU, careful init, BatchNorm let deep nets train from random init |
| Regularization | features reflect $p(x)$, biasing toward structure that aids $p(y\mid x)$ | with enough labeled data the supervised signal alone finds better features |
| Data efficiency | uses abundant unlabeled $x$ when labels are scarce | when labels are plentiful, unsupervised init is wasted effort |

> **Remark (Why it faded).** Greedy pretraining was a fix for a broken
> optimizer. Once [ReLU activations](/deep-learning/neural-networks/activation-functions),
> [variance-preserving initialization](/deep-learning/optimization/initialization),
> [normalization](/deep-learning/regularization/normalization), and
> [adaptive optimizers](/deep-learning/optimization/momentum-and-adaptive-methods)
> made deep stacks trainable from scratch, the unsupervised stage became
> unnecessary _for vision and supervised tasks_. Its core idea — pretrain a
> representation on cheap data, then specialize — persisted, migrating into
> transfer learning and self-supervision.

### A worked count: how far the distributed advantage reaches

To put numbers on the exponential gap: suppose an encoder produces $n = 20$ binary
features. A one-hot code over those
$20$ units names exactly $20$ concepts. A distributed code, if the features are in
general position in an input space of dimension $d \ge 20$, names up to $2^{20}
\approx 1.05 \times 10^6$: fifty thousand times more, from the same $20$ numbers.

The caveat is the phrase "general position in dimension $d \ge n$." When the features
live in a lower-dimensional input, $d < n$, the region count is capped by the
partition formula from above,

$$
\#\text{regions} \;=\; \sum_{i=0}^{d} \binom{n}{i},
$$

which is the number of cells $n$ hyperplanes cut $\mathbb{R}^d$ into. With $n = 20$
features but only $d = 3$ input dimensions, this is $\binom{20}{0} + \binom{20}{1} +
\binom{20}{2} + \binom{20}{3} = 1 + 20 + 190 + 1140 = 1351$ regions, not $2^{20}$. The
distributed code still far exceeds the one-hot count of $20$, but the ambient dimension,
not the unit count alone, sets the ceiling. This is why width _and_ depth both matter:
depth composes features so that later layers act as if they see a higher effective
dimension, lifting the cap toward the full $2^n$.

## Measuring and provably-limiting disentanglement

Goodfellow states disentanglement as a Jacobian property and leaves it there; the
decade since sharpened both how to _pursue_ it and how to _measure_ it, and also
established a hard limit.

The pursuit turned disentanglement into an explicit training objective. **β-VAE**
(Higgins et al., 2017, _ICLR_) reweights the KL term of the
[variational autoencoder](/deep-learning/generative-models/variational-autoencoders)
by a factor $\beta > 1$, pressuring the posterior toward a factorized prior so that
each latent coordinate captures one factor; **FactorVAE** (Kim & Mnih, 2018, _ICML_)
adds a total-correlation penalty that directly punishes statistical dependence between
coordinates. Both make the block-diagonal Jacobian of the definition an explicit
optimization target.

The measurement side produced scores that operationalize the definition on synthetic
data whose true factors $\mathbf{z}$ are known: the **MIG** (mutual-information gap,
Chen et al., 2018, _NeurIPS_) measures how much more one latent coordinate tells you about
a factor than the runner-up coordinate does, and the **DCI** triple (disentanglement,
completeness, informativeness; Eastwood & Williams, 2018, _ICLR_) reads the same idea
off the weights of a probe regressor. A high MIG means one factor maps cleanly to one
coordinate — exactly $\partial h_j/\partial z_i \approx 0$ for $i \neq \pi(j)$ made
into a number.

The limit is the important part. Locatello et al. (2019, _ICML_, best-paper) trained
thousands of models across six methods and proved that **unsupervised disentanglement
is impossible without inductive biases or supervision**: for any generative process
there exist infinitely many equally-good encoders whose latents are entangled, so no
purely unsupervised objective can prefer the disentangled one. Their experiments bore
it out — random seeds, not the choice of method, dominated the disentanglement scores.
The practical reading is that disentanglement is real and useful, but it requires
priors (the factorized prior, the augmentations, a few labels); it does not come from
the data alone. That is the same lesson the manifold hypothesis teaches geometrically: the
structure a good representation exposes is structure you must assume, then verify.

## Takeaways

- **A good representation is defined by three properties.** It _disentangles_ the
  independent factors of variation (a block-diagonal encoder Jacobian), spends its
  dimensions as a _distributed_ code (patterns of units, not one unit per concept),
  and respects the low-dimensional _manifold_ the data occupies.
- **Distributed codes generalize by construction.** $n$ features in general position
  name up to $2^n$ concepts against a one-hot code's $n$, and an unseen concept lands
  at a new pattern of already-trained features rather than an unlit unit — the
  statistical root of why neural features transfer.
- **Natural data lives on a thin manifold.** Uniform samples are never natural;
  representation learning is the search for intrinsic coordinates that flatten the
  curved data sheet, turning a nonlinear input-space boundary into a linear one in
  code space, which is why depth (repeated partial flattening) helps.
- **Greedy layer-wise pretraining was the first practical application** of these ideas: stage-wise
  unsupervised init that briefly made depth trainable. Better activations,
  initialization, normalization, and optimizers superseded it — but its core move,
  _pretrain a representation on cheap data then specialize_, migrated into transfer
  learning and self-supervision.
- **Disentanglement is not free.** Metrics (MIG, DCI) can measure it, objectives
  (β-VAE, FactorVAE) can pursue it, but Locatello et al. proved no _unsupervised_
  method can guarantee it without inductive biases or labels.

This continues in [Transfer Learning](/deep-learning/practical/transfer-learning),
which turns these three properties into the engineering of feature reuse: feature
extraction and fine-tuning, domain adaptation, and the modern arc to self-supervised
foundation models.

[^gf-repr]: **Goodfellow**, _Deep Learning_, Ch. 15 — Representation Learning: the framing of a good representation around disentangled factors, distributed codes, and manifold structure.
[^gf-distributed]: **Goodfellow**, _Deep Learning_, §15.4 — Distributed Representations: the region-counting argument by which $n$ features distinguish exponentially more configurations than a one-hot code, and why this drives generalization.
[^gf-manifold]: **Goodfellow**, _Deep Learning_, §5.11.3 / §15.6 — Manifold Learning: natural data concentrates near a low-dimensional manifold, and representation learning finds intrinsic coordinates on it.
[^gf-greedy]: **Goodfellow**, _Deep Learning_, §15.1 — Greedy Layer-Wise Unsupervised Pretraining: the stage-wise unsupervised init that briefly made depth trainable, later superseded by better optimization.