Representation Learning

A learning algorithm never sees the world; it sees a representation of it. A $28 \times 28$ digit is a point in $R^{784}$ , an utterance is a waveform, a sentence is a sequence of token ids, and the difficulty of every downstream task is set less by the task than by the coordinates the input arrives in. The thesis of this lesson, and of deep learning as a whole, is that the representation is itself learnable, and that a good one is characterized by three concrete properties.

The entire content of the field's progress is in what makes one $f_{θ}$ better than another. We isolate three answers (disentanglement, distributedness, and manifold structure) and then show how they are reused across tasks.¹

What makes a good representation

Disentangling the factors of variation

Natural data is generated by a small number of independent causes. A photograph of a face is determined by identity, pose, expression, lighting direction, and camera distance; these vary independently in the world but arrive entangled in pixel space, where changing the lighting alters every coordinate at once. A good representation undoes the entanglement.

Formally, disentanglement is a statement about the Jacobian of the encoder composed with the generator. Write $h = f_{θ} (g (z))$ ; the encoder disentangles when

\frac{\partial h _{j}}{\partial z _{i}} \approx 0 for i \neq = π (j), \frac{\partial h _{π (j)}}{\partial z _{i}} \neq = 0,

for some assignment $π$ of factors to coordinates: the Jacobian is (up to permutation) block-diagonal. Contrast pixel space, where $\partial x / \partial z_{light}$ is dense: every pixel responds to the light.

Factors of variation disentangled: each axis of the learned grid moves one independent factor (pose, lighting) while the other is held fixed.

The practical consequence: when factors are separated, the task-relevant ones are read off by a linear classifier and the nuisance ones are ignored — the very invariance a deep net needs to learn. We return to disentanglement as the explicit objective of variational autoencoders.

Distributed representations and the exponential advantage

How a representation uses its dimensions matters as much as what it encodes. A symbolic or one-hot code uses one active unit per concept: $n$ units distinguish $n$ concepts, and nothing is shared between them. A distributed code lets concepts be patterns of activity across units, so $n$ units carve the input space into exponentially many regions.

The advantage is a counting argument. Each binary feature $h_{j} \in {0, 1}$ is a hyperplane splitting the input space in two; $n$ features in general position partition $R^{d}$ into a number of regions that grows as a polynomial of degree $d$ in $n$ ,

# regions = i = 0 \sum d (i n) = O (n^{d}) (n ≫ d),

and when $n \leq d$ this reaches the full $2^{n}$ . Each region is a distinct concept the representation can label differently — so $n$ features distinguish exponentially more configurations than the $n$ a one-hot code would manage.

One-hot versus distributed:

n

units name

n

concepts, but

n

bits as a pattern name

2^{n}

— an exponential gap.

Property	One-hot / local	Distributed
Concepts per $n$ units	$n$	up to $2^{n}$
Sharing between concepts	none	features reused across concepts
Generalization to novel combos	none (unseen $=$ off)	interpolates in feature space
Statistical efficiency	one example per concept	examples share statistical strength
Example	a lookup table, word ids	word embeddings, hidden units

The deeper consequence is generalization. A one-hot model has no representation for an unseen concept; a distributed model places it at a new pattern of already-trained features and so generalizes to configurations absent from the training set: the statistical core of why neural features transfer.²

Smoothness and manifold structure

The third property is geometric. Useful representations are smooth: nearby inputs map to nearby codes, and the encoder respects the low-dimensional surface the data actually occupies. This is the manifold hypothesis, which deserves its own section.

The manifold hypothesis

High-dimensional natural data does not fill its ambient space. Sample a $32 \times 32$ RGB image uniformly at random (choose each of the $3072$ values independently) and you get television static with probability arbitrarily close to one; you will never stumble onto a face, a digit, or any natural image. The set of natural images is a vanishingly thin subset of $R^{3072}$ , and it is structured: it concentrates near a smooth surface of far lower dimension.³

The intrinsic dimension $k$ is the number of independent factors of variation: a fixed digit under translation, rotation, and stroke-width has an intrinsic dimension of a handful, even though it lives in $R^{784}$ . Three observations support the hypothesis: uniform samples are never natural; natural data admits smooth deformations (slide, rotate, dim the light) that trace paths staying on the data; and nearest-neighbor interpolation between two examples in pixel space leaves the manifold (yielding ghosts), whereas interpolation in a learned code stays on it.

The manifold hypothesis: 3D data lies on a curved 2D sheet, and representation learning finds coordinates that unroll it to a flat plane.

Representation learning, geometrically, is the search for a chart on $M$ : a map $f_{θ} : M \to R^{k}$ assigning intrinsic coordinates so that geodesic distance along the sheet becomes ordinary Euclidean distance in code space. The encoder's Jacobian $J = \partial f_{θ} / \partial x$ should have rank $k$ at each point, with its row space aligned to the tangent plane $T_{x} M$ and a null space spanning the off-manifold directions the representation is free to ignore. This is the precise version of the coordinate-change argument from the machine-learning refresher: on raw pixels a digit shifted by a few columns lands far from itself; on manifold coordinates the shift is a short move along one tangent direction.

This is why depth helps: each layer is a partial flattening, and the composition straightens a manifold that no single affine map could.

Greedy layer-wise pretraining

Before end-to-end supervised training of deep nets was reliable (roughly 2006–2011), gradients through many layers vanished or exploded, and a deep stack initialized randomly would not train. The breakthrough that briefly made depth work was greedy layer-wise unsupervised pretraining: build the representation one layer at a time, each layer trained unsupervised to model the output of the layer below, then fine-tune the whole stack with labels.⁴

Algorithm:

\textsc{GreedyPretrain}(\mathcal{D}, L)

— layer-wise unsupervised init, then supervised fine-tune

1
$h^{(0)} \gets x$ for each $x$ in $\mathcal{D}$
raw inputs are layer-0 codes
2
for $\ell \gets 1$ to $L$ do
3
fit unsupervised model $f_\ell$ to codes $h^{(\ell-1)}$
RBM or autoencoder, layers below frozen
4
$h^{(\ell)} \gets f_\ell(h^{(\ell-1)})$
encode upward, freeze, move on
5
add supervised head $g$ on top of $h^{(L)}$
6
fine-tune all of $f_1, \dots, f_L, g$ jointly by gradient descent
backprop end-to-end
7
return the trained network

It mattered for two reasons, both now understood as side effects of better optimization and regularization rather than a deep necessity.

Why it helped (then)	Mechanism	Why it faded
Optimization	unsupervised init lands weights in a basin from which backprop descends	ReLU, careful init, BatchNorm let deep nets train from random init
Regularization	features reflect $p (x)$ , biasing toward structure that aids $p (y ∣ x)$	with enough labeled data the supervised signal alone finds better features
Data efficiency	uses abundant unlabeled $x$ when labels are scarce	when labels are plentiful, unsupervised init is wasted effort

A worked count: how far the distributed advantage reaches

To put numbers on the exponential gap: suppose an encoder produces $n = 20$ binary features. A one-hot code over those $20$ units names exactly $20$ concepts. A distributed code, if the features are in general position in an input space of dimension $d \geq 20$ , names up to $2^{20} \approx 1.05 \times 1 0^{6}$ : fifty thousand times more, from the same $20$ numbers.

The caveat is the phrase general position in dimension $d \geq n$ . When the features live in a lower-dimensional input, $d < n$ , the region count is capped by the partition formula from above,

# regions = i = 0 \sum d (i n),

which is the number of cells $n$ hyperplanes cut $R^{d}$ into. With $n = 20$ features but only $d = 3$ input dimensions, this is $(0 20) + (1 20) + (2 20) + (3 20) = 1 + 20 + 190 + 1140 = 1351$ regions, not $2^{20}$ . The distributed code still far exceeds the one-hot count of $20$ , but the ambient dimension, not the unit count alone, sets the ceiling. This is why width and depth both matter: depth composes features so that later layers act as if they see a higher effective dimension, lifting the cap toward the full $2^{n}$ .

Measuring and provably-limiting disentanglement

Goodfellow states disentanglement as a Jacobian property and leaves it there; the decade since sharpened both how to pursue it and how to measure it, and also established a hard limit.

The pursuit turned disentanglement into an explicit training objective. β-VAE (Higgins et al., 2017, ICLR) reweights the KL term of the variational autoencoder by a factor $β > 1$ , pressuring the posterior toward a factorized prior so that each latent coordinate captures one factor; FactorVAE (Kim & Mnih, 2018, ICML) adds a total-correlation penalty that directly punishes statistical dependence between coordinates. Both make the block-diagonal Jacobian of the definition an explicit optimization target.

The measurement side produced scores that operationalize the definition on synthetic data whose true factors $z$ are known: the MIG (mutual-information gap, Chen et al., 2018, NeurIPS) measures how much more one latent coordinate tells you about a factor than the runner-up coordinate does, and the DCI triple (disentanglement, completeness, informativeness; Eastwood & Williams, 2018, ICLR) reads the same idea off the weights of a probe regressor. A high MIG means one factor maps cleanly to one coordinate — exactly $\partial h_{j} / \partial z_{i} \approx 0$ for $i \neq = π (j)$ made into a number.

The limit is the important part. Locatello et al. (2019, ICML, best-paper) trained thousands of models across six methods and proved that unsupervised disentanglement is impossible without inductive biases or supervision: for any generative process there exist infinitely many equally-good encoders whose latents are entangled, so no purely unsupervised objective can prefer the disentangled one. Their experiments bore it out — random seeds, not the choice of method, dominated the disentanglement scores. The practical reading is that disentanglement is real and useful, but it requires priors (the factorized prior, the augmentations, a few labels); it does not come from the data alone. That is the same lesson the manifold hypothesis teaches geometrically: the structure a good representation exposes is structure you must assume, then verify.

Takeaways

A good representation is defined by three properties. It disentangles the independent factors of variation (a block-diagonal encoder Jacobian), spends its dimensions as a distributed code (patterns of units, not one unit per concept), and respects the low-dimensional manifold the data occupies.
Distributed codes generalize by construction. $n$ features in general position name up to $2^{n}$ concepts against a one-hot code's $n$ , and an unseen concept lands at a new pattern of already-trained features rather than an unlit unit — the statistical root of why neural features transfer.
Natural data lives on a thin manifold. Uniform samples are never natural; representation learning is the search for intrinsic coordinates that flatten the curved data sheet, turning a nonlinear input-space boundary into a linear one in code space, which is why depth (repeated partial flattening) helps.
Greedy layer-wise pretraining was the first practical application of these ideas: stage-wise unsupervised init that briefly made depth trainable. Better activations, initialization, normalization, and optimizers superseded it — but its core move, pretrain a representation on cheap data then specialize, migrated into transfer learning and self-supervision.
Disentanglement is not free. Metrics (MIG, DCI) can measure it, objectives (β-VAE, FactorVAE) can pursue it, but Locatello et al. proved no unsupervised method can guarantee it without inductive biases or labels.

This continues in Transfer Learning, which turns these three properties into the engineering of feature reuse: feature extraction and fine-tuning, domain adaptation, and the modern arc to self-supervised foundation models.

Goodfellow, Deep Learning, Ch. 15 — Representation Learning: the framing of a good representation around disentangled factors, distributed codes, and manifold structure. ↩
Goodfellow, Deep Learning, §15.4 — Distributed Representations: the region-counting argument by which $n$ features distinguish exponentially more configurations than a one-hot code, and why this drives generalization. ↩
Goodfellow, Deep Learning, §5.11.3 / §15.6 — Manifold Learning: natural data concentrates near a low-dimensional manifold, and representation learning finds intrinsic coordinates on it. ↩
Goodfellow, Deep Learning, §15.1 — Greedy Layer-Wise Unsupervised Pretraining: the stage-wise unsupervised init that briefly made depth trainable, later superseded by better optimization. ↩