Autoencoders · study.

An autoencoder is a network trained to copy its input to its output. Stated that baldly the task is absurd: the identity function copies perfectly and learns nothing. The task becomes useful when the identity is forbidden: route the copy through a channel too narrow, too sparse, or too constrained to pass the input verbatim, so the network must instead discover a compressed code from which the input can be approximately rebuilt. The reconstruction is a pretext; the code is what matters.¹

The model factors into two maps. An encoder $f$ compresses the input to a code (or latent representation) $h$ , and a decoder $g$ expands the code back to a reconstruction $\overset{x}{^}$ :

h = f (x), \overset{x}{^} = g (h) = g (f (x)) .

The autoencoder hourglass: encoder

f

compresses

x

to a code

h

, decoder

g

expands it back to

\overset{x}{^}

The training objective averages the reconstruction loss over the data,

f, g min E_{x \sim p_{data}} [∥ x - g (f (x)) ∥_{2}^{2}],

and is optimized by the same minibatch gradient descent that trains any other feedforward net: an autoencoder is a feedforward net whose target is its own input. What distinguishes the variants below is not the loss but the constraint that stops $g \circ f$ from collapsing to the identity. We organize them by that constraint.

variant	constraint on the code	what it learns
undercomplete	$dim h < dim x$ (bottleneck)	a compressing subspace; PCA when linear
sparse	penalty $Ω (h) = λ ∥ h ∥_{1}$	a few active features per input
denoising	reconstruct clean $x$ from corrupted $\tilde{x}$	the data manifold / a score field
contractive	penalty $λ ∥ \partial h / \partial x ∥_{F}^{2}$	a locally flat, noise-robust code

Undercomplete autoencoders

The most direct way to forbid the identity is to make the code smaller than the input. If $h \in R^{k}$ and $x \in R^{d}$ with $k < d$ , then $f$ must discard $d - k$ dimensions' worth of information, and the only way to reconstruct well is to discard the dimensions that matter least. The code is a bottleneck.

An undercomplete autoencoder squeezes input

x \in R^{6}

through a narrow code

h \in R^{2}

and expands it back to

\overset{x}{^} \in R^{6}

The linear case is PCA

When both maps are linear and the loss is squared error, the autoencoder is not merely like principal component analysis: it spans the same subspace. This is the foundational result of the chapter and ties the autoencoder back to the linear factor models that precede it.

The autoencoder recovers the PCA subspace, not necessarily an orthonormal, variance-ordered basis: $W$ can be any matrix whose rows span $U_{k}$ , since $V W$ (and therefore the loss) is invariant to an invertible change of basis $W \mapsto A W$ , $V \mapsto V A^{- 1}$ inside the code. The latent coordinates are fixed only up to a linear transformation. The value of the autoencoder framing is what comes next: make $f$ and $g$ nonlinear and the model learns a curved, $k$ -dimensional manifold rather than a flat $k$ -dimensional subspace, a strict generalization of PCA.²

	linear AE / PCA	nonlinear AE
encoder $f$	$W x$	deep nonlinear net
learned set	a flat $k$ -plane through the origin	a curved $k$ -manifold
code meaning	projection onto principal axes	nonlinear coordinates on the manifold
capacity risk	none (linear)	can memorize if over-powered

That last cell is the catch. A sufficiently powerful nonlinear encoder can route the input through a 1-D code by abusing precision, encoding the index $i$ of a training example as a single real number and having the decoder look it up, copying the data without learning any useful structure. Capacity must be constrained, which the bottleneck does only crudely. The regularized autoencoders below constrain it directly, and in doing so can afford codes that are as large as, or larger than, the input.

Regularized autoencoders

An overcomplete autoencoder has $dim h \geq dim x$ . Without a bottleneck nothing stops it from learning the identity, so we add an explicit penalty $Ω$ to the loss. The general regularized objective is

f, g min E_{x} [L (x, g (f (x))) + Ω (h, x)],

and the choice of $Ω$ decides what kind of structure the code acquires. Three choices dominate.

Sparse autoencoders

A sparse autoencoder penalizes the magnitude of the code, typically with an $L^{1}$ term $Ω (h) = λ ∥ h ∥_{1} = λ \sum_{i} ∣ h_{i} ∣$ . Most code units are then driven to zero on any given input, and each input is explained by a small active subset of an otherwise large dictionary of features.

The penalty admits a clean probabilistic reading. If we treat $h$ as a latent variable with a factorial prior $p (h) \propto \prod_{i} e^{- λ ∣ h_{i} ∣}$ (a Laplace prior, which is sharply peaked at zero), then maximizing the joint log-likelihood $log p (x ∣ h) + log p (h)$ recovers exactly a Gaussian reconstruction term plus the $L^{1}$ penalty:

- log p (h) = λ i \sum ∣ h_{i} ∣ + const, - log p (x ∣ h) \propto ∥ x - g (h) ∥_{2}^{2} .

Sparsity is thus a MAP estimate under a sparse prior, not an arbitrary add-on: the same logic that makes $L^{1}$ regression select few features.³

A sparse code over an overcomplete dictionary: most units are off (light), a few fire (filled) per input, and different inputs recruit different active subsets.

Denoising autoencoders

The denoising autoencoder (DAE) changes the task rather than adding a penalty. Corrupt each input with noise, $\tilde{x} \sim C (\tilde{x} ∣ x)$ , feed the corrupted $\tilde{x}$ to the network, and require it to reconstruct the clean $x$ :

f, g min E_{x \sim p_{data}} E_{\tilde{x} \sim C (\tilde{x} ∣ x)} [∥ x - g (f (\tilde{x})) ∥_{2}^{2}] .

Copying is now impossible, since the input it sees is not the target it must produce, so even an overcomplete DAE cannot learn the identity. To undo the corruption it must learn where clean data plausibly lies and pull a corrupted point back toward it.

A denoising autoencoder maps a corrupted input

\tilde{x} = x + ε

back toward the clean target

x

; it cannot copy, so it learns structure.

The geometry is the payoff. If clean data concentrates near a low-dimensional manifold, corruption with small Gaussian noise $\tilde{x} = x + ε$ , $ε \sim N (0, σ^{2} I)$ , pushes points off the manifold; the optimal denoiser maps them back onto it. Under squared-error loss the reconstruction is the conditional mean $g (f (\tilde{x})) = E [x ∣ \tilde{x}]$ , and a short expansion connects the residual to the score, the gradient of the log data density:

g (f (\tilde{x})) - \tilde{x} \approx σ^{2} \nabla_{\tilde{x}} log p_{data} (\tilde{x}) .

The denoiser's correction vector points up the density gradient, toward higher- probability regions, that is, back toward the manifold. A DAE trained with Gaussian noise is, implicitly, a score estimator: it learns the vector field that flows any point onto the data manifold.⁴

Contractive autoencoders

The contractive autoencoder (CAE) penalizes the sensitivity of the code to its input, the squared Frobenius norm of the encoder's Jacobian:

Ω (h, x) = λ ∥ \frac{\partial f ( x )}{\partial x} ∥_{F}^{2} = λ i, j \sum (\frac{\partial h _{i}}{\partial x _{j}})^{2} .

This pressures the encoder to be locally constant: small moves in $x$ should barely move $h$ . The encoder cannot be constant everywhere — it must still vary enough to reconstruct — so the optimum stays flat in directions that leave the manifold (noise directions, which reconstruction does not need) and varies only along the manifold (the directions that carry data). The Jacobian's singular values collapse except along the tangent space of the data manifold.⁵

penalty target	sparse AE	contractive AE
what is penalized	the code value $∥ h ∥_{1}$	the code's derivative $∥ \partial h / \partial x ∥_{F}^{2}$
effect on code	few units active	units locally insensitive to $x$
manifold reading	sparse dictionary	flat across normal directions

The DAE and CAE converge on the same idea from opposite ends. Denoising forces robustness to noise explicitly, by training against corrupted inputs; the contractive penalty forces it analytically, by shrinking the Jacobian. For small Gaussian noise the two are equivalent to first order (both penalize how much the representation reacts to perturbation), and both, as the next section shows, are really statements about a manifold.

The manifold perspective

The unifying picture behind every variant is the manifold hypothesis: real high-dimensional data (natural images, speech, text) does not fill its ambient space but concentrates near a much lower-dimensional, curved manifold. A $d$ -pixel image lives in $R^{d}$ , yet the set of natural images is a vanishingly thin sheet inside that cube; almost every point in $R^{d}$ is noise.

An autoencoder is a parameterization of such a manifold: the decoder $g$ maps the $k$ -dimensional code space onto a $k$ -dimensional surface in $R^{d}$ , the image ${g (h) : h \in R^{k}}$ , and the encoder $f$ assigns intrinsic coordinates to points near it. Reconstruction error is small exactly for points on the manifold and large for points off it, so minimizing it shapes the manifold to hug the data.

The manifold view of denoising: reconstruction vectors (red) point corrupted points back onto the data manifold (blue), tracing the score

\nabla_{x} log p_{data}

This is why denoising is so natural a training signal. Corruption is a sample of off the manifold; the clean target is on the manifold; learning to reconstruct is learning the projection onto $M$ . The contractive penalty arrives at the same surface analytically, by flattening the encoder normal to $M$ , and the undercomplete bottleneck approximates it crudely with a flat $k$ -plane. All four variants are estimating the same object: the manifold the data lives on.

variant	how it finds the manifold
undercomplete	restricts the code to $k$ dimensions; linear case is the best $k$ -plane
sparse	tiles the manifold with a sparse, overcomplete dictionary of local features
denoising	learns to project off-manifold points back on — estimates the score
contractive	flattens the encoder normal to the manifold, varies only along it

Training a denoising autoencoder

The training loop is ordinary minibatch gradient descent with one extra step: draw fresh corruption for each example on each pass, so the network sees a different $\tilde{x}$ for the same $x$ every epoch and cannot memorize a fixed noise pattern.

Algorithm:

\textsc{TrainDAE}(f_\theta, g_\phi, \mathcal{D}, C, \eta)

— denoising autoencoder training

1
initialize encoder params $\theta$ , decoder params $\phi$
2
repeat
3
sample a minibatch $\{x^{(1)}, \dots, x^{(m)}\} \sim \mathcal{D}$
4
for each example $x^{(i)}$ do
5
draw corruption $\tilde x^{(i)} \sim C(\tilde x \mid x^{(i)})$
e.g. add Gaussian noise / mask pixels
6
$h^{(i)} \gets f_\theta(\tilde x^{(i)})$
encode the corrupted input
7
$\hat x^{(i)} \gets g_\phi(h^{(i)})$
decode toward the clean target
8
$L \gets \frac{1}{m}\sum_i \lVert x^{(i)} - \hat x^{(i)}\rVert_2^2$
reconstruct CLEAN x
9
$(\theta, \phi) \gets (\theta, \phi) - \eta\,\nabla_{\theta,\phi} L$
joint gradient step
10
until converged
11
return $\theta, \phi$

The only line that differs from a vanilla autoencoder is the corruption draw; set $C (\tilde{x} ∣ x) = δ (\tilde{x} - x)$ (no corruption) and the loop reduces to the plain reconstruction objective. Common corruption processes are additive Gaussian noise, masking a random subset of inputs to zero (as in masked pretraining), and salt-and-pepper flips.

The latent space

What does the learned code actually look like? Train an autoencoder on clustered data and the encoder lays the codes out so that the structure reconstruction needs is preserved: inputs that are alike map to nearby codes, distinct groups separate. The bottleneck's geometry becomes a usable feature space: the codes are what you feed a downstream classifier, retrieval index, or visualizer.⁶

A 2-D latent space: each point is a code

h = f (x)

colored by cluster; a good autoencoder maps similar inputs to nearby codes.

This is the bridge to generative modeling. An ordinary autoencoder organizes its latent space but leaves gaps: sample a random $h$ and the decoder may produce garbage, because nothing forced the codes to fill the space according to a known distribution. Imposing such a distribution on the code, so that sampling $h$ and decoding yields a fresh, plausible $x$ , is the step the variational autoencoder takes — turning the autoencoder from a compressor into a generator.

Autoencoders in current practice

The autoencoder outlived its 2006-era role as a deep-net pretrainer and became a staple in three modern settings, each a direct descendant of a variant above.

Masked autoencoding is self-supervised pretraining at scale. The denoising idea — reconstruct a clean signal from a corrupted one — became the dominant pretraining objective when the corruption is masking: BERT masks tokens and predicts them, and the masked autoencoder (MAE) masks a large fraction of image patches and reconstructs the pixels, learning representations that transfer across vision tasks.⁷ Masking to zero is the corruption process $C$ named in the training loop above, scaled to Transformers and very high mask ratios.

The score connection became a generative model. The theorem that a Gaussian-denoising autoencoder's residual estimates $\nabla_{x} log p_{data}$ is the seed of denoising score matching and, through it, of diffusion models: a diffusion model is a denoising autoencoder trained at many noise levels at once, and sampling walks the learned score field back toward the data manifold.⁸ The manifold picture drawn above — reconstruction arrows pointing off-manifold points back on — is literally the reverse diffusion trajectory.

Discrete-code autoencoders compress for downstream generators. The VQ-VAE's learned discrete codebook turned the bottleneck into a compact token language, and modern image and audio generators (including latent diffusion) run their expensive generative model on these autoencoder codes rather than on raw pixels, using the autoencoder purely as a perceptual compressor.⁹ The bottleneck that began as a way to force compression is now the standard front-end that makes large-scale generation affordable.

Takeaways

An autoencoder is encoder $h = f (x)$ plus decoder $\overset{x}{^} = g (h)$ , trained to minimize reconstruction error $∥ x - g (f (x)) ∥^{2}$ . The copy is a pretext; the code $h$ is the point.
An undercomplete autoencoder ( $dim h < dim x$ ) compresses through a bottleneck. The linear case with squared error recovers the PCA subspace, the top $k$ eigenvectors of the data covariance (Eckart–Young).
Regularized autoencoders drop the bottleneck for an explicit constraint: sparse ( $L^{1}$ on $h$ , a MAP estimate under a Laplace prior), denoising (reconstruct clean $x$ from corrupted $\tilde{x}$ ), contractive (penalize the encoder Jacobian $∥ \partial h / \partial x ∥_{F}^{2}$ ).
A denoising autoencoder learns the data manifold: its reconstruction residual estimates the score $\nabla_{x} log p_{data}$ , the field that flows off-manifold points back on.
The unifying view is the manifold hypothesis: data lives near a low-dimensional curved surface, and every autoencoder variant is a way of estimating that surface.
Organizing the latent space generatively (sampling codes from a known prior) is the leap to the variational autoencoder.

Goodfellow, Deep Learning, Ch. 14 — autoencoders as feedforward nets trained to reconstruct their input through a constrained code; the representation, not the copy, is the object of interest. ↩
Goodfellow, Deep Learning, §14.1 — an undercomplete autoencoder with linear maps and squared-error loss spans the same subspace as PCA (the top eigenvectors of the data covariance). ↩
Goodfellow, Deep Learning, §14.2.1 — sparse autoencoders, where the $L^{1}$ code penalty is read as a MAP estimate under a factorial Laplace prior on the latent units. ↩
Goodfellow, Deep Learning, §14.5 — denoising autoencoders, and the result that the reconstruction residual estimates the score $\nabla_{x} log p_{data}$ of the data-generating density. ↩
Goodfellow, Deep Learning, §14.2.3 / §14.7 — contractive autoencoders penalize the Frobenius norm of the encoder Jacobian, flattening the code off the manifold's tangent directions. ↩
Chollet, Deep Learning with Python, Ch. 8 — using a learned bottleneck code as a downstream feature space, and the gap between a plain autoencoder's latent space and a samplable generative one. ↩
He et al., Masked Autoencoders Are Scalable Vision Learners, CVPR 2022, and Devlin et al., BERT, NAACL 2019 — masking as the corruption process, scaled to Transformer pretraining. ↩
Vincent, A Connection Between Score Matching and Denoising Autoencoders, Neural Computation 2011 — the denoising residual as a score estimator, the basis of denoising score matching and diffusion. ↩
Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022 — running a generative model on an autoencoder's compressed latent codes rather than raw pixels. ↩