---
title: Autoencoders
module: Generative Models
moduleNumber: 7
lessonNumber: 2
order: 702
summary: >
  An autoencoder is a network trained to copy its input to its output through a
  narrow channel; the useful product is the bottleneck representation $h$, not
  the reconstruction. We derive the undercomplete autoencoder and prove its linear case
  recovers PCA, then trade the bottleneck for explicit regularization (sparse,
  denoising, contractive) and show how a denoising autoencoder learns the
  low-dimensional manifold the data lives on.
topics: [Generative Models]
sources:
  - book: Goodfellow
    ref: "Ch. 14 — Autoencoders"
  - book: Goodfellow
    ref: "§14.2 Regularized Autoencoders; §14.5–14.6 Denoising & Manifolds"
---

An **autoencoder** is a network trained to copy its input to its output. Stated
that baldly the task is absurd: the identity function copies perfectly and learns
nothing. The task becomes useful when the identity is forbidden: route the copy
through a channel too
narrow, too sparse, or too constrained to pass the input verbatim, so the network
must instead discover a _compressed code_ from which the input can be approximately
rebuilt. The reconstruction is a pretext; the code is what matters.[^gf-ae]

The model factors into two maps. An **encoder** $f$ compresses the input to a
**code** (or latent representation) $h$, and a **decoder** $g$ expands the code
back to a reconstruction $\hat x$:

$$
h = f(x), \qquad \hat x = g(h) = g(f(x)).
$$

$$
% caption: The autoencoder hourglass: encoder $f$ compresses $x$ to a code $h$, decoder $g$ expands it back to $\hat x$.
\begin{tikzpicture}[>=stealth, font=\small,
  io/.style={draw, minimum width=11mm, minimum height=30mm, align=center, font=\footnotesize},
  net/.style={draw, trapezium, trapezium stretches=true, minimum width=20mm, minimum height=28mm, align=center, font=\footnotesize, inner sep=2pt},
  code/.style={draw, thick, minimum width=9mm, minimum height=12mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[io] (x) at (0,0) {input\\$x$};
  \node[net, trapezium angle=70, draw=acc, text=acc] (enc) at (2.6,0) {encoder\\$f$};
  \node[code, draw=acc, text=acc] (h) at (4.9,0) {code\\$h$};
  \node[net, trapezium angle=110, draw=acc, text=acc] (dec) at (7.2,0) {decoder\\$g$};
  \node[io] (xh) at (9.8,0) {output\\(recon.)};
  \draw[->, acc, thick] (x) -- (enc);
  \draw[->, acc, thick] (enc) -- (h);
  \draw[->, acc, thick] (h) -- (dec);
  \draw[->, acc, thick] (dec) -- (xh);
  \draw[<->, black] (0,-1.9) -- (9.8,-1.9);
  \node[black, font=\footnotesize, anchor=north] at (4.9,-1.92) {reconstruct: minimize distance from $x$ to output};
\end{tikzpicture}
$$

> **Definition (Autoencoder).** A pair of parameterized maps, an encoder
> $h = f(x)$ and a decoder $\hat x = g(h)$, trained jointly to minimize a
> **reconstruction loss** $L(x, g(f(x)))$ that penalizes the discrepancy between
> the input and its reconstruction. For real-valued $x$ the loss is squared error,
> $L = \lVert x - g(f(x))\rVert_2^2$; for binary $x$ it is cross-entropy.

The training objective averages the reconstruction loss over the data,

$$
\min_{f, g}\; \mathbb{E}_{x \sim p_{\text{data}}}\brackets{\, \lVert x - g(f(x)) \rVert_2^2 \,},
$$

and is optimized by the same minibatch gradient descent that trains any other
feedforward net: an autoencoder _is_ a feedforward net whose target is its own
input. What distinguishes the variants below is not the loss but the **constraint**
that stops $g \circ f$ from collapsing to the identity. We organize them by that
constraint.

| variant | constraint on the code | what it learns |
| --- | --- | --- |
| undercomplete | $\dim h < \dim x$ (bottleneck) | a compressing subspace; PCA when linear |
| sparse | penalty $\Omega(h) = \lambda\lVert h\rVert_1$ | a few active features per input |
| denoising | reconstruct clean $x$ from corrupted $\tilde x$ | the data manifold / a score field |
| contractive | penalty $\lambda\lVert \partial h/\partial x\rVert_F^2$ | a locally flat, noise-robust code |

## Undercomplete autoencoders

The most direct way to forbid the identity is to make the code _smaller_ than the
input. If $h \in \mathbb{R}^k$ and $x \in \mathbb{R}^d$ with $k < d$, then $f$ must
discard $d - k$ dimensions' worth of information, and the only way to reconstruct
well is to discard the dimensions that matter least. The code is a **bottleneck**.

> **Definition (Undercomplete autoencoder).** An autoencoder whose code dimension
> is strictly smaller than the input dimension, $\dim h < \dim x$. The bottleneck
> forces the encoder to compress; minimizing reconstruction error then drives it to
> keep the most salient factors of variation and drop the rest.

$$
% caption: An undercomplete autoencoder squeezes input $x \in \mathbb{R}^{6}$ through a narrow code $h \in \mathbb{R}^{2}$ and expands it back to $\hat x \in \mathbb{R}^{6}$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  u/.style={circle, draw, minimum size=4mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  % layer x-positions and half-heights (units stacked vertically)
  % input (6), enc (4), code (2), dec (4), output (6)
  \foreach \i in {1,...,6} \node[u] (in\i) at (0, {1.5 - 0.6*(\i-1)}) {};
  \foreach \i in {1,...,4} \node[u] (e\i) at (2.1, {0.9 - 0.6*(\i-1)}) {};
  \foreach \i in {1,2} \node[u, draw=acc, thick] (c\i) at (4.2, {0.3 - 0.6*(\i-1)}) {};
  \foreach \i in {1,...,4} \node[u] (d\i) at (6.3, {0.9 - 0.6*(\i-1)}) {};
  \foreach \i in {1,...,6} \node[u] (o\i) at (8.4, {1.5 - 0.6*(\i-1)}) {};
  % edges input -> enc (neutral scaffold; must stay visible in both themes)
  \foreach \a in {1,...,6} \foreach \b in {1,...,4} \draw[black] (in\a) -- (e\b);
  \foreach \a in {1,...,4} \foreach \b in {1,2} \draw[black] (e\a) -- (c\b);
  \foreach \a in {1,2} \foreach \b in {1,...,4} \draw[black] (c\a) -- (d\b);
  \foreach \a in {1,...,4} \foreach \b in {1,...,6} \draw[black] (d\a) -- (o\b);
  % labels (R^n / \in render as garbage in tikz nodes; keep them in the caption)
  \node[font=\footnotesize] at (0,-2.6) {input: 6 dims};
  \node[acc, font=\footnotesize] at (4.2,-2.6) {code: 2 dims};
  \node[font=\footnotesize] at (8.4,-2.6) {output: 6 dims};
  \draw[<-, black] (3.3,2.1) -- (1.1,2.1) node[midway, above, black, font=\footnotesize] {compress};
  \draw[->, black] (5.1,2.1) -- (7.3,2.1) node[midway, above, black, font=\footnotesize] {expand};
\end{tikzpicture}
$$

### The linear case is PCA

When both maps are linear and the loss is squared error, the autoencoder is not
merely _like_ principal component analysis: it spans the same subspace. This is
the foundational result of the chapter and ties the autoencoder back to the
[linear factor models](/deep-learning/generative-models/linear-factor-models) that
precede it.

> **Theorem (Linear undercomplete autoencoder $\equiv$ PCA).** Let $f(x) = W x$
> and $g(h) = V h$ with $W \in \mathbb{R}^{k \times d}$, $V \in \mathbb{R}^{d \times k}$,
> and let the data be centered, $\mathbb{E}[x] = 0$, with covariance
> $\Sigma = \mathbb{E}[x x^{T}]$. The minimizer of
> $\mathbb{E}\lVert x - V W x\rVert_2^2$ projects $x$ onto the subspace spanned by
> the top $k$ eigenvectors of $\Sigma$, the same subspace PCA selects.

> **Proof.** Write the reconstruction as $\hat x = P x$ with $P = V W$, a rank-$\le k$
> matrix (a product of a $d\times k$ and a $k\times d$ matrix). The objective is
>
> $$
> \mathbb{E}\lVert x - P x\rVert_2^2
> = \mathbb{E}\brackets{\tr\!\parens{(I - P)\,x x^{T}\,(I - P)^{T}}}
> = \tr\!\parens{(I - P)\,\Sigma\,(I - P)^{T}}.
> $$
>
> We minimize over all matrices $P$ of rank at most $k$. The Eckart–Young theorem
> states that the best rank-$k$ approximation of $\Sigma$ in this trace (Frobenius)
> sense is its truncated eigendecomposition: writing $\Sigma = U \Lambda U^{T}$ with
> eigenvalues sorted $\lambda_1 \ge \cdots \ge \lambda_d$, the optimal $P$ is the
> orthogonal projector $P^\star = U_k U_k^{T}$ onto the leading $k$ eigenvectors
> $U_k = [u_1, \dots, u_k]$. Any factorization $V W = U_k U_k^{T}$ (e.g.
> $W = U_k^{T}$, $V = U_k$) attains it. The residual error equals the discarded
> spectral mass $\sum_{i>k} \lambda_i$, exactly PCA's reconstruction error. $\qed$

The autoencoder recovers the PCA _subspace_, not necessarily an orthonormal,
variance-ordered basis: $W$ can be any matrix whose rows span $U_k$, since
$V W$ (and therefore the loss) is invariant to an invertible change of basis
$W \mapsto A W$, $V \mapsto V A^{-1}$ inside the code. The latent coordinates are
fixed only up to a linear transformation. The value of the autoencoder framing is
what comes _next_: make $f$ and $g$ nonlinear and the model learns a curved,
$k$-dimensional **manifold** rather than a flat $k$-dimensional subspace, a strict
generalization of PCA.[^gf-pca]

| | linear AE / PCA | nonlinear AE |
| --- | --- | --- |
| encoder $f$ | $W x$ | deep nonlinear net |
| learned set | a flat $k$-plane through the origin | a curved $k$-manifold |
| code meaning | projection onto principal axes | nonlinear coordinates on the manifold |
| capacity risk | none (linear) | can memorize if over-powered |

That last cell is the catch. A sufficiently powerful nonlinear encoder can route the
input through a 1-D code by abusing precision, encoding the index $i$ of a training
example as a single real number and having the decoder look it up, copying the data
without learning any useful structure. Capacity must be _constrained_, which the
bottleneck does only crudely. The regularized autoencoders below constrain it
directly, and in doing so can afford codes that are as large as, or larger than, the
input.

## Regularized autoencoders

An **overcomplete** autoencoder has $\dim h \ge \dim x$. Without a bottleneck nothing
stops it from learning the identity, so we add an explicit penalty $\Omega$ to the
loss. The general regularized objective is

$$
\min_{f, g}\; \mathbb{E}_{x}\brackets{\, L(x, g(f(x))) + \Omega(h, x) \,},
$$

and the choice of $\Omega$ decides what kind of structure the code acquires. Three
choices dominate.

### Sparse autoencoders

A **sparse autoencoder** penalizes the magnitude of the code, typically with an
$L^1$ term $\Omega(h) = \lambda \lVert h \rVert_1 = \lambda \sum_i |h_i|$. Most code
units are then driven to zero on any given input, and each input is explained by a
_small_ active subset of an otherwise large dictionary of features.

> **Definition (Sparse autoencoder).** An autoencoder with code penalty
> $\Omega(h) = \lambda \lVert h\rVert_1$ ($\lambda > 0$) added to the reconstruction
> loss, so that only a few of the (possibly many) code units are active per example.
> The code is overcomplete but **sparse**: high capacity, low usage.

The penalty admits a clean probabilistic reading. If we treat $h$ as a latent
variable with a factorial prior $p(h) \propto \prod_i e^{-\lambda |h_i|}$ (a
Laplace prior, which is sharply peaked at zero), then maximizing the joint
log-likelihood $\log p(x \mid h) + \log p(h)$ recovers exactly a Gaussian
reconstruction term plus the $L^1$ penalty:

$$
-\log p(h) = \lambda \sum_i |h_i| + \text{const},
\qquad
-\log p(x \mid h) \propto \lVert x - g(h)\rVert_2^2.
$$

Sparsity is thus a MAP estimate under a sparse prior, not an arbitrary add-on: the
same logic that makes $L^1$ regression select few features.[^gf-sparse]

$$
% caption: A sparse code over an overcomplete dictionary: most units are off (light), a few fire (filled) per input, and different inputs recruit different active subsets.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  off/.style={draw, black, minimum size=5mm, inner sep=0pt},
  on/.style={draw, thick, minimum size=5mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  % three inputs, each recruiting a different sparse subset of an 8-unit code
  \foreach \r/\lab in {0/A, 1/B, 2/C} {
    \node[font=\footnotesize, anchor=east] at (-0.4, {-1.3*\r}) {input \lab};
  }
  % row A: units 2,6 active
  \foreach \c in {1,...,8} \node[off] at ({0.7*\c}, 0) {};
  \foreach \c in {2,6} \node[on, fill=acc!25, draw=acc] at ({0.7*\c}, 0) {};
  % row B: units 4,5 active
  \foreach \c in {1,...,8} \node[off] at ({0.7*\c}, -1.3) {};
  \foreach \c in {4,5} \node[on, fill=acc!25, draw=acc] at ({0.7*\c}, -1.3) {};
  % row C: units 1,8 active
  \foreach \c in {1,...,8} \node[off] at ({0.7*\c}, -2.6) {};
  \foreach \c in {1,8} \node[on, fill=acc!25, draw=acc] at ({0.7*\c}, -2.6) {};
  \node[acc, anchor=west] at (6.3, -1.3) {f\/illed = on};
  \node[black, anchor=north] at (3.15, -3.2) {8-unit overcomplete code (2 on per input)};
\end{tikzpicture}
$$

### Denoising autoencoders

The **denoising autoencoder** (DAE) changes the task rather than adding a penalty.
Corrupt each input with noise, $\tilde x \sim C(\tilde x \mid x)$, feed the corrupted
$\tilde x$ to the network, and require it to reconstruct the _clean_ $x$:

$$
\min_{f, g}\; \mathbb{E}_{x \sim p_{\text{data}}}\;
\mathbb{E}_{\tilde x \sim C(\tilde x \mid x)}
\brackets{\, \lVert x - g(f(\tilde x)) \rVert_2^2 \,}.
$$

Copying is now impossible, since the input it sees is not the target it must
produce, so even an overcomplete DAE cannot learn the identity. To undo the
corruption it must
learn where clean data plausibly lies and pull a corrupted point back toward it.

$$
% caption: A denoising autoencoder maps a corrupted input $\tilde x = x + \varepsilon$ back toward the clean target $x$; it cannot copy, so it learns structure.
\begin{tikzpicture}[>=stealth, font=\small,
  io/.style={draw, minimum width=13mm, minimum height=14mm, align=center, font=\footnotesize},
  net/.style={draw, minimum width=20mm, minimum height=14mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  \node[io] (x) at (0,0) {clean\\input $x$};
  \node[io, draw=red, text=red] (xt) at (3.1,0) {corrupt\\input};
  \node[net, draw=acc, text=acc, thick] (ae) at (6.4,0) {encoder $f$\\+ decoder $g$};
  \node[io, draw=green, text=green] (xh) at (9.7,0) {clean\\recon.};
  \draw[->, red, thick] (x) -- (xt) node[midway, above, font=\footnotesize, text=red] {add noise};
  \draw[->, acc, thick] (xt) -- (ae);
  \draw[->, acc, thick] (ae) -- (xh);
  % target arrow: train output toward clean x
  \draw[->, green, thick, dashed] (xh.south) .. controls (9.7,-2.0) and (0,-2.0) .. (x.south)
    node[midway, below, font=\footnotesize, text=green] {target is the clean input};
\end{tikzpicture}
$$

The geometry is the payoff. If clean data concentrates near a low-dimensional
manifold, corruption with small Gaussian noise $\tilde x = x + \varepsilon$,
$\varepsilon \sim \mathcal{N}(0, \sigma^2 I)$, pushes points _off_ the manifold; the
optimal denoiser maps them back _onto_ it. Under squared-error loss the
reconstruction is the conditional mean $g(f(\tilde x)) = \mathbb{E}[x \mid \tilde x]$,
and a short expansion connects the residual to the **score**, the gradient of the
log data density:

$$
g(f(\tilde x)) - \tilde x \;\approx\; \sigma^2 \, \nabla_{\tilde x} \log p_{\text{data}}(\tilde x).
$$

The denoiser's correction vector points _up the density gradient_, toward higher-
probability regions, that is, back toward the manifold. A DAE trained with Gaussian
noise is, implicitly, a score estimator: it learns the vector field that flows any
point onto the data manifold.[^gf-dae]

> **Theorem (Denoising learns the score).** For a denoising autoencoder trained with
> additive Gaussian corruption $\tilde x = x + \varepsilon$, $\varepsilon \sim
> \mathcal{N}(0, \sigma^2 I)$, under squared-error reconstruction, the optimal
> reconstruction minus the input estimates the score of the noised density up to the
> noise variance: $\;r(\tilde x) - \tilde x \approx \sigma^2\, \nabla_{\tilde x}\log p_{\text{data}}(\tilde x)$.

> **Proof.** The squared-error minimizer is the posterior mean
> $r(\tilde x) = \mathbb{E}[x \mid \tilde x]$. With Gaussian noise the joint density
> of $(\tilde x, x)$ has $p(\tilde x \mid x) \propto \exp(-\lVert \tilde x - x\rVert^2 / 2\sigma^2)$,
> so $\nabla_{\tilde x} p(\tilde x) = \int \nabla_{\tilde x} p(\tilde x\mid x)\,p(x)\,dx
> = \int \frac{x - \tilde x}{\sigma^2}\,p(\tilde x\mid x)\,p(x)\,dx$. Dividing by
> $p(\tilde x)$ turns the integral into a posterior expectation,
> $\nabla_{\tilde x}\log p(\tilde x) = \frac{1}{\sigma^2}\parens{\mathbb{E}[x\mid \tilde x] - \tilde x}
> = \frac{1}{\sigma^2}\parens{r(\tilde x) - \tilde x}$, where $p(\tilde x)$ is the
> noised density that approaches $p_{\text{data}}$ as $\sigma \to 0$. Rearranging
> gives the claim. $\qed$

### Contractive autoencoders

The **contractive autoencoder** (CAE) penalizes the sensitivity of the code to its
input, the squared Frobenius norm of the encoder's Jacobian:

$$
\Omega(h, x) = \lambda \norm{ \frac{\partial f(x)}{\partial x} }_F^2
= \lambda \sum_{i,j} \parens{ \frac{\partial h_i}{\partial x_j} }^2.
$$

This pressures the encoder to be _locally constant_: small moves in $x$ should
barely move $h$. The encoder cannot be constant everywhere — it must still vary
enough to reconstruct — so the optimum stays flat in directions
that _leave_ the manifold (noise directions, which reconstruction does not need) and
varies only _along_ the manifold (the directions that carry data). The Jacobian's
singular values collapse except along the tangent space of the data manifold.[^gf-cae]

> **Definition (Contractive autoencoder).** An autoencoder with penalty
> $\lambda\lVert \partial f / \partial x\rVert_F^2$ on the encoder Jacobian. It learns
> a representation that is **insensitive** to input perturbations off the data
> manifold and sensitive only along it, explicitly carving out the manifold's
> tangent directions.

| penalty target | sparse AE | contractive AE |
| --- | --- | --- |
| what is penalized | the code value $\lVert h\rVert_1$ | the code's derivative $\lVert \partial h/\partial x\rVert_F^2$ |
| effect on code | few units active | units locally insensitive to $x$ |
| manifold reading | sparse dictionary | flat across normal directions |

The DAE and CAE converge on the same idea from opposite ends. Denoising forces
robustness to noise _explicitly_, by training against corrupted inputs; the
contractive penalty forces it _analytically_, by shrinking the Jacobian. For small
Gaussian noise the two are equivalent to first order (both penalize how much the
representation reacts to perturbation), and both, as the next section shows, are
really statements about a manifold.

## The manifold perspective

The unifying picture behind every variant is the **manifold hypothesis**: real
high-dimensional data (natural images, speech, text) does not fill its ambient
space but concentrates near a much lower-dimensional, curved manifold. A $d$-pixel
image lives in $\mathbb{R}^d$, yet the set of _natural_ images is a vanishingly thin
sheet inside that cube; almost every point in $\mathbb{R}^d$ is noise.

> **Definition (Manifold hypothesis).** The assumption that the data distribution
> $p_{\text{data}}$ places nearly all its mass on (or near) a $k$-dimensional manifold
> $\mathcal{M} \subset \mathbb{R}^d$ with $k \ll d$. Directions tangent to
> $\mathcal{M}$ are the **factors of variation** the data actually exhibits;
> directions normal to it correspond to perturbations no real sample displays.

An autoencoder _is_ a parameterization of such a manifold: the decoder $g$ maps the
$k$-dimensional code space onto a $k$-dimensional surface in $\mathbb{R}^d$, the
image $\{g(h) : h \in \mathbb{R}^k\}$, and the encoder $f$ assigns intrinsic
coordinates to points near it. Reconstruction error is small exactly for points on
the manifold and large for points off it, so minimizing it _shapes_ the manifold to
hug the data.

$$
% caption: The manifold view of denoising: reconstruction vectors (red) point corrupted points back onto the data manifold (blue), tracing the score $\nabla_x \log p_{\text{data}}$.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % the data manifold: a curved arc
  \draw[acc, very thick] plot[domain=-3:3, samples=60] (\x, {0.42*\x*\x - 1.2});
  \node[acc, font=\footnotesize, anchor=west] at (3.0,2.4) {data manifold};
  % clean data points ON the manifold
  \foreach \x in {-2.6,-1.7,-0.8,0.2,1.1,2.0,2.7}
    \fill[acc] (\x, {0.42*\x*\x - 1.2}) circle (1.7pt);
  % corrupted points off the manifold, with reconstruction arrows back to nearest point
  % point 1 above the arm
  \fill[red] (-1.9,0.7) circle (1.7pt);
  \draw[->, red, thick] (-1.9,0.7) -- (-1.78,{0.42*1.78*1.78 - 1.2 + 0.08});
  % point 2 inside the bowl
  \fill[red] (0.3,-0.55) circle (1.7pt);
  \draw[->, red, thick] (0.3,-0.55) -- (0.28,{0.42*0.28*0.28 - 1.2 - 0.04});
  % point 3 right side
  \fill[red] (1.9,0.05) circle (1.7pt);
  \draw[->, red, thick] (1.9,0.05) -- (1.78,{0.42*1.78*1.78 - 1.2 + 0.06});
  % point 4 left outer
  \fill[red] (-2.7,1.6) circle (1.7pt);
  \draw[->, red, thick] (-2.7,1.6) -- (-2.5,{0.42*2.5*2.5 - 1.2 + 0.1});
  \node[red, font=\footnotesize, anchor=north] at (0.3,-1.55) {corrupted points projected back};
\end{tikzpicture}
$$

This is why denoising is so natural a training signal. Corruption is a sample of
"off the manifold"; the clean target is "on the manifold"; learning to reconstruct
is learning the _projection_ onto $\mathcal{M}$. The contractive penalty arrives at
the same surface analytically, by flattening the encoder normal to $\mathcal{M}$,
and the undercomplete bottleneck approximates it crudely with a flat $k$-plane. All
four variants are estimating the same object: the manifold the data lives on.

| variant | how it finds the manifold |
| --- | --- |
| undercomplete | restricts the code to $k$ dimensions; linear case is the best $k$-plane |
| sparse | tiles the manifold with a sparse, overcomplete dictionary of local features |
| denoising | learns to project off-manifold points back on — estimates the score |
| contractive | flattens the encoder normal to the manifold, varies only along it |

## Training a denoising autoencoder

The training loop is ordinary minibatch gradient descent with one extra step: draw
fresh corruption for each example on each pass, so the network sees a different
$\tilde x$ for the same $x$ every epoch and cannot memorize a fixed noise pattern.

```algorithm
caption: $\textsc{TrainDAE}(f_\theta, g_\phi, \mathcal{D}, C, \eta)$ — denoising autoencoder training
initialize encoder params $\theta$, decoder params $\phi$
repeat
  sample a minibatch $\{x^{(1)}, \dots, x^{(m)}\} \sim \mathcal{D}$
  for each example $x^{(i)}$ do
    draw corruption $\tilde x^{(i)} \sim C(\tilde x \mid x^{(i)})$ // e.g. add Gaussian noise / mask pixels
    $h^{(i)} \gets f_\theta(\tilde x^{(i)})$ // encode the corrupted input
    $\hat x^{(i)} \gets g_\phi(h^{(i)})$ // decode toward the clean target
  $L \gets \frac{1}{m}\sum_i \lVert x^{(i)} - \hat x^{(i)}\rVert_2^2$ // reconstruct CLEAN x
  $(\theta, \phi) \gets (\theta, \phi) - \eta\,\nabla_{\theta,\phi} L$ // joint gradient step
until converged
return $\theta, \phi$
```

The only line that differs from a vanilla autoencoder is the corruption draw; set
$C(\tilde x \mid x) = \delta(\tilde x - x)$ (no corruption) and the loop reduces to
the plain reconstruction objective. Common corruption processes are additive
Gaussian noise, masking a random subset of inputs to zero (as in masked
pretraining), and salt-and-pepper flips.

## The latent space

What does the learned code actually look like? Train an autoencoder on clustered
data and the encoder lays the codes out so that the structure reconstruction needs
is preserved: inputs that are alike map to nearby codes, distinct groups separate.
The bottleneck's geometry becomes a usable feature space: the codes are what you
feed a downstream classifier, retrieval index, or visualizer.[^chollet-ae]

$$
% caption: A 2-D latent space: each point is a code $h = f(x)$ colored by cluster; a good autoencoder maps similar inputs to nearby codes.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % axes for the 2-D code
  \draw[->, thick] (-0.3,0) -- (5.2,0) node[right, font=\footnotesize] {$h_1$};
  \draw[->, thick] (0,-0.3) -- (0,4.2) node[above, font=\footnotesize] {$h_2$};
  % cluster A (blue) lower-left
  \foreach \p in {(0.9,0.8),(1.2,1.2),(0.7,1.4),(1.4,0.7),(1.0,1.0),(1.5,1.3),(0.6,0.9)}
    \fill[acc] \p circle (2.4pt);
  \node[acc, font=\footnotesize] at (1.05,0.25) {class A};
  % cluster B (green) upper-middle
  \foreach \p in {(2.4,3.2),(2.8,3.5),(2.2,2.9),(3.0,3.1),(2.6,3.3),(2.9,2.8),(2.3,3.5)}
    \fill[green] \p circle (2.4pt);
  \node[green, font=\footnotesize] at (2.6,3.95) {class B};
  % cluster C (red) right
  \foreach \p in {(4.0,1.4),(4.4,1.8),(3.8,1.1),(4.5,1.3),(4.1,1.7),(4.3,0.9),(3.9,1.6)}
    \fill[red] \p circle (2.4pt);
  \node[red, font=\footnotesize] at (4.2,0.45) {class C};
\end{tikzpicture}
$$

This is the bridge to generative modeling. An ordinary autoencoder organizes its
latent space but leaves _gaps_: sample a random $h$ and the decoder may produce
garbage, because nothing forced the codes to fill the space according to a known
distribution. Imposing such a distribution on the code, so that sampling $h$ and
decoding yields a fresh, plausible $x$, is the step the
[variational autoencoder](/deep-learning/generative-models/variational-autoencoders)
takes — turning the autoencoder from a compressor into a generator.

## Autoencoders in current practice

The autoencoder outlived its 2006-era role as a deep-net pretrainer and became a
staple in three modern settings, each a direct descendant of a variant above.

**Masked autoencoding is self-supervised pretraining at scale.** The denoising
idea — reconstruct a clean signal from a corrupted one — became the dominant
pretraining objective when the corruption is _masking_: **BERT** masks tokens and
predicts them, and the **masked autoencoder** (MAE) masks a large fraction of image
patches and reconstructs the pixels, learning representations that transfer across
vision tasks.[^mae] Masking to zero is the corruption process $C$ named in
the training loop above, scaled to Transformers and very high mask ratios.

**The score connection became a generative model.** The theorem that a
Gaussian-denoising autoencoder's residual estimates $\nabla_x \log p_{\text{data}}$
is the seed of **denoising score matching** and, through it, of
[diffusion models](/deep-learning/generative-models/diffusion-and-score-based-models):
a diffusion model is a denoising autoencoder trained at _many_ noise levels at once,
and sampling walks the learned score field back toward the data manifold.[^dsm] The
manifold picture drawn above — reconstruction arrows pointing off-manifold points
back on — is literally the reverse diffusion trajectory.

**Discrete-code autoencoders compress for downstream generators.** The
[VQ-VAE's](/deep-learning/generative-models/variational-autoencoders) learned
discrete codebook turned the bottleneck into a compact token language, and modern
image and audio generators (including latent diffusion) run their expensive
generative model on these autoencoder codes rather than on raw pixels, using the
autoencoder purely as a perceptual compressor.[^latentdiff] The bottleneck that
began as a way to force compression is now the standard front-end that makes
large-scale generation affordable.

## Takeaways

- An **autoencoder** is encoder $h = f(x)$ plus decoder $\hat x = g(h)$, trained to
  minimize reconstruction error $\lVert x - g(f(x))\rVert^2$. The copy is a pretext;
  the **code** $h$ is the point.
- An **undercomplete** autoencoder ($\dim h < \dim x$) compresses through a
  bottleneck. The **linear** case with squared error recovers the **PCA** subspace,
  the top $k$ eigenvectors of the data covariance (Eckart–Young).
- **Regularized** autoencoders drop the bottleneck for an explicit constraint:
  **sparse** ($L^1$ on $h$, a MAP estimate under a Laplace prior), **denoising**
  (reconstruct clean $x$ from corrupted $\tilde x$), **contractive** (penalize the
  encoder Jacobian $\lVert \partial h/\partial x\rVert_F^2$).
- A **denoising** autoencoder learns the data **manifold**: its reconstruction
  residual estimates the **score** $\nabla_x \log p_{\text{data}}$, the field that
  flows off-manifold points back on.
- The unifying view is the **manifold hypothesis**: data lives near a low-dimensional
  curved surface, and every autoencoder variant is a way of estimating that surface.
- Organizing the latent space _generatively_ (sampling codes from a known prior)
  is the leap to the
  [variational autoencoder](/deep-learning/generative-models/variational-autoencoders).

[^gf-ae]: **Goodfellow**, _Deep Learning_, Ch. 14 — autoencoders as feedforward nets trained to reconstruct their input through a constrained code; the representation, not the copy, is the object of interest.
[^gf-pca]: **Goodfellow**, _Deep Learning_, §14.1 — an undercomplete autoencoder with linear maps and squared-error loss spans the same subspace as PCA (the top eigenvectors of the data covariance).
[^gf-sparse]: **Goodfellow**, _Deep Learning_, §14.2.1 — sparse autoencoders, where the $L^1$ code penalty is read as a MAP estimate under a factorial Laplace prior on the latent units.
[^gf-dae]: **Goodfellow**, _Deep Learning_, §14.5 — denoising autoencoders, and the result that the reconstruction residual estimates the score $\nabla_x\log p_{\text{data}}$ of the data-generating density.
[^gf-cae]: **Goodfellow**, _Deep Learning_, §14.2.3 / §14.7 — contractive autoencoders penalize the Frobenius norm of the encoder Jacobian, flattening the code off the manifold's tangent directions.
[^chollet-ae]: **Chollet**, _Deep Learning with Python_, Ch. 8 — using a learned bottleneck code as a downstream feature space, and the gap between a plain autoencoder's latent space and a samplable generative one.
[^mae]: **He et al.**, "Masked Autoencoders Are Scalable Vision Learners," CVPR 2022, and **Devlin et al.**, "BERT," NAACL 2019 — masking as the corruption process, scaled to Transformer pretraining.
[^dsm]: **Vincent**, "A Connection Between Score Matching and Denoising Autoencoders," Neural Computation 2011 — the denoising residual as a score estimator, the basis of denoising score matching and diffusion.
[^latentdiff]: **Rombach et al.**, "High-Resolution Image Synthesis with Latent Diffusion Models," CVPR 2022 — running a generative model on an autoencoder's compressed latent codes rather than raw pixels.