---
title: Variational Autoencoders
module: Generative Models
moduleNumber: 7
lessonNumber: 3
order: 703
summary: >
  An autoencoder compresses, but its latent space has gaps: sample a
  point between two encodings and the decoder produces noise. The variational
  autoencoder fixes this by training a probabilistic encoder against a prior, so the
  latent space becomes a smooth, samplable density. We derive the evidence lower
  bound it maximizes, the reparameterization trick that lets gradients flow through a
  random sample, and the closed-form Gaussian regularizer that pulls the posterior
  toward the prior.
topics: [Generative Models]
sources:
  - book: Goodfellow
    ref: "§20.10.3 — Variational Autoencoders"
  - book: Goodfellow
    ref: "Ch. 19 — Approximate Inference; §19.1 Inference as Optimization"
  - book: Chollet
    ref: "§8.4 — Generating images with variational autoencoders"
---

An ordinary [autoencoder](/deep-learning/generative-models/autoencoders) learns a
deterministic code $z = f(x)$ and a reconstruction $\hat x = g(z)$, and nothing
constrains _where_ in code-space the encodings land. The trained codes form
isolated islands; the gaps between them decode to garbage, so the autoencoder
compresses but cannot _generate_: there is no distribution to sample from. The
**variational autoencoder** (VAE) repairs exactly this. It makes the encoder
output a _distribution_ over codes, trains that distribution to match a fixed
prior $p(z)$, and thereby turns the latent space into a smooth density we can
sample. The cost is a detour through **variational inference**; the result is
a principled, gradient-trainable generative model.[^gf-vae]

## The latent-variable model

A VAE is a **latent-variable generative model**: each datum $x$ is generated by
first drawing a hidden code $z$ from a simple prior, then mapping it through a
decoder $p_\theta(x \mid z)$.

> **Definition (Latent-variable model).** A generative model that explains data
> through unobserved variables $z$. It fixes a prior $p(z)$ and a conditional
> $p_\theta(x \mid z)$ (the **decoder**) and defines the data density as the
> marginal $p_\theta(x) = \int p_\theta(x \mid z)\,p(z)\,dz$.

We take the standard Gaussian prior and a decoder network $g_\theta$ that maps a
code to the parameters of an output distribution:

$$
p(z) = \mathcal N(0, I),
\qquad
p_\theta(x \mid z) = \mathcal N\!\parens{x;\, g_\theta(z),\, I},
$$

so the data density is the marginal over all codes,

$$
p_\theta(x) = \int p_\theta(x \mid z)\,p(z)\,dz.
$$

$$
% caption: The generative model: a code $z$ drawn from the prior is mapped by decoder $g_\theta$ to the parameters of $p_\theta(x\mid z)$.
\begin{tikzpicture}[>=stealth, font=\small,
  v/.style={circle, draw, minimum size=11mm, inner sep=0pt},
  net/.style={draw, minimum width=16mm, minimum height=11mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[v, draw=acc, text=acc, thick] (z) at (0,0) {z};
  \node[net] (dec) at (3.0,0) {decoder\\g};
  \node[v] (x) at (6.2,0) {x};
  \node[font=\footnotesize, text=acc, anchor=south] at (0,0.8) {prior};
  \node[font=\footnotesize, anchor=north] at (0,-0.8) {N(0, I)};
  \draw[->, acc, thick] (z) -- (dec);
  \draw[->, acc, thick] (dec) -- (x) node[midway, above, font=\scriptsize] {sample};
  \node[font=\footnotesize, anchor=north] at (6.2,-0.8) {data};
\end{tikzpicture}
$$

To train this by maximum likelihood we would maximize $\log p_\theta(x)$. The
integral, however, is **intractable**: it ranges over the entire latent space, and
the decoder $g_\theta$ is a nonlinear network, so there is no closed form. The
posterior we would need to invert the generative process,

$$
p_\theta(z \mid x) = \frac{p_\theta(x \mid z)\,p(z)}{p_\theta(x)},
$$

inherits the same intractable normalizer $p_\theta(x)$ in its denominator. We
cannot compute it, and so we cannot directly do maximum likelihood. Variational
inference works around this.[^gf-inference]

## Variational inference and the ELBO

The idea of [variational inference](/deep-learning/probabilistic-methods/approximate-inference)
is to give up on the exact posterior and instead introduce a tractable
**approximate posterior** $q_\phi(z \mid x)$ (a second network, the **encoder**)
chosen from a family we can sample and differentiate. We pick a Gaussian with
diagonal covariance whose mean and variance are emitted by the encoder network
$f_\phi$:

$$
q_\phi(z \mid x) = \mathcal N\!\parens{z;\, \mu_\phi(x),\, \diag\sigma_\phi^2(x)}.
$$

The encoder is also called the **inference network** because it performs
approximate inference: given $x$, it guesses which codes $z$ could have produced
it. The decoder is the **generative network**. We now derive a tractable objective
that bounds the log-evidence from below.

> **Theorem (Evidence lower bound).** For any approximate posterior $q_\phi(z\mid x)$ with the same support as $p_\theta(z \mid x)$,
> $$
> \log p_\theta(x) \;\ge\; \mathbb{E}_{q_\phi(z\mid x)}\!\brackets{\log p_\theta(x\mid z)} - D_{KL}\!\parens{q_\phi(z\mid x)\,\|\,p(z)},
> $$
> and the gap between the two sides equals $D_{KL}(q_\phi(z\mid x)\,\|\,p_\theta(z\mid x)) \ge 0$.

> **Proof.** Multiply and divide inside the log by $q_\phi(z \mid x)$ and write the
> evidence as an expectation under $q_\phi$ (legal because $\log p_\theta(x)$ does
> not depend on $z$):
> $$
> \log p_\theta(x)
> = \mathbb{E}_{q_\phi(z\mid x)}\!\brackets{\log \frac{p_\theta(x, z)}{q_\phi(z\mid x)}}
> + \mathbb{E}_{q_\phi(z\mid x)}\!\brackets{\log \frac{q_\phi(z\mid x)}{p_\theta(z\mid x)}}.
> $$
> The first identity follows from $p_\theta(x,z) = p_\theta(z\mid x)\,p_\theta(x)$:
> the two $\log$ ratios telescope to $\log p_\theta(x)$, whose expectation under
> $q_\phi$ is itself. The second term is precisely $D_{KL}(q_\phi(z\mid x)\,\|\,p_\theta(z\mid x)) \ge 0$
> by Gibbs' inequality, so dropping it can only decrease the right side:
> $$
> \log p_\theta(x) \;\ge\; \mathbb{E}_{q_\phi(z\mid x)}\!\brackets{\log \frac{p_\theta(x, z)}{q_\phi(z\mid x)}} \;=:\; \mathcal{L}(\theta,\phi;x).
> $$
> Finally split the joint $p_\theta(x,z) = p_\theta(x\mid z)\,p(z)$ inside the
> bound:
> $$
> \begin{aligned}
> \mathcal{L} &= \mathbb{E}_{q_\phi}\!\brackets{\log p_\theta(x\mid z)} + \mathbb{E}_{q_\phi}\!\brackets{\log \frac{p(z)}{q_\phi(z\mid x)}} \\
> &= \mathbb{E}_{q_\phi}\!\brackets{\log p_\theta(x\mid z)} - D_{KL}\!\parens{q_\phi(z\mid x)\,\|\,p(z)}. \qquad\qed
> \end{aligned}
> $$

The bound $\mathcal{L}(\theta, \phi; x)$ is the **evidence lower bound** (ELBO).
Maximizing it does double duty: because $\log p_\theta(x) = \mathcal{L} + D_{KL}(q_\phi \,\|\, p_\theta(z\mid x))$ with the evidence fixed in $\phi$, pushing $\mathcal{L}$ up
simultaneously raises a lower bound on the data likelihood _and_ squeezes the
inference gap, tightening $q_\phi$ toward the true posterior.[^gf-elbo]

$$
% caption: The log-evidence splits into the ELBO plus the inference gap; maximizing the ELBO raises the bound and shrinks the gap, tight when $q_\phi$ matches the posterior.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  % total evidence bar
  \draw[black] (0,0) -- (0,3.4);
  \node[anchor=east, font=\footnotesize] at (-0.15,1.7) {log-evidence};
  % stacked bar: ELBO (bottom) + gap (top) — light tint + crisp outline
  \draw[acc, very thick, fill=acc!15] (0.6,0) rectangle (2.0,2.5);
  \draw[black, very thick, fill=black!8] (0.6,2.5) rectangle (2.0,3.4);
  \node[acc, font=\footnotesize] at (1.3,1.25) {ELBO};
  \node[font=\scriptsize] at (1.3,2.95) {gap};
  % bracket for total height
  \draw[black] (2.25,0) -- (2.55,0) -- (2.55,3.4) -- (2.25,3.4);
  \node[anchor=west, font=\footnotesize] at (2.65,0.7) {f\/ixed};
  % a second bar: tighter q raises ELBO, shrinks gap
  \draw[acc, very thick, fill=acc!15] (4.4,0) rectangle (5.8,3.15);
  \draw[black, very thick, fill=black!8] (4.4,3.15) rectangle (5.8,3.4);
  \node[acc, font=\footnotesize] at (5.1,1.55) {ELBO};
  \draw[->, black, thick] (2.9,1.7) -- (4.1,1.7) node[midway, above, font=\scriptsize] {train};
  \node[font=\scriptsize, anchor=north] at (5.1,-0.1) {tigh\/ter q};
\end{tikzpicture}
$$

### Reading the two terms

The ELBO reads as a **reconstruction term** minus a **regularizer**, and this is
the entire intuition behind the architecture:

$$
\mathcal{L}(\theta, \phi; x) = \underbrace{\mathbb{E}_{q_\phi(z\mid x)}\!\brackets{\log p_\theta(x\mid z)}}_{\text{reconstruction}} - \underbrace{D_{KL}\!\parens{q_\phi(z\mid x)\,\|\,p(z)}}_{\text{KL regularizer}}.
$$

| Term | Encourages | Pulls toward | Without it |
| --- | --- | --- | --- |
| $\mathbb{E}_{q}[\log p_\theta(x\mid z)]$ | codes that decode back to $x$ | faithful reconstruction | useless codes, blurry output |
| $D_{KL}(q_\phi \,\|\, p)$ | posteriors near the prior $\mathcal N(0,I)$ | a smooth, samplable latent space | holey latent space (a plain autoencoder) |

With a unit-variance Gaussian decoder, $-\log p_\theta(x\mid z)$ is squared error up
to a constant, so the reconstruction term _is_ the familiar autoencoder loss. The
KL term is the new ingredient: it is precisely what an ordinary
autoencoder lacks, the term that packs the encodings against the prior so the gaps
close.

## The reparameterization trick

The ELBO contains an expectation over $z \sim q_\phi(z \mid x)$, and we must
differentiate it with respect to $\phi$ — but $\phi$ controls the very
distribution we are sampling from. The naive estimator differentiates the sampling
density directly (the **score-function** or REINFORCE estimator),

$$
\nabla_\phi \,\mathbb{E}_{q_\phi}[h(z)] = \mathbb{E}_{q_\phi}\!\brackets{h(z)\,\nabla_\phi \log q_\phi(z\mid x)},
$$

which is unbiased but notoriously **high-variance**: $h(z)$ swings widely and the
$\nabla_\phi \log q_\phi$ weight does not cancel that swing, so estimates are noisy
and training crawls.

The **reparameterization trick** removes the randomness from the gradient path. A
Gaussian sample can be written as a deterministic, differentiable transform of a
_fixed_, parameter-free noise source:

$$
z = \mu_\phi(x) + \sigma_\phi(x) \odot \epsilon,
\qquad \epsilon \sim \mathcal N(0, I).
$$

Now the only stochastic node is $\epsilon$, which carries no parameters; the
sample $z$ is a smooth function of $\mu_\phi$ and $\sigma_\phi$. The expectation
becomes an expectation over the _fixed_ distribution of $\epsilon$, and the
gradient slips inside it:

$$
\nabla_\phi \,\mathbb{E}_{q_\phi}[h(z)] = \mathbb{E}_{\epsilon \sim \mathcal N(0,I)}\!\brackets{\nabla_\phi\, h\parens{\mu_\phi(x) + \sigma_\phi(x)\odot\epsilon}},
$$

a low-variance estimate we can compute with a single sample of $\epsilon$ and
ordinary backpropagation.[^gf-reparam]

$$
% caption: Reparameterization. Before, the stochastic sample blocks the gradient; after, $z=\mu + \sigma\odot\epsilon$ moves randomness into $\epsilon$, leaving a deterministic path the gradient follows.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  v/.style={circle, draw, minimum size=9mm, inner sep=0pt},
  rnd/.style={circle, draw, minimum size=9mm, inner sep=0pt, draw=black, dashed},
  op/.style={draw, minimum width=9mm, minimum height=9mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % --- BEFORE ---
  \node[v, draw=acc, text=acc] (p1) at (0,0) {f} ;
  \node[rnd] (s1) at (1.9,0) {z};
  \node[v] (o1) at (3.8,0) {loss};
  \node[font=\footnotesize, anchor=south] at (0,0.65) {encoder};
  \node[font=\scriptsize, text=black, anchor=north] at (1.9,-0.6) {random};
  \draw[->, thick] (p1) -- (s1);
  \draw[->, thick] (s1) -- (o1);
  % blocked gradient
  \draw[->, red, thick] (o1.south) .. controls (2.8,-1.3) and (2.4,-1.3) .. (s1.south);
  \node[red, font=\footnotesize] at (3.2,-1.45) {\texttt{blocked}};
  \node[font=\footnotesize, anchor=north] at (1.9,-2.1) {stochastic node};
  % --- AFTER ---
  \begin{scope}[xshift=7.4cm]
    \node[v, draw=acc, text=acc] (p2) at (0,0) {f};
    \node[op] (s2) at (1.9,0) {+};
    \node[v] (o2) at (3.8,0) {loss};
    \node[rnd] (eps) at (1.9,1.6) {noise};
    \node[font=\footnotesize, anchor=south] at (0,0.65) {encoder};
    \draw[->, thick] (p2) -- (s2);
    \draw[->, thick] (eps) -- (s2);
    \draw[->, thick] (s2) -- (o2);
    \node[font=\scriptsize, anchor=south, text=acc] at (0.95,0.9) {mean, scale};
    \node[font=\scriptsize, anchor=west] at (2.45,0.45) {z};
    % flowing gradient
    \draw[->, green, thick] (o2.south) .. controls (2.8,-1.3) and (0.9,-1.3) .. (p2.south);
    \node[green, font=\footnotesize] at (1.9,-1.55) {\texttt{gradient} passes};
    \node[font=\footnotesize, anchor=north] at (1.9,-2.1) {deterministic path + noise};
  \end{scope}
\end{tikzpicture}
$$

> **Definition (Reparameterization trick).** Expressing a sample from a
> parameterized distribution $q_\phi$ as a deterministic, differentiable function
> $z = T_\phi(x, \epsilon)$ of the parameters and an independent, parameter-free
> noise variable $\epsilon$. The randomness is pushed off the gradient path, so
> $\nabla_\phi \mathbb{E}_{q_\phi}[h(z)] = \mathbb{E}_\epsilon[\nabla_\phi h(T_\phi(x,\epsilon))]$
> can be estimated by ordinary backpropagation through $T_\phi$.

| Estimator | Form | Variance | Requires |
| --- | --- | --- | --- |
| Score-function (REINFORCE) | $\mathbb{E}_{q}[h(z)\,\nabla_\phi \log q_\phi]$ | high | only that $\log q_\phi$ be differentiable |
| Reparameterized (pathwise) | $\mathbb{E}_{\epsilon}[\nabla_\phi h(T_\phi(x,\epsilon))]$ | low | $q_\phi$ reparameterizable, $h$ differentiable in $z$ |

The trick is what makes the VAE trainable end-to-end: it lets a
single gradient pass through the sampled code and into the encoder's weights.

## The closed-form KL term

The KL regularizer between two Gaussians has a closed form, so the second ELBO
term needs no sampling at all. For a diagonal posterior $q = \mathcal N(\mu, \diag\sigma^2)$
against the prior $p = \mathcal N(0, I)$ in $k$ latent dimensions,

$$
D_{KL}\!\parens{q_\phi(z\mid x)\,\|\,p(z)} = \frac{1}{2}\sum_{j=1}^{k}\parens{\mu_j^2 + \sigma_j^2 - \log \sigma_j^2 - 1}.
$$

Each coordinate is penalized for drifting its mean off $0$ ($\mu_j^2$) and for a
variance that strays from $1$ (the $\sigma_j^2 - \log\sigma_j^2 - 1$ term, which is
$\ge 0$ and vanishes only at $\sigma_j = 1$). Substituting the reparameterized
reconstruction term, the per-example objective the network maximizes is fully
explicit:

$$
\mathcal{L}(\theta, \phi; x) = \underbrace{\log p_\theta\!\parens{x \mid \mu_\phi(x) + \sigma_\phi(x)\odot\epsilon}}_{\text{one-sample reconstruction}} \;-\; \frac{1}{2}\sum_{j=1}^{k}\parens{\mu_j^2 + \sigma_j^2 - \log\sigma_j^2 - 1}.
$$

In practice the encoder emits $\log\sigma^2$ rather than $\sigma$ (the log keeps
the variance positive without a constraint and stabilizes the gradient), and the
expectation is approximated by the single noise draw $\epsilon$ per example. The
whole VAE is now a single differentiable graph.

### A worked ELBO computation

For example, take a
two-dimensional latent space ($k = 2$) and an encoder that, for a particular input
$x$, emits

$$
\mu = (0.8,\; -0.3),
\qquad
\sigma^2 = (0.5,\; 1.2),
\qquad\text{so}\quad
\log\sigma^2 = (-0.693,\; 0.182).
$$

**The KL term first**, coordinate by coordinate, using
$\tfrac12(\mu_j^2 + \sigma_j^2 - \log\sigma_j^2 - 1)$:

$$
\begin{aligned}
j = 1:&\quad \tfrac12\parens{0.64 + 0.5 - (-0.693) - 1} = \tfrac12(0.833) = 0.417, \\
j = 2:&\quad \tfrac12\parens{0.09 + 1.2 - 0.182 - 1} = \tfrac12(0.108) = 0.054.
\end{aligned}
$$

So $D_{KL}(q_\phi \,\|\, p) = 0.417 + 0.054 = 0.471$ nats. Coordinate $1$ pays most
of it: its mean sits well off the origin ($0.8$) _and_ its variance is squeezed to
$0.5$, both of which the prior penalizes. Coordinate $2$ is nearly free because its
mean is small and its variance is close to $1$ — almost exactly the prior already.

**Now the reconstruction term.** Draw one noise sample $\epsilon = (1.0, -0.5)$ and
reparameterize:

$$
z = \mu + \sigma\odot\epsilon
= \parens{0.8 + \sqrt{0.5}\cdot1.0,\;\; -0.3 + \sqrt{1.2}\cdot(-0.5)}
= (1.507,\; -0.848).
$$

Push $z$ through the decoder to get $g_\theta(z) = \hat x$. With a unit-variance
Gaussian decoder over, say, a $D = 4$-pixel output, the reconstruction log-density
is $\log p_\theta(x\mid z) = -\tfrac12\lVert x - \hat x\rVert^2 - \tfrac{D}{2}\log(2\pi)$.
If the squared reconstruction error came out to $\lVert x - \hat x\rVert^2 = 0.6$,
then

$$
\log p_\theta(x\mid z) = -0.30 - 2\log(2\pi) = -0.30 - 3.676 = -3.976.
$$

**The ELBO is their difference:**

$$
\mathcal{L} = \underbrace{-3.976}_{\text{reconstruction}} - \underbrace{0.471}_{\text{KL}} = -4.447 \text{ nats}.
$$

The optimizer minimizes $-\mathcal{L} = 4.447$. The KL
of $0.471$ is small next to the reconstruction cost, which is typical early in
training — the network first learns to reconstruct, and the KL only bites once the
codes are informative enough that packing them against the prior costs fidelity.
The $\beta$ weight below controls exactly that ratio.

$$
% caption: The full VAE: encoder $f_\phi$ produces $(\mu,\sigma)$, reparameterization samples $z=\mu+\sigma\odot\epsilon$, and decoder $g_\theta$ reconstructs $\hat x$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  v/.style={circle, draw, minimum size=10mm, inner sep=0pt},
  net/.style={draw, minimum width=15mm, minimum height=14mm, align=center},
  pr/.style={draw, minimum width=13mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[v] (x) at (0,0) {x};
  \node[net, draw=acc, text=acc, thick] (enc) at (2.2,0) {encoder\\f};
  \node[pr] (mu) at (4.5,0.95) {mean};
  \node[pr] (sg) at (4.5,-0.95) {scale};
  \node[v] (z) at (6.7,0) {z};
  \node[net] (dec) at (8.9,0) {decoder\\g};
  \node[v] (xh) at (11.1,0) {x'};
  \draw[->, acc, thick] (x) -- (enc);
  \draw[->, acc, thick] (enc) -- (mu);
  \draw[->, acc, thick] (enc) -- (sg);
  \draw[->, thick] (mu) -- (z);
  \draw[->, thick] (sg) -- (z);
  \draw[->, acc, thick] (z) -- (dec);
  \draw[->, acc, thick] (dec) -- (xh);
  \node[font=\scriptsize, text=acc, anchor=south] at (10.0,0.95) {recon\/struct};
  % noise into z
  \node[draw=black, dashed, minimum width=11mm, minimum height=7mm, font=\scriptsize] (eps) at (6.7,1.9) {noise};
  \draw[->, black, thick] (eps) -- (z) node[midway, right, font=\scriptsize] {sample};
  % prior pull
  \node[pr, draw=black] (prior) at (6.7,-2.0) {N(0, I)};
  \draw[<->, black, thick, dashed] (z) -- (prior) node[midway, right, font=\scriptsize] {KL pull};
  \node[font=\scriptsize, text=black, anchor=east] at (5.9,-2.0) {prior};
\end{tikzpicture}
$$

## Training the VAE

Training maximizes the ELBO by minimizing its negation, averaged over minibatches,
with a single reparameterized sample per example. Every line is an ordinary
differentiable operation, so a standard optimizer trains both networks jointly.

```algorithm
caption: $\textsc{TrainVAE}(f_\phi, g_\theta, \mathcal{D}, \eta)$ — amortized variational inference
initialize encoder $\phi$, decoder $\theta$
repeat
  sample a minibatch $\{x_i\} \sim \mathcal{D}$
  for each $x_i$ do
    $(\mu, s) \gets f_\phi(x_i)$ // encoder emits mean and $\log$-variance $s$
    $\epsilon \gets$ sample from $\mathcal{N}(0, I)$ // parameter-free noise
    $z \gets \mu + \exp(s / 2) \odot \epsilon$ // reparameterized code
    $L_{\text{rec}} \gets -\log p_\theta(x_i \mid z)$ // reconstruction (e.g. squared error)
    $L_{\text{kl}} \gets \tfrac12 \sum_j (\mu_j^2 + \exp(s_j) - s_j - 1)$ // closed-form KL
    $\mathcal{L}_i \gets L_{\text{rec}} + L_{\text{kl}}$ // negative ELBO
  $g \gets \nabla_{\phi,\theta}\, \tfrac{1}{m}\sum_i \mathcal{L}_i$ // backprop through the sample
  $(\phi, \theta) \gets (\phi, \theta) - \eta \cdot g$ // joint gradient step
until converged
return $\phi, \theta$
```

> **Remark (Amortized inference).** Classical variational inference optimizes a
> separate $q$ for every datum. The VAE instead trains one **inference network**
> $f_\phi$ to predict the posterior parameters from $x$ in a single forward pass —
> it _amortizes_ the per-example optimization into shared weights, so inference at
> test time is one cheap encoder evaluation.

## Sampling, interpolation, and the smooth latent space

Once trained, the encoder can be discarded for generation: because $q_\phi$ was
pressed against $p(z) = \mathcal N(0, I)$, an _unconditional_ sample from the prior
lands where the decoder expects data.

$$
z \sim \mathcal N(0, I), \qquad x \sim p_\theta(x \mid z) = g_\theta(z).
$$

This is what the KL term buys. In a plain autoencoder the codes occupy an
unknown, holey region, and a prior sample almost surely misses it; the VAE's
regularizer guarantees the prior _is_ the code distribution, so sampling works. The
same smoothness makes **latent-space interpolation** meaningful: a straight line
between two codes $z_a, z_b$ decodes to a continuous morph because nearby codes
decode to nearby data.[^chollet-vae]

$$
% caption: Latent-space interpolation: decoding $g_\theta(z)$ on a grid across the 2D latent space, icons morph continuously because the KL term made the space smooth.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  % axes for the 2D latent grid
  \draw[->, black, thick] (-0.55,-0.5) -- (4.7,-0.5) node[right, font=\scriptsize, black] {$z_1$};
  \draw[->, black, thick] (-0.55,-0.5) -- (-0.55,4.7) node[above, font=\scriptsize, black] {$z_2$};
  % 5x5 grid of morphing icons: radius shrinks left->right, squash grows bottom->top
  \foreach \i in {0,1,2,3,4} {
    \foreach \j in {0,1,2,3,4} {
      \pgfmathsetmacro{\rx}{0.34 - 0.045*\i}
      \pgfmathsetmacro{\ry}{0.16 + 0.045*\j}
      \draw[acc, thick, fill=acc!15] (\i,\j) ellipse ({\rx} and {\ry});
    }
  }
\end{tikzpicture}
$$

The grid is the canonical VAE diagnostic: sweeping the two latent coordinates and
decoding each point traces a continuous atlas of the data manifold, with no torn
seams between encodings.

## Two distributions, one pull

It helps to picture the KL term geometrically. The approximate posterior
$q_\phi(z \mid x)$ is a bell centered at $\mu_\phi(x)$ with width $\sigma_\phi(x)$;
the prior $p(z)$ is the unit bell at the origin. The KL regularizer pulls every
per-example bell toward the prior; the reconstruction term opposes it,
placing each $x$ at a distinct, confident code.

$$
% caption: The KL term pulls the posterior $q_\phi(z\mid x)$ toward the prior while the reconstruction term pushes its mean off-origin; their balance sets the learned posterior.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  % axis
  \draw[->, thick] (-3.6,0) -- (4.0,0) node[right, font=\footnotesize] {z};
  \draw[->, thick] (0,-0.2) -- (0,2.9) node[above, font=\footnotesize] {density};
  % prior p(z): unit bell at 0 (neutral dashed, light tint under it)
  \draw[black, very thick, dashed] plot[domain=-3.5:3.8, samples=120]
    (\x, {2.0*exp(-0.5*\x*\x)});
  \node[font=\footnotesize, anchor=south] at (-1.6,1.25) {prior p};
  % approximate posterior q: narrower bell shifted to mean ~ 2 (accent, tint fill)
  \draw[acc, very thick, fill=acc!15] plot[domain=-3.5:3.8, samples=140]
    (\x, {2.55*exp(-0.5*(\x-2)*(\x-2)/0.4)}) -- (3.8,0) -- (-3.5,0) -- cycle;
  \node[acc, font=\footnotesize, anchor=south west] at (2.15,2.05) {posterior q};
  % the KL pull arrow from q toward p
  \draw[->, black, thick] (1.9,0.55) -- (0.55,0.55);
  \node[font=\scriptsize, anchor=south] at (1.25,0.6) {KL pull};
\end{tikzpicture}
$$

> **Remark (Posterior collapse).** When the decoder is powerful enough to model
> $x$ on its own, the optimizer can drive $D_{KL}(q_\phi \,\|\, p)$ to zero by
> setting $q_\phi(z\mid x) = p(z)$ for every $x$: the encoder stops using $z$ and
> the latent code carries no information. Common remedies are **KL warm-up**
> (annealing a weight $\beta$ on the KL term from $0$ upward) and weakening or
> rate-limiting the decoder so it must rely on the code.

The weight $\beta$ generalizes the objective to $\mathcal{L}_\beta = \mathbb{E}_q[\log p_\theta(x\mid z)] - \beta\,D_{KL}(q_\phi \,\|\, p)$:
$\beta < 1$ favors reconstruction (sharper samples, looser prior match), $\beta > 1$
favors disentangled, prior-aligned codes (the $\beta$-VAE), and $\beta = 1$ is the
exact ELBO.[^gf-collapse]

$$
% caption: The $\beta$ tradeoff: raising $\beta$ tightens the code's match to the prior (KL falls) at the cost of reconstruction fidelity; $\beta<1$ sharpens samples, large $\beta$ risks posterior collapse.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \draw[->, thick] (0,0) -- (6.2,0) node[right] {KL weight};
  \draw[->, thick] (0,0) -- (0,3.6) node[above] {loss term};
  % reconstruction error rises with beta (decoder ignores code)
  \draw[acc, very thick] plot[domain=0.3:5.7, samples=40] (\x, {0.5 + 0.42*\x});
  \node[acc, anchor=south west] at (2.7,2.35) {recon. loss};
  % KL falls with beta
  \draw[green, very thick] plot[domain=0.3:5.7, samples=40] (\x, {3.0*exp(-0.55*\x)});
  \node[green, anchor=west] at (2.4,1.15) {KL to prior};
  % collapse zone
  \draw[black, dashed] (4.7,0) -- (4.7,3.4);
  \node[black, font=\scriptsize, anchor=south, align=center] at (5.35,2.9) {collapse\\risk};
\end{tikzpicture}
$$

## What VAEs became

The VAE of Kingma and Welling was published in 2014 alongside an independent
derivation by Rezende, Mohamed, and Wierstra, who called the same object a **deep
latent Gaussian model** and its reparameterized gradient **stochastic
backpropagation**.[^rezende] Both papers solved the same problem — how to
backpropagate through a sampled latent — and the trick has since become standard
machinery well outside generative modeling, anywhere an expectation over a learned
distribution appears in a loss.

**Discrete latents needed a new trick.** Reparameterization as stated requires a
continuous, differentiable transform, which rules out categorical codes. The
**Gumbel-softmax** (or Concrete) relaxation fixed this by reparameterizing a
categorical draw as a temperature-controlled softmax over Gumbel noise, recovering
a differentiable path to discrete latents.[^gumbel] The **VQ-VAE** took the opposite
route — a hard nearest-neighbor lookup into a learned codebook, with a
straight-through gradient — and its discrete codes are what let autoregressive and,
later, diffusion models operate in a compact latent space rather than on raw
pixels.[^vqvae]

**The $\beta$-VAE turned the KL weight into a research program.** Raising $\beta$
above $1$ was found to encourage **disentangled** representations, where individual
latent coordinates track independent factors of variation such as pose or lighting;
the same paper reframed the whole objective as a constrained optimization with the
KL as an information bottleneck.[^betavae] That information-theoretic reading —
the KL term is the number of nats the code is _allowed_ to carry about the input —
is the cleanest way to understand both disentanglement and posterior collapse.

**The blur problem drove the field toward other models.** VAEs reconstruct through
a likelihood term that, with a Gaussian decoder, is squared error, and averaging
over the posterior produces the characteristic soft, blurry samples. This is the
practical reason [GANs](/deep-learning/generative-models/generative-adversarial-networks)
and [diffusion models](/deep-learning/generative-models/diffusion-and-score-based-models)
overtook the plain VAE for high-fidelity image synthesis — though the VAE's
encoder-decoder latent space survives inside them, most visibly as the compression
front-end of latent diffusion.

## Takeaways

- A VAE is a **latent-variable model** $p_\theta(x) = \int p_\theta(x\mid z)\,p(z)\,dz$
  with prior $p(z) = \mathcal N(0, I)$; the integral and the posterior
  $p_\theta(z\mid x)$ are intractable, so we cannot do direct maximum likelihood.
- **Variational inference** introduces an encoder $q_\phi(z\mid x)$ and the
  **ELBO** $\log p_\theta(x) \ge \mathbb{E}_q[\log p_\theta(x\mid z)] - D_{KL}(q_\phi \,\|\, p)$,
  whose gap is exactly $D_{KL}(q_\phi \,\|\, p_\theta(z\mid x))$ — maximizing the
  bound both fits the data and tightens the approximation.
- The ELBO reads as **reconstruction minus a KL regularizer**; the KL term is the
  force a plain [autoencoder](/deep-learning/generative-models/autoencoders) lacks,
  packing codes against the prior so the latent space becomes smooth and samplable.
- The **reparameterization trick** $z = \mu_\phi(x) + \sigma_\phi(x)\odot\epsilon$
  moves randomness onto a parameter-free $\epsilon$, replacing the high-variance
  score-function estimator with a low-variance pathwise gradient that backpropagates
  through the sample.
- The Gaussian-vs-Gaussian KL is **closed-form**, sampling from the prior generates
  data, latent interpolation morphs smoothly, and **posterior collapse** is the
  characteristic failure when the decoder ignores the code.

[^gf-vae]: **Goodfellow**, _Deep Learning_, §20.10.3 — the variational autoencoder as a directed latent-variable model trained by maximizing the ELBO with an amortized inference network.
[^gf-inference]: **Goodfellow**, _Deep Learning_, Ch. 19 (§19.1) — inference as optimization: the true posterior $p_\theta(z\mid x)$ is intractable because of the evidence integral, motivating a variational surrogate.
[^gf-elbo]: **Goodfellow**, _Deep Learning_, §19.1 — the evidence lower bound $\mathcal L = \log p(x) - D_{KL}(q\,\|\,p(z\mid x))$, and why maximizing it both bounds the likelihood and tightens the posterior approximation.
[^gf-reparam]: **Goodfellow**, _Deep Learning_, §20.9 — the reparameterization (pathwise) gradient estimator, contrasted with the high-variance score-function/REINFORCE estimator of §20.9.1.
[^chollet-vae]: **Chollet**, _Deep Learning with Python_, §8.4 — generating images with a VAE in practice: the encoder emits $\log\sigma^2$, the sampling layer applies $z=\mu+\sigma\odot\epsilon$, and decoding a latent grid traces a continuous manifold.
[^gf-collapse]: **Goodfellow**, _Deep Learning_, §20.10.3 — the failure mode where a sufficiently powerful decoder ignores $z$ and drives the KL to zero (posterior collapse), and the role of a weight on the KL term.
[^rezende]: **Rezende, Mohamed & Wierstra**, "Stochastic Backpropagation and Approximate Inference in Deep Generative Models," ICML 2014 — the independent derivation of the reparameterized gradient for deep latent Gaussian models.
[^gumbel]: **Jang, Gu & Poole**, "Categorical Reparameterization with Gumbel-Softmax," ICLR 2017 (and **Maddison, Mnih & Teh**, "The Concrete Distribution," ICLR 2017) — a differentiable relaxation that reparameterizes discrete categorical latents.
[^vqvae]: **van den Oord, Vinyals & Kavukcuoglu**, "Neural Discrete Representation Learning" (VQ-VAE), NeurIPS 2017 — discrete latents via a learned codebook and straight-through gradients.
[^betavae]: **Higgins et al.**, "$\beta$-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework," ICLR 2017 — weighting the KL term above one to encourage disentangled representations.