---
title: Approximate Inference
module: Probabilistic Methods
moduleNumber: 8
lessonNumber: 3
order: 803
summary: >
  In a latent-variable model the quantity we need, the posterior $p(h\mid v)$ over
  hidden causes, is almost never computable, because its normalizer is an
  intractable sum over configurations. Approximate inference reframes the problem as
  optimization: maximize the evidence lower bound, a tractable functional whose gap
  to the true log-evidence equals a KL divergence. From that single bound fall
  expectation–maximization, mean-field variational inference, MAP, and the learned
  encoders behind variational autoencoders.
topics: [Probabilistic Methods]
sources:
  - book: Goodfellow
    ref: "Ch. 19 — Approximate Inference"
  - book: Goodfellow
    ref: "§19.1 Inference as Optimization; §19.2 Expectation Maximization; §19.4 Mean Field"
  - book: Goodfellow
    ref: "§19.3 MAP Inference; §20.10.3 — Amortized Inference (VAEs)"
---

A [structured probabilistic model](/deep-learning/probabilistic-methods/structured-probabilistic-models)
with latent variables explains each observation $v$ as generated from a
hidden cause $h$: a joint $p(v, h\mid\theta)$ factorizes through the latents, and
training maximizes the **marginal likelihood** $p(v) = \sum_h p(v, h)$. Both
learning and prediction route through one object, the **posterior** over the
hidden cause,

$$
p(h \mid v) = \frac{p(v, h)}{p(v)} = \frac{p(v, h)}{\sum_{h'} p(v, h')}.
$$

The numerator is a single, cheap product of model factors. The denominator is the
problem: a sum over _every_ configuration of $h$. For $h \in \{0,1\}^m$ that is
$2^m$ terms, and for the graphs that make a model interesting the sum admits no
factorization.

> **Definition (Inference).** Given an observation $v$, computing (or summarizing)
> the posterior $p(h\mid v)$ over the latent variables; equivalently, computing
> expectations $\mathbb{E}_{h\sim p(h\mid v)}[f(h)]$ under it. Exact inference means
> evaluating the normalizer $p(v) = \sum_h p(v,h)$.

> **Theorem (Intractability of exact inference).** Computing $p(h\mid v)$ exactly is
> $\mathsf{NP}$-hard for general graphical models; for a fully connected latent
> layer the marginal $p(v)=\sum_h p(v,h)$ has no sub-exponential evaluation.

> **Proof (sketch).** Exact marginalization over a layer of $m$ binary latents with
> dense pairwise couplings computes the partition function of an Ising model, whose
> evaluation is $\#\mathsf{P}$-complete; a polynomial algorithm would collapse the
> counting hierarchy. The sum's $2^m$ terms do not factor when every $h_i$ couples to
> $v$ and to its neighbors, so dynamic programming over a low-treewidth structure is
> unavailable. $\qed$

Instead of computing $p(h\mid v)$, we _search_ for a tractable distribution $q(h)$
that stands in for it: inference as **optimization**.[^gf-inference]

## Inference as optimization: the evidence lower bound

Introduce an arbitrary distribution $q(h)$ over the latents. We do not yet say what
$q$ is; we only multiply and divide the log-evidence by it and apply Jensen's
inequality. Start from $\log p(v) = \log\sum_h p(v,h)$ and insert $q$:

$$
\log p(v)
= \log \sum_h q(h)\,\frac{p(v,h)}{q(h)}
= \log \mathbb{E}_{h\sim q}\!\brackets{\frac{p(v,h)}{q(h)}}.
$$

The logarithm is concave, so by **Jensen's inequality** the log of the expectation
dominates the expectation of the log:

$$
\log p(v) \;\ge\; \mathbb{E}_{h\sim q}\!\brackets{\log \frac{p(v,h)}{q(h)}}
\;=:\; \mathcal{L}(v,\theta,q).
$$

This functional $\mathcal{L}$ is the **evidence lower bound** (ELBO). It is
tractable whenever $q$ is: it requires only expectations under our chosen $q$, never
the intractable sum. The gap between $\log p(v)$ and $\mathcal{L}$ equals a KL
divergence, which the next identity makes precise.

> **Definition (Evidence lower bound).** For any distribution $q$ over the latents,
> $\mathcal{L}(v,\theta,q) = \mathbb{E}_{h\sim q}[\log p(v,h\mid\theta)] - \mathbb{E}_{h\sim q}[\log q(h)]$.
> The second term is the entropy $H(q)$, so $\mathcal{L} = \mathbb{E}_q[\log p(v,h)] + H(q)$.

Expand the joint as $p(v,h) = p(h\mid v)\,p(v)$ inside $\mathcal{L}$ and the bound
splits:

$$
\mathcal{L}(v,\theta,q)
= \mathbb{E}_{h\sim q}\!\brackets{\log \frac{p(h\mid v)\,p(v)}{q(h)}}
= \log p(v) + \mathbb{E}_{h\sim q}\!\brackets{\log \frac{p(h\mid v)}{q(h)}}.
$$

The remaining expectation is the negative KL divergence from $q$ to the true
posterior, giving the **central decomposition** of approximate inference:

$$
\;\mathcal{L}(v,\theta,q) = \log p(v) - D_{\mathrm{KL}}\!\parens{q(h)\,\|\,p(h\mid v)}\;
$$

Because $D_{\mathrm{KL}} \ge 0$ always, $\mathcal{L} \le \log p(v)$ (the bound is
confirmed), and the slack is _precisely_ how far $q$ sits from the posterior. Two
consequences drive everything below.

> **Theorem (Tightness of the ELBO).** For fixed $\theta$ and $v$, the bound
> $\mathcal{L}(v,\theta,q) \le \log p(v)$ holds for every $q$, with equality **iff**
> $q(h) = p(h\mid v)$. Maximizing $\mathcal{L}$ over $q$ minimizes
> $D_{\mathrm{KL}}(q\,\|\,p(h\mid v))$ and tightens the bound.

> **Proof.** From the decomposition, $\log p(v) - \mathcal{L}(v,\theta,q) = D_{\mathrm{KL}}(q\,\|\,p(h\mid v)) \ge 0$,
> so $\mathcal{L} \le \log p(v)$. Gibbs' inequality gives $D_{\mathrm{KL}}(q\,\|\,p)=0$
> iff $q=p$ almost everywhere; since $\log p(v)$ does not depend on $q$, maximizing
> $\mathcal{L}$ over $q$ is identical to minimizing that KL. $\qed$

The figure below shows this: the log-evidence is a fixed ceiling,
and the ELBO rises toward it as the KL gap closes.

$$
% caption: The log-evidence is fixed and splits as ELBO plus a KL gap; as $q$ approaches the posterior, the gap shrinks and the bound rises.
\begin{tikzpicture}[font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % fixed ceiling = log evidence
  \draw[dashed, black] (-0.3,4.2) -- (10.6,4.2);
  \node[anchor=west, text=black] at (10.6,4.2) {log-evidence};
  % three stacked bars: bound (blue, solid outline) + gap (lighter), gap shrinking left to right
  % bar 1: loose
  \draw[acc, very thick, fill=acc!22] (0.4,0) rectangle (1.9,1.7);
  \draw[acc, very thick, fill=acc!13] (0.4,1.7) rectangle (1.9,4.2);
  \node[anchor=north, text=acc] at (1.15,-0.1) {loose $q$};
  % bar 2: better
  \draw[acc, very thick, fill=acc!22] (3.4,0) rectangle (4.9,3.0);
  \draw[acc, very thick, fill=acc!13] (3.4,3.0) rectangle (4.9,4.2);
  \node[anchor=north, text=acc] at (4.15,-0.1) {better $q$};
  % bar 3: tight
  \draw[acc, very thick, fill=acc!22] (6.4,0) rectangle (7.9,4.05);
  \draw[acc, very thick, fill=acc!13] (6.4,4.05) rectangle (7.9,4.2);
  \node[anchor=north, text=acc] at (7.15,-0.1) {$q = $ posterior};
  % labels for the two pieces on bar 1
  \node[anchor=west, text=acc] at (8.6,1.0) {bound};
  \node[anchor=west, text=acc] at (8.6,2.9) {KL gap};
  % baseline
  \draw[->, thick] (0,0) -- (10.4,0);
  \node[green, anchor=south west] at (6.3,4.25) {gap vanishes};
\end{tikzpicture}
$$

The one bound supports two distinct optimizations:

- **Learning**: hold $q$ fixed and maximize over $\theta$, raising a tractable
  surrogate for the likelihood.
- **Inference**: hold $\theta$ fixed and maximize over $q$, tightening the bound
  toward the likelihood.

Alternating them is the entire expectation–maximization algorithm.[^gf-elbo]

## Expectation–maximization

Suppose the posterior $p(h\mid v;\theta)$ _is_ tractable for fixed $\theta$ (it is
for mixtures, factor analyzers, HMMs). Then the inference step is exact: the $q$ that
maximizes $\mathcal{L}$ is the posterior itself, driving the KL gap to zero.
**Expectation–maximization** (EM) alternates this exact inference with a likelihood
climb in $\theta$.

> **Definition (EM steps).** Given current parameters $\theta^{(t)}$:
> the **E-step** sets $q^{(t)}(h) = p(h\mid v;\theta^{(t)})$, maximizing
> $\mathcal{L}$ over $q$, so $\mathcal{L}(v,\theta^{(t)},q^{(t)}) = \log p(v;\theta^{(t)})$;
> the **M-step** sets $\theta^{(t+1)} = \arg\max_\theta \mathbb{E}_{h\sim q^{(t)}}[\log p(v,h\mid\theta)]$,
> the only $\theta$-dependent part of $\mathcal{L}$, since $H(q^{(t)})$ is constant.

The M-step maximizes the **expected complete-data log-likelihood**, often written
$Q(\theta\mid\theta^{(t)}) = \mathbb{E}_{h\sim q^{(t)}}[\log p(v,h\mid\theta)]$. With
the latents' responsibilities $q^{(t)}$ frozen, this is usually a closed-form
weighted fit: for a Gaussian mixture it reduces to the weighted means and
covariances. The essential guarantee is that the true marginal likelihood never
decreases.

> **Theorem (Monotone ascent of EM).** Each EM iteration satisfies
> $\log p(v;\theta^{(t+1)}) \ge \log p(v;\theta^{(t)})$, with equality only at a
> stationary point of the likelihood.

> **Proof.** After the E-step the bound is tight:
> $\mathcal{L}(v,\theta^{(t)},q^{(t)}) = \log p(v;\theta^{(t)})$. The M-step chooses
> $\theta^{(t+1)}$ to maximize $\mathcal{L}(v,\cdot,q^{(t)})$, so
> $\mathcal{L}(v,\theta^{(t+1)},q^{(t)}) \ge \mathcal{L}(v,\theta^{(t)},q^{(t)})$.
> The bound still holds at the new parameters,
> $\log p(v;\theta^{(t+1)}) \ge \mathcal{L}(v,\theta^{(t+1)},q^{(t)})$. Chaining,
>
> $$
> \log p(v;\theta^{(t+1)}) \;\ge\; \mathcal{L}(v,\theta^{(t+1)},q^{(t)})
> \;\ge\; \mathcal{L}(v,\theta^{(t)},q^{(t)}) \;=\; \log p(v;\theta^{(t)}).
> $$
>
> The likelihood is non-decreasing and, bounded above, converges. $\qed$

Geometrically, the E-step lifts a tractable lower-bound curve until
it _touches_ the likelihood at $\theta^{(t)}$; the M-step climbs that surrogate to
its peak; the next E-step builds a fresh tangent bound there. The likelihood rises
along a staircase of tangent lower bounds.

$$
% caption: EM as alternating optimization: the E-step builds a lower bound (black) tangent to $\log p(v)$ (blue), the M-step jumps to its maximum, and likelihood never decreases.
\begin{tikzpicture}[font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % axes
  \draw[->, thick] (-0.2,0) -- (9.4,0) node[right] {parameter};
  \draw[->, thick] (0,-0.2) -- (0,4.6) node[above] {log-prob};
  % the true log-likelihood curve (a broad hill)
  \draw[acc, very thick] plot[domain=0.3:9.0, samples=80]
    (\x, {3.7*exp(-0.10*(\x-6.4)*(\x-6.4)) + 0.5});
  \node[acc, anchor=south west] at (7.2,3.9) {$\log p(v)$};
  % first tangent lower bound, touching at x=2.6
  \draw[black, thick] plot[domain=1.0:4.4, samples=40]
    (\x, {-0.45*(\x-2.6)*(\x-2.6) + 1.66});
  \fill[acc] (2.6,1.66) circle (2.2pt);
  \draw[black, dashed] (2.6,0) -- (2.6,1.66);
  \node[anchor=north] at (2.6,-0.05) {\texttt{theta(t)}};
  % second tangent lower bound, touching at x=4.6 (M-step peak of first bound)
  \draw[black, thick] plot[domain=3.0:6.3, samples=40]
    (\x, {-0.45*(\x-4.6)*(\x-4.6) + 2.96});
  \fill[acc] (4.6,2.96) circle (2.2pt);
  \draw[black, dashed] (4.6,0) -- (4.6,2.96);
  \node[anchor=north] at (4.6,-0.05) {\texttt{theta(t+1)}};
  % climb arrow
  \draw[->, green, very thick] (2.75,1.78) .. controls (3.6,2.7) .. (4.45,2.86);
  \node[green, anchor=north] at (3.7,0.95) {M-step maximize};
  \node[black, anchor=west] at (0.3,3.7) {E-step: \texttt{tangent bound}};
\end{tikzpicture}
$$

```algorithm
caption: $\textsc{ExpectationMaximization}(v, p, \theta_0)$ — alternate exact inference and a likelihood climb
initialize $\theta \gets \theta_0$
repeat
  $q(h) \gets p(h \mid v;\ \theta)$ // E-step: exact posterior, bound becomes tight
  $\theta \gets \arg\max_{\theta'}\ \mathbb{E}_{h\sim q}[\log p(v, h \mid \theta')]$ // M-step: maximize expected complete-data log-likelihood
until $\log p(v;\ \theta)$ stops increasing
return $\theta$
```

#### A worked E-step: responsibilities in a two-component mixture

For example, take a one-dimensional mixture of two
Gaussians with equal mixing weights $\pi_1 = \pi_2 = 0.5$, means $\mu_1 = 0$,
$\mu_2 = 4$, and unit variances, and ask for the posterior over the latent component
$h \in \{1, 2\}$ that generated an observed point $v = 1$. The E-step is Bayes' rule on
two numbers. The unnormalized responsibilities are the prior times the likelihood,

$$
\tilde r_1 = 0.5 \cdot \mathcal{N}(1; 0, 1) = 0.5 \cdot \frac{e^{-1/2}}{\sqrt{2\pi}}
\approx 0.5 \cdot 0.2420 = 0.1210,
$$
$$
\tilde r_2 = 0.5 \cdot \mathcal{N}(1; 4, 1) = 0.5 \cdot \frac{e^{-9/2}}{\sqrt{2\pi}}
\approx 0.5 \cdot 0.0044 = 0.0022,
$$

and normalizing gives the posterior $q(h) = p(h \mid v)$ that the E-step returns,

$$
r_1 = \frac{0.1210}{0.1210 + 0.0022} \approx 0.982,
\qquad
r_2 \approx 0.018.
$$

The point at $v = 1$ is assigned almost entirely to component $1$, consistent with its
proximity to $\mu_1 = 0$. The M-step then refits each Gaussian's mean as the
responsibility-weighted average of the data, $\mu_k \gets \sum_n r_{k}(v_n)\, v_n /
\sum_n r_{k}(v_n)$ — a soft $k$-means, where the responsibilities are the soft
assignments. This is the entire EM loop: the E-step turns each point into
a distribution over components, the M-step refits the components to those soft counts,
and the marginal likelihood climbs at every round.

EM is the special case where the E-step is _exact_.[^gf-em] When the posterior is itself
intractable, the E-step can only be approximated, and that is variational
inference. The simplest approximation is a point mass.

## MAP inference as a point-mass $q$

The crudest approximation collapses $q$ onto a single point: a **maximum a
posteriori** (MAP) estimate keeps only the most probable latent configuration and
discards the rest of the posterior.

> **Definition (MAP inference).** Restrict $q$ to Dirac point masses
> $q(h) = \delta(h - \mu)$. The ELBO term $\mathbb{E}_q[\log p(v,h)] = \log p(v,\mu)$
> and the (degenerate) entropy is dropped as a constant, so maximizing the bound over
> $\mu$ reduces to $\mu^\star = \arg\max_h \log p(v, h) = \arg\max_h \log p(h\mid v)$.

MAP is the limit of a variational $q$ whose spread is forced to zero: a Gaussian
$q = \mathcal{N}(h;\mu,\beta^{-1}I)$ as the precision $\beta \to \infty$. The bound it
maximizes keeps the joint at the mode but throws away all posterior uncertainty —
fast and often good enough for a point prediction, but it ignores multimodality and
variance.[^gf-map] The variational families below retain more of the posterior.

## Variational inference and the mean field

When $p(h\mid v)$ is intractable, restrict $q$ to a family $\mathcal{Q}$ simple
enough to optimize, and find the closest member. This is **variational inference**:
minimize $D_{\mathrm{KL}}(q\,\|\,p(h\mid v))$ over $q\in\mathcal{Q}$, equivalently
maximize $\mathcal{L}$ over the family.

> **Definition (Variational inference).** Choose a tractable family $\mathcal{Q}$ and
> set $q^\star = \arg\max_{q\in\mathcal{Q}} \mathcal{L}(v,\theta,q) = \arg\min_{q\in\mathcal{Q}} D_{\mathrm{KL}}(q\,\|\,p(h\mid v))$.
> The bound is now generally loose: the residual KL is the price of the restriction.

Geometrically, the true posterior sits outside the family, and inference projects it
onto $\mathcal{Q}$ in the (reverse) KL geometry, landing on the member that best
matches it where $q$ places its mass.

$$
% caption: The intractable posterior $p(h\mid v)$ (red) lies outside the tractable family $\mathcal{Q}$; variational inference returns its closest member $q^\star$ in KL.
\begin{tikzpicture}[font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  % the family as a curved manifold (a long shallow arc)
  \draw[black, very thick] plot[domain=-0.2:7.2, samples=60]
    (\x, {1.0 + 0.18*(\x-3.5)*(\x-3.5)*0.18});
  \node[black, anchor=west] at (6.6,1.9) {family $Q$};
  % a few members on the manifold
  \foreach \x in {1.0,2.3,3.5,4.7,6.0}
    \fill[black] (\x, {1.0 + 0.18*(\x-3.5)*(\x-3.5)*0.18}) circle (1.6pt);
  % true posterior off the manifold
  \fill[red] (3.5,4.2) circle (3pt);
  \node[red, anchor=south] at (3.5,4.35) {posterior};
  % projection point q*
  \fill[green] (3.5,1.0) circle (3pt);
  \node[green, anchor=north] at (3.5,0.78) {best member};
  % projection arrow (KL)
  \draw[->, acc, very thick] (3.5,4.05) -- (3.5,1.2);
  \node[acc, anchor=west] at (3.62,2.6) {minimize KL};
\end{tikzpicture}
$$

The standard choice for $\mathcal{Q}$ is the **mean-field** family: assume the
latents are independent under $q$,

$$
q(h) = \prod_{i=1}^m q_i(h_i).
$$

The true posterior generally couples the latents: knowing $h_1$ shifts the belief
about $h_2$. Mean field severs every such edge and keeps only the marginals. The
diagram contrasts the two graphs.

$$
% caption: Mean-field severs the posterior's couplings: the true $p(h\mid v)$ (left) links the latents, while the factorized $q=\prod_i q_i$ (right) keeps only independent marginals.
\begin{tikzpicture}[font=\footnotesize,
  lat/.style={circle, draw, minimum size=8mm, inner sep=1pt}]
  \definecolor{acc}{HTML}{2348F2}
  % left: coupled posterior
  \node[lat] (a1) at (0,1.4)   {$h_1$};
  \node[lat] (a2) at (1.5,1.4) {$h_2$};
  \node[lat] (a3) at (0.75,0)  {$h_3$};
  \draw[thick] (a1) -- (a2);
  \draw[thick] (a1) -- (a3);
  \draw[thick] (a2) -- (a3);
  \node[anchor=north] at (0.75,-0.7) {true posterior};
  \node[anchor=north] at (0.75,-1.2) {(coupled)};
  % arrow
  \node[acc] at (3.6,0.7) {\texttt{--->}};
  \node[acc, anchor=south] at (3.6,0.95) {\texttt{mean field}};
  % right: factorized q
  \node[lat, draw=acc, text=acc] (b1) at (5.4,1.4)   {$h_1$};
  \node[lat, draw=acc, text=acc] (b2) at (6.9,1.4) {$h_2$};
  \node[lat, draw=acc, text=acc] (b3) at (6.15,0)  {$h_3$};
  \node[anchor=north, text=acc] at (6.15,-0.7) {factorized $q$};
  \node[anchor=north, text=acc] at (6.15,-1.2) {\texttt{(independent)}};
\end{tikzpicture}
$$

No couplings, so each factor optimizes almost independently. To derive the update
for one factor $q_j$, isolate its contribution to $\mathcal{L}$. Writing
$\tilde p(v,h) = \log p(v,h)$ and holding all other factors fixed,

$$
\mathcal{L} = \mathbb{E}_{q}[\log p(v,h)] - \sum_i \mathbb{E}_{q_i}[\log q_i]
= \mathbb{E}_{q_j}\!\brackets{\mathbb{E}_{q_{-j}}[\log p(v,h)]} - \mathbb{E}_{q_j}[\log q_j] + \text{const},
$$

where $q_{-j}$ denotes all factors except the $j$-th and the constant absorbs the
other entropies. Define the partial expectation
$\log \tilde q_j(h_j) := \mathbb{E}_{q_{-j}}[\log p(v,h)]$. Then the $q_j$-dependent
part is

$$
\mathcal{L}(q_j) = \mathbb{E}_{q_j}\!\brackets{\log \frac{\tilde q_j(h_j)}{q_j(h_j)}} + \text{const}
= -\,D_{\mathrm{KL}}\!\parens{q_j \,\|\, \tilde q_j} + \text{const},
$$

a single KL in $q_j$. It is maximized when $q_j \propto \tilde q_j$, giving the
**mean-field coordinate update**:

$$
\;\log q_j^\star(h_j) = \mathbb{E}_{q_{-j}}\!\brackets{\log p(v,h)} + \text{const}\;
\qquad
q_j^\star(h_j) \propto \exp\!\parens{\mathbb{E}_{q_{-j}}[\log p(v,h)]}.
$$

Each factor is the exponentiated expected log-joint, averaged over the _current_
estimates of every other factor: a fixed-point iteration that cycles through the
coordinates and is guaranteed to raise $\mathcal{L}$ at every step.[^gf-meanfield]

```algorithm
caption: $\textsc{MeanField}(v, p)$ — coordinate ascent on the factorized ELBO
initialize each factor $q_i$ (e.g. uniform)
repeat
  for each latent $j \gets 1$ to $m$ do
    $\log q_j(h_j) \gets \mathbb{E}_{q_{-j}}[\log p(v, h)] + c$ // hold other factors fixed
    normalize $q_j$ so $\sum_{h_j} q_j(h_j) = 1$
until $\mathcal{L}(v, \theta, q)$ converges
return $q = \prod_i q_i$
```

The bound is genuinely loose here, and the _direction_ of the KL we minimize
determines the bias of the answer.

> **Theorem (Mode-seeking bias of variational inference).** Minimizing
> $D_{\mathrm{KL}}(q\,\|\,p)$ — the reverse KL — penalizes $q$ for placing mass where
> $p$ is small but not for missing mass where $p$ is large. The optimal $q$ is
> **mode-seeking**: it concentrates on one mode of a multimodal $p$ and underestimates
> its variance, rather than averaging across modes.

> **Proof (sketch).** The integrand $q(h)\log\frac{q(h)}{p(h)}$ blows up wherever
> $q>0$ but $p\approx 0$, so the minimizer forces $q\to 0$ there; conversely regions
> with $p>0$ but $q\approx 0$ contribute $\approx 0$, incurring no penalty. Thus $q$
> is driven to sit _inside_ the support of one mode and avoid the low-density valleys
> between modes — it shrinks rather than spreads. (Minimizing the _forward_
> $D_{\mathrm{KL}}(p\,\|\,q)$ instead is mean-seeking, but requires expectations under
> the intractable $p$.) $\qed$

The two KL directions are not interchangeable; the choice is forced by tractability
and it determines the qualitative failure mode of the approximation.

| Objective | Direction | Expectations under | Behaviour | Tractable? |
| --- | --- | --- | --- | --- |
| $D_{\mathrm{KL}}(q\,\|\,p)$ | reverse | $q$ (chosen, simple) | mode-seeking, under-dispersed | yes — used in VI |
| $D_{\mathrm{KL}}(p\,\|\,q)$ | forward | $p$ (intractable posterior) | mean-seeking, mass-covering | no — needs $p(h\mid v)$ |

Variational inference is committed to the reverse KL precisely because it requires
only expectations under the $q$ we control, the same reason it inherits the
mode-seeking bias.

For a correlated true posterior, the mean-field factorization cannot represent the
correlation at all: its independence assumption forces an axis-aligned $q$, which
fits inside the tilted true density and underestimates the variance along the
correlated directions.

$$
% caption: A correlated posterior $p(h\mid v)$ (red, tilted) approximated by a factorized $q$ (blue, axis-aligned): mean-field cannot tilt, so it underestimates variance.
\begin{tikzpicture}[font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, thick] (-2.7,0) -- (2.9,0) node[right] {$h_1$};
  \draw[->, thick] (0,-2.5) -- (0,2.7) node[above] {$h_2$};
  % true posterior: tilted ellipse (correlated)
  \begin{scope}[rotate=33]
    \draw[red, very thick] (0,0) ellipse (2.3 and 0.85);
    \draw[red!75, thick] (0,0) ellipse (1.5 and 0.55);
  \end{scope}
  \node[red, anchor=west] at (1.7,1.85) {true posterior};
  % mean-field q: axis-aligned ellipse, smaller (inside)
  \draw[acc, very thick] (0,0) ellipse (1.05 and 1.05);
  \draw[acc, thick] (0,0) ellipse (0.65 and 0.65);
  \node[acc, anchor=west] at (1.15,-1.55) {factorized $q$};
  % center
  \fill[black] (0,0) circle (1.4pt);
\end{tikzpicture}
$$

## Amortized inference: learn the encoder

Mean-field solves a _fresh_ optimization for every datapoint $v$, wasteful when
millions of $v$ share structure. **Amortized inference** replaces per-example
optimization with a single learned function: an **inference network** (encoder)
$f_\phi$ that maps any $v$ directly to the parameters of its approximate posterior.

> **Definition (Amortized inference).** Train one network $q_\phi(h\mid v)$ whose
> output is the variational distribution for $v$ — e.g. $f_\phi(v) = (\mu, \log\sigma^2)$
> parameterizing $\mathcal{N}(h;\mu,\sigma^2 I)$. Inference for a new $v$ is one
> forward pass; the cost of optimization is _amortized_ across the dataset by sharing
> $\phi$.

The ELBO becomes the training objective for $\phi$ (and jointly $\theta$): maximize
$\mathbb{E}_{v}\brackets{\mathcal{L}(v,\theta,q_\phi(\cdot\mid v))}$ by gradient ascent.
This recovers the encoder of a
[variational autoencoder](/deep-learning/generative-models/variational-autoencoders):
the inference network is the encoder, the generative model $p(v\mid h)$ is the
decoder, and the **reparameterization trick** lets the gradient of $\mathcal{L}$ flow
back through the sampled $h$ into $\phi$.[^chollet-vae]

### The reparameterization trick

Gradient ascent on the amortized ELBO needs $\nabla_\phi \mathcal{L}$, and $\mathcal{L}$
contains an expectation over $h\sim q_\phi(h\mid v)$ whose _distribution_ depends on
$\phi$. Differentiating through a sample is the obstacle: the naive score-function
estimator $\nabla_\phi\mathbb{E}_{q_\phi}[g(h)] = \mathbb{E}_{q_\phi}[g(h)\,\nabla_\phi\log q_\phi(h\mid v)]$
is unbiased but high-variance, because it never uses the derivative of $g$ and instead
weights raw samples by a log-density gradient.

The trick removes $\phi$ from the source of randomness. For a Gaussian
$q_\phi(h\mid v) = \mathcal{N}(h;\mu_\phi(v),\sigma_\phi^2(v)I)$, sample a
parameter-free noise variable and push it through a deterministic map,

$$
\varepsilon \sim \mathcal{N}(0, I), \qquad h = \mu_\phi(v) + \sigma_\phi(v)\odot\varepsilon,
$$

so $h$ is now a differentiable function of $\phi$ with the randomness isolated in
$\varepsilon$. The expectation moves onto the fixed base distribution and the gradient
passes straight through the sample,

$$
\nabla_\phi\,\mathbb{E}_{h\sim q_\phi}[g(h)]
= \nabla_\phi\,\mathbb{E}_{\varepsilon\sim\mathcal{N}(0,I)}\!\brackets{g\!\parens{\mu_\phi(v) + \sigma_\phi(v)\odot\varepsilon}}
= \mathbb{E}_{\varepsilon}\!\brackets{\nabla_\phi\, g\!\parens{\mu_\phi(v) + \sigma_\phi(v)\odot\varepsilon}}.
$$

The interchange of $\nabla_\phi$ and $\mathbb{E}_\varepsilon$ is valid because the
measure no longer carries $\phi$; a single Monte Carlo sample of $\varepsilon$ gives a
low-variance, unbiased gradient. Backpropagation flows through the two deterministic
paths $\mu_\phi$ and $\sigma_\phi$ and stops at the noise input, which has no gradient.

$$
% caption: The reparameterization trick. The encoder emits $\mu$ and $\sigma$; a sample $h = \mu + \sigma\odot\varepsilon$ is deterministic given the external noise $\varepsilon$, so the gradient (dashed) flows back through $\mu$ and $\sigma$ but not through $\varepsilon$.
\begin{tikzpicture}[font=\footnotesize,
  box/.style={draw, minimum width=13mm, minimum height=8mm, align=center},
  op/.style={circle, draw, minimum size=6mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % input
  \node[box] (v) at (0,0) {input $v$};
  % encoder
  \node[box, draw=acc, text=acc] (enc) at (2.3,0) {encoder\\$f$};
  % mu and sigma outputs
  \node[box] (mu) at (4.7,0.9) {mean};
  \node[box] (sig) at (4.7,-0.9) {std};
  % noise (external)
  \node[box, draw=red, text=red] (eps) at (4.7,-2.6) {noise};
  % combine node
  \node[op] (comb) at (7.0,0) {$h$};
  % decoder
  \node[box, draw=acc, text=acc] (dec) at (9.2,0) {decoder\\$g$};
  % forward edges
  \draw[->, thick] (v) -- (enc);
  \draw[->, thick] (enc) -- (mu);
  \draw[->, thick] (enc) -- (sig);
  \draw[->, thick] (mu) -- (comb);
  \draw[->, thick] (sig) -- (comb);
  \draw[->, red, thick] (eps) -- (comb);
  \draw[->, thick] (comb) -- (dec);
  % gradient path (dashed, back through mu/sigma only)
  \draw[<-, green, dashed, thick] (mu.south east) to[bend left=12] (comb.north);
  \draw[<-, green, dashed, thick] (sig.north east) to[bend right=12] (comb.south);
  \node[green, anchor=west] at (6.9,1.7) {gradient path};
  \node[red, anchor=north] at (4.1,-3.15) {no gradient};
  \node[anchor=north] at (7.9,-3.15) {$h = $ mean $+$ std \texttt{x} noise};
\end{tikzpicture}
$$

Without this reroute the VAE could not train by ordinary backpropagation; with it, the
encoder, the sampling step, and the decoder form one differentiable graph.[^chollet-vae]

$$
% caption: Amortized inference: one shared encoder $f_\phi$ maps each input $v$ to its posterior parameters, replacing per-datapoint optimization with a single forward pass.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  io/.style={circle, draw, minimum size=9mm, inner sep=1pt},
  net/.style={draw, thick, minimum width=20mm, minimum height=16mm, align=center},
  par/.style={draw, minimum width=15mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  % three inputs sharing one encoder
  \node[io] (v1) at (0,1.7)  {$v_1$};
  \node[io] (v2) at (0,0)    {$v_2$};
  \node[io] (v3) at (0,-1.7) {$v_3$};
  % the shared encoder
  \node[net, draw=acc, text=acc] (enc) at (3.2,0) {encoder\\$f$};
  % outputs: per-input posterior params
  \node[par] (p1) at (6.4,1.7)  {mean, var};
  \node[par] (p2) at (6.4,0)    {mean, var};
  \node[par] (p3) at (6.4,-1.7) {mean, var};
  % edges in
  \draw[->, thick] (v1) -- (enc);
  \draw[->, thick] (v2) -- (enc);
  \draw[->, thick] (v3) -- (enc);
  % edges out
  \draw[->, acc, thick] (enc) -- (p1);
  \draw[->, acc, thick] (enc) -- (p2);
  \draw[->, acc, thick] (enc) -- (p3);
  \node[anchor=west, text=acc] at (5.7,2.6) {posterior params};
  \node[anchor=east] at (0.5,2.6) {inputs};
  \node[text=acc, anchor=north] at (3.2,-1.1) {shared weights};
\end{tikzpicture}
$$

## Variational inference versus MCMC

Variational methods are not the only route to an intractable posterior. **Markov chain
Monte Carlo** (MCMC) makes the opposite trade-off: instead of optimizing a simple $q$, it
builds a Markov chain whose stationary distribution is exactly $p(h\mid v)$, then draws
correlated samples from it. The two families sit at opposite ends of a bias–variance
and speed trade-off.

> **Takeaway.** Variational inference is **biased but fast** — it commits to a restricted
> family $\mathcal{Q}$, so a residual KL gap never closes, but each estimate is a
> deterministic optimization that returns a usable $q$ in bounded time, differentiable and
> friendly to gradient training. MCMC is **asymptotically exact but slow** — with enough
> samples its estimates are unbiased for the true posterior, but convergence is only
> guaranteed in the limit, mixing can be slow, and there is no differentiable objective to
> plug into an encoder.

For training deep latent-variable models on millions of examples, the variational
side wins: a biased gradient available in one forward and backward pass beats an
unbiased estimate that needs a long chain per datapoint. MCMC remains the reference when
an unbiased answer is worth the cost.[^gf-meanfield]

## The methods of approximate inference

Every method above is one choice of $q$ optimized against the same ELBO; they differ
only in _what $q$ is allowed to be_, how it is computed, and therefore how exact and
how costly the resulting inference is.

| Method | Form of $q$ | Inference cost | KL gap (exactness) |
| --- | --- | --- | --- |
| Exact / EM E-step | $q = p(h\mid v)$, full posterior | tractable only for special models | zero — bound is tight |
| MAP | point mass $\delta(h-\mu)$ | one optimization for the mode | large — discards all variance |
| Mean-field VI | factorized $\prod_i q_i(h_i)$ | coordinate ascent per datapoint | nonzero — misses correlations |
| Amortized VI | $q_\phi(h\mid v)$, one shared net | one forward pass after training | nonzero $+$ amortization gap |

EM is the exact corner; MAP the degenerate corner; mean-field trades exactness for a
tractable factorization; amortization trades a little more exactness for inference
in a single forward pass. Across the table, one quantity varies: how much of the
true posterior $q$ is permitted to capture, with cost and fidelity trading off
against each other.

## Richer posteriors and tighter bounds

The mean-field family is deliberately crude — its independence assumption is the source
of the mode-seeking, variance-shrinking bias diagrammed above. Three developments since
Goodfellow (2016) relax that restriction while keeping the ELBO machinery intact.

**Normalizing-flow posteriors** widen the family $\mathcal{Q}$. Rezende & Mohamed (2015,
_ICML_) replace the factorized Gaussian $q_\phi(h \mid v)$ with a simple base
distribution pushed through a chain of invertible maps $h = f_K \circ \cdots \circ f_1(z_0)$,
whose change-of-variables Jacobian keeps the density tractable. The resulting $q$ can be
correlated and multimodal, so the axis-aligned ellipse from the correlated-posterior
figure can tilt to match the true density — the KL gap shrinks because the family
is no longer forced to be independent.

**Tighter bounds** attack the slack directly. The importance-weighted autoencoder (IWAE;
Burda et al., 2016, _ICLR_) averages $K$ importance samples _inside_ the log,

$$
\mathcal{L}_K = \mathbb{E}\!\brackets{\log \frac{1}{K}\sum_{k=1}^{K}
\frac{p(v, h_k)}{q_\phi(h_k \mid v)}}
\;\ge\; \mathcal{L}_1,
$$

and this multi-sample bound is provably tighter than the single-sample ELBO, approaching
$\log p(v)$ as $K \to \infty$. It trades compute for exactness: $K$ posterior samples per
gradient step instead of one.

**Black-box / stochastic VI** removes the last hand-derivation. Ranganath et al. (2014,
_AISTATS_) and Hoffman et al. (2013, _JMLR_) showed the ELBO gradient can be estimated by
Monte Carlo for _any_ model — using the score-function estimator, or the
reparameterization gradient this lesson derived — so variational inference no longer needs
a model-specific mean-field update worked out by hand. This is what makes the
probabilistic-programming systems of the previous lesson possible: write the generative
model, and the ELBO is optimized automatically. All three developments (richer $q$,
tighter bound, automatic gradient) still maximize the same evidence lower bound this
lesson is built on.

## Takeaways

- **Inference is intractable** because $p(h\mid v)$ needs the normalizer
  $p(v)=\sum_h p(v,h)$, an exponential sum with no factorization for interesting
  models; the solution is to reframe inference as **optimization** over a surrogate $q$.
- The **ELBO** $\mathcal{L}(v,\theta,q) = \log p(v) - D_{\mathrm{KL}}(q\,\|\,p(h\mid v))$
  lower-bounds the log-evidence with slack equal to a KL; maximizing over $q$ tightens
  it, maximizing over $\theta$ learns the model.
- **EM** is exact inference (E-step: $q=p(h\mid v)$) alternated with a likelihood
  climb (M-step: maximize $\mathbb{E}_q[\log p(v,h)]$); the likelihood is **monotone
  non-decreasing**, a staircase of tangent lower bounds.
- **MAP** is the point-mass $q$; **mean-field VI** factorizes $q=\prod_i q_i$ with the
  coordinate update $q_j^\star\propto\exp(\mathbb{E}_{q_{-j}}[\log p(v,h)])$, biased
  **mode-seeking** because it minimizes the reverse KL.
- **Amortized inference** learns one encoder $q_\phi(h\mid v)$ that emits posterior
  parameters in a forward pass, exactly the encoder of a
  [variational autoencoder](/deep-learning/generative-models/variational-autoencoders).

[^gf-inference]: **Goodfellow**, _Deep Learning_, §19.1 — Inference as Optimization: recasting the intractable posterior computation as maximizing the evidence lower bound over a surrogate $q$.
[^gf-elbo]: **Goodfellow**, _Deep Learning_, §19.1 — the ELBO decomposition $\mathcal{L} = \log p(v) - D_{\mathrm{KL}}(q\,\|\,p(h\mid v))$ and its two optimizations (learning in $\theta$, inference in $q$).
[^gf-em]: **Goodfellow**, _Deep Learning_, §19.2 — Expectation Maximization: the exact-E-step special case and the monotone-ascent guarantee on the marginal likelihood.
[^gf-map]: **Goodfellow**, _Deep Learning_, §19.3 — MAP Inference and Sparse Coding: the point-mass $q=\delta(h-\mu)$ as the zero-variance limit of a variational family.
[^gf-meanfield]: **Goodfellow**, _Deep Learning_, §19.4 — Variational Inference and Learning: the mean-field factorization, the coordinate-ascent fixed point, and the mode-seeking bias of the reverse KL.
[^chollet-vae]: **Chollet**, _Deep Learning with Python_, Ch. 12 — the VAE encoder as an amortized inference network emitting $(\mu, \log\sigma^2)$, trained end-to-end through the reparameterization trick.