Approximate Inference

A structured probabilistic model with latent variables explains each observation $v$ as generated from a hidden cause $h$: a joint $p(v,h\mid \theta )$ factorizes through the latents, and training maximizes the marginal likelihood $p(v)={\sum }_{h}p(v,h)$. Both learning and prediction route through one object, the posterior over the hidden cause,

The numerator is a single, cheap product of model factors. The denominator is the problem: a sum over every configuration of $h$. For $h\in \{0,1{\}}^m$ that is $2^m$ terms, and for the graphs that make a model interesting the sum admits no factorization.

Instead of computing $p(h\mid v)$, we search for a tractable distribution $q(h)$ that stands in for it: inference as optimization.¹

Inference as optimization: the evidence lower bound

Introduce an arbitrary distribution $q(h)$ over the latents. We do not yet say what $q$ is; we only multiply and divide the log-evidence by it and apply Jensen's inequality. Start from $\log p(v)=\log {\sum }_{h}p(v,h)$ and insert $q$:

The logarithm is concave, so by Jensen's inequality the log of the expectation dominates the expectation of the log:

This functional $\mathcal{L}$ is the evidence lower bound (ELBO). It is tractable whenever $q$ is: it requires only expectations under our chosen $q$, never the intractable sum. The gap between $\log p(v)$ and $\mathcal{L}$ equals a KL divergence, which the next identity makes precise.

Expand the joint as $p(v,h)=p(h\mid v)p(v)$ inside $\mathcal{L}$ and the bound splits:

The remaining expectation is the negative KL divergence from $q$ to the true posterior, giving the central decomposition of approximate inference:

Because $D_{\mathrm{KL}}\ge 0$ always, $\mathcal{L}\le \log p(v)$ (the bound is confirmed), and the slack is precisely how far $q$ sits from the posterior. Two consequences drive everything below.

The figure below shows this: the log-evidence is a fixed ceiling, and the ELBO rises toward it as the KL gap closes.

The one bound supports two distinct optimizations:

• Learning: hold $q$ fixed and maximize over $\theta$, raising a tractable surrogate for the likelihood.
• Inference: hold $\theta$ fixed and maximize over $q$, tightening the bound toward the likelihood.

Alternating them is the entire expectation–maximization algorithm.²

Expectation–maximization

Suppose the posterior $p(h\mid v;\theta )$ is tractable for fixed $\theta$ (it is for mixtures, factor analyzers, HMMs). Then the inference step is exact: the $q$ that maximizes $\mathcal{L}$ is the posterior itself, driving the KL gap to zero. Expectation–maximization (EM) alternates this exact inference with a likelihood climb in $\theta$.

The M-step maximizes the expected complete-data log-likelihood, often written $Q(\theta \mid {\theta }^{(t)})={\mathbb{E}}_{h\sim q^{(t)}}[\log p(v,h\mid \theta )]$. With the latents' responsibilities $q^{(t)}$ frozen, this is usually a closed-form weighted fit: for a Gaussian mixture it reduces to the weighted means and covariances. The essential guarantee is that the true marginal likelihood never decreases.

Geometrically, the E-step lifts a tractable lower-bound curve until it touches the likelihood at ${\theta }^{(t)}$; the M-step climbs that surrogate to its peak; the next E-step builds a fresh tangent bound there. The likelihood rises along a staircase of tangent lower bounds.

A worked E-step: responsibilities in a two-component mixture

For example, take a one-dimensional mixture of two Gaussians with equal mixing weights ${\pi }_1={\pi }_2=0.5$, means ${\mu }_1=0$, ${\mu }_2=4$, and unit variances, and ask for the posterior over the latent component $h\in \{1,2\}$ that generated an observed point $v=1$. The E-step is Bayes' rule on two numbers. The unnormalized responsibilities are the prior times the likelihood,

and normalizing gives the posterior $q(h)=p(h\mid v)$ that the E-step returns,

The point at $v=1$ is assigned almost entirely to component $1$, consistent with its proximity to ${\mu }_1=0$. The M-step then refits each Gaussian's mean as the responsibility-weighted average of the data, ${\mu }_k\leftarrow {\sum }_n{r}_k(v_n)v_n/{\sum }_n{r}_k(v_n)$ — a soft $k$-means, where the responsibilities are the soft assignments. This is the entire EM loop: the E-step turns each point into a distribution over components, the M-step refits the components to those soft counts, and the marginal likelihood climbs at every round.

EM is the special case where the E-step is exact.³ When the posterior is itself intractable, the E-step can only be approximated, and that is variational inference. The simplest approximation is a point mass.

MAP inference as a point-mass $q$

The crudest approximation collapses $q$ onto a single point: a maximum a posteriori (MAP) estimate keeps only the most probable latent configuration and discards the rest of the posterior.

MAP is the limit of a variational $q$ whose spread is forced to zero: a Gaussian $q=\mathcal{N}(h;\mu ,{\beta }^{-1}I)$ as the precision $\beta \to \infty$. The bound it maximizes keeps the joint at the mode but throws away all posterior uncertainty — fast and often good enough for a point prediction, but it ignores multimodality and variance.⁴ The variational families below retain more of the posterior.

Variational inference and the mean field

When $p(h\mid v)$ is intractable, restrict $q$ to a family $\mathcal{Q}$ simple enough to optimize, and find the closest member. This is variational inference: minimize $D_{\mathrm{KL}}(q\Vert p(h\mid v))$ over $q\in \mathcal{Q}$, equivalently maximize $\mathcal{L}$ over the family.

Geometrically, the true posterior sits outside the family, and inference projects it onto $\mathcal{Q}$ in the (reverse) KL geometry, landing on the member that best matches it where $q$ places its mass.

The standard choice for $\mathcal{Q}$ is the mean-field family: assume the latents are independent under $q$,

The true posterior generally couples the latents: knowing $h_1$ shifts the belief about $h_2$. Mean field severs every such edge and keeps only the marginals. The diagram contrasts the two graphs.

No couplings, so each factor optimizes almost independently. To derive the update for one factor $q_j$, isolate its contribution to $\mathcal{L}$. Writing $\tilde{p}(v,h)=\log p(v,h)$ and holding all other factors fixed,

where $q_{-j}$ denotes all factors except the $j$-th and the constant absorbs the other entropies. Define the partial expectation $\log {\tilde{q}}_j(h_j):={\mathbb{E}}_{q_{-j}}[\log p(v,h)]$. Then the $q_j$-dependent part is

a single KL in $q_j$. It is maximized when $q_j\propto {\tilde{q}}_j$, giving the mean-field coordinate update:

Each factor is the exponentiated expected log-joint, averaged over the current estimates of every other factor: a fixed-point iteration that cycles through the coordinates and is guaranteed to raise $\mathcal{L}$ at every step.⁵

The bound is genuinely loose here, and the direction of the KL we minimize determines the bias of the answer.

The two KL directions are not interchangeable; the choice is forced by tractability and it determines the qualitative failure mode of the approximation.

Variational inference is committed to the reverse KL precisely because it requires only expectations under the $q$ we control, the same reason it inherits the mode-seeking bias.

For a correlated true posterior, the mean-field factorization cannot represent the correlation at all: its independence assumption forces an axis-aligned $q$, which fits inside the tilted true density and underestimates the variance along the correlated directions.

Amortized inference: learn the encoder

Mean-field solves a fresh optimization for every datapoint $v$, wasteful when millions of $v$ share structure. Amortized inference replaces per-example optimization with a single learned function: an inference network (encoder) $f_{\varphi }$ that maps any $v$ directly to the parameters of its approximate posterior.

The ELBO becomes the training objective for $\varphi$ (and jointly $\theta$): maximize ${\mathbb{E}}_v\left[\mathcal{L}(v,\theta ,q_{\varphi }(\cdot \mid v))\right]$ by gradient ascent. This recovers the encoder of a variational autoencoder: the inference network is the encoder, the generative model $p(v\mid h)$ is the decoder, and the reparameterization trick lets the gradient of $\mathcal{L}$ flow back through the sampled $h$ into $\varphi$.⁶

The reparameterization trick

Gradient ascent on the amortized ELBO needs ${\nabla }_{\varphi }\mathcal{L}$, and $\mathcal{L}$ contains an expectation over $h\sim q_{\varphi }(h\mid v)$ whose distribution depends on $\varphi$. Differentiating through a sample is the obstacle: the naive score-function estimator ${\nabla }_{\varphi }{\mathbb{E}}_{q_{\varphi }}[g(h)]={\mathbb{E}}_{q_{\varphi }}[g(h){\nabla }_{\varphi }\log q_{\varphi }(h\mid v)]$ is unbiased but high-variance, because it never uses the derivative of $g$ and instead weights raw samples by a log-density gradient.

The trick removes $\varphi$ from the source of randomness. For a Gaussian $q_{\varphi }(h\mid v)=\mathcal{N}(h;{\mu }_{\varphi }(v),{\sigma }_{\varphi }^2(v)I)$, sample a parameter-free noise variable and push it through a deterministic map,

so $h$ is now a differentiable function of $\varphi$ with the randomness isolated in $\epsilon$. The expectation moves onto the fixed base distribution and the gradient passes straight through the sample,

The interchange of ${\nabla }_{\varphi }$ and ${\mathbb{E}}_{\epsilon }$ is valid because the measure no longer carries $\varphi$; a single Monte Carlo sample of $\epsilon$ gives a low-variance, unbiased gradient. Backpropagation flows through the two deterministic paths ${\mu }_{\varphi }$ and ${\sigma }_{\varphi }$ and stops at the noise input, which has no gradient.

Without this reroute the VAE could not train by ordinary backpropagation; with it, the encoder, the sampling step, and the decoder form one differentiable graph.⁶

Variational inference versus MCMC

Variational methods are not the only route to an intractable posterior. Markov chain Monte Carlo (MCMC) makes the opposite trade-off: instead of optimizing a simple $q$, it builds a Markov chain whose stationary distribution is exactly $p(h\mid v)$, then draws correlated samples from it. The two families sit at opposite ends of a bias–variance and speed trade-off.

For training deep latent-variable models on millions of examples, the variational side wins: a biased gradient available in one forward and backward pass beats an unbiased estimate that needs a long chain per datapoint. MCMC remains the reference when an unbiased answer is worth the cost.⁵

The methods of approximate inference

Every method above is one choice of $q$ optimized against the same ELBO; they differ only in what $q$ is allowed to be, how it is computed, and therefore how exact and how costly the resulting inference is.

EM is the exact corner; MAP the degenerate corner; mean-field trades exactness for a tractable factorization; amortization trades a little more exactness for inference in a single forward pass. Across the table, one quantity varies: how much of the true posterior $q$ is permitted to capture, with cost and fidelity trading off against each other.

Richer posteriors and tighter bounds

The mean-field family is deliberately crude — its independence assumption is the source of the mode-seeking, variance-shrinking bias diagrammed above. Three developments since Goodfellow (2016) relax that restriction while keeping the ELBO machinery intact.

Normalizing-flow posteriors widen the family $\mathcal{Q}$. Rezende & Mohamed (2015, ICML) replace the factorized Gaussian $q_{\varphi }(h\mid v)$ with a simple base distribution pushed through a chain of invertible maps $h=f_K\circ \cdots \circ f_1(z_0)$, whose change-of-variables Jacobian keeps the density tractable. The resulting $q$ can be correlated and multimodal, so the axis-aligned ellipse from the correlated-posterior figure can tilt to match the true density — the KL gap shrinks because the family is no longer forced to be independent.

Tighter bounds attack the slack directly. The importance-weighted autoencoder (IWAE; Burda et al., 2016, ICLR) averages $K$ importance samples inside the log,

and this multi-sample bound is provably tighter than the single-sample ELBO, approaching $\log p(v)$ as $K\to \infty$. It trades compute for exactness: $K$ posterior samples per gradient step instead of one.

Black-box / stochastic VI removes the last hand-derivation. Ranganath et al. (2014, AISTATS) and Hoffman et al. (2013, JMLR) showed the ELBO gradient can be estimated by Monte Carlo for any model — using the score-function estimator, or the reparameterization gradient this lesson derived — so variational inference no longer needs a model-specific mean-field update worked out by hand. This is what makes the probabilistic-programming systems of the previous lesson possible: write the generative model, and the ELBO is optimized automatically. All three developments (richer $q$, tighter bound, automatic gradient) still maximize the same evidence lower bound this lesson is built on.

Takeaways

• Inference is intractable because $p(h\mid v)$ needs the normalizer $p(v)={\sum }_{h}p(v,h)$, an exponential sum with no factorization for interesting models; the solution is to reframe inference as optimization over a surrogate $q$.
• The ELBO $\mathcal{L}(v,\theta ,q)=\log p(v)-D_{\mathrm{KL}}(q\Vert p(h\mid v))$ lower-bounds the log-evidence with slack equal to a KL; maximizing over $q$ tightens it, maximizing over $\theta$ learns the model.
• EM is exact inference (E-step: $q=p(h\mid v)$) alternated with a likelihood climb (M-step: maximize ${\mathbb{E}}_q[\log p(v,h)]$); the likelihood is monotone non-decreasing, a staircase of tangent lower bounds.
• MAP is the point-mass $q$; mean-field VI factorizes $q={\prod }_i{q}_i$ with the coordinate update $q_j^{\star }\propto \exp ({\mathbb{E}}_{q_{-j}}[\log p(v,h)])$, biased mode-seeking because it minimizes the reverse KL.
• Amortized inference learns one encoder $q_{\varphi }(h\mid v)$ that emits posterior parameters in a forward pass, exactly the encoder of a variational autoencoder.