Autoregressive Models & Normalizing Flows

The variational autoencoder optimizes a bound on $\log p(x)$; the GAN never writes a density at all. This lesson covers the two families that compute $\log p(x)$ exactly (no bound, no adversary) by constraining the architecture so the likelihood stays tractable. Autoregressive models accept slow sampling; normalizing flows accept a restricted class of invertible maps. Both are trained by plain maximum likelihood.¹

Autoregressive models: the chain rule as architecture

Any joint distribution over an ordered vector $x=(x_1,\dots ,x_d)$ factors exactly, with no assumptions, by the probability chain rule:

Taking logs turns the product into a sum, and the training objective is the negative log-likelihood averaged over the data, one term per coordinate:

Generation walks the same factorization left to right: sample $x_1\sim p(x_1)$, feed it back to get $p(x_2\mid x_1)$, sample $x_2$, and so on. Each draw conditions on every coordinate already produced.²

The defining tension is built into this picture. The likelihood is a single forward pass (every conditional evaluated in parallel because the true $x_{<i}$ are known at training time), but sampling is inherently sequential: $x_i$ cannot be drawn until $x_{i-1}$ exists.

Enforcing the ordering with masks

The architectural problem is leakage: a naive network mixing all inputs would let the prediction of $x_i$ peek at $x_i$ itself or at future coordinates, making the conditional ill-defined. The fix across the whole family is the same: zero out the forbidden connections so output $i$ is a function of inputs $<i$ only.

MADE (Masked Autoencoder for Distribution Estimation) realizes this in a plain autoencoder. Assign each hidden unit a number $m$ in $\{1,\dots ,d-1\}$; keep a weight from input $k$ to a unit only if $k\le m$, and from a unit to output $i$ only if $m<i$. Element-wise multiplying each weight matrix by a binary mask $M$ enforces the constraint with no change to the forward cost:³

and the masks are chosen so that $\partial {\hat{x}}_i/\partial x_k=0$ whenever $k\ge i$ — exactly the autoregressive property.

PixelRNN and PixelCNN apply the same idea to images, generating pixels in raster order. PixelRNN threads an LSTM along rows; PixelCNN is faster, replacing the recurrence with a stack of masked convolutions. The convolution kernel is masked so the receptive field covers only pixels above the current one, and to its left within the current row; the future half of the kernel is zeroed.

WaveNet carries the construction to raw audio, where $d$ is tens of thousands of samples. It stacks dilated causal convolutions: causal so output $t$ depends only on samples $\le t$, dilated so the receptive field grows exponentially with depth: a stack of $L$ layers with dilations $1,2,4,\dots ,2^{L-1}$ reaches back $2^L$ samples while keeping the parameter count linear in $L$.

Normalizing flows: change of variables

Flows take a different route to an exact likelihood. Start from a simple base density $p_z$ (a standard Gaussian) and push a sample through an invertible, differentiable map $f$:

The density of $x$ follows from conservation of probability mass. A region of $z$-space of volume $dz$ carries probability $p_z(z)dz$; the map $f$ stretches it to a region of volume $|\det J_f|dz$ in $x$-space, where $J_f=\partial f/\partial z$ is the Jacobian. Mass is preserved, so the density must scale inversely with the volume.⁴

Writing $p_x(x)|\det J_f|=p_z(z)$ and substituting $z=f^{-1}(x)$ gives the change-of-variables formula, on which the whole family rests:

In log space, products become sums and the inverse becomes a sign flip:

The objective is now plain maximum likelihood, computed exactly:

The Jacobian-determinant bottleneck

The formula is exact but useless unless three operations are cheap. Computing $\det J_f$ for a general dense $J$ costs $O(d^3)$, fatal for image-sized $d$. Every flow architecture is therefore engineered around making the Jacobian structured so its determinant is trivial.

The recurring trick: if $J_f$ is triangular, its determinant is just the product of its diagonal entries, $\det J={\prod }_i{J}_{ii}$ — an $O(d)$ computation. The question becomes how to build an expressive bijection whose Jacobian is triangular.

Coupling layers: NICE and RealNVP

The answer is the coupling layer. Partition the coordinates into two blocks, $x=(x_a,x_b)$. Copy the first block unchanged; transform the second block with a function whose parameters are produced by a network reading only the first block:

where $s$ and $t$ (scale and translation) are arbitrary neural networks; they need not be invertible themselves. This is the affine coupling of RealNVP; dropping the scale ($s\equiv 0$) recovers the additive coupling of NICE.

Two properties make the coupling layer the standard building block of flows. First, it is trivially invertible without inverting $s$ or $t$ — given $(y_a,y_b)$, recover $x_a=y_a$, recompute $s(x_a),t(x_a)$, and solve:

Second, the Jacobian is triangular. Since $y_a=x_a$ and $y_b$ depends on $x_b$ only through the diagonal scaling $\exp (s(x_a))$:

The block-triangular structure means the determinant ignores the messy off-diagonal block entirely:

The expensive determinant has collapsed to a sum of the scale outputs, an $O(d)$ operation, independent of $s$ and $t$'s complexity.⁵

A worked Jacobian and likelihood

Trace a single point through one affine coupling layer with real numbers. Work in $d=2$, split as $x_a=x_1$ and $x_b=x_2$, and suppose the scale and shift networks happen to output $s(x_a)=0.5x_a$ and $t(x_a)=x_a$. Take the input $x=(2,3)$, so $x_a=2$ gives $s=1.0$ and $t=2.0$. The forward map is

The Jacobian of this map is lower-triangular,

and its log-determinant ignores the off-diagonal $\star$ entirely: $\log |\det J|={\sum }_k{s}_k=1.0$. Confirm the lemma numerically — the determinant is $1\cdot e^{1.0}=2.718$, whose log is $1.0$, exactly the sum of the scale outputs, no matter what $\star$ equals.

Now read off the likelihood. Under a standard Gaussian base, the point maps back to $z=f^{-1}(x)$ during evaluation; for a single-layer flow the base point is the pre-image, but to make the change-of-variables arithmetic concrete, suppose an input whose inverse is $z=(0.5,-1.0)$ with accumulated log-determinant $\ell =1.0$. The base log-density of a 2-D unit Gaussian is $\log p_z(z)=-\frac{1}{2}\Vert z{\Vert }^2-\frac{d}{2}\log (2\pi )=-\frac{1}{2}(0.25+1.0)-\log (2\pi )=-0.625-1.838=-2.463$. The change-of-variables formula then gives

The stretch cost $\ell$ is subtracted directly: because this layer expanded volume (positive log-scale $s=1.0$), it spread the base mass thinner, so the data density is lower than the base density at the corresponding point. A contracting layer ($s<0$) would add to the log-likelihood. Stacking $K$ layers just accumulates $\ell ={\sum }_k{\ell }_k$, and every term is an $O(d)$ sum of scale outputs.

Composing flows

A single coupling layer leaves half its coordinates untouched, so flows alternate which block is transformed and stack many layers. Because the composition of diffeomorphisms is a diffeomorphism, the whole stack is one valid flow, and its log-determinant is just the sum of the per-layer terms, the chain rule applied to the determinant:

with $h_0=z$ and $h_k=f_k(h_{k-1})$. Each term is the $O(d)$ coupling determinant above, so the total cost stays linear.

The exact-likelihood families, side by side

Autoregressive models and flows are duals: an autoregressive model is itself a flow whose Jacobian is triangular by ordering rather than by partition, which is why inverse autoregressive flows and masked autoregressive flows unify the two.⁶ Each family makes a different trade among the three things a generative model is asked to do — score a sample, draw a sample, and draw a good sample.

Each family trades one axis for the other two. Autoregressive models get exactness and quality at the cost of slow sampling; flows get exactness and fast sampling at the cost of a constrained, less expressive map; the VAE gets fast sampling and a flexible decoder at the cost of exactness (a bound); the GAN gets sharp samples at the cost of the likelihood entirely. No single family wins on all three axes, which is why the field keeps all four, and why hybrids (flow-decoded VAEs, autoregressive flows) combine one family's strength with another's. The next lesson turns to a fifth route, the energy-based and Boltzmann machines that model an unnormalized density, with the difficulty moved to the partition function.

Flows and autoregressive models today

Both families kept advancing after Goodfellow's text, and their two headline weaknesses — slow autoregressive sampling, weak flow expressiveness — were the targets.

Autoregressive models became the backbone of modern generation. WaveNet's dilated-causal design set the template, and the Transformer is now the dominant autoregressive architecture: a masked (causal) self-attention layer applies the leakage-prevention mask of this lesson to a sequence, and every large language model generates by the same left-to-right chain rule.⁷ The slow-sampling problem drove parallel WaveNet, which distilled a trained autoregressive teacher into an inverse-autoregressive-flow student that samples in one pass — a direct use of the AR/flow duality noted above.⁸

Flows gained expressiveness and exact-inverse attention. Kingma and Dhariwal's Glow added invertible $1\times 1$ convolutions (a learned channel permutation with a cheap determinant) and actnorm, producing the first flow with GAN-competitive high-resolution samples.⁹ Continuous normalizing flows then replaced the discrete stack with an ordinary differential equation, computing the log-determinant as an integral of a trace via the Hutchinson estimator — the FFJORD model — which removed the architectural constraints on the Jacobian entirely.¹⁰

The families converged with diffusion. A diffusion model can be read as a deep, fixed flow whose per-step Jacobian is trivial, and score-based models share the continuous-flow ODE formulation; the next lesson's energy-based models and the diffusion models after them inherit the exact-likelihood ambitions of this lesson while relaxing the invertibility that made flows rigid.

Takeaways

• Autoregressive models factor the joint exactly by the chain rule $p(x)={\prod }_{i}p(x_i\mid x_{<i})$ and learn each conditional with a network; likelihood is one parallel pass, but sampling is $d$ sequential steps.
• Masked connectivity (MADE's $W\odot M$, PixelCNN's zeroed kernel half, WaveNet's dilated causal convolutions) enforces the ordering so output $i$ cannot see input $\ge i$.
• Normalizing flows push a Gaussian base through an invertible map and read off $\log p_x(x)=\log p_z(z)-\log |\det J_f|$: exact likelihood, contingent on a cheap Jacobian determinant.
• Coupling layers (NICE/RealNVP) split coordinates and transform one half conditioned on the other, yielding a triangular Jacobian whose log-determinant is just ${\sum }_{k}s(x_a{)}_k$; composing many such layers keeps the cost $O(d)$.
• The four families trade among exact likelihood, sampling speed, and sample quality; no family dominates, which is the reason all of them persist.