---
title: Autoregressive Models & Normalizing Flows
module: Generative Models
moduleNumber: 7
lessonNumber: 5
order: 705
summary: >
  Two families that provide exact likelihoods, each at a cost. Autoregressive models
  factor the joint by the probability chain rule and learn each conditional with
  a masked network: exact $\log p(x)$, but sampling proceeds one coordinate at a
  time. Normalizing flows push a simple base density through an invertible map
  and read $\log p(x)$ off the change-of-variables formula, trading architectural
  freedom for a cheap Jacobian determinant via triangular coupling layers.
topics: [Generative Models]
sources:
  - book: Goodfellow
    ref: "§20.10.7 — Autoregressive Networks; §20.10.8 NADE"
  - book: Goodfellow
    ref: "§20.10.9 — Generative Stochastic Networks; §3.9.2 change of variables"
  - book: Goodfellow
    ref: "§20.10.10 — Other Generation Schemes (flows, inverse autoregressive)"
---

The [variational autoencoder](/deep-learning/generative-models/variational-autoencoders)
optimizes a _bound_ on $\log p(x)$; the
[GAN](/deep-learning/generative-models/generative-adversarial-networks) never
writes a density at all. This lesson covers the two families that compute
$\log p(x)$ **exactly** (no bound, no adversary) by constraining the
architecture so the likelihood stays tractable. **Autoregressive models** accept
slow sampling; **normalizing flows** accept a restricted class of
invertible maps. Both are trained by plain maximum likelihood.[^gf-exact]

## Autoregressive models: the chain rule as architecture

Any joint distribution over an ordered vector $x = (x_1, \dots, x_d)$ factors
**exactly**, with no assumptions, by the probability chain rule:

$$
p(x) = \prod_{i=1}^{d} p(x_i \mid x_1, \dots, x_{i-1}) = \prod_{i=1}^{d} p(x_i \mid x_{<i}).
$$

> **Definition (Autoregressive model).** A density model that fixes an ordering
> of the coordinates and parameterizes each conditional $p(x_i \mid x_{<i})$ by a
> neural network sharing parameters across positions. The factorization is exact,
> so the only modeling choice is the form of each conditional.

Taking logs turns the product into a sum, and the training objective is the
negative log-likelihood averaged over the data, one term per coordinate:

$$
\log p_\theta(x) = \sum_{i=1}^{d} \log p_\theta(x_i \mid x_{<i}),
\qquad
\mathcal{L}(\theta) = -\frac{1}{n}\sum_{j=1}^{n} \sum_{i=1}^{d} \log p_\theta\parens{x_i^{(j)} \mid x_{<i}^{(j)}}.
$$

Generation walks the same factorization left to right: sample $x_1 \sim p(x_1)$,
feed it back to get $p(x_2 \mid x_1)$, sample $x_2$, and so on. Each draw
conditions on every coordinate already produced.[^gf-ar]

$$
% caption: Autoregressive generation in raster order: pixel $x_i$ (green) is sampled from $p(x_i \mid x_{<i})$, conditioned on the committed pixels (blue).
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % 4x4 grid; cells (row r from top, col c)
  \foreach \r in {0,1,2,3} {
    \foreach \c in {0,1,2,3} {
      \pgfmathsetmacro{\x}{\c*1.0}
      \pgfmathsetmacro{\y}{-\r*1.0}
      \node[draw, black, minimum size=10mm, inner sep=0pt] (n\r\c) at (\x,\y) {};
    }
  }
  % committed cells (raster order up to current): row 0 all, row 1 cols 0,1
  \foreach \r/\c in {0/0,0/1,0/2,0/3,1/0,1/1} {
    \draw[acc, thick, fill=acc!15] (n\r\c.south west) rectangle (n\r\c.north east);
  }
  % current cell being drawn: row 1 col 2
  \draw[green, very thick, fill=green!15] (n12.south west) rectangle (n12.north east);
  \node[green] at (n12) {$x_i$};
  % dependency arrows into the current cell from a few prior cells
  \draw[->, acc, thick] (n11.east) -- (n12.west);
  \draw[->, acc, thick] (n02.south) -- (n12.north);
  \draw[->, acc, thick] (n10.south east) .. controls +(0.5,-0.4) .. (n12.south west);
  % raster-order sweep arrow below
  \draw[->, black, thick] (-0.5,-3.9) -- (3.5,-3.9);
  \node[black, anchor=north, font=\footnotesize] at (1.5,-4.0) {raster scan order};
\end{tikzpicture}
$$

The defining tension is built into this picture. The likelihood is a single
forward pass (every conditional evaluated in parallel because the true
$x_{<i}$ are known at training time), but **sampling is inherently
sequential**: $x_i$ cannot be drawn until $x_{i-1}$ exists.

| Operation | Cost | Why |
| --- | --- | --- |
| Likelihood $\log p(x)$ | one parallel forward pass | all $x_{<i}$ are observed, so every conditional evaluates at once |
| Sampling | $d$ sequential passes | $x_i$ depends on the freshly sampled $x_{<i}$ |
| Training | parallel (teacher forcing) | condition on ground-truth $x_{<i}$, not model samples |

### Enforcing the ordering with masks

The architectural problem is **leakage**: a naive network mixing all inputs
would let the prediction of $x_i$ peek at $x_i$ itself or at future coordinates,
making the conditional ill-defined. The fix across the whole family is the same:
**zero out the forbidden connections** so output $i$ is a function of inputs
$< i$ only.

**MADE** (Masked Autoencoder for Distribution Estimation) realizes this in a
plain autoencoder. Assign each hidden unit a number $m$ in $\{1, \dots, d-1\}$;
keep a weight from input $k$ to a unit only if $k \le m$, and from a unit to
output $i$ only if $m < i$. Element-wise multiplying each weight matrix by a
binary mask $M$ enforces the constraint with no change to the forward cost:[^gf-made]

$$
W' = W \odot M, \qquad
\hat{x} = \sigma\parens{W'^{(2)}\,h + b^{(2)}},
\quad
h = \sigma\parens{W'^{(1)}\,x + b^{(1)}},
$$

and the masks are chosen so that $\partial \hat{x}_i / \partial x_k = 0$ whenever
$k \ge i$ — exactly the autoregressive property.

> **Definition (Autoregressive mask).** A binary matrix $M$ multiplied
> element-wise into a weight matrix, $W' = W \odot M$, chosen so that the path
> from input $x_k$ to output $\hat{x}_i$ exists only when $k < i$. It encodes the
> conditioning structure $p(x_i \mid x_{<i})$ directly in the connectivity.

**PixelRNN** and **PixelCNN** apply the same idea to images, generating pixels in
raster order. PixelRNN threads an LSTM along rows; **PixelCNN** is faster,
replacing the recurrence with a stack of **masked convolutions**. The
convolution kernel is masked so the receptive field covers only pixels above the
current one, and to its left within the current row; the future half of the
kernel is zeroed.

$$
% caption: A masked PixelCNN kernel: weights over already-generated pixels are kept, while the center and future positions are zeroed so the convolution cannot see ahead.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  % 5x5 kernel grid
  \foreach \r in {0,1,2,3,4} {
    \foreach \c in {0,1,2,3,4} {
      \pgfmathsetmacro{\x}{\c*1.0}
      \pgfmathsetmacro{\y}{-\r*1.0}
      \node[draw, black, minimum size=10mm, inner sep=0pt] (k\r\c) at (\x,\y) {};
    }
  }
  % kept (context) cells: all rows above center (rows 0,1), and row 2 cols 0,1
  \foreach \r/\c in {0/0,0/1,0/2,0/3,0/4,1/0,1/1,1/2,1/3,1/4,2/0,2/1} {
    \draw[acc, thick, fill=acc!15] (k\r\c.south west) rectangle (k\r\c.north east);
    \node[acc, font=\footnotesize] at (k\r\c) {$1$};
  }
  % center pixel (being predicted): masked to 0
  \draw[green, very thick, fill=green!15] (k22.south west) rectangle (k22.north east);
  \node[green, font=\footnotesize] at (k22) {$0$};
  % future cells: row 2 cols 3,4 and rows 3,4 -> masked 0 (red)
  \foreach \r/\c in {2/3,2/4,3/0,3/1,3/2,3/3,3/4,4/0,4/1,4/2,4/3,4/4} {
    \node[red, font=\footnotesize] at (k\r\c) {$0$};
  }
  % legend
  \node[acc, anchor=west, font=\footnotesize] at (5.4,0) {kept (past)};
  \node[green, anchor=west, font=\footnotesize] at (5.4,-2.0) {center (masked)};
  \node[red, anchor=west, font=\footnotesize] at (5.4,-4.0) {future (masked)};
\end{tikzpicture}
$$

**WaveNet** carries the construction to raw audio, where $d$ is tens of thousands
of samples. It stacks **dilated causal convolutions**: causal so output $t$
depends only on samples $\le t$, dilated so the receptive field grows
_exponentially_ with depth: a stack of $L$ layers with dilations
$1, 2, 4, \dots, 2^{L-1}$ reaches back $2^L$ samples while keeping the parameter
count linear in $L$.

| Model | Conditional network | Domain | Mask mechanism |
| --- | --- | --- | --- |
| MADE | masked MLP / autoencoder | vectors | masked weight matrices $W \odot M$ |
| PixelRNN | row LSTM | images | recurrence is causal by construction |
| PixelCNN | masked CNN | images | future half of the kernel zeroed |
| WaveNet | dilated causal CNN | audio | causal conv, exponential dilation |

> **Theorem (Exactness of the autoregressive likelihood).** For any
> autoregressive model the training objective equals the exact negative
> log-likelihood; no variational gap is incurred.

> **Proof.** The chain-rule factorization $p_\theta(x) = \prod_i p_\theta(x_i
> \mid x_{<i})$ is an identity for _any_ joint, so it holds for the model's joint
> with no approximation. Summing $\log$ of both sides gives $\log p_\theta(x) =
> \sum_i \log p_\theta(x_i \mid x_{<i})$, which is precisely the per-coordinate
> objective. The masks guarantee each network output $\hat{x}_i$ depends only on
> $x_{<i}$, so each factor is a genuine normalized conditional and their product
> is a normalized density. Hence the objective _is_ $-\log p_\theta(x)$, exactly.
> $\qed$

## Normalizing flows: change of variables

Flows take a different route to an exact likelihood. Start from a simple base
density $p_z$ (a standard Gaussian) and push a sample through an **invertible,
differentiable** map $f$:

$$
z \sim p_z(z), \qquad x = f(z), \qquad z = f^{-1}(x).
$$

> **Definition (Normalizing flow).** A generative model $x = f(z)$ where $f$ is a
> diffeomorphism (smooth with a smooth inverse) and $z$ has a tractable base
> density $p_z$. The density $p_x$ is obtained exactly from $p_z$ by the
> change-of-variables formula; sampling is a forward pass of $f$, and likelihood
> a forward pass of $f^{-1}$.

The density of $x$ follows from conservation of probability mass. A region of
$z$-space of volume $dz$ carries probability $p_z(z)\,dz$; the map $f$ stretches
it to a region of volume $\lvert \det J_f \rvert\,dz$ in $x$-space, where
$J_f = \partial f / \partial z$ is the Jacobian. Mass is preserved, so the
density must scale inversely with the volume.[^gf-cov]

$$
% caption: Change of variables: the invertible map $f$ warps a simple base density into a complex one, with density divided by the local stretch $\abs{\det J_f}$.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % base space (left): regular grid + a highlighted square
  \begin{scope}[xshift=0cm]
    \foreach \i in {0,1,2,3} \draw[black] (\i*0.7,0) -- (\i*0.7,2.1);
    \foreach \j in {0,1,2,3} \draw[black] (0,\j*0.7) -- (2.1,\j*0.7);
    \draw[acc, thick, fill=acc!15] (0.7,0.7) rectangle (1.4,1.4);
    \node[acc, font=\footnotesize] at (2.5,2.35) {$dz$};
    \draw[acc, ->] (2.3,2.2) -- (1.5,1.42);
    \node[anchor=north, font=\footnotesize] at (1.05,-0.25) {base $p_z$ (Gaussian)};
  \end{scope}
  % arrow with f
  \draw[->, acc, very thick] (2.9,1.05) -- (4.4,1.05);
  \node[acc, anchor=south, font=\footnotesize] at (3.65,1.15) {$x = f(z)$};
  % warped space (right): curved grid + warped square
  \begin{scope}[xshift=5.0cm]
    \foreach \i in {0,1,2,3}
      \draw[black] plot[domain=0:2.1, samples=20] ({\i*0.7 + 0.18*sin(\x*120)},\x);
    \foreach \j in {0,1,2,3}
      \draw[black] plot[domain=0:2.1, samples=20] (\x,{\j*0.7 + 0.18*sin(\x*120)});
    % a stretched parallelogram region
    \draw[green, thick, fill=green!15] (0.95,0.7) -- (1.85,0.95) -- (1.7,1.75) -- (0.8,1.5) -- cycle;
    \node[green, font=\footnotesize, anchor=south] at (1.25,2.45) {area $= \det J \, dz$};
    \draw[green, ->] (1.3,2.4) -- (1.3,1.72);
    \node[anchor=north, font=\footnotesize] at (1.05,-0.25) {density $p_x$ (complex)};
  \end{scope}
\end{tikzpicture}
$$

Writing $p_x(x)\,\lvert\det J_f\rvert = p_z(z)$ and substituting $z = f^{-1}(x)$
gives the change-of-variables formula, on which the whole family rests:

$$
p_x(x) = p_z\parens{f^{-1}(x)}\,\abs{\det \frac{\partial f^{-1}}{\partial x}}
= p_z(z)\,\abs{\det \frac{\partial f}{\partial z}}^{-1}.
$$

In log space, products become sums and the inverse becomes a sign flip:

$$
\log p_x(x) = \log p_z(z) - \log\abs{\det \frac{\partial f}{\partial z}},
\qquad z = f^{-1}(x).
$$

> **Theorem (Change of variables).** Let $f : \mathbb{R}^d \to \mathbb{R}^d$ be a
> diffeomorphism and $z = f^{-1}(x)$. Then
> $\log p_x(x) = \log p_z(z) - \log\lvert\det J_f(z)\rvert$, where
> $J_f = \partial f / \partial z$.

> **Proof.** Probability mass is invariant under reparameterization:
> $\int_A p_x(x)\,dx = \int_{f^{-1}(A)} p_z(z)\,dz$ for every measurable $A$. The
> multivariate substitution $x = f(z)$ has $dx = \lvert\det J_f(z)\rvert\,dz$, so
> $\int_{f^{-1}(A)} p_x(f(z))\,\lvert\det J_f(z)\rvert\,dz = \int_{f^{-1}(A)}
> p_z(z)\,dz$. As this holds for all $A$, the integrands agree pointwise:
> $p_x(f(z))\,\lvert\det J_f(z)\rvert = p_z(z)$. Solving for $p_x$ and taking
> logs yields the stated identity. $\qed$

The objective is now plain maximum likelihood, computed exactly:

$$
\mathcal{L}(\theta) = -\frac{1}{n}\sum_{j=1}^{n}\brackets{\log p_z\parens{f_\theta^{-1}(x^{(j)})} - \log\abs{\det J_{f_\theta}(z^{(j)})}}.
$$

### The Jacobian-determinant bottleneck

The formula is exact but useless unless three operations are cheap. Computing
$\det J_f$ for a general dense $J$ costs $O(d^3)$, fatal for image-sized $d$.
Every flow architecture is therefore engineered around making the Jacobian
**structured** so its determinant is trivial.

| Requirement | Why | Constraint on $f$ |
| --- | --- | --- |
| $f^{-1}$ exists | evaluate $z = f^{-1}(x)$ for the likelihood | $f$ bijective |
| $f^{-1}$ cheap | likelihood at scale | invert in $O(d)$, not by solving a system |
| $\det J_f$ cheap | the log-likelihood term | triangular / diagonal Jacobian → $O(d)$ |

The recurring trick: if $J_f$ is **triangular**, its determinant is just the
product of its diagonal entries, $\det J = \prod_i J_{ii}$ — an $O(d)$
computation. The question becomes how to build an expressive bijection whose
Jacobian is triangular.

## Coupling layers: NICE and RealNVP

The answer is the **coupling layer**. Partition the coordinates into two blocks,
$x = (x_a, x_b)$. Copy the first block unchanged; transform the second block with
a function whose parameters are produced by a network reading _only_ the first
block:

$$
\begin{aligned}
y_a &= x_a, \\
y_b &= x_b \odot \exp\parens{s(x_a)} + t(x_a),
\end{aligned}
$$

where $s$ and $t$ (scale and translation) are arbitrary neural networks; they
need not be invertible themselves. This is the **affine coupling** of RealNVP;
dropping the scale ($s \equiv 0$) recovers the additive coupling of NICE.

$$
% caption: An affine coupling layer: block $x_a$ passes through untouched and drives networks $s,t$ that scale and shift $x_b$, the only block transformed.
\begin{tikzpicture}[>=stealth, font=\small,
  blk/.style={draw, minimum width=15mm, minimum height=9mm, align=center},
  net/.style={draw, acc, thick, minimum width=12mm, minimum height=8mm}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % inputs
  \node[blk] (xa) at (0,1.4)  {$x_a$};
  \node[blk] (xb) at (0,-1.4) {$x_b$};
  % s,t networks fed by xa
  \node[net] (s) at (3.0,0.7)  {$s$};
  \node[net] (t) at (3.0,-0.5) {$t$};
  % combine op
  \node[draw, circle, minimum size=11mm, font=\scriptsize, align=center] (op) at (5.4,-1.4) {scale\\shift};
  % outputs
  \node[blk, draw=green, text=green] (ya) at (8.0,1.4)  {$y_a = x_a$};
  \node[blk, draw=green, text=green] (yb) at (8.0,-1.4) {$y_b$};
  % wires: xa passes through
  \draw[->, thick] (xa) -- (ya);
  % xa drives s and t
  \draw[->, acc, thick] (xa) -- (s);
  \draw[->, acc, thick] (xa) -- (t);
  % s,t feed the op
  \draw[->, acc, thick] (s) -- (op);
  \draw[->, acc, thick] (t) -- (op);
  % xb into op, op to yb
  \draw[->, thick] (xb) -- (op);
  \draw[->, thick] (op) -- (yb);
\end{tikzpicture}
$$

Two properties make the coupling layer the standard building block of flows. First, it is **trivially
invertible** without inverting $s$ or $t$ — given $(y_a, y_b)$, recover $x_a =
y_a$, recompute $s(x_a), t(x_a)$, and solve:

$$
x_a = y_a, \qquad x_b = \parens{y_b - t(y_a)} \odot \exp\parens{-s(y_a)}.
$$

Second, the **Jacobian is triangular**. Since $y_a = x_a$ and $y_b$ depends on
$x_b$ only through the diagonal scaling $\exp(s(x_a))$:

$$
\frac{\partial y}{\partial x} =
\begin{bmatrix}
I & 0 \\[4pt]
\dfrac{\partial y_b}{\partial x_a} & \diag\!\parens{\exp(s(x_a))}
\end{bmatrix}.
$$

The block-triangular structure means the determinant ignores the messy
off-diagonal block entirely:

$$
\det \frac{\partial y}{\partial x} = \prod_{k} \exp\parens{s(x_a)_k} = \exp\parens{\textstyle\sum_k s(x_a)_k},
\qquad
\log\abs{\det \frac{\partial y}{\partial x}} = \sum_k s(x_a)_k.
$$

The expensive determinant has collapsed to a **sum of the scale outputs**, an
$O(d)$ operation, independent of $s$ and $t$'s complexity.[^gf-flow]

> **Lemma (Coupling-layer determinant).** For an affine coupling layer the
> log-determinant of the Jacobian equals $\sum_k s(x_a)_k$, the sum of the
> log-scale network's outputs, regardless of the architecture of $s$ and $t$.

> **Proof.** The Jacobian is block lower-triangular with blocks $I$ and
> $\diag(\exp(s(x_a)))$ on the diagonal. The determinant of a block
> triangular matrix is the product of the determinants of its diagonal blocks, so
> $\det J = \det(I)\cdot\det(\diag(\exp(s))) = \prod_k \exp(s_k)$.
> Taking $\log$ gives $\sum_k s_k$. The off-diagonal block $\partial y_b /
> \partial x_a$ never enters the determinant. $\qed$

### A worked Jacobian and likelihood

Trace a single point through one affine coupling layer with real numbers. Work in
$d = 2$, split as $x_a = x_1$ and $x_b = x_2$, and suppose the scale and shift
networks happen to output $s(x_a) = 0.5\,x_a$ and $t(x_a) = x_a$. Take the input
$x = (2,\; 3)$, so $x_a = 2$ gives $s = 1.0$ and $t = 2.0$. The forward map is

$$
y_a = x_a = 2,
\qquad
y_b = x_b\,e^{s} + t = 3\cdot e^{1.0} + 2 = 3(2.718) + 2 = 10.155.
$$

The Jacobian of this map is lower-triangular,

$$
\frac{\partial y}{\partial x}
= \begin{bmatrix} 1 & 0 \\[2pt] \dfrac{\partial y_b}{\partial x_a} & e^{s} \end{bmatrix}
= \begin{bmatrix} 1 & 0 \\ \star & 2.718 \end{bmatrix},
$$

and its log-determinant ignores the off-diagonal $\star$ entirely:
$\log\lvert\det J\rvert = \sum_k s_k = 1.0$. Confirm the lemma numerically — the
determinant is $1\cdot e^{1.0} = 2.718$, whose log is $1.0$, exactly the sum of the
scale outputs, no matter what $\star$ equals.

Now read off the likelihood. Under a standard Gaussian base, the point maps back to
$z = f^{-1}(x)$ during evaluation; for a _single-layer_ flow the base point is the
pre-image, but to make the change-of-variables arithmetic concrete, suppose an input
whose inverse is $z = (0.5,\; -1.0)$ with accumulated log-determinant $\ell = 1.0$.
The base log-density of a 2-D unit Gaussian is
$\log p_z(z) = -\tfrac12\lVert z\rVert^2 - \tfrac{d}{2}\log(2\pi) = -\tfrac12(0.25 + 1.0) - \log(2\pi) = -0.625 - 1.838 = -2.463$.
The change-of-variables formula then gives

$$
\log p_x(x) = \log p_z(z) - \ell = -2.463 - 1.0 = -3.463 \text{ nats}.
$$

The stretch cost $\ell$ is subtracted directly: because this layer _expanded_
volume (positive log-scale $s = 1.0$), it spread the base mass thinner, so the
data density is lower than the base density at the corresponding point. A
contracting layer ($s < 0$) would _add_ to the log-likelihood. Stacking $K$ layers
just accumulates $\ell = \sum_k \ell_k$, and every term is an $O(d)$ sum of scale
outputs.

### Composing flows

A single coupling layer leaves half its coordinates untouched, so flows
**alternate** which block is transformed and stack many layers. Because the
composition of diffeomorphisms is a diffeomorphism, the whole stack is one valid
flow, and its log-determinant is just the **sum** of the per-layer terms, the
chain rule applied to the determinant:

$$
f = f_K \circ \cdots \circ f_1,
\qquad
\log\abs{\det \frac{\partial f}{\partial z}} = \sum_{k=1}^{K} \log\abs{\det \frac{\partial f_k}{\partial h_{k-1}}},
$$

with $h_0 = z$ and $h_k = f_k(h_{k-1})$. Each term is the $O(d)$ coupling
determinant above, so the total cost stays linear.

$$
% caption: A normalizing flow stacks invertible maps $f_1,\dots,f_K$ that push a Gaussian base $p_z$ step by step toward the complex target $p_x$.
\begin{tikzpicture}[>=stealth, font=\small,
  d/.style={draw, minimum width=16mm, minimum height=14mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % base: a single round blob
  \node[d, label={below:base $p_z$}] (z) at (0,0) {};
  \draw[acc, thick, fill=acc!15] (z) circle (4mm);
  % intermediate 1: slightly stretched
  \node[d, label={below:$h_1$}] (h1) at (3.0,0) {};
  \draw[acc, thick, fill=acc!15] (h1.center) ellipse (5mm and 3mm);
  % intermediate 2: bent
  \node[d, label={below:$h_2$}] (h2) at (6.0,0) {};
  \draw[acc, thick, fill=acc!15] (h2.center) ++(-1mm,1mm) ellipse (4mm and 4.5mm);
  \draw[acc, thick, fill=acc!15] (h2.center) ++(2mm,-1.5mm) circle (2.2mm);
  % target: complex shape (crescent-ish, palette only)
  \node[d, label={below:target $p_x$}] (x) at (9.0,0) {};
  \draw[green, thick, fill=green!15] (x.center) ++(-1mm,0) ellipse (5mm and 3mm);
  \fill[white] (x.center) ++(1.5mm,1mm) ellipse (3.5mm and 2.5mm);
  % arrows with f_k
  \draw[->, acc, very thick] (z) -- (h1) node[midway, above, font=\footnotesize, text=acc] {$f_1$};
  \draw[->, acc, very thick] (h1) -- (h2) node[midway, above, font=\footnotesize, text=acc] {$f_2$};
  \draw[->, acc, very thick] (h2) -- (x) node[midway, above, font=\footnotesize, text=acc] {$f_K$};
\end{tikzpicture}
$$

```algorithm
caption: $\textsc{FlowLogLikelihood}(x, \{f_k\})$ — exact $\log p_x(x)$ via change of variables
$h \gets x$
$\ell \gets 0$ // accumulated log-determinant
for $k \gets K$ down to $1$ do
  $h \gets f_k^{-1}(h)$ // invert one coupling layer
  $\ell \gets \ell + \log|\det \partial f_k / \partial h|$ // sum of scale outputs, $O(d)$
$z \gets h$
return $\log p_z(z) - \ell$
```

## The exact-likelihood families, side by side

Autoregressive models and flows are **duals**: an autoregressive model is itself
a flow whose Jacobian is triangular by ordering rather than by partition, which
is why _inverse autoregressive flows_ and _masked autoregressive flows_ unify
the two.[^gf-iaf] Each family makes a different trade among the three things a generative
model is asked to do — score a sample, draw a sample, and draw a _good_ sample.

| Family | Exact likelihood | Sampling speed | Sample quality | Mechanism |
| --- | --- | --- | --- | --- |
| [Autoregressive](#autoregressive-models-the-chain-rule-as-architecture) | yes (chain rule) | slow ($d$ sequential steps) | high | masked conditionals $p(x_i \mid x_{<i})$ |
| Normalizing flow | yes (change of variables) | fast (one parallel pass) | medium | invertible map, triangular Jacobian |
| [VAE](/deep-learning/generative-models/variational-autoencoders) | lower bound (ELBO) | fast | medium (blurry) | amortized variational inference |
| [GAN](/deep-learning/generative-models/generative-adversarial-networks) | none | fast | high (sharp) | adversarial min–max |

$$
% caption: The generative trilemma. Each family scores strongly on two of three axes (exact likelihood, fast sampling, high sample quality) and weakly on the third; no family fills the whole triangle.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % triangle of the three desiderata
  \coordinate (L) at (90:3.4);   % likelihood (top)
  \coordinate (S) at (210:3.4);  % speed (bottom-left)
  \coordinate (Q) at (330:3.4);  % quality (bottom-right)
  \draw[black, thick] (L) -- (S) -- (Q) -- cycle;
  \node[anchor=south, font=\footnotesize] at (L) {exact \texttt{likelihood}};
  \node[anchor=north east, font=\scriptsize] at (S) {fast sampling};
  \node[anchor=north west, font=\scriptsize] at (Q) {sharp samples};
  % place each family near the edge it owns (the two vertices it scores on)
  \node[acc, font=\footnotesize] at (0.0,0.6) {AR: \texttt{likelihood} + \texttt{quality}};
  \node[green, font=\footnotesize] at (0.0,-0.35) {\texttt{flow}: \texttt{likelihood} + speed};
  \node[red, font=\footnotesize] at (0.0,-1.3) {GAN: speed + \texttt{quality}};
\end{tikzpicture}
$$

> **Definition (Exact-likelihood model).** A generative model that evaluates
> $\log p_\theta(x)$ exactly, without a variational bound or an implicit
> adversarial objective. Autoregressive models and normalizing flows are the two
> principal families; both are trained by direct maximum likelihood.

Each family trades one axis for the other two. Autoregressive models get
exactness and quality at the cost of slow sampling; flows get exactness and fast
sampling at the cost of a
constrained, less expressive map; the
[VAE](/deep-learning/generative-models/variational-autoencoders) gets fast
sampling and a flexible decoder at the cost of exactness (a bound); the
[GAN](/deep-learning/generative-models/generative-adversarial-networks) gets
sharp samples at the cost of the likelihood entirely. No single family wins on all
three axes, which is why the field keeps all four, and why hybrids
(flow-decoded VAEs, autoregressive flows) combine one family's strength with
another's. The next lesson turns to a fifth route, the
[energy-based and Boltzmann machines](/deep-learning/generative-models/energy-based-and-boltzmann-machines)
that model an _unnormalized_ density, with the difficulty moved to the partition
function.

## Flows and autoregressive models today

Both families kept advancing after Goodfellow's text, and their two headline
weaknesses — slow autoregressive sampling, weak flow expressiveness — were the
targets.

**Autoregressive models became the backbone of modern generation.** WaveNet's
dilated-causal design set the template, and the **Transformer** is now the dominant
autoregressive architecture: a masked (causal) self-attention layer applies the
leakage-prevention mask of this lesson to a sequence, and every
large language model generates by the same left-to-right chain rule.[^transformer]
The slow-sampling problem drove **parallel WaveNet**, which distilled a trained
autoregressive teacher into an inverse-autoregressive-flow student that samples in
one pass — a direct use of the AR/flow duality noted above.[^parallelwavenet]

**Flows gained expressiveness and exact-inverse attention.** Kingma and Dhariwal's
**Glow** added invertible $1\times1$ convolutions (a learned channel permutation
with a cheap determinant) and actnorm, producing the first flow with
GAN-competitive high-resolution samples.[^glow] **Continuous normalizing flows**
then replaced the discrete stack with an ordinary differential equation, computing
the log-determinant as an integral of a trace via the Hutchinson estimator — the
**FFJORD** model — which removed the architectural constraints on the Jacobian
entirely.[^ffjord]

**The families converged with diffusion.** A diffusion model can be read as a deep,
fixed flow whose per-step Jacobian is trivial, and score-based models share the
continuous-flow ODE formulation; the next lesson's
[energy-based models](/deep-learning/generative-models/energy-based-and-boltzmann-machines)
and the
[diffusion models](/deep-learning/generative-models/diffusion-and-score-based-models)
after them inherit the exact-likelihood ambitions of this lesson while relaxing the
invertibility that made flows rigid.

## Takeaways

- **Autoregressive models** factor the joint exactly by the chain rule
  $p(x) = \prod_i p(x_i \mid x_{<i})$ and learn each conditional with a network;
  likelihood is one parallel pass, but sampling is $d$ sequential steps.
- **Masked connectivity** (MADE's $W \odot M$, PixelCNN's zeroed kernel half,
  WaveNet's dilated causal convolutions) enforces the ordering so output $i$
  cannot see input $\ge i$.
- **Normalizing flows** push a Gaussian base through an invertible map and read
  off $\log p_x(x) = \log p_z(z) - \log\lvert\det J_f\rvert$: exact likelihood,
  contingent on a cheap Jacobian determinant.
- **Coupling layers** (NICE/RealNVP) split coordinates and transform one half
  conditioned on the other, yielding a triangular Jacobian whose log-determinant
  is just $\sum_k s(x_a)_k$; composing many such layers keeps the cost $O(d)$.
- The four families trade among **exact likelihood**, **sampling speed**, and
  **sample quality**; no family dominates, which is the reason all of them
  persist.

[^gf-exact]: **Goodfellow**, _Deep Learning_, §20.10.7–20.10.10 — autoregressive networks and flow-based generation as the exact-likelihood alternatives to the VAE's bound and the GAN's implicit objective.
[^gf-ar]: **Goodfellow**, _Deep Learning_, §20.10.7 — fully visible belief networks and autoregressive factorization $p(x)=\prod_i p(x_i\mid x_{<i})$, with parallel training and sequential sampling.
[^gf-made]: **Goodfellow**, _Deep Learning_, §20.10.8–20.10.10 — masked autoregressive networks (NADE/MADE) and masked/dilated convolutional variants (PixelCNN, WaveNet) that enforce the conditioning order through connectivity.
[^gf-cov]: **Goodfellow**, _Deep Learning_, §3.9.2 — the change-of-variables formula $p_x(x)=p_z(z)\,|\det \partial f/\partial z|^{-1}$ that converts a base density through an invertible map.
[^gf-flow]: **Goodfellow**, _Deep Learning_, §20.10.10 — normalizing flows and coupling layers (NICE/RealNVP) whose triangular Jacobian makes the log-determinant a cheap sum of scale outputs.
[^gf-iaf]: **Goodfellow**, _Deep Learning_, §20.10.10 — the duality between autoregressive models and flows, and the inverse/masked autoregressive flows that interpolate between fast sampling and fast density evaluation.
[^transformer]: **Vaswani et al.**, "Attention Is All You Need," NeurIPS 2017 — masked (causal) self-attention as the autoregressive architecture behind modern language models.
[^parallelwavenet]: **van den Oord et al.**, "Parallel WaveNet: Fast High-Fidelity Speech Synthesis," ICML 2018 — probability-density distillation of an autoregressive teacher into an inverse-autoregressive-flow student for single-pass sampling.
[^glow]: **Kingma & Dhariwal**, "Glow: Generative Flow with Invertible 1×1 Convolutions," NeurIPS 2018 — invertible 1×1 convolutions and actnorm for high-resolution flow synthesis.
[^ffjord]: **Grathwohl et al.**, "FFJORD: Free-Form Continuous Dynamics for Scalable Reversible Generative Models," ICLR 2019, building on **Chen et al.**, "Neural Ordinary Differential Equations," NeurIPS 2018 — continuous normalizing flows with an ODE-integrated log-determinant.