---
title: "Diffusion and Score-Based Models"
module: Generative Models
moduleNumber: 7
lessonNumber: 7
order: 707
summary: >
  Corrupt a data point with Gaussian noise in small steps until only noise remains,
  then train a network to undo one step at a time. We derive the forward process
  and its closed-form marginal, reduce the variational bound to the single
  noise-prediction objective that makes diffusion trainable, and show the
  score-matching view that unifies it with Langevin sampling and the continuous
  SDE. The lesson closes with DDIM fast sampling, classifier-free guidance, and the
  latent diffusion that powers modern text-to-image systems.
topics: [Generative Models]
sources:
  - book: Goodfellow
    ref: "Ch. 20 — deep generative models (diffusion postdates the 2016 text)"
---

The [variational autoencoder](/deep-learning/generative-models/variational-autoencoders)
learns one encoder that maps data to a latent code in a single jump, and a single
decoder that jumps back. **Diffusion models** instead replace the one
hard jump with a long chain of easy ones. Corrupt a datum by adding a little
Gaussian noise, repeat hundreds of times until it is indistinguishable from white
noise, then learn to reverse one small step at a time. Each step is a tiny,
well-conditioned denoising problem, and the model is the same network applied over
and over. The forward corruption has no parameters and a closed form; all the
learning is in the reverse chain.[^sohl]

## The two chains

Diffusion defines a fixed **forward process** that gradually turns data $x_0$ into
noise $x_T$, and a learned **reverse process** that turns noise back into data.

> **Definition (Diffusion model).** A latent-variable model with latents
> $x_1, \dots, x_T$ of the same dimension as the data $x_0$. A fixed forward Markov
> chain $q(x_t \mid x_{t-1})$ adds Gaussian noise until $x_T \approx \mathcal N(0, I)$;
> a learned reverse Markov chain $p_\theta(x_{t-1} \mid x_t)$ removes it. Generation
> draws $x_T \sim \mathcal N(0, I)$ and runs the reverse chain to $x_0$.

$$
% caption: Forward chain $q$ destroys the datum into Gaussian noise; the learned reverse chain $p_\theta$ rebuilds a sample from noise.
\begin{tikzpicture}[>=stealth, font=\small,
  v/.style={circle, draw, black, thick, minimum size=12mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[v] (x0) at (0,0) {$x_0$};
  \node[v] (x1) at (2.6,0) {$x_1$};
  \node[black] (dots) at (5.0,0) {. . .};
  \node[v] (xt1) at (7.4,0) {$x_{t}$};
  \node[v] (xt) at (10.0,0) {$x_T$};
  % forward arrows (top), accent
  \draw[->, acc, thick] (x0) to[bend left=38] node[above, font=\scriptsize, text=acc] {q} (x1);
  \draw[->, acc, thick] (x1) to[bend left=38] node[above, font=\scriptsize, text=acc] {q} (dots);
  \draw[->, acc, thick] (dots) to[bend left=38] node[above, font=\scriptsize, text=acc] {q} (xt1);
  \draw[->, acc, thick] (xt1) to[bend left=38] node[above, font=\scriptsize, text=acc] {q} (xt);
  % reverse arrows (bottom), red
  \draw[->, red, thick] (x1) to[bend left=38] node[below, font=\scriptsize, text=red] {p} (x0);
  \draw[->, red, thick] (dots) to[bend left=38] node[below, font=\scriptsize, text=red] {p} (x1);
  \draw[->, red, thick] (xt1) to[bend left=38] node[below, font=\scriptsize, text=red] {p} (dots);
  \draw[->, red, thick] (xt) to[bend left=38] node[below, font=\scriptsize, text=red] {p} (xt1);
  % labels
  \node[acc, font=\footnotesize, anchor=south] at (5.0,1.55) {forward: add noise};
  \node[red, font=\footnotesize, anchor=north] at (5.0,-1.55) {reverse: denoise};
  \node[font=\footnotesize, anchor=north] at (0,-0.95) {data};
  \node[font=\footnotesize, anchor=north] at (10.0,-0.95) {noise};
\end{tikzpicture}
$$

The forward chain supplies the training data. Because $q$ is fixed and tractable,
at any step we can produce a noised sample $x_t$ from a clean $x_0$ and train the
network to predict what was added. Training never simulates the reverse chain; it only
denoises samples drawn from the forward chain.

## The forward process

Each forward step adds Gaussian noise scaled by a small **variance schedule**
$\beta_t \in (0, 1)$, shrinking the signal by $\sqrt{1 - \beta_t}$ so the variance
stays bounded:

$$
q(x_t \mid x_{t-1}) = \mathcal N\!\parens{x_t;\, \sqrt{1 - \beta_t}\, x_{t-1},\; \beta_t I}.
$$

The $\sqrt{1 - \beta_t}$ on the mean is what keeps the marginal from exploding: as
$t$ grows the original signal is attenuated and replaced by noise, and after enough
steps $x_T$ is essentially $\mathcal N(0, I)$ regardless of $x_0$.

> **Definition (Variance schedule).** The sequence $\beta_1 < \cdots < \beta_T$
> setting the per-step noise. Write $\alpha_t = 1 - \beta_t$ and
> $\bar\alpha_t = \prod_{s=1}^{t} \alpha_s$; then $\bar\alpha_t$ is the fraction of
> the original signal's variance surviving to step $t$, decreasing from $\approx 1$
> to $\approx 0$.

### The closed-form marginal

Sampling $x_t$ by running $t$ steps of the chain would be wasteful. The chain of
Gaussians collapses into a single Gaussian: $x_t$ given $x_0$ has a closed form, so
we jump to any step in one shot.

> **Theorem (Diffusion marginal).** Under the forward process,
> $$
> q(x_t \mid x_0) = \mathcal N\!\parens{x_t;\, \sqrt{\bar\alpha_t}\, x_0,\; (1 - \bar\alpha_t) I},
> \qquad \bar\alpha_t = \prod_{s=1}^{t}(1 - \beta_s).
> $$

> **Proof.** Reparameterize one step with $\epsilon_{t-1} \sim \mathcal N(0, I)$:
> $$
> x_t = \sqrt{\alpha_t}\, x_{t-1} + \sqrt{1 - \alpha_t}\, \epsilon_{t-1}.
> $$
> Expand $x_{t-1}$ the same way and substitute:
> $$
> x_t = \sqrt{\alpha_t}\parens{\sqrt{\alpha_{t-1}}\, x_{t-2} + \sqrt{1 - \alpha_{t-1}}\, \epsilon_{t-2}} + \sqrt{1 - \alpha_t}\, \epsilon_{t-1}.
> $$
> The two trailing terms are independent zero-mean Gaussians with variances
> $\alpha_t(1 - \alpha_{t-1})$ and $(1 - \alpha_t)$. The sum of independent
> Gaussians is Gaussian with variance the sum, so they merge into a single
> $\sqrt{1 - \alpha_t \alpha_{t-1}}\, \bar\epsilon$ with $\bar\epsilon \sim \mathcal N(0,I)$:
> $$
> x_t = \sqrt{\alpha_t \alpha_{t-1}}\, x_{t-2} + \sqrt{1 - \alpha_t \alpha_{t-1}}\, \bar\epsilon.
> $$
> Induction over all $t$ steps replaces the product $\alpha_t \cdots \alpha_1$ with
> $\bar\alpha_t$, giving $x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon$,
> which is the stated Gaussian. $\qed$

This identity is what makes training practical: it lets us sample $x_t$
directly from $x_0$ with one noise draw,

$$
x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon,
\qquad \epsilon \sim \mathcal N(0, I),
$$

so a training step picks a random $t$, corrupts $x_0$ in one operation, and trains
the network to recover $\epsilon$.

$$
% caption: The surviving signal $\sqrt{\bar\alpha_t}$ decays as the noise level $\sqrt{1-\bar\alpha_t}$ grows; their crossover sets where structure dissolves.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, black, thick] (0,0) -- (6.6,0) node[right, font=\scriptsize] {step t};
  \draw[->, black, thick] (0,0) -- (0,3.4) node[above, font=\scriptsize] {scale};
  \draw[black, thick] (0,3.0) -- (-0.12,3.0) node[left, font=\scriptsize] {1};
  % signal sqrt(abar): cosine-like decay 1 -> 0  (acc)
  \draw[acc, very thick] plot[domain=0:6, samples=80]
    (\x, {3.0*cos(deg(0.262*\x))});
  % noise sqrt(1-abar): 0 -> 1 (red)
  \draw[red, very thick] plot[domain=0:6, samples=80]
    (\x, {3.0*sin(deg(0.262*\x))});
  \node[acc, anchor=south west, font=\scriptsize] at (0.2,2.5) {signal};
  \node[red, anchor=north east, font=\scriptsize] at (6.0,1.7) {noise};
  % crossover marker
  \draw[black, thick, dashed] (3.0,0) -- (3.0,2.12);
  \node[black, anchor=north, font=\scriptsize] at (3.0,-0.05) {crossover};
\end{tikzpicture}
$$

A common choice is the **cosine schedule**, which keeps $\bar\alpha_t$ from
collapsing too fast near $t = 0$ and spends more steps where the image still holds
structure; the original DDPM used a linear $\beta_t$ ramp.[^nichol]

### A worked forward and reverse step

For example, work with a single scalar coordinate and
a schedule where, at some step $t$, the accumulated signal fraction is
$\bar\alpha_t = 0.36$, so $\sqrt{\bar\alpha_t} = 0.6$ and
$\sqrt{1 - \bar\alpha_t} = 0.8$. **Forward:** corrupt a clean value $x_0 = 2.0$ with a
noise draw $\epsilon = 1.0$ in one shot,

$$
x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon = 0.6\cdot2.0 + 0.8\cdot1.0 = 1.2 + 0.8 = 2.0.
$$

The signal has shrunk to $1.2$ and $0.8$ of noise now rides on top. **Reverse:** the
network sees $x_t = 2.0$ and (having been trained on exactly this construction)
predicts the noise; suppose $\epsilon_\theta(x_t, t) = 0.9$, close to the true $1.0$.
With single-step parameters $\alpha_t = 0.9$ (so $\beta_t = 0.1$,
$\sqrt{\alpha_t} = 0.949$) the DDPM reverse mean is

$$
\mu_\theta = \frac{1}{\sqrt{\alpha_t}}\parens{x_t - \frac{\beta_t}{\sqrt{1 - \bar\alpha_t}}\,\epsilon_\theta}
= \frac{1}{0.949}\parens{2.0 - \frac{0.1}{0.8}\cdot0.9}
= \frac{1}{0.949}(2.0 - 0.1125) = 1.988.
$$

The reverse step nudges the value from $2.0$ toward $1.988$, subtracting a sliver of
the predicted noise; ancestral sampling then adds a small $\sigma_t z$ jitter (except
on the final step). Reading the same prediction through the score view,
$s_\theta = -\epsilon_\theta / \sqrt{1 - \bar\alpha_t} = -0.9/0.8 = -1.125$: the score
points in the $-x$ direction here, indicating that density rises as $x$
decreases, which is the same drift the reverse mean applied. The two readings are
equivalent views of one prediction.

## The reverse process and the variational bound

To generate, we need $q(x_{t-1} \mid x_t)$, the true reverse step. It is
intractable because it depends on the whole data distribution through Bayes' rule.
We approximate it with a Gaussian whose mean is a network and whose variance is
fixed (or learned):

$$
p_\theta(x_{t-1} \mid x_t) = \mathcal N\!\parens{x_{t-1};\, \mu_\theta(x_t, t),\; \Sigma_\theta(x_t, t)},
\qquad
p(x_T) = \mathcal N(0, I).
$$

A Gaussian reverse step is justified: when $\beta_t$ is small, the true reverse
$q(x_{t-1} \mid x_t)$ is itself approximately Gaussian, so matching it with a
Gaussian is a tight approximation rather than a crude one.

### The ELBO over the chain

As with the [VAE](/deep-learning/generative-models/variational-autoencoders), the
marginal $p_\theta(x_0)$ is intractable, so we maximize a variational bound. The
forward chain $q$ plays the role of the approximate posterior, but it is fixed and
parameter-free, which makes the bound far simpler to train.

> **Theorem (Diffusion ELBO).** With $q(x_{1:T}\mid x_0)$ the forward chain,
> $$
> \mathbb E_q\!\brackets{-\log p_\theta(x_0)} \le \mathbb E_q\!\brackets{ -\log p(x_T) - \sum_{t \ge 1} \log \frac{p_\theta(x_{t-1}\mid x_t)}{q(x_t \mid x_{t-1})} } =: L.
> $$

> **Proof.** Jensen's inequality on $-\log p_\theta(x_0) = -\log \mathbb E_q\brackets{p_\theta(x_{0:T})/q(x_{1:T}\mid x_0)}$
> gives $-\log p_\theta(x_0) \le \mathbb E_q[-\log\parens{p_\theta(x_{0:T})/q(x_{1:T}\mid x_0)}]$.
> Both chains factor as Markov products, $p_\theta(x_{0:T}) = p(x_T)\prod_t p_\theta(x_{t-1}\mid x_t)$
> and $q(x_{1:T}\mid x_0) = \prod_t q(x_t\mid x_{t-1})$. Substituting the products
> and splitting the log of the ratio gives the stated $L$. $\qed$

Training this raw bound is high-variance because $q(x_t \mid x_{t-1})$ is noisy. The
fix is to condition on $x_0$. The forward posterior $q(x_{t-1} \mid x_t, x_0)$ is
**tractable and Gaussian** by Bayes' rule combined with the two closed-form
marginals:

$$
q(x_{t-1}\mid x_t, x_0) = \mathcal N\!\parens{x_{t-1};\, \tilde\mu_t(x_t, x_0),\; \tilde\beta_t I},
$$

$$
\tilde\mu_t(x_t, x_0) = \frac{\sqrt{\bar\alpha_{t-1}}\,\beta_t}{1 - \bar\alpha_t}\, x_0 + \frac{\sqrt{\alpha_t}\,(1 - \bar\alpha_{t-1})}{1 - \bar\alpha_t}\, x_t,
\qquad
\tilde\beta_t = \frac{1 - \bar\alpha_{t-1}}{1 - \bar\alpha_t}\,\beta_t.
$$

Rewriting $L$ in terms of this posterior groups it into per-step KL divergences
between two Gaussians, each computable in closed form:

$$
L = \underbrace{D_{KL}\!\parens{q(x_T\mid x_0)\,\|\,p(x_T)}}_{L_T}
+ \sum_{t > 1} \underbrace{D_{KL}\!\parens{q(x_{t-1}\mid x_t, x_0)\,\|\,p_\theta(x_{t-1}\mid x_t)}}_{L_{t-1}}
\underbrace{-\,\log p_\theta(x_0\mid x_1)}_{L_0}.
$$

$L_T$ has no parameters (both ends are fixed Gaussians), $L_0$ is a reconstruction
term, and the work is in the $L_{t-1}$ terms: match the learned reverse Gaussian to
the tractable forward posterior at every step.[^ho]

$$
% caption: The bound decomposes into a fixed prior-matching term $L_T$, per-step denoising KLs $L_{t-1}$, and a reconstruction term $L_0$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  b/.style={draw, thick, minimum width=20mm, minimum height=11mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[b, draw=red, text=red] (lt) at (0,0) {$L_T$\\prior match};
  \node[b, draw=acc, text=acc] (lk) at (3.4,0) {denoise KL\\step t};
  \node[b, draw=acc, text=acc] (l2) at (6.8,0) {denoise KL\\step 2};
  \node[black] (dots) at (8.9,0) {. . .};
  \node[b] (l0) at (11.3,0) {$L_0$\\recon};
  \draw[->, black, thick] (lt) -- (lk);
  \draw[->, black, thick] (lk) -- (l2);
  \draw[->, black, thick] (l2) -- (dots);
  \draw[->, black, thick] (dots) -- (l0);
  \node[red, font=\scriptsize, anchor=north] at (0,-0.95) {no parameters};
  \node[acc, font=\scriptsize, anchor=north] at (5.1,-0.95) {the training signal};
\end{tikzpicture}
$$

### Reducing to noise prediction

Each $L_{t-1}$ is a KL between two Gaussians with the **same** fixed variance, so it
reduces to the squared distance between their means,
$L_{t-1} = \tfrac{1}{2\sigma_t^2}\norm{\tilde\mu_t(x_t, x_0) - \mu_\theta(x_t, t)}^2 + C$.
The key step is to reparameterize the mean. From the marginal,
$x_0 = (x_t - \sqrt{1 - \bar\alpha_t}\,\epsilon)/\sqrt{\bar\alpha_t}$; substituting
into $\tilde\mu_t$ rewrites the target in terms of the noise $\epsilon$ that
produced $x_t$:

$$
\tilde\mu_t(x_t, x_0) = \frac{1}{\sqrt{\alpha_t}}\parens{x_t - \frac{\beta_t}{\sqrt{1 - \bar\alpha_t}}\,\epsilon}.
$$

So instead of predicting the mean we let the network predict the noise,
$\epsilon_\theta(x_t, t)$, and set
$\mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}}\parens{x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\epsilon_\theta(x_t, t)}$.
The mean difference collapses to a noise difference, and the whole bound reduces to a
single regression target:[^ho]

$$
L_{t-1} - C = \frac{\beta_t^2}{2\sigma_t^2\,\alpha_t\,(1 - \bar\alpha_t)}\,\norm{\epsilon - \epsilon_\theta(x_t, t)}^2.
$$

> **Theorem (Simple DDPM objective).** Dropping the $t$-dependent weight in front of
> each term yields the unweighted training loss
> $$
> L_{\text{simple}}(\theta) = \mathbb E_{t, x_0, \epsilon}\,\norm{\epsilon - \epsilon_\theta\!\parens{\sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon,\; t}}^2,
> $$
> with $t \sim \mathrm{Unif}\{1, \dots, T\}$ and $\epsilon \sim \mathcal N(0, I)$.

The objective is mean-squared error between true and predicted noise. Ho et al.
found the **unweighted** loss trains better than the variational weighting, because
dropping the $1/(1-\bar\alpha_t)$-style factor up-weights the harder, high-noise
steps relative to the trivial low-noise ones.[^ho]

```algorithm
caption: $\textsc{TrainDDPM}(\epsilon_\theta, \mathcal{D})$ — minimize the noise-prediction loss
repeat
  $x_0 \gets$ sample from $\mathcal{D}$
  $t \gets$ sample from $\mathrm{Unif}\{1, \dots, T\}$ // a random diffusion step
  $\epsilon \gets$ sample from $\mathcal{N}(0, I)$ // the target noise
  $x_t \gets \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon$ // one-shot corruption
  take a gradient step on $\nabla_\theta \, \|\epsilon - \epsilon_\theta(x_t, t)\|^2$
until converged
return $\theta$
```

Sampling runs the learned reverse step from $x_T \sim \mathcal N(0, I)$ down to
$x_0$, injecting fresh noise $z$ at every step except the last:

```algorithm
caption: $\textsc{SampleDDPM}(\epsilon_\theta)$ — ancestral sampling of the reverse chain
$x_T \gets$ sample from $\mathcal{N}(0, I)$
for $t \gets T$ down to $1$ do
  $z \gets$ sample from $\mathcal{N}(0, I)$ if $t > 1$ else $0$ // no noise on the last step
  $x_{t-1} \gets \dfrac{1}{\sqrt{\alpha_t}}\parens{x_t - \dfrac{\beta_t}{\sqrt{1 - \bar\alpha_t}}\, \epsilon_\theta(x_t, t)} + \sigma_t\, z$
return $x_0$
```

## The score-based view

The same algorithm can be derived from a different starting point:
model the **score** of the data density, $\nabla_x \log p(x)$, the direction in
which probability rises fastest.

> **Definition (Score function).** The gradient of the log-density with respect to
> the input, $s(x) = \nabla_x \log p(x)$. It is a vector field pointing toward
> higher-density regions and does not involve the intractable
> normalizing constant $Z$: since $\log p(x) = \log\tilde p(x) - \log Z$ and $Z$ is
> constant in $x$, the gradient $\nabla_x \log p(x) = \nabla_x \log\tilde p(x)$ drops
> it.

This is the score's advantage over the density itself: a model of $\nabla_x \log p$
sidesteps the partition function that makes
[energy-based models](/deep-learning/generative-models/energy-based-and-boltzmann-machines)
intractable.
We cannot fit a score by regressing on $\nabla_x \log p$ because we never observe it;
**denoising score matching** supplies a tractable surrogate. Perturb $x$ with
Gaussian noise of scale $\sigma$ to get $\tilde x = x + \sigma\epsilon$. The score of
the noised density has a closed form, and matching it reduces to predicting the
noise:

$$
\nabla_{\tilde x} \log q_\sigma(\tilde x \mid x) = -\frac{\tilde x - x}{\sigma^2} = -\frac{\epsilon}{\sigma}.
$$

> **Theorem (Score equals scaled noise).** For the diffusion marginal
> $x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon$, the score of
> $q(x_t \mid x_0)$ is
> $$
> \nabla_{x_t} \log q(x_t \mid x_0) = -\frac{x_t - \sqrt{\bar\alpha_t}\, x_0}{1 - \bar\alpha_t} = -\frac{\epsilon}{\sqrt{1 - \bar\alpha_t}}.
> $$

> **Proof.** $q(x_t \mid x_0)$ is the Gaussian
> $\mathcal N(\sqrt{\bar\alpha_t}\, x_0, (1 - \bar\alpha_t) I)$, whose log-density is
> $-\norm{x_t - \sqrt{\bar\alpha_t}\, x_0}^2 / (2(1 - \bar\alpha_t))$ up to a
> constant. Differentiating in $x_t$ gives $-(x_t - \sqrt{\bar\alpha_t}\, x_0)/(1 - \bar\alpha_t)$.
> Substituting $x_t - \sqrt{\bar\alpha_t}\, x_0 = \sqrt{1 - \bar\alpha_t}\,\epsilon$
> yields $-\epsilon/\sqrt{1 - \bar\alpha_t}$. $\qed$

A noise-predictor and a score-model are therefore the **same object** up to a known
scale. If $s_\theta(x_t, t) \approx \nabla_{x_t} \log q(x_t)$, then
$\epsilon_\theta(x_t, t) = -\sqrt{1 - \bar\alpha_t}\; s_\theta(x_t, t)$. DDPM's
$\epsilon$-prediction loss is denoising score matching at all noise scales at once,
which is why the two literatures describe one method.[^song-ncsn]

| Quantity | $\epsilon$-prediction view | Score view |
| --- | --- | --- |
| Network output | predicted noise $\epsilon_\theta(x_t,t)$ | score $s_\theta(x_t,t)$ |
| Relation | $\epsilon_\theta = -\sqrt{1-\bar\alpha_t}\; s_\theta$ | $s_\theta = -\epsilon_\theta / \sqrt{1-\bar\alpha_t}$ |
| Training | regress on injected noise | denoising score matching |
| Sampling | ancestral reverse step | Langevin / reverse SDE |

### Langevin dynamics

Given a score field, **Langevin dynamics** samples from $p$ by gradient ascent on
$\log p$ with calibrated noise, a stochastic walk that climbs density while the
noise prevents collapse to the single mode:

$$
x_{k+1} = x_k + \frac{\eta}{2}\, s_\theta(x_k) + \sqrt{\eta}\, z_k,
\qquad z_k \sim \mathcal N(0, I).
$$

As $\eta \to 0$ and the chain length grows, $x_k$ converges to a sample from $p$.
The noised-score view explains why a single scale fails and a schedule of scales is
needed: in low-density regions the data score is badly estimated, so sampling starts
at a large noise scale (smooth, well-estimated field) and anneals to small scales
— the same annealing the reverse diffusion chain performs.[^song-ncsn]

$$
% caption: Annealed sampling across noise scales. A large $\sigma$ blurs the two modes into one broad basin with a reliable score; shrinking $\sigma$ sharpens the field back to the true bimodal density.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  % large sigma: single broad bump, well-estimated everywhere
  \begin{scope}
    \draw[->, black] (-0.2,0) -- (3.4,0) node[right, font=\scriptsize]{$x$};
    \draw[acc, very thick] plot[domain=0:3.1, samples=60]
      (\x, {1.5*exp(-0.5*(\x-1.55)*(\x-1.55)/0.7)});
    \node[font=\scriptsize, anchor=north] at (1.55,-0.15) {large scale};
  \end{scope}
  % medium sigma: modes emerging
  \begin{scope}[xshift=4.2cm]
    \draw[->, black] (-0.2,0) -- (3.4,0) node[right, font=\scriptsize]{$x$};
    \draw[acc, very thick] plot[domain=0:3.1, samples=80]
      (\x, {1.1*exp(-0.5*(\x-0.95)*(\x-0.95)/0.25) + 1.2*exp(-0.5*(\x-2.15)*(\x-2.15)/0.25)});
    \node[font=\scriptsize, anchor=north] at (1.55,-0.15) {medium scale};
  \end{scope}
  % small sigma: sharp bimodal true density
  \begin{scope}[xshift=8.4cm]
    \draw[->, black] (-0.2,0) -- (3.4,0) node[right, font=\scriptsize]{$x$};
    \draw[acc, very thick] plot[domain=0:3.1, samples=100]
      (\x, {1.4*exp(-0.5*(\x-0.85)*(\x-0.85)/0.06) + 1.5*exp(-0.5*(\x-2.25)*(\x-2.25)/0.06)});
    \node[font=\scriptsize, anchor=north] at (1.55,-0.15) {small scale};
  \end{scope}
  % anneal arrow
  \draw[->, black, thick] (3.5,1.7) -- (4.1,1.7);
  \draw[->, black, thick] (7.7,1.7) -- (8.3,1.7);
  \node[font=\scriptsize, anchor=south] at (5.9,1.7) {anneal};
\end{tikzpicture}
$$

$$
% caption: Langevin dynamics follows the score field $\nabla \log p$ (arrows) from a random start, drifting toward the high-density mode while injected noise jitters the path.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % density contours: two concentric circles around the mode (light tint fill + outline)
  \draw[black, thick, fill=acc!12] (3.2,1.6) circle (1.7);
  \draw[black, thick, fill=acc!15] (3.2,1.6) circle (0.95);
  \node[acc, font=\footnotesize, anchor=south] at (3.2,2.0) {\texttt{mode}};
  % score field arrows pointing inward toward the mode
  \foreach \a in {0,45,...,315} {
    \draw[->, acc, thick]
      ({3.2+2.2*cos(\a)},{1.6+2.2*sin(\a)}) -- ({3.2+1.55*cos(\a)},{1.6+1.55*sin(\a)});
  }
  % Langevin path: jittered walk from a corner start to the mode
  \draw[red, very thick]
    (0.2,0.2) -- (0.95,0.95) -- (1.4,0.7) -- (2.05,1.45) -- (2.4,1.1) -- (2.9,1.65) -- (3.1,1.55);
  \node[red, circle, fill=red, inner sep=1.3pt] at (0.2,0.2) {};
  \node[red, font=\scriptsize, anchor=west] at (0.3,0.05) {start};
  \node[green, circle, fill=green, inner sep=1.6pt] at (3.1,1.55) {};
\end{tikzpicture}
$$

## The continuous limit: SDEs and the probability-flow ODE

Sending the step size to zero turns the discrete chain into a **stochastic
differential equation**. The forward corruption is the SDE
$dx = f(x, t)\,dt + g(t)\,dw$ with $w$ a Wiener process; the discrete schedule is its
Euler discretization. This SDE has an exact **reverse-time
SDE**, written entirely in terms of the score:

$$
dx = \brackets{f(x, t) - g(t)^2\, \nabla_x \log p_t(x)}\,dt + g(t)\,d\bar w.
$$

Plug in the learned score $s_\theta$ and integrate backward from noise to data: this
is diffusion sampling in continuous time, with DDPM a particular discretization.
Every such SDE has a deterministic counterpart with the **same marginals** at every
$t$, the **probability-flow ODE**:

$$
\frac{dx}{dt} = f(x, t) - \tfrac{1}{2}\, g(t)^2\, \nabla_x \log p_t(x).
$$

Because it is an ODE, sampling is deterministic given $x_T$, the trajectory is
unique, and any black-box ODE solver applies; it also gives an exact log-likelihood
through the change-of-variables / continuous-flow identity, tying diffusion back to
[normalizing flows](/deep-learning/generative-models/autoregressive-and-normalizing-flows).[^song-sde]

## Fast sampling with DDIM

DDPM sampling needs hundreds to thousands of sequential network passes, one per step.
**DDIM** reduces this cost by abandoning the Markov assumption. It defines a family of
non-Markovian forward processes that share the **same marginals** $q(x_t \mid x_0)$,
so a network trained with the DDPM objective can be reused unchanged, yet whose
reverse step is deterministic and can **skip** steps.[^song-ddim] The DDIM update
first predicts the clean datum, then re-noises it to the next (possibly distant) step:

$$
\hat x_0 = \frac{x_t - \sqrt{1 - \bar\alpha_t}\, \epsilon_\theta(x_t, t)}{\sqrt{\bar\alpha_t}},
\qquad
x_{t-1} = \sqrt{\bar\alpha_{t-1}}\, \hat x_0 + \sqrt{1 - \bar\alpha_{t-1}}\, \epsilon_\theta(x_t, t).
$$

There is no added noise term, so the map from $x_T$ to $x_0$ is deterministic; it is
in fact an Euler discretization of the probability-flow ODE. Determinism allows large
step skips with little quality loss, cutting sampling to tens of steps.

| Property | DDPM (ancestral) | DDIM |
| --- | --- | --- |
| Reverse process | Markov, stochastic | non-Markov, deterministic |
| Added noise per step | $\sigma_t z$ | none |
| Typical steps | $1000$ | $20$–$50$ |
| Same trained network | yes | yes (identical objective) |
| Latent meaning | noise reservoir | encodes the sample (invertible) |

$$
% caption: DDPM walks every step with re-injected noise; DDIM follows a deterministic trajectory that skips steps, reaching $x_0$ in far fewer passes.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  s/.style={circle, draw, black, thick, minimum size=8mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % DDPM row: many small dense steps
  \node[red, font=\scriptsize, anchor=east] at (-0.3,1.4) {DDPM};
  \foreach \i in {0,1,2,3,4,5,6,7,8} {
    \node[s, minimum size=5mm] (d\i) at (\i*1.05,1.4) {};
  }
  \foreach \i [evaluate=\i as \j using int(\i+1)] in {0,1,2,3,4,5,6,7} {
    \draw[->, red, thick] (d\i) -- (d\j);
  }
  \node[red, font=\scriptsize, anchor=west] at (8.9,1.4) {many steps};
  % DDIM row: few large steps
  \node[acc, font=\scriptsize, anchor=east] at (-0.3,0) {DDIM};
  \foreach \i in {0,1,2,3} {
    \node[s] (m\i) at (\i*2.8,0) {};
  }
  \foreach \i [evaluate=\i as \j using int(\i+1)] in {0,1,2} {
    \draw[->, acc, thick] (m\i) -- (m\j);
  }
  \node[acc, font=\scriptsize, anchor=west] at (8.9,0) {few steps};
  % shared endpoints labels
  \node[black, font=\scriptsize, anchor=north] at (0,-0.5) {noise};
  \node[black, font=\scriptsize, anchor=north] at (8.4,-0.5) {data};
\end{tikzpicture}
$$

## Conditioning and guidance

To make $p(x \mid y)$ (an image given a label or caption), we steer the reverse chain
toward the condition $y$. Bayes' rule on the score splits the conditional score into
the unconditional score plus the gradient of the classifier likelihood:

$$
\nabla_x \log p(x \mid y) = \nabla_x \log p(x) + \nabla_x \log p(y \mid x).
$$

**Classifier guidance** instantiates this directly: train a separate classifier
$p_\phi(y \mid x_t)$ on noised inputs and add a scaled version of its gradient to the
score at sampling time, with a **guidance scale** $w$ that exaggerates the steer:[^dhariwal]

$$
\tilde s_\theta(x_t, t) = s_\theta(x_t, t) + w\,\nabla_{x_t} \log p_\phi(y \mid x_t).
$$

The drawback is the extra classifier, trained on noisy data, whose gradients are
unreliable. **Classifier-free guidance** removes it. Train one network to be *both*
conditional and unconditional by randomly dropping $y$ during training (replacing it
with a null token), then at sampling time combine the two predictions to synthesize
the same steer without any classifier:[^hosalimans]

$$
\tilde\epsilon_\theta(x_t, t, y) = (1 + w)\,\epsilon_\theta(x_t, t, y) - w\,\epsilon_\theta(x_t, t, \varnothing).
$$

> **Definition (Classifier-free guidance).** Extrapolating the noise prediction along
> the line from the unconditional estimate $\epsilon_\theta(\cdot, \varnothing)$
> through the conditional one $\epsilon_\theta(\cdot, y)$ by a factor $1 + w$. Larger
> $w$ tightens adherence to $y$ (sharper, more on-prompt samples) at the cost of
> diversity. It is the standard conditioning method in modern text-to-image models.

$$
% caption: Guidance extrapolates past the conditional prediction along the line from the unconditional one, scale $w$ controlling how far the steer is pushed.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % baseline axis
  \draw[black, thick] (0,0) -- (9.0,0);
  % unconditional point
  \node[black, circle, fill=black, inner sep=1.6pt] (u) at (1.2,0) {};
  \node[black, font=\scriptsize, anchor=north] at (1.2,-0.2) {uncond.};
  % conditional point
  \node[acc, circle, fill=acc, inner sep=1.8pt] (c) at (4.2,0) {};
  \node[acc, font=\scriptsize, anchor=north] at (4.2,-0.2) {cond. y};
  % guided point (extrapolated)
  \node[green, circle, fill=green, inner sep=1.8pt] (g) at (7.8,0) {};
  \node[green, font=\scriptsize, anchor=north] at (7.8,-0.2) {guided};
  % arrows showing the extrapolation
  \draw[->, acc, thick] (u) -- (c) node[midway, above, font=\scriptsize, text=acc] {base steer};
  \draw[->, green, thick] (c) -- (g) node[midway, above, font=\scriptsize, text=green] {push by w};
\end{tikzpicture}
$$

## Latent diffusion

Diffusing in pixel space is expensive: every one of hundreds of reverse steps runs a
full-resolution network. **Latent diffusion** (the model behind Stable Diffusion)
moves the entire process into the compact latent space of a pretrained autoencoder.
An encoder $E$ compresses $x$ to a latent $z = E(x)$ at a fraction of the spatial
size; diffusion trains and samples on $z$; a decoder $D$ maps the generated latent
back to a pixel image $D(z)$.[^rombach]

$$
z = E(x), \qquad z \text{ diffuses and denoises}, \qquad x \approx D(z_0).
$$

The autoencoder strips imperceptible high-frequency detail so the diffusion model
spends its capacity on the semantically meaningful structure, and the smaller spatial
resolution makes each denoising step cheap. The denoiser itself is a **U-Net**: a
convolutional encoder-decoder with skip connections that join matching-resolution
feature maps, so fine spatial detail is preserved across the bottleneck. Two ingredients adapt it
to diffusion. The step index $t$ enters through a **time embedding** (a sinusoidal
encoding fed to every block), telling the network the current noise level. The
condition $y$ (e.g. a text embedding) enters through **cross-attention** layers, so
the prompt modulates features throughout the network.

$$
% caption: The U-Net denoiser: a downsampling encoder and upsampling decoder joined by skip connections, with time embedding $t$ and condition $y$ injected at every block.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  b/.style={draw, thick, align=center, minimum width=12mm}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % encoder (down) blocks, decreasing height
  \node[b, draw=acc, text=acc, minimum height=20mm] (e1) at (0,0) {enc};
  \node[b, draw=acc, text=acc, minimum height=14mm] (e2) at (1.8,0) {enc};
  \node[b, draw=acc, text=acc, minimum height=9mm]  (e3) at (3.6,0) {enc};
  % bottleneck
  \node[b, draw=black, minimum height=7mm] (mid) at (5.4,0) {mid};
  % decoder (up) blocks, increasing height
  \node[b, draw=acc, text=acc, minimum height=9mm]  (d3) at (7.2,0) {dec};
  \node[b, draw=acc, text=acc, minimum height=14mm] (d2) at (9.0,0) {dec};
  \node[b, draw=acc, text=acc, minimum height=20mm] (d1) at (10.8,0) {dec};
  % forward path
  \draw[->, black, thick] (e1) -- (e2);
  \draw[->, black, thick] (e2) -- (e3);
  \draw[->, black, thick] (e3) -- (mid);
  \draw[->, black, thick] (mid) -- (d3);
  \draw[->, black, thick] (d3) -- (d2);
  \draw[->, black, thick] (d2) -- (d1);
  % skip connections (green, curved over the top)
  \draw[->, green, thick] (e1.north) to[bend left=32] node[above, font=\scriptsize, text=green] {skip} (d1.north);
  \draw[->, green, thick] (e2.north) to[bend left=30] (d2.north);
  \draw[->, green, thick] (e3.north) to[bend left=28] (d3.north);
  % time embedding and condition injected at bottleneck
  \node[b, draw=red, text=red, minimum height=6mm] (te) at (5.4,-2.4) {time t};
  \node[b, draw=red, text=red, minimum height=6mm] (cy) at (8.3,-2.4) {cond. y};
  \draw[->, red, thick] (te) -- (mid);
  \draw[->, red, thick] (cy) -- (d3);
  % io labels
  \node[black, font=\scriptsize, anchor=east] at (-0.7,0) {$x_t$};
  \node[black, font=\scriptsize, anchor=west] at (11.5,0) {noise};
\end{tikzpicture}
$$

## Takeaways

- A **diffusion model** fixes a parameter-free forward chain
  $q(x_t \mid x_{t-1}) = \mathcal N(\sqrt{1-\beta_t}\,x_{t-1}, \beta_t I)$ that
  destroys data into Gaussian noise, and learns a reverse chain
  $p_\theta(x_{t-1}\mid x_t)$ that rebuilds it.
- The forward marginal is **closed-form**,
  $q(x_t \mid x_0) = \mathcal N(\sqrt{\bar\alpha_t}\,x_0, (1-\bar\alpha_t)I)$, derived
  by merging the chain's independent Gaussian increments, so any $x_t$ is one noise
  draw from $x_0$.
- The variational bound decomposes into per-step Gaussian KLs, and reparameterizing
  the mean reduces it to the **simple objective**
  $\norm{\epsilon - \epsilon_\theta(x_t, t)}^2$: the network predicts the injected
  noise.
- The **score view** identifies $\epsilon_\theta = -\sqrt{1-\bar\alpha_t}\,s_\theta$,
  so $\epsilon$-prediction is denoising score matching, sampled by Langevin dynamics
  or, in continuous time, by the **reverse SDE** and its deterministic
  **probability-flow ODE**.
- **DDIM** reuses the same network with a non-Markov deterministic reverse step that
  skips steps, cutting sampling from $\sim 1000$ to tens of passes; **classifier-free
  guidance** $\tilde\epsilon = (1+w)\epsilon_\theta(y) - w\,\epsilon_\theta(\varnothing)$
  steers samples toward a condition without an external classifier.
- **Latent diffusion** runs the whole process in a pretrained autoencoder's latent
  space with a **U-Net** denoiser, time embedding, and cross-attention conditioning,
  the architecture behind modern text-to-image generation.

[^sohl]: **Sohl-Dickstein, Weiss, Maheswaranathan & Ganguli** (ICML 2015), _Deep Unsupervised Learning using Nonequilibrium Thermodynamics_ — the original diffusion model: a learned reverse of a fixed Gaussian noising chain.
[^ho]: **Ho, Jain & Abbeel** (NeurIPS 2020), _Denoising Diffusion Probabilistic Models_ — DDPM; the reparameterization that turns the ELBO into the simple $\norm{\epsilon-\epsilon_\theta}^2$ objective.
[^nichol]: **Nichol & Dhariwal** (ICML 2021), _Improved Denoising Diffusion Probabilistic Models_ — the cosine variance schedule and learned reverse variances that sharpen samples and log-likelihood.
[^song-ncsn]: **Song & Ermon** (NeurIPS 2019), _Generative Modeling by Estimating Gradients of the Data Distribution_ — NCSN; denoising score matching across noise scales sampled by annealed Langevin dynamics.
[^song-sde]: **Song, Sohl-Dickstein, Kingma, Kumar, Ermon & Poole** (ICLR 2021), _Score-Based Generative Modeling through Stochastic Differential Equations_ — the forward/reverse SDE and the probability-flow ODE unifying diffusion and score models.
[^song-ddim]: **Song, Meng & Ermon** (ICLR 2021), _Denoising Diffusion Implicit Models_ — DDIM; a non-Markov deterministic reverse process sharing DDPM's marginals, enabling few-step sampling.
[^dhariwal]: **Dhariwal & Nichol** (NeurIPS 2021), _Diffusion Models Beat GANs on Image Synthesis_ — classifier guidance, adding a noised-input classifier's gradient to the score with a scale $w$.
[^hosalimans]: **Ho & Salimans** (NeurIPS Workshop 2022), _Classifier-Free Diffusion Guidance_ — jointly training a conditional and unconditional model and extrapolating between their predictions.
[^rombach]: **Rombach, Blattmann, Lorenz, Esser & Ommer** (CVPR 2022), _High-Resolution Image Synthesis with Latent Diffusion Models_ — Stable Diffusion; diffusion in a pretrained autoencoder latent with a cross-attention U-Net.