Diffusion and Score-Based Models

The variational autoencoder learns one encoder that maps data to a latent code in a single jump, and a single decoder that jumps back. Diffusion models instead replace the one hard jump with a long chain of easy ones. Corrupt a datum by adding a little Gaussian noise, repeat hundreds of times until it is indistinguishable from white noise, then learn to reverse one small step at a time. Each step is a tiny, well-conditioned denoising problem, and the model is the same network applied over and over. The forward corruption has no parameters and a closed form; all the learning is in the reverse chain.¹

The two chains

Diffusion defines a fixed forward process that gradually turns data $x_0$ into noise $x_T$, and a learned reverse process that turns noise back into data.

The forward chain supplies the training data. Because $q$ is fixed and tractable, at any step we can produce a noised sample $x_t$ from a clean $x_0$ and train the network to predict what was added. Training never simulates the reverse chain; it only denoises samples drawn from the forward chain.

The forward process

Each forward step adds Gaussian noise scaled by a small variance schedule${\beta }_t\in (0,1)$, shrinking the signal by $\sqrt{1-{\beta }_t}$ so the variance stays bounded:

The $\sqrt{1-{\beta }_t}$ on the mean is what keeps the marginal from exploding: as $t$ grows the original signal is attenuated and replaced by noise, and after enough steps $x_T$ is essentially $\mathcal{N}(0,I)$ regardless of $x_0$.

The closed-form marginal

Sampling $x_t$ by running $t$ steps of the chain would be wasteful. The chain of Gaussians collapses into a single Gaussian: $x_t$ given $x_0$ has a closed form, so we jump to any step in one shot.

This identity is what makes training practical: it lets us sample $x_t$ directly from $x_0$ with one noise draw,

so a training step picks a random $t$, corrupts $x_0$ in one operation, and trains the network to recover $ϵ$.

A common choice is the cosine schedule, which keeps ${\bar{\alpha }}_t$ from collapsing too fast near $t=0$ and spends more steps where the image still holds structure; the original DDPM used a linear ${\beta }_t$ ramp.²

A worked forward and reverse step

For example, work with a single scalar coordinate and a schedule where, at some step $t$, the accumulated signal fraction is ${\bar{\alpha }}_t=0.36$, so $\sqrt{{\bar{\alpha }}_t}=0.6$ and $\sqrt{1-{\bar{\alpha }}_t}=0.8$. Forward: corrupt a clean value $x_0=2.0$ with a noise draw $ϵ=1.0$ in one shot,

The signal has shrunk to $1.2$ and $0.8$ of noise now rides on top. Reverse: the network sees $x_t=2.0$ and (having been trained on exactly this construction) predicts the noise; suppose ${ϵ}_{\theta }(x_t,t)=0.9$, close to the true $1.0$. With single-step parameters ${\alpha }_t=0.9$ (so ${\beta }_t=0.1$, $\sqrt{{\alpha }_t}=0.949$) the DDPM reverse mean is

The reverse step nudges the value from $2.0$ toward $1.988$, subtracting a sliver of the predicted noise; ancestral sampling then adds a small ${\sigma }_{t}z$ jitter (except on the final step). Reading the same prediction through the score view, $s_{\theta }=-{ϵ}_{\theta }/\sqrt{1-{\bar{\alpha }}_t}=-0.9/0.8=-1.125$: the score points in the $-x$ direction here, indicating that density rises as $x$ decreases, which is the same drift the reverse mean applied. The two readings are equivalent views of one prediction.

The reverse process and the variational bound

To generate, we need $q(x_{t-1}\mid x_t)$, the true reverse step. It is intractable because it depends on the whole data distribution through Bayes' rule. We approximate it with a Gaussian whose mean is a network and whose variance is fixed (or learned):

A Gaussian reverse step is justified: when ${\beta }_t$ is small, the true reverse $q(x_{t-1}\mid x_t)$ is itself approximately Gaussian, so matching it with a Gaussian is a tight approximation rather than a crude one.

The ELBO over the chain

As with the VAE, the marginal $p_{\theta }(x_0)$ is intractable, so we maximize a variational bound. The forward chain $q$ plays the role of the approximate posterior, but it is fixed and parameter-free, which makes the bound far simpler to train.

Training this raw bound is high-variance because $q(x_t\mid x_{t-1})$ is noisy. The fix is to condition on $x_0$. The forward posterior $q(x_{t-1}\mid x_t,x_0)$ is tractable and Gaussian by Bayes' rule combined with the two closed-form marginals:

Rewriting $L$ in terms of this posterior groups it into per-step KL divergences between two Gaussians, each computable in closed form:

$L_T$ has no parameters (both ends are fixed Gaussians), $L_0$ is a reconstruction term, and the work is in the $L_{t-1}$ terms: match the learned reverse Gaussian to the tractable forward posterior at every step.³

Reducing to noise prediction

Each $L_{t-1}$ is a KL between two Gaussians with the same fixed variance, so it reduces to the squared distance between their means, $L_{t-1}=\frac{1}{2{\sigma }_t^2}\Vert {\tilde{\mu }}_t(x_t,x_0)-{\mu }_{\theta }(x_t,t){\Vert }^2+C$. The key step is to reparameterize the mean. From the marginal, $x_0=(x_t-\sqrt{1-{\bar{\alpha }}_t}ϵ)/\sqrt{{\bar{\alpha }}_t}$; substituting into ${\tilde{\mu }}_t$ rewrites the target in terms of the noise $ϵ$ that produced $x_t$:

So instead of predicting the mean we let the network predict the noise, ${ϵ}_{\theta }(x_t,t)$, and set ${\mu }_{\theta }(x_t,t)=\frac{1}{\sqrt{{\alpha }_t}}\left(x_t-\frac{{\beta }_t}{\sqrt{1-{\bar{\alpha }}_t}}{ϵ}_{\theta }(x_t,t)\right)$. The mean difference collapses to a noise difference, and the whole bound reduces to a single regression target:³

The objective is mean-squared error between true and predicted noise. Ho et al. found the unweighted loss trains better than the variational weighting, because dropping the $1/(1-{\bar{\alpha }}_t)$-style factor up-weights the harder, high-noise steps relative to the trivial low-noise ones.³

Sampling runs the learned reverse step from $x_T\sim \mathcal{N}(0,I)$ down to $x_0$, injecting fresh noise $z$ at every step except the last:

The score-based view

The same algorithm can be derived from a different starting point: model the score of the data density, ${\nabla }_x\log p(x)$, the direction in which probability rises fastest.

This is the score's advantage over the density itself: a model of ${\nabla }_x\log p$ sidesteps the partition function that makes energy-based models intractable. We cannot fit a score by regressing on ${\nabla }_x\log p$ because we never observe it; denoising score matching supplies a tractable surrogate. Perturb $x$ with Gaussian noise of scale $\sigma$ to get $\tilde{x}=x+\sigma ϵ$. The score of the noised density has a closed form, and matching it reduces to predicting the noise:

A noise-predictor and a score-model are therefore the same object up to a known scale. If $s_{\theta }(x_t,t)\approx {\nabla }_{x_t}\log q(x_t)$, then ${ϵ}_{\theta }(x_t,t)=-\sqrt{1-{\bar{\alpha }}_t}{s}_{\theta }(x_t,t)$. DDPM's $ϵ$-prediction loss is denoising score matching at all noise scales at once, which is why the two literatures describe one method.⁴

Langevin dynamics

Given a score field, Langevin dynamics samples from $p$ by gradient ascent on $\log p$ with calibrated noise, a stochastic walk that climbs density while the noise prevents collapse to the single mode:

As $\eta \to 0$ and the chain length grows, $x_k$ converges to a sample from $p$. The noised-score view explains why a single scale fails and a schedule of scales is needed: in low-density regions the data score is badly estimated, so sampling starts at a large noise scale (smooth, well-estimated field) and anneals to small scales — the same annealing the reverse diffusion chain performs.⁴

The continuous limit: SDEs and the probability-flow ODE

Sending the step size to zero turns the discrete chain into a stochastic differential equation. The forward corruption is the SDE $dx=f(x,t)dt+g(t)dw$ with $w$ a Wiener process; the discrete schedule is its Euler discretization. This SDE has an exact reverse-time SDE, written entirely in terms of the score:

Plug in the learned score $s_{\theta }$ and integrate backward from noise to data: this is diffusion sampling in continuous time, with DDPM a particular discretization. Every such SDE has a deterministic counterpart with the same marginals at every $t$, the probability-flow ODE:

Because it is an ODE, sampling is deterministic given $x_T$, the trajectory is unique, and any black-box ODE solver applies; it also gives an exact log-likelihood through the change-of-variables / continuous-flow identity, tying diffusion back to normalizing flows.⁵

Fast sampling with DDIM

DDPM sampling needs hundreds to thousands of sequential network passes, one per step. DDIM reduces this cost by abandoning the Markov assumption. It defines a family of non-Markovian forward processes that share the same marginals $q(x_t\mid x_0)$, so a network trained with the DDPM objective can be reused unchanged, yet whose reverse step is deterministic and can skip steps.⁶ The DDIM update first predicts the clean datum, then re-noises it to the next (possibly distant) step:

There is no added noise term, so the map from $x_T$ to $x_0$ is deterministic; it is in fact an Euler discretization of the probability-flow ODE. Determinism allows large step skips with little quality loss, cutting sampling to tens of steps.

Conditioning and guidance

To make $p(x\mid y)$ (an image given a label or caption), we steer the reverse chain toward the condition $y$. Bayes' rule on the score splits the conditional score into the unconditional score plus the gradient of the classifier likelihood:

Classifier guidance instantiates this directly: train a separate classifier $p_{\varphi }(y\mid x_t)$ on noised inputs and add a scaled version of its gradient to the score at sampling time, with a guidance scale $w$ that exaggerates the steer:⁷

The drawback is the extra classifier, trained on noisy data, whose gradients are unreliable. Classifier-free guidance removes it. Train one network to be both conditional and unconditional by randomly dropping $y$ during training (replacing it with a null token), then at sampling time combine the two predictions to synthesize the same steer without any classifier:⁸

Latent diffusion

Diffusing in pixel space is expensive: every one of hundreds of reverse steps runs a full-resolution network. Latent diffusion (the model behind Stable Diffusion) moves the entire process into the compact latent space of a pretrained autoencoder. An encoder $E$ compresses $x$ to a latent $z=E(x)$ at a fraction of the spatial size; diffusion trains and samples on $z$; a decoder $D$ maps the generated latent back to a pixel image $D(z)$.⁹

The autoencoder strips imperceptible high-frequency detail so the diffusion model spends its capacity on the semantically meaningful structure, and the smaller spatial resolution makes each denoising step cheap. The denoiser itself is a U-Net: a convolutional encoder-decoder with skip connections that join matching-resolution feature maps, so fine spatial detail is preserved across the bottleneck. Two ingredients adapt it to diffusion. The step index $t$ enters through a time embedding (a sinusoidal encoding fed to every block), telling the network the current noise level. The condition $y$ (e.g. a text embedding) enters through cross-attention layers, so the prompt modulates features throughout the network.

Takeaways

• A diffusion model fixes a parameter-free forward chain $q(x_t\mid x_{t-1})=\mathcal{N}(\sqrt{1-{\beta }_t}{x}_{t-1},{\beta }_{t}I)$ that destroys data into Gaussian noise, and learns a reverse chain $p_{\theta }(x_{t-1}\mid x_t)$ that rebuilds it.
• The forward marginal is closed-form, $q(x_t\mid x_0)=\mathcal{N}(\sqrt{{\bar{\alpha }}_t}{x}_0,(1-{\bar{\alpha }}_t)I)$, derived by merging the chain's independent Gaussian increments, so any $x_t$ is one noise draw from $x_0$.
• The variational bound decomposes into per-step Gaussian KLs, and reparameterizing the mean reduces it to the simple objective$\Vert ϵ-{ϵ}_{\theta }(x_t,t){\Vert }^2$: the network predicts the injected noise.
• The score view identifies ${ϵ}_{\theta }=-\sqrt{1-{\bar{\alpha }}_t}{s}_{\theta }$, so $ϵ$-prediction is denoising score matching, sampled by Langevin dynamics or, in continuous time, by the reverse SDE and its deterministic probability-flow ODE.
• DDIM reuses the same network with a non-Markov deterministic reverse step that skips steps, cutting sampling from $\sim 1000$ to tens of passes; classifier-free guidance $\widetilde{ϵ}=(1+w){ϵ}_{\theta }(y)-w{ϵ}_{\theta }(\varnothing )$ steers samples toward a condition without an external classifier.
• Latent diffusion runs the whole process in a pretrained autoencoder's latent space with a U-Net denoiser, time embedding, and cross-attention conditioning, the architecture behind modern text-to-image generation.