---
title: Energy-Based & Boltzmann Machines
module: Generative Models
moduleNumber: 7
lessonNumber: 6
order: 706
summary: >
  Energy-based models replace an explicit density with a scalar energy and a
  Boltzmann normalization, $p(x) = e^{-E(x)}/Z$: simple to specify, but with an
  intractable partition function $Z$. The Boltzmann machine and its restricted
  variant make the energy bilinear so the hidden units factorize, and contrastive
  divergence sidesteps $Z$ by replacing the model expectation with a few Gibbs
  steps started at the data. We close on the undirected deep models (DBNs and
  DBMs) and how they differ from the directed VAE.
topics: [Generative Models]
sources:
  - book: Goodfellow
    ref: "§16.2.4 — Energy-Based Models"
  - book: Goodfellow
    ref: "Ch. 18 — Confronting the Partition Function"
  - book: Goodfellow
    ref: "§20.1–20.4 — Boltzmann Machines, RBMs, DBNs, DBMs"
---

Every generative model so far has paid for tractability with structure. An
[autoregressive model](/deep-learning/generative-models/autoregressive-and-normalizing-flows)
factorizes $p(x)$ into a chain of conditionals; a
[normalizing flow](/deep-learning/generative-models/autoregressive-and-normalizing-flows)
restricts itself to invertible maps with cheap Jacobians; a
[VAE](/deep-learning/generative-models/variational-autoencoders) optimizes a
bound rather than the likelihood itself. **Energy-based models** make the
opposite trade. They specify the density in the most flexible way possible, any
scalar function of $x$, and concentrate the entire difficulty in a single
intractable normalizing constant.[^gf-ebm]

## The energy and the partition function

An energy-based model assigns each configuration $x$ a scalar **energy** $E(x)$
and turns it into a probability by the **Boltzmann distribution**: exponentiate
the negative energy and normalize.

$$
p(x) = \frac{\exp\!\parens{-E(x)}}{Z}, \qquad
Z = \sum_{x} \exp\!\parens{-E(x)},
$$

with $Z$ the integral $\int \exp(-E(x))\,dx$ in the continuous case. The energy is
unconstrained: _any_ $E_\theta : \mathcal{X} \to \mathbb{R}$ yields a valid
density, because the exponential is positive and $Z$ rescales it to integrate to
one. That freedom is the source of both the model's flexibility and its central
difficulty.

> **Definition (Energy-based model).** A model that parameterizes an unnormalized
> log-density $-E_\theta(x)$ and defines $p_\theta(x) = \exp(-E_\theta(x))/Z(\theta)$,
> where the **partition function** $Z(\theta) = \sum_x \exp(-E_\theta(x))$
> normalizes it. Low energy means high probability; the model says nothing about
> the absolute scale of $E$, only its _differences_.

Three facts fall straight out of the definition and frame everything below.

| Quantity | Expression | Reading |
| --- | --- | --- |
| Probability | $p(x) = e^{-E(x)}/Z$ | low energy $\Rightarrow$ high probability |
| Log-probability | $\log p(x) = -E(x) - \log Z$ | $-E$ up to an additive constant |
| Energy difference | $\log\dfrac{p(x_1)}{p(x_2)} = E(x_2) - E(x_1)$ | $Z$ cancels — _ratios are free_ |

The third row is the key fact. Comparing two configurations, scoring, or running
a Markov chain that only ever looks at ratios — none of these need $Z$. Only an
_absolute_ probability, or the likelihood's gradient, requires it.

For example, suppose a model assigns
energies $E(x_1) = 1.5$ and $E(x_2) = 3.0$ to two configurations. Their probability
ratio needs no $Z$:

$$
\frac{p(x_1)}{p(x_2)} = e^{-(E(x_1) - E(x_2))} = e^{-(1.5 - 3.0)} = e^{1.5} = 4.48.
$$

The lower-energy configuration $x_1$ is $4.48$ times more probable — a number we
computed without ever summing over the $2^d$ configurations. To turn either into an
_absolute_ probability, though, we would need $Z = \sum_x e^{-E(x)}$: even for a
modest binary vector $x \in \{0,1\}^{30}$ that is a sum over $2^{30} \approx 10^9$
terms, and the count doubles with every added unit. That gap — cheap ratios,
intractable absolutes — is the reason the training methods below exist.

> **Definition (Partition function).** The normalizer
> $Z(\theta) = \sum_x \exp(-E_\theta(x))$, a sum over _every_ configuration of
> $x$. For $x \in \{0,1\}^d$ it has $2^d$ terms; for continuous $x$ it is a
> generally intractable high-dimensional integral. Computing $Z$ exactly is the
> central obstruction of the entire energy-based family.[^gf-partition]

The energy surface is the object to picture. Data modes are basins, regions the
model deems probable, carved as wells in $E$; everything else is a high-energy
plateau. The map $x \mapsto e^{-E(x)}$ flips the picture upside down, turning deep
wells into tall probability peaks.

$$
% caption: An energy landscape $E(x)$ (black) and its induced density $p(x) \propto e^{-E(x)}$ (blue): deep energy wells become tall probability peaks.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  % axes
  \draw[->, thick] (-0.2,0) -- (9.4,0) node[right] {$x$};
  \draw[->, thick] (0,-0.2) -- (0,3.6);
  % energy curve E(x): two wells (low energy) at x~2.4 and x~6.4, barrier between
  \draw[black, very thick]
    plot[domain=0.3:9.0, samples=120]
    (\x, {1.7 + 0.95*cos((\x*40)-20) + 0.18*\x - 0.9*exp(-((\x-2.4)^2)/0.7) - 1.0*exp(-((\x-6.4)^2)/0.9)});
  % legend in the clear band above the central barrier, away from both curves
  \node[black, anchor=west] at (3.25,3.32) {energy $E(x)$};
  % density curve p(x) ~ e^{-E}: peaks where E is low
  \draw[acc, very thick]
    plot[domain=0.3:9.0, samples=120]
    (\x, {0.35 + 2.0*exp(-((\x-2.4)^2)/0.55) + 2.3*exp(-((\x-6.4)^2)/0.7)});
  \node[acc, anchor=west] at (3.25,2.83) {density $p(x)$};
  % mark the two modes
  \draw[black, dashed] (2.4,0) -- (2.4,2.3);
  \draw[black, dashed] (6.4,0) -- (6.4,2.6);
  \node[anchor=north, font=\footnotesize] at (2.4,-0.05) {\texttt{mode} 1};
  \node[anchor=north, font=\footnotesize] at (6.4,-0.05) {\texttt{mode} 2};
\end{tikzpicture}
$$

## The Boltzmann machine

A **Boltzmann machine** is the energy-based model whose energy is a _quadratic_
form over binary units. Split the units into a **visible** vector $v \in \{0,1\}^{d}$
(the data) and a **hidden** vector $h \in \{0,1\}^{m}$ (latent features), and the
general Boltzmann machine couples every pair of units. The **restricted** Boltzmann
machine (RBM) drops all within-layer couplings, keeping only the visible–hidden
weights $W$:

$$
E(v, h) = -\,v^\top W h \;-\; b^\top v \;-\; c^\top h,
$$

where $W \in \mathbb{R}^{d \times m}$ are the interaction weights and $b, c$ the
visible and hidden biases. The joint density is $p(v,h) = e^{-E(v,h)}/Z$ as before.
The defining structural fact is that the RBM's interaction graph is **bipartite**:
visible units connect only to hidden units, never to each other.

$$
% caption: The restricted Boltzmann machine: visible units $v_i$ connect to hidden units $h_j$ with no within-layer edges, so the bipartite graph makes conditionals factorize.
\begin{tikzpicture}[>=stealth, font=\small,
  vis/.style={circle, draw, thick, minimum size=8mm, inner sep=0pt},
  hid/.style={circle, draw, thick, fill=acc!15, minimum size=8mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  % hidden layer (top)
  \node[hid] (h1) at (1.0,2.2) {$h_1$};
  \node[hid] (h2) at (3.0,2.2) {$h_2$};
  \node[hid] (h3) at (5.0,2.2) {$h_3$};
  % visible layer (bottom)
  \node[vis] (v1) at (0.0,0) {$v_1$};
  \node[vis] (v2) at (2.0,0) {$v_2$};
  \node[vis] (v3) at (4.0,0) {$v_3$};
  \node[vis] (v4) at (6.0,0) {$v_4$};
  % all visible-hidden edges (bipartite, complete) — neutral scaffold, stays visible
  \foreach \h in {h1,h2,h3}
    \foreach \v in {v1,v2,v3,v4}
      \draw[black] (\v) -- (\h);
  % one highlighted weight, labelled off to the right clear of all edges
  \draw[acc, very thick] (v2) -- (h2);
  \node[font=\footnotesize, text=acc, anchor=west] at (6.7,1.1) {weight $W_{22}$};
  \draw[acc, thin, ->] (6.65,1.1) .. controls (4.6,1.0) .. (2.6,1.25);
  % layer labels
  \node[anchor=east, font=\footnotesize] at (-0.4,2.2) {hidden};
  \node[anchor=east, font=\footnotesize] at (-0.4,0) {visible};
\end{tikzpicture}
$$

### Conditional independence

The bipartite structure pays off immediately: given the visible units, the hidden
units are **conditionally independent of one another**, and vice versa. This is
what makes the RBM practical, so we derive it. Fix $v$ and expand the conditional
$p(h \mid v) = p(v,h)/p(v)$. Every term in the energy that does not involve $h$
(namely $-b^\top v$ and the constant $\log Z$) is absorbed into the normalizer over
$h$, leaving

$$
p(h \mid v) \;\propto\; \exp\!\parens{v^\top W h + c^\top h}
= \exp\!\parens{\textstyle\sum_{j} \parens{c_j + v^\top W_{:,j}} h_j}
= \prod_{j} \exp\!\parens{(c_j + v^\top W_{:,j})\,h_j}.
$$

The exponential of a sum factorizes into a product over $j$, and there is no term
coupling $h_j$ to $h_{j'}$, precisely because the RBM has no hidden–hidden edge.
Normalizing each factor over $h_j \in \{0,1\}$ gives an independent product of
logistic conditionals:

$$
p(h \mid v) = \prod_{j=1}^{m} p(h_j \mid v),
\qquad
p(h_j = 1 \mid v) = \sigma\!\parens{c_j + v^\top W_{:,j}},
$$

where $\sigma(z) = 1/(1+e^{-z})$ is the logistic sigmoid. By the symmetry of the
energy in $v$ and $h$, the same argument run the other way gives
$p(v \mid h) = \prod_i p(v_i \mid h)$ with
$p(v_i = 1 \mid h) = \sigma(b_i + W_{i,:}\,h)$.[^gf-rbm]

> **Theorem (RBM conditional factorization).** In a restricted Boltzmann machine,
> $p(h \mid v) = \prod_{j} p(h_j \mid v)$ and $p(v \mid h) = \prod_{i} p(v_i \mid h)$,
> with each unit a logistic function of the opposite layer. Both full-layer
> conditionals can therefore be sampled in one parallel pass.

This is the engine of **block Gibbs sampling**: because a whole layer is
conditionally independent given the other, we resample all of $h$ at once, then all
of $v$ at once, alternating — an exact Gibbs sampler whose blocks are entire layers.
We lean on this in the negative phase below, and it rests on the
[Monte Carlo and MCMC](/deep-learning/probabilistic-methods/monte-carlo-and-mcmc)
machinery.

### A worked Gibbs step

For example, take a tiny RBM with $d = 3$
visible and $m = 2$ hidden units, weights and biases

$$
W = \begin{bmatrix} 1.0 & -0.5 \\ 0.2 & 0.8 \\ -0.3 & 0.4 \end{bmatrix},
\qquad
c = (0.1,\; -0.2),
\qquad
b = (0,\; 0,\; 0).
$$

Clamp a data point $v = (1, 0, 1)$ and compute the hidden activations. Each hidden
unit's input is $c_j + v^\top W_{:,j}$:

$$
\begin{aligned}
h_1:&\quad 0.1 + (1\cdot1.0 + 0\cdot0.2 + 1\cdot(-0.3)) = 0.1 + 0.7 = 0.8, \\
h_2:&\quad -0.2 + (1\cdot(-0.5) + 0\cdot0.8 + 1\cdot0.4) = -0.2 - 0.1 = -0.3.
\end{aligned}
$$

Through the sigmoid, $p(h_1 = 1 \mid v) = \sigma(0.8) = 0.690$ and
$p(h_2 = 1 \mid v) = \sigma(-0.3) = 0.426$. Say the Bernoulli draws give
$h = (1, 0)$. Now sample the visible layer back from this $h$, using
$p(v_i = 1 \mid h) = \sigma(b_i + W_{i,:}h)$:

$$
\begin{aligned}
v_1:&\quad \sigma(1.0\cdot1 + (-0.5)\cdot0) = \sigma(1.0) = 0.731, \\
v_2:&\quad \sigma(0.2\cdot1 + 0.8\cdot0) = \sigma(0.2) = 0.550, \\
v_3:&\quad \sigma((-0.3)\cdot1 + 0.4\cdot0) = \sigma(-0.3) = 0.426.
\end{aligned}
$$

Suppose these draw to $v^{(1)} = (1, 1, 0)$. That single up-then-down pass is one
CD-1 step: the reconstruction $v^{(1)}$ differs from the data $v^{(0)} = (1,0,1)$ at
two coordinates, and the weight update $\eta(v^{(0)}h^{(0)\top} - v^{(1)}h^{(1)\top})$
nudges $W$ to make the data more probable and the reconstruction less so. The whole
layer resampled in one parallel pass, exactly as the factorization allows.

## The partition-function problem

To train by maximum likelihood we need the gradient of $\log p_\theta(v)$. Writing
the **free energy** $F(v) = -\log \sum_h e^{-E(v,h)}$, the marginal is
$p(v) = e^{-F(v)}/Z$, so $\log p(v) = -F(v) - \log Z$. Both pieces depend on
$\theta$. Differentiate:

$$
\nabla_\theta \log p(v)
= -\,\nabla_\theta F(v) \;-\; \nabla_\theta \log Z.
$$

The second term is where $Z$ bites. Using $\nabla_\theta \log Z = \tfrac{1}{Z}\nabla_\theta Z$
and $\nabla_\theta Z = \sum_x \nabla_\theta e^{-E(x)} = -\sum_x e^{-E(x)} \nabla_\theta E(x)$,

$$
\nabla_\theta \log Z
= -\sum_{x} \frac{e^{-E(x)}}{Z}\,\nabla_\theta E(x)
= -\,\mathbb{E}_{x \sim p_\theta}\!\brackets{\nabla_\theta E(x)}.
$$

The intractable sum has collapsed into an **expectation under the model**. Putting
the two terms together (with $F$ likewise an expectation over the hidden units
given the data), the log-likelihood gradient splits cleanly into two pieces of
opposite sign:

$$
\nabla_\theta \log p(v)
= \underbrace{-\,\mathbb{E}_{h \sim p(h\mid v)}\brackets{\nabla_\theta E(v,h)}}_{\text{positive phase (data)}}
\;+\; \underbrace{\mathbb{E}_{(v,h) \sim p_\theta}\brackets{\nabla_\theta E(v,h)}}_{\text{negative phase (model)}}.
$$

> **Theorem (Positive and negative phases).** The maximum-likelihood gradient of an
> energy-based model is the difference of two expectations of $\nabla_\theta E$: one
> over the data distribution (clamping $v$ to an observed example), and one over the
> model's own distribution. Training **lowers** the energy of observed data and
> **raises** the energy of samples the model currently favors.[^gf-phases]

For the RBM's bilinear energy the abstract gradient becomes concrete. With
$\nabla_W E = -v h^\top$, the weight update is the difference of two correlation
matrices:

$$
\frac{\partial \log p(v)}{\partial W}
= \underbrace{\mathbb{E}_{h \sim p(h\mid v)}\!\brackets{v\,h^\top}}_{\text{data correlations}}
\;-\;
\underbrace{\mathbb{E}_{(v,h)\sim p_\theta}\!\brackets{v\,h^\top}}_{\text{model correlations}}
\;=\; \langle v h^\top\rangle_{\text{data}} - \langle v h^\top\rangle_{\text{model}}.
$$

The positive phase is cheap: clamp $v$ to a data point and use
$p(h \mid v) = \prod_j \sigma(c_j + v^\top W_{:,j})$ in closed form. The negative
phase is the hard one: it needs samples from $p_\theta$, which requires running
the Markov chain to equilibrium, every gradient step.

$$
% caption: The two phases of the log-likelihood gradient: the positive phase pulls energy down at data, the negative phase pushes it up at the model's samples.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % a single energy curve
  \draw[black, very thick]
    plot[domain=0.3:8.4, samples=120]
    (\x, {1.9 - 0.95*exp(-((\x-2.6)^2)/0.7) - 0.6*exp(-((\x-6.0)^2)/0.8)});
  \draw[->, thick] (-0.1,0) -- (8.7,0) node[right] {$x$};
  \node[black, anchor=west] at (6.7,2.35) {energy $E(x)$};
  % data point: pull energy DOWN (green, downward arrow)
  \fill[green] (2.6,0.95) circle (2.6pt);
  \draw[->, green, very thick] (2.6,1.5) -- (2.6,0.6);
  \node[green, anchor=south, font=\scriptsize, align=center] at (2.6,1.55) {data\\push down};
  % model sample: push energy UP (red, upward arrow)
  \fill[red] (6.0,1.3) circle (2.6pt);
  \draw[->, red, very thick] (6.0,0.85) -- (6.0,1.75);
  \node[red, anchor=north, font=\footnotesize, align=center] at (6.0,0.7) {\texttt{model} sample\\push up};
\end{tikzpicture}
$$

## Contrastive divergence

The negative phase asks for samples from $p_\theta$, and reaching equilibrium can
take an enormous number of Gibbs sweeps — far too many to afford on every gradient
step. **Contrastive divergence** (CD-$k$) makes a brutal but effective
approximation: instead of running the chain to equilibrium from a random start,
start it _at the data_ and run it for only $k$ steps (typically $k = 1$). The data
already lives near the modes we care about, so a handful of steps produces a
**negative sample** good enough to estimate the gradient direction.[^gf-cd]

$$
% caption: Block Gibbs sampling for CD-$k$, started at data $v^{(0)}$: each step resamples a whole layer in parallel, and after $k$ rounds $v^{(k)}$ is the negative sample.
\begin{tikzpicture}[>=stealth, font=\small,
  st/.style={circle, draw, thick, minimum size=9mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % chain of states alternating visible (bottom) / hidden (top)
  \node[st] (v0) at (0,0)   {$v^{0}$};
  \node[st, fill=acc!15] (h0) at (2.2,1.5) {$h^{0}$};
  \node[st] (v1) at (4.4,0) {$v^{1}$};
  \node[st, fill=acc!15] (h1) at (6.6,1.5) {$h^{1}$};
  \node[st] (vk) at (8.8,0) {$v^{k}$};
  % sampling arrows
  \draw[->, acc, thick] (v0) -- (h0) node[midway, anchor=south east, font=\scriptsize] {sample $h$};
  \draw[->, acc, thick] (h0) -- (v1) node[midway, anchor=south west, font=\scriptsize] {sample $v$};
  \draw[->, acc, thick] (v1) -- (h1);
  \draw[->, acc, thick] (h1) -- (vk) node[midway, anchor=south west, font=\scriptsize] {after $k$ steps};
  % labels
  \node[anchor=north, font=\scriptsize, text=green] at (0,-0.65) {data start};
  \node[anchor=north, font=\footnotesize, text=red] at (8.8,-0.65) {\texttt{model} sample};
\end{tikzpicture}
$$

The two correlation matrices are then formed from the clamped data ($v^{(0)}$, with
$h^{(0)}$ its expected hiddens) and from the short-chain sample ($v^{(k)}$ with its
hiddens), and the weights step by their difference.

```algorithm
caption: $\textsc{ContrastiveDivergence}(v^{(0)}, W, b, c, k, \eta)$ — one CD-$k$ update of an RBM
$h^{(0)} \gets$ sample from $p(h \mid v^{(0)})$, each $h_j \sim \text{Bernoulli}(\sigma(c_j + v^{(0)\top} W_{:,j}))$
$v \gets v^{(0)}$, $h \gets h^{(0)}$
for $t \gets 1$ to $k$ do // $k$ steps of block Gibbs
  $v \gets$ sample from $p(v \mid h)$, each $v_i \sim \text{Bernoulli}(\sigma(b_i + W_{i,:}\,h))$
  $h \gets$ sample from $p(h \mid v)$, each $h_j \sim \text{Bernoulli}(\sigma(c_j + v^\top W_{:,j}))$
$v^{(k)} \gets v$, $h^{(k)} \gets h$ // the negative sample
$W \gets W + \eta\,\parens{v^{(0)} h^{(0)\top} - v^{(k)} h^{(k)\top}}$ // data minus model
$b \gets b + \eta\,\parens{v^{(0)} - v^{(k)}}$
$c \gets c + \eta\,\parens{h^{(0)} - h^{(k)}}$
return $W, b, c$
```

> **Remark (Why CD works at all).** CD-$k$ does not follow the true likelihood
> gradient: it is a biased estimator. But the bias is in a direction that still
> separates data from non-data: it lowers energy where the data sits and raises it
> just _downhill_ of the data, which is enough to carve wells in the right places.
> Larger $k$ reduces the bias at the cost of more sampling; **persistent CD** keeps
> the chain alive across updates instead of restarting at the data, tracking the
> model distribution more faithfully.

$$
% caption: CD-$k$ restarts the Markov chain at each data batch and runs $k$ short steps (biased, cheap); persistent CD keeps one long-running chain across updates, drifting closer to the true model samples.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  dot/.style={circle, fill=acc, minimum size=2mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % --- CD-k: three short chains each restarting at the data line ---
  \draw[green, thick, dashed] (0,2.3) -- (7.0,2.3);
  \node[green, anchor=east, font=\scriptsize] at (-0.15,2.3) {data};
  \node[anchor=east, font=\scriptsize] at (-0.15,1.55) {CD-$k$};
  \foreach \sx in {0.6,3.0,5.4} {
    \fill[green] (\sx,2.3) circle (1.4pt);
    \draw[->, acc, thick] (\sx,2.3) -- (\sx+0.55,1.55) -- (\sx+1.0,1.35);
  }
  \node[anchor=west, font=\footnotesize, text=acc] at (7.1,1.55) {restart each \texttt{batch}};
  % --- persistent CD: one continuous chain wandering down toward model region ---
  \node[anchor=east, font=\scriptsize] at (-0.15,0.35) {PCD};
  \draw[->, acc, thick] (0.6,0.75) -- (1.8,0.5) -- (3.0,0.6) -- (4.2,0.35) -- (5.4,0.45) -- (6.6,0.3);
  \fill[green] (0.6,0.75) circle (1.4pt);
  \node[anchor=west, font=\scriptsize, text=acc] at (7.1,0.4) {one chain, never reset};
\end{tikzpicture}
$$

The three ways to supply the negative phase trade bias against cost along one axis:
how faithfully the chain reaches the model distribution before each update.

| Method | Chain start | Steps per update | Negative-phase quality | Cost |
| --- | --- | --- | --- | --- |
| Exact MLE | random | run to equilibrium | unbiased | impractical |
| CD-$k$ | the data batch | $k$ (often $1$) | biased, short-range | cheap |
| Persistent CD | previous chain state | $1$, never reset | lower bias over time | cheap, less stable |

## Deep belief networks and deep Boltzmann machines

Stacking turns a single RBM into a deep generative model. A **deep belief network**
(DBN) is trained **greedily, layer by layer**: fit an RBM to the data, freeze it,
treat its hidden activations as the "data" for a second RBM, and repeat. Each new
layer models the distribution of the features below it.

$$
% caption: A deep belief network as stacked RBMs trained greedily bottom-up: each hidden layer feeds the next RBM; lower connections become directed, the top RBM stays undirected.
\begin{tikzpicture}[>=stealth, font=\small,
  u/.style={circle, draw, thick, minimum size=6mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  % visible layer
  \foreach \i in {1,2,3,4} \node[u] (v\i) at (\i*1.3,0) {};
  % h1
  \foreach \i in {1,2,3} \node[u, fill=acc!15] (a\i) at (\i*1.3+0.65,1.6) {};
  % h2
  \foreach \i in {1,2,3} \node[u, draw=acc, fill=acc!15] (b\i) at (\i*1.3+0.65,3.2) {};
  % edges RBM1: v - h1 (neutral scaffold, stays visible)
  \foreach \v in {1,2,3,4} \foreach \h in {1,2,3} \draw[black] (v\v) -- (a\h);
  % edges RBM2: h1 - h2
  \foreach \v in {1,2,3} \foreach \h in {1,2,3} \draw[black] (a\v) -- (b\h);
  % brace-ish labels
  \node[anchor=west, font=\footnotesize] at (5.9,0.8) {RBM 1};
  \node[anchor=west, font=\footnotesize] at (5.9,2.4) {RBM 2};
  \node[anchor=east, font=\footnotesize] at (0.9,0) {$v$};
  \node[anchor=east, font=\footnotesize] at (0.9,1.6) {$h_1$};
  \node[anchor=east, font=\footnotesize] at (0.9,3.2) {$h_2$};
\end{tikzpicture}
$$

A **deep Boltzmann machine** (DBM) keeps _every_ layer undirected: it is a single
energy-based model over all layers at once, $E(v, h_1, h_2) = -v^\top W_1 h_1 -
h_1^\top W_2 h_2$, trained jointly rather than greedily. Both drove the
2006–2009 revival of deep learning: greedy RBM pretraining was the first reliable
way to initialize deep nets, a story we pick up under
[representation learning](/deep-learning/practical/representation-learning).
Modern practice has largely replaced them with directed latent-variable models and
better initialization, but the conceptual lineage runs straight through here.[^gf-dbn]

## Directed versus undirected deep generative models

The energy-based family is the **undirected** branch of deep generative modeling.
Set it beside the **directed** branch (the VAE and its
autoregressive kin) and the two trade exactly opposite difficulties.

| | Directed (VAE) | Undirected (RBM / DBM) |
| --- | --- | --- |
| Factorization | $p(x,z) = p(z)\,p(x \mid z)$ | $p(v,h) = e^{-E(v,h)}/Z$ |
| Normalizer | each conditional self-normalizes | one global $Z$, intractable |
| Sampling | ancestral, one forward pass | MCMC (Gibbs), iterative |
| Inference $p(z \mid x)$ | intractable $\to$ variational bound | exact & factorized (RBM only) |
| Training | maximize the ELBO | CD / persistent CD on the gradient |
| Hard part | the posterior | the partition function |

The directed model self-normalizes but has an intractable
posterior, so it leans on
[approximate inference](/deep-learning/probabilistic-methods/approximate-inference);
the undirected model has an exact, factorized posterior but an intractable
normalizer, so it leans on
[Monte Carlo](/deep-learning/probabilistic-methods/monte-carlo-and-mcmc). Each
is tractable exactly where the other is not.

## Energy models after the RBM

The RBM and DBM faded from practice, but the energy-based idea did not — it
resurfaced with modern neural energies and a new set of tricks for dodging the
partition function.

**Score matching sidesteps $Z$ entirely.** Hyvärinen's insight was that the gradient
of the log-density, $\nabla_x \log p(x) = -\nabla_x E(x)$, does _not_ depend on $Z$
because the constant vanishes under the derivative. Fitting this **score** rather
than the density avoids the partition function outright, and **denoising score
matching** made it practical by learning the score of noise-corrupted data.[^scorematch]
This is the direct ancestor of score-based generative models.

**Deep EBMs train by short-run MCMC.** Replacing the bilinear RBM energy with a
convolutional network gives a modern energy-based model over images, trained with
Langevin-dynamics sampling in place of block Gibbs — persistent contrastive
divergence with a neural energy.[^deepebm] The negative-phase problem of this lesson
is unchanged; only the sampler and the energy parameterization modernized.

**The line runs straight into diffusion.** A
[diffusion model](/deep-learning/generative-models/diffusion-and-score-based-models)
learns exactly the noise-conditional score $\nabla_x \log p_t(x)$ that denoising
score matching targets, and sampling by reversing the noising process is Langevin
dynamics on a sequence of smoothed energies.[^songermon] The partition function that
blocked the Boltzmann machine is avoided rather than computed: the score never
involves $Z$. The undirected, energy-surface picture
of generation that began with the Boltzmann machine is, in this sense, the one that
prevailed.

## Takeaways

- An **energy-based model** writes $p(x) = e^{-E(x)}/Z$ with the **partition
  function** $Z = \sum_x e^{-E(x)}$. Any scalar energy is a valid model; low energy
  means high probability, and probability _ratios_ are free of $Z$.
- The **RBM** uses the bilinear energy $E(v,h) = -v^\top W h - b^\top v - c^\top h$
  on a **bipartite** visible–hidden graph. Bipartiteness forces
  $p(h \mid v) = \prod_j \sigma(c_j + v^\top W_{:,j})$ to factorize, enabling
  parallel **block Gibbs** sampling.
- The log-likelihood gradient splits into a **positive phase** (lower energy at the
  data) and a **negative phase** $\mathbb{E}_{\text{model}}[\nabla_\theta E]$ that
  needs samples from the model: the partition-function problem in disguise.
- **Contrastive divergence** approximates the negative phase with $k$ Gibbs steps
  started at the data; it is biased but carves wells in the right places.
  **Persistent CD** improves the estimate by not restarting the chain.
- **DBNs** stack RBMs greedily; **DBMs** keep all layers undirected and train
  jointly. Both seeded deep learning's revival before directed models took over.
- Directed (VAE) and undirected (RBM/DBM) deep generative models trade opposite
  intractabilities — the **posterior** versus the **partition function**.

[^gf-ebm]: **Goodfellow**, _Deep Learning_, §16.2.4 — energy-based models: an unnormalized log-density $-E_\theta(x)$ turned into $p_\theta(x)=e^{-E_\theta(x)}/Z$ by the Boltzmann normalization.
[^gf-partition]: **Goodfellow**, _Deep Learning_, Ch. 18 — confronting the partition function $Z=\sum_x e^{-E(x)}$, the central intractability of undirected models and the reason ratios are cheap but absolute likelihoods are not.
[^gf-rbm]: **Goodfellow**, _Deep Learning_, §20.2 — the restricted Boltzmann machine: a bipartite visible–hidden graph whose bilinear energy makes $p(h\mid v)$ and $p(v\mid h)$ factorize into logistic conditionals.
[^gf-phases]: **Goodfellow**, _Deep Learning_, §18.1 — the positive/negative phase decomposition of the log-likelihood gradient, lowering energy at the data and raising it under the model's own samples.
[^gf-cd]: **Goodfellow**, _Deep Learning_, §18.2 — contrastive divergence and persistent CD: approximating the negative phase with a few Gibbs steps started at the data rather than running the chain to equilibrium.
[^gf-dbn]: **Goodfellow**, _Deep Learning_, §20.3–20.4 — deep belief networks (greedy layerwise RBM stacking) and deep Boltzmann machines (jointly trained, fully undirected), and their role in the deep-learning revival.
[^scorematch]: **Hyvärinen**, "Estimation of Non-Normalized Statistical Models by Score Matching," JMLR 2005, and **Vincent**, "A Connection Between Score Matching and Denoising Autoencoders," Neural Computation 2011 — fitting $\nabla_x \log p(x)$ to avoid the partition function.
[^deepebm]: **Du & Mordatch**, "Implicit Generation and Modeling with Energy-Based Models," NeurIPS 2019 — deep convolutional energy functions trained with Langevin-dynamics negative sampling.
[^songermon]: **Song & Ermon**, "Generative Modeling by Estimating Gradients of the Data Distribution," NeurIPS 2019 — score-based generation via denoising score matching and annealed Langevin sampling, the bridge to diffusion models.