Energy-Based & Boltzmann Machines

Every generative model so far has paid for tractability with structure. An autoregressive model factorizes $p(x)$ into a chain of conditionals; a normalizing flow restricts itself to invertible maps with cheap Jacobians; a VAE optimizes a bound rather than the likelihood itself. Energy-based models make the opposite trade. They specify the density in the most flexible way possible, any scalar function of $x$, and concentrate the entire difficulty in a single intractable normalizing constant.¹

The energy and the partition function

An energy-based model assigns each configuration $x$ a scalar energy $E(x)$ and turns it into a probability by the Boltzmann distribution: exponentiate the negative energy and normalize.

with $Z$ the integral $\int \exp (-E(x))dx$ in the continuous case. The energy is unconstrained: any $E_{\theta }:\mathcal{X}\to \mathbb{R}$ yields a valid density, because the exponential is positive and $Z$ rescales it to integrate to one. That freedom is the source of both the model's flexibility and its central difficulty.

Three facts fall straight out of the definition and frame everything below.

The third row is the key fact. Comparing two configurations, scoring, or running a Markov chain that only ever looks at ratios — none of these need $Z$. Only an absolute probability, or the likelihood's gradient, requires it.

For example, suppose a model assigns energies $E(x_1)=1.5$ and $E(x_2)=3.0$ to two configurations. Their probability ratio needs no $Z$:

The lower-energy configuration $x_1$ is $4.48$ times more probable — a number we computed without ever summing over the $2^d$ configurations. To turn either into an absolute probability, though, we would need $Z={\sum }_x{e}^{-E(x)}$: even for a modest binary vector $x\in \{0,1{\}}^{30}$ that is a sum over $2^{30}\approx 10^9$ terms, and the count doubles with every added unit. That gap — cheap ratios, intractable absolutes — is the reason the training methods below exist.

The energy surface is the object to picture. Data modes are basins, regions the model deems probable, carved as wells in $E$; everything else is a high-energy plateau. The map $x\mapsto e^{-E(x)}$ flips the picture upside down, turning deep wells into tall probability peaks.

The Boltzmann machine

A Boltzmann machine is the energy-based model whose energy is a quadratic form over binary units. Split the units into a visible vector $v\in \{0,1{\}}^d$ (the data) and a hidden vector $h\in \{0,1{\}}^m$ (latent features), and the general Boltzmann machine couples every pair of units. The restricted Boltzmann machine (RBM) drops all within-layer couplings, keeping only the visible–hidden weights $W$:

where $W\in {\mathbb{R}}^{d\times m}$ are the interaction weights and $b,c$ the visible and hidden biases. The joint density is $p(v,h)=e^{-E(v,h)}/Z$ as before. The defining structural fact is that the RBM's interaction graph is bipartite: visible units connect only to hidden units, never to each other.

Conditional independence

The bipartite structure pays off immediately: given the visible units, the hidden units are conditionally independent of one another, and vice versa. This is what makes the RBM practical, so we derive it. Fix $v$ and expand the conditional $p(h\mid v)=p(v,h)/p(v)$. Every term in the energy that does not involve $h$ (namely $-b^{\top }v$ and the constant $\log Z$) is absorbed into the normalizer over $h$, leaving

The exponential of a sum factorizes into a product over $j$, and there is no term coupling $h_j$ to $h_{j^{\prime }}$, precisely because the RBM has no hidden–hidden edge. Normalizing each factor over $h_j\in \{0,1\}$ gives an independent product of logistic conditionals:

where $\sigma (z)=1/(1+e^{-z})$ is the logistic sigmoid. By the symmetry of the energy in $v$ and $h$, the same argument run the other way gives $p(v\mid h)={\prod }_{i}p(v_i\mid h)$ with $p(v_i=1\mid h)=\sigma (b_i+W_{i,:}h)$.³

This is the engine of block Gibbs sampling: because a whole layer is conditionally independent given the other, we resample all of $h$ at once, then all of $v$ at once, alternating — an exact Gibbs sampler whose blocks are entire layers. We lean on this in the negative phase below, and it rests on the Monte Carlo and MCMC machinery.

A worked Gibbs step

For example, take a tiny RBM with $d=3$ visible and $m=2$ hidden units, weights and biases

Clamp a data point $v=(1,0,1)$ and compute the hidden activations. Each hidden unit's input is $c_j+v^{\top }{W}_{:,j}$:

Through the sigmoid, $p(h_1=1\mid v)=\sigma (0.8)=0.690$ and $p(h_2=1\mid v)=\sigma (-0.3)=0.426$. Say the Bernoulli draws give $h=(1,0)$. Now sample the visible layer back from this $h$, using $p(v_i=1\mid h)=\sigma (b_i+W_{i,:}h)$:

Suppose these draw to $v^{(1)}=(1,1,0)$. That single up-then-down pass is one CD-1 step: the reconstruction $v^{(1)}$ differs from the data $v^{(0)}=(1,0,1)$ at two coordinates, and the weight update $\eta (v^{(0)}{h}^{(0)\top }-v^{(1)}{h}^{(1)\top })$ nudges $W$ to make the data more probable and the reconstruction less so. The whole layer resampled in one parallel pass, exactly as the factorization allows.

The partition-function problem

To train by maximum likelihood we need the gradient of $\log p_{\theta }(v)$. Writing the free energy $F(v)=-\log {\sum }_h{e}^{-E(v,h)}$, the marginal is $p(v)=e^{-F(v)}/Z$, so $\log p(v)=-F(v)-\log Z$. Both pieces depend on $\theta$. Differentiate:

The second term is where $Z$ bites. Using ${\nabla }_{\theta }\log Z=\frac{1}{Z}{\nabla }_{\theta }Z$ and ${\nabla }_{\theta }Z={\sum }_x{\nabla }_{\theta }{e}^{-E(x)}=-{\sum }_x{e}^{-E(x)}{\nabla }_{\theta }E(x)$,

The intractable sum has collapsed into an expectation under the model. Putting the two terms together (with $F$ likewise an expectation over the hidden units given the data), the log-likelihood gradient splits cleanly into two pieces of opposite sign:

For the RBM's bilinear energy the abstract gradient becomes concrete. With ${\nabla }_{W}E=-v{h}^{\top }$, the weight update is the difference of two correlation matrices:

The positive phase is cheap: clamp $v$ to a data point and use $p(h\mid v)={\prod }_j\sigma (c_j+v^{\top }{W}_{:,j})$ in closed form. The negative phase is the hard one: it needs samples from $p_{\theta }$, which requires running the Markov chain to equilibrium, every gradient step.

Contrastive divergence

The negative phase asks for samples from $p_{\theta }$, and reaching equilibrium can take an enormous number of Gibbs sweeps — far too many to afford on every gradient step. Contrastive divergence (CD-$k$) makes a brutal but effective approximation: instead of running the chain to equilibrium from a random start, start it at the data and run it for only $k$ steps (typically $k=1$). The data already lives near the modes we care about, so a handful of steps produces a negative sample good enough to estimate the gradient direction.⁵

The two correlation matrices are then formed from the clamped data ($v^{(0)}$, with $h^{(0)}$ its expected hiddens) and from the short-chain sample ($v^{(k)}$ with its hiddens), and the weights step by their difference.

The three ways to supply the negative phase trade bias against cost along one axis: how faithfully the chain reaches the model distribution before each update.

Deep belief networks and deep Boltzmann machines

Stacking turns a single RBM into a deep generative model. A deep belief network (DBN) is trained greedily, layer by layer: fit an RBM to the data, freeze it, treat its hidden activations as the data for a second RBM, and repeat. Each new layer models the distribution of the features below it.

A deep Boltzmann machine (DBM) keeps every layer undirected: it is a single energy-based model over all layers at once, $E(v,h_1,h_2)=-v^{\top }{W}_1{h}_1-h_1^{\top }{W}_2{h}_2$, trained jointly rather than greedily. Both drove the 2006–2009 revival of deep learning: greedy RBM pretraining was the first reliable way to initialize deep nets, a story we pick up under representation learning. Modern practice has largely replaced them with directed latent-variable models and better initialization, but the conceptual lineage runs straight through here.⁶

Directed versus undirected deep generative models

The energy-based family is the undirected branch of deep generative modeling. Set it beside the directed branch (the VAE and its autoregressive kin) and the two trade exactly opposite difficulties.

The directed model self-normalizes but has an intractable posterior, so it leans on approximate inference; the undirected model has an exact, factorized posterior but an intractable normalizer, so it leans on Monte Carlo. Each is tractable exactly where the other is not.

Energy models after the RBM

The RBM and DBM faded from practice, but the energy-based idea did not — it resurfaced with modern neural energies and a new set of tricks for dodging the partition function.

Score matching sidesteps $Z$ entirely. Hyvärinen's insight was that the gradient of the log-density, ${\nabla }_x\log p(x)=-{\nabla }_{x}E(x)$, does not depend on $Z$ because the constant vanishes under the derivative. Fitting this score rather than the density avoids the partition function outright, and denoising score matching made it practical by learning the score of noise-corrupted data.⁷ This is the direct ancestor of score-based generative models.

Deep EBMs train by short-run MCMC. Replacing the bilinear RBM energy with a convolutional network gives a modern energy-based model over images, trained with Langevin-dynamics sampling in place of block Gibbs — persistent contrastive divergence with a neural energy.⁸ The negative-phase problem of this lesson is unchanged; only the sampler and the energy parameterization modernized.

The line runs straight into diffusion. A diffusion model learns exactly the noise-conditional score ${\nabla }_x\log p_t(x)$ that denoising score matching targets, and sampling by reversing the noising process is Langevin dynamics on a sequence of smoothed energies.⁹ The partition function that blocked the Boltzmann machine is avoided rather than computed: the score never involves $Z$. The undirected, energy-surface picture of generation that began with the Boltzmann machine is, in this sense, the one that prevailed.

Takeaways

• An energy-based model writes $p(x)=e^{-E(x)}/Z$ with the partition function $Z={\sum }_x{e}^{-E(x)}$. Any scalar energy is a valid model; low energy means high probability, and probability ratios are free of $Z$.
• The RBM uses the bilinear energy $E(v,h)=-v^{\top }Wh-b^{\top }v-c^{\top }h$ on a bipartite visible–hidden graph. Bipartiteness forces $p(h\mid v)={\prod }_j\sigma (c_j+v^{\top }{W}_{:,j})$ to factorize, enabling parallel block Gibbs sampling.
• The log-likelihood gradient splits into a positive phase (lower energy at the data) and a negative phase ${\mathbb{E}}_{\text{model}}[{\nabla }_{\theta }E]$ that needs samples from the model: the partition-function problem in disguise.
• Contrastive divergence approximates the negative phase with $k$ Gibbs steps started at the data; it is biased but carves wells in the right places. Persistent CD improves the estimate by not restarting the chain.
• DBNs stack RBMs greedily; DBMs keep all layers undirected and train jointly. Both seeded deep learning's revival before directed models took over.
• Directed (VAE) and undirected (RBM/DBM) deep generative models trade opposite intractabilities — the posterior versus the partition function.