---
title: "Statistical Entropy: Boltzmann and Gibbs"
module: Microstates, Phase Space, and Statistical Entropy
moduleNumber: 2
lessonNumber: 4
order: 204
summary: >
  Entropy is the logarithm of the number of accessible microstates. This lesson
  builds the two statistical entropies — Boltzmann's S = k ln Omega for an
  isolated system and Gibbs's S = -k sum p ln p for any ensemble — proves they
  agree for a uniform distribution, and connects both to Shannon's measure of
  missing information. The second law emerges as the drift toward maximum
  multiplicity, and maximizing the Gibbs entropy under constraints previews the
  canonical distribution.
topics: ["Microstates, Phase Space, and Statistical Entropy"]
sources:
  - book: Reif
    ref: "Ch. 3 Statistical Thermodynamics; §3.3–3.6"
  - book: Schroeder
    ref: "An Introduction to Thermal Physics, Ch. 2 The Second Law; §2.1–2.6; Ch. 3 §3.1"
  - book: Kardar
    ref: "Statistical Physics of Particles, Ch. 4 Classical Statistical Mechanics; §4.3"
  - book: Pathria & Beale
    ref: "Statistical Mechanics, Ch. 1 §1.2; Ch. 3 §3.3"
draft: false
---

The equal-probability postulate made the equilibrium of an isolated system the
macrostate of largest multiplicity. Turning that counting principle into
thermodynamics requires a single quantity whose maximization is equivalent to
maximizing the multiplicity and which is additive across independent systems, so
that it matches the extensive entropy of the macroscopic theory. The logarithm of
the multiplicity is that quantity. This lesson constructs Boltzmann's entropy for
an isolated system, generalizes it to Gibbs's entropy for an arbitrary ensemble,
shows the two coincide when the distribution is uniform, and reads both as a
measure of missing information.[^reif-entropy]

## The Boltzmann entropy

Let $\Omega$ be the number of microstates accessible to an isolated system at
fixed energy, volume, and particle number — the count of states in the energy
shell of the previous lessons. Two demands single out its functional form as the
entropy. The entropy must be a monotonic function of $\Omega$, since more
accessible states should mean more entropy, and it must be **additive** for
independent systems, to match the extensivity of thermodynamic entropy.

Consider two independent systems with multiplicities $\Omega_1$ and $\Omega_2$.
Because any microstate of the first can be combined with any microstate of the
second, the joint system has multiplicity

$$
\Omega_{12} = \Omega_1\,\Omega_2.
$$

Additivity demands a function $S(\Omega)$ with
$S(\Omega_1\Omega_2) = S(\Omega_1) + S(\Omega_2)$. The only continuous solution is
the logarithm, and the constant of
proportionality is fixed by matching to the thermodynamic temperature scale.

> **Definition (Boltzmann entropy).** The entropy of an isolated system with
> $\Omega$ accessible microstates (Boltzmann's multiplicity $W$) is
> $$
> S = k \ln \Omega,
> $$
> where $k = 1.381 \times 10^{-23}\ \mathrm{J/K}$ is the Boltzmann constant. The
> logarithm converts the multiplicative multiplicity into an additive entropy;
> the constant $k$ fixes the unit so that $1/T = \partial S/\partial E$ agrees
> with the absolute temperature.[^schroeder-boltz]

The multiplicative-to-additive conversion is the whole content of the logarithm,
and it is worth seeing drawn: two boxes whose state-counts multiply have
entropies that add.

$$
% caption: The multiplicities (the state-counts, written Omega in the text and
% W here) of independent systems multiply while their entropies add; the
% logarithm converts the product of counts into a sum of entropies.
\begin{tikzpicture}[scale=1.0, font=\footnotesize]
\definecolor{acc}{HTML}{4A6FA5}
% system 1
\draw[acc, thick] (0,0) rectangle (2.6,1.8);
\node[anchor=south, font=\scriptsize] at (1.3,1.85) {system 1};
\node[anchor=center] at (1.3,1.15) {$W_1$};
\node[text=acc, anchor=center, font=\scriptsize] at (1.3,0.45) {$S_1 = k\ln W_1$};
% times
\node[font=\large] at (3.15,0.9) {$\times$};
% system 2
\begin{scope}[xshift=3.7cm]
\draw[acc, thick] (0,0) rectangle (2.6,1.8);
\node[anchor=south, font=\scriptsize] at (1.3,1.85) {system 2};
\node[anchor=center] at (1.3,1.15) {$W_2$};
\node[text=acc, anchor=center, font=\scriptsize] at (1.3,0.45) {$S_2 = k\ln W_2$};
\end{scope}
% equals combined
\node[font=\large] at (6.85,0.9) {$=$};
\begin{scope}[xshift=7.5cm]
\draw[acc, very thick, fill=acc!10] (0,0) rectangle (3.0,1.8);
\node[anchor=south, font=\scriptsize] at (1.5,1.85) {combined};
\node[anchor=center] at (1.5,1.15) {$W_1 W_2$};
\node[text=acc, anchor=center, font=\scriptsize] at (1.5,0.45) {$S = S_1 + S_2$};
\end{scope}
\end{tikzpicture}
$$

The counting is done with Stirling's approximation, which is what makes $\ln\Omega$
extensive. For the combinatorial multiplicities of many-body systems $\Omega$
involves factorials of numbers of order $N$, and

$$
\ln N! = N\ln N - N + \tfrac{1}{2}\ln(2\pi N) + \cdots
\;\approx\; N\ln N - N
$$

for large $N$, with the correction terms subextensive and negligible. Every
entropy computed in the following modules — the two-state paramagnet, the Einstein
solid, the ideal gas — runs through this approximation.

## Multiplicity of a two-state system

The paramagnet of a later module is the cleanest illustration and supplies a
second figure. Take $N$ independent spins, each up or down, with $n$ up. The
multiplicity is the binomial coefficient

$$
\Omega(n) = \binom{N}{n} = \frac{N!}{n!\,(N-n)!},
$$

the number of ways to choose which spins point up. Its logarithm, via Stirling
with $f = n/N$, is

$$
\ln\Omega(n) = -N\big[f\ln f + (1-f)\ln(1-f)\big],
$$

the binary entropy function times $N$. It is maximal at $f = \tfrac{1}{2}$
(equal numbers up and down), where $\Omega$ takes its largest value
$\binom{N}{N/2}$, and it falls off as a Gaussian of width $\Delta n = \tfrac{1}{2}
\sqrt{N}$ about the peak — the sharpness established in the last lesson, here for a
concrete count.

$$
% caption: The multiplicity of N spins with n pointing up peaks sharply at the
% equal split n = N/2, with a Gaussian width proportional to the square root of
% N; the vast majority of arrangements sit near equal populations.
\begin{tikzpicture}[scale=1.0, font=\footnotesize]
\definecolor{acc}{HTML}{4A6FA5}
\draw[->, black] (0,0) -- (7.2,0) node[right] {n / N};
\draw[->, black] (0,0) -- (0,3.3) node[above] {number of ways};
\draw[black] (3.5,0.06) -- (3.5,-0.06) node[anchor=north, font=\scriptsize] {$\frac{1}{2}$};
\draw[black] (0.2,0.06) -- (0.2,-0.06) node[anchor=north, font=\scriptsize] {0};
\draw[black] (6.8,0.06) -- (6.8,-0.06) node[anchor=north, font=\scriptsize] {1};
\draw[acc, very thick, smooth] plot coordinates
  {(0.2,0.02)(1.0,0.08)(1.7,0.28)(2.3,0.85)(2.9,1.9)(3.3,2.8)(3.5,3.0)(3.7,2.8)(4.1,1.9)(4.7,0.85)(5.3,0.28)(6.0,0.08)(6.8,0.02)};
\draw[acc, dashed] (3.5,0) -- (3.5,3.0);
\node[text=acc, anchor=west, font=\scriptsize] at (3.85,2.4) {width shrinks as N grows};
\end{tikzpicture}
$$

> **Worked example.** Take $N = 100$ spins with $n = 50$ up. The exact
> multiplicity is $\binom{100}{50} \approx 1.01\times 10^{29}$, so the entropy is
> $$
> S = k\ln\binom{100}{50} = k\,(66.78) = 9.22\times 10^{-22}\ \mathrm{J/K}.
> $$
> The Stirling formula $\ln\Omega \approx N\ln 2 = 69.31$ for the equal split
> overshoots by the subextensive $\tfrac{1}{2}\ln(\pi N/2) = 2.53$, an error of
> $3.8\%$ at $N = 100$ that falls to order $10^{-21}$ relative at $N = 10^{22}$. The
> leading term $N\ln 2$ is the entropy per spin $k\ln 2$ times $N$: each spin,
> free to be up or down, contributes one bit, and $k\ln 2 = 9.57\times
> 10^{-24}\ \mathrm{J/K}$ is the entropy of one bit of missing information.

## The Gibbs entropy

Boltzmann's formula presumes a uniform distribution over the accessible states —
the microcanonical situation. A system in contact with a reservoir is not
uniform: its microstates carry unequal probabilities $p_i$. Gibbs's entropy
extends the definition to any probability distribution over microstates.

> **Definition (Gibbs entropy).** For an ensemble assigning probability $p_i$ to
> microstate $i$, the entropy is
> $$
> S = -k \sum_i p_i \ln p_i,
> $$
> the sum running over all microstates, with the convention $0\ln 0 = 0$. For a
> continuous phase-space density the sum becomes an integral,
> $S = -k\int \rho \ln\rho\; \d^{3N}q\,\d^{3N}p$ over suitably normalized
> cells.[^kardar-gibbs]

The Gibbs entropy reduces to the Boltzmann entropy exactly when the distribution
is uniform over $\Omega$ accessible states. Setting $p_i = 1/\Omega$ for each of
the $\Omega$ accessible microstates and zero otherwise,

$$
S = -k \sum_{i=1}^{\Omega} \frac{1}{\Omega}\ln\frac{1}{\Omega}
= -k\,\Omega \cdot \frac{1}{\Omega}\,(-\ln\Omega)
= k \ln\Omega.
$$

The two entropies agree on the microcanonical ensemble, and the Gibbs form is the
general one, valid for the canonical and grand-canonical ensembles where the
probabilities are not uniform. It is additive in the same sense: for two
independent systems the joint distribution factorizes, $p_{ij} = p_i^{(1)}
p_j^{(2)}$, and

$$
S_{12} = -k\sum_{ij} p_i^{(1)}p_j^{(2)}
\big(\ln p_i^{(1)} + \ln p_j^{(2)}\big) = S_1 + S_2.
$$

## Entropy as missing information

The Gibbs form is identical, up to the constant $k$, to the quantity Shannon
introduced as the measure of the information missing from a probability
distribution — the average number of yes/no questions needed to determine the
microstate. A distribution concentrated on one microstate ($p_1 = 1$, the rest
zero) has $S = 0$: the state is known, nothing is missing. A distribution spread
evenly over $\Omega$ states has the maximum entropy $k\ln\Omega$ available to that
support: the state is maximally uncertain. Between these extremes the entropy
measures how spread the distribution is.

$$
% caption: The Gibbs entropy measures the spread of a distribution. A broad
% distribution over many microstates has large entropy; as probability
% concentrates onto a few states the entropy falls, reaching zero for a
% distribution supported on a single state.
\begin{tikzpicture}[scale=1.0, font=\footnotesize]
\definecolor{acc}{HTML}{4A6FA5}
% left: broad distribution
\draw[->, black] (0,0) -- (3.4,0) node[right, font=\scriptsize] {state};
\draw[->, black] (0,0) -- (0,2.6) node[above, font=\scriptsize] {$p_i$};
\foreach \x/\h in {0.35/0.9, 0.85/1.05, 1.35/1.15, 1.85/1.05, 2.35/0.95, 2.85/0.8}
  \fill[acc!18, draw=acc] (\x,0) rectangle (\x+0.4,\h);
\node[text=acc, anchor=south, font=\scriptsize] at (1.6,1.9) {large $S$};
\node[anchor=north, font=\scriptsize] at (1.6,-0.35) {spread out};
% right: peaked distribution
\begin{scope}[xshift=5.0cm]
\draw[->, black] (0,0) -- (3.4,0) node[right, font=\scriptsize] {state};
\draw[->, black] (0,0) -- (0,2.6) node[above, font=\scriptsize] {$p_i$};
\foreach \x/\h in {0.35/0.12, 0.85/0.22, 1.35/2.3, 1.85/0.22, 2.35/0.1, 2.85/0.06}
  \fill[acc!18, draw=acc] (\x,0) rectangle (\x+0.4,\h);
\node[text=acc, anchor=west, font=\scriptsize] at (2.45,1.7) {small $S$};
\node[anchor=north, font=\scriptsize] at (1.6,-0.35) {concentrated};
\end{scope}
\draw[->, black, very thick] (3.7,1.1) -- (4.7,1.1);
\node[anchor=south, font=\scriptsize] at (4.2,1.15) {concentrate};
\end{tikzpicture}
$$

This reading resolves the apparent subjectivity of the equal-probability
postulate. The uniform distribution is the one that assumes the least beyond the
constraints — it maximizes the missing information consistent with what is known.
Assigning any other distribution would encode information about the microstate
that the macroscopic constraints do not supply.

## The second law as the drift to maximum multiplicity

Bring two isolated systems into contact so they exchange energy at fixed total
$E = E_1 + E_2$. Before contact each was separately in equilibrium; the combined
constraint now allows the energy to redistribute. By the equal-probability
postulate the joint system explores all microstates compatible with the total
energy, and the probability of a split $(E_1, E - E_1)$ is proportional to the
joint multiplicity

$$
\Omega_{12}(E_1) = \Omega_1(E_1)\,\Omega_2(E - E_1).
$$

Taking the logarithm, the equilibrium split maximizes

$$
\ln\Omega_{12} = \ln\Omega_1(E_1) + \ln\Omega_2(E - E_1),
$$

that is, it maximizes the total entropy $S_1 + S_2$. Setting the derivative with
respect to $E_1$ to zero,

$$
\frac{\partial \ln\Omega_1}{\partial E_1} = \frac{\partial \ln\Omega_2}{\partial E_2}
\quad\Longleftrightarrow\quad
\frac{\partial S_1}{\partial E_1} = \frac{\partial S_2}{\partial E_2},
$$

which is the equality $1/T_1 = 1/T_2$ once temperature is defined by $1/T =
\partial S/\partial E$. The systems reach a common temperature, energy flows so as
to increase the total number of accessible microstates, and the second law is the
statement that this number does not decrease. The next module derives temperature,
pressure, and the chemical potential in full from this maximization; the point
here is that the second law is a counting statement about multiplicity.

## Maximizing the Gibbs entropy under constraints

The same maximization principle, applied to the Gibbs entropy with constraints
imposed by Lagrange multipliers, generates the equilibrium distributions of the
following modules. Seek the distribution $\{p_i\}$ that maximizes $S = -k\sum_i
p_i\ln p_i$ subject to normalization and a fixed mean energy,

$$
\sum_i p_i = 1, \qquad \sum_i p_i E_i = \langle E \rangle.
$$

Introduce multipliers $\alpha$ for normalization and $\beta$ for the energy, and
extremize

$$
\mathcal{L} = -\sum_i p_i \ln p_i
- \alpha\Big(\sum_i p_i - 1\Big)
- \beta\Big(\sum_i p_i E_i - \langle E\rangle\Big),
$$

where $k$ has been absorbed into the multipliers. Differentiating with respect to
$p_i$,

$$
\frac{\partial \mathcal{L}}{\partial p_i}
= -\ln p_i - 1 - \alpha - \beta E_i = 0
\quad\Longrightarrow\quad
p_i = e^{-1-\alpha}\,e^{-\beta E_i}.
$$

Normalization fixes the prefactor and gives the **canonical distribution**

$$
p_i = \frac{e^{-\beta E_i}}{Z}, \qquad Z = \sum_i e^{-\beta E_i},
$$

with $Z$ the partition function. The multiplier $\beta$ is identified with
$1/kT$ by matching the resulting entropy to the thermodynamic $\partial S/\partial
E$, recovering the Boltzmann factor of the first lesson from a maximum-entropy
principle rather than a reservoir argument. The two derivations meet in the
canonical module.

$$
% caption: Maximum-entropy inference. With no constraint beyond normalization the
% flat distribution maximizes the Gibbs entropy; adding a mean-energy constraint
% tilts it into the exponential Boltzmann form.
\begin{tikzpicture}[scale=1.0, font=\footnotesize]
\definecolor{acc}{HTML}{4A6FA5}
% left: flat distribution
\draw[->, black] (0,0) -- (3.6,0) node[right, font=\scriptsize] {$E_i$};
\draw[->, black] (0,0) -- (0,2.4) node[above, font=\scriptsize] {$p_i$};
\foreach \x in {0.35,0.85,1.35,1.85,2.35,2.85}
  \fill[acc!18, draw=acc] (\x,0) rectangle (\x+0.4,1.1);
\node[anchor=north, font=\scriptsize] at (1.7,-0.35) {no constraint: uniform};
% right: exponential
\begin{scope}[xshift=5.0cm]
\draw[->, black] (0,0) -- (3.6,0) node[right, font=\scriptsize] {$E_i$};
\draw[->, black] (0,0) -- (0,2.4) node[above, font=\scriptsize] {$p_i$};
\foreach \x/\h in {0.35/2.0, 0.85/1.5, 1.35/1.12, 1.85/0.84, 2.35/0.63, 2.85/0.47}
  \fill[acc!18, draw=acc] (\x,0) rectangle (\x+0.4,\h);
\draw[black, thick, smooth] plot coordinates {(0.55,2.0)(1.05,1.5)(1.55,1.12)(2.05,0.84)(2.55,0.63)(3.05,0.47)};
\node[anchor=north, font=\scriptsize] at (1.7,-0.35) {held mean $E$: Boltzmann};
\end{scope}
\draw[->, black, very thick] (3.9,1.0) -- (4.7,1.0);
\end{tikzpicture}
$$

## The entropies in one view

The three notions of entropy that recur through the course are one quantity seen
under different constraints.

| Entropy | Form | Applies to | Reduces to |
| --- | --- | --- | --- |
| Boltzmann | $S = k\ln\Omega$ | isolated system, uniform $p_i$ | itself (microcanonical) |
| Gibbs | $S = -k\sum_i p_i\ln p_i$ | any ensemble | $k\ln\Omega$ when uniform |
| Shannon | $H = -\sum_i p_i\log_2 p_i$ | any distribution | $S/(k\ln 2)$ |

The Boltzmann entropy counts states for the isolated system, the Gibbs entropy
generalizes to systems in contact, and the Shannon measure supplies the
information-theoretic reading that justifies the equal-probability postulate as
least-biased inference. All three grow toward the same maximum as a distribution
spreads, and the second law is their common increase. The following module puts
the Boltzmann form to work, deriving the microcanonical thermodynamics of the
ideal gas and the two-state paramagnet.

[^reif-entropy]: **Reif**, _Fundamentals of Statistical and Thermal Physics_, §3.3–3.5 — the definition of entropy as $k\ln\Omega$, its additivity, and the approach to equilibrium as the increase of the total number of accessible states.
[^schroeder-boltz]: **Schroeder**, _An Introduction to Thermal Physics_, §2.6 and §3.1 — the multiplicity of combinatorial systems, Stirling's approximation, and the definition $S = k\ln\Omega$ with $1/T = \partial S/\partial E$.
[^kardar-gibbs]: **Kardar**, _Statistical Physics of Particles_, §4.3, and **Pathria & Beale**, _Statistical Mechanics_, §3.3 — the Gibbs entropy $-k\sum p_i\ln p_i$, its reduction to $k\ln\Omega$ for a uniform distribution, and the maximum-entropy derivation of the canonical distribution by Lagrange multipliers.