Probability & Information Theory

A trained network does not output a number; it outputs a distribution. A classifier emits $p_{\theta }(y\mid x)$ over labels, a language model $p_{\theta }(x_t\mid x_{<t})$ over tokens, a generative model $p_{\theta }(x)$ over images. Training bends that distribution toward the data, and the gap between the two is measured information-theoretically. This lesson is the probabilistic substrate every later loss stands on, the companion to linear algebra and numerical computation.

Random variables and their distributions

A random variable is a variable whose value is uncertain; a probability distribution says how likely each value is. The form of the distribution depends on whether the variable is discrete or continuous.

The distinction matters in code: a PMF value is a probability and lies in $[0,1]$; a PDF value is a density and may exceed $1$ (a narrow Gaussian peaks above $1$), only the integral is constrained.¹

Marginal and conditional probability

Given a joint distribution over several variables, two operations recover the pieces. Marginalization sums (or integrates) a variable away; conditioning fixes one variable's value and renormalizes.

Geometrically, a joint density over $(x,y)$ is a surface; the marginal $p(x)$ is its shadow projected onto the $x$-axis, while the conditional $p(y\mid x_0)$ is a vertical slice at $x=x_0$ rescaled to integrate to one.

The chain rule of probability

Any joint distribution factors into a product of conditionals. Apply the conditional-probability definition repeatedly:

This is the identity an autoregressive model exploits directly: a language model writes $p(x_1,\dots ,x_T)={\prod }_{t}p(x_t\mid x_{<t})$ and learns each factor with the same network applied left to right.²

Independence and conditional independence

Two variables are independent when their joint factors into marginals: knowing one tells you nothing about the other. They are conditionally independent when that factorization holds only after a third variable is fixed.

Conditional independence is strictly weaker than independence and the more useful of the two: a child's height and vocabulary are correlated marginally, but conditionally independent given age. Whole families of models (naive Bayes, hidden Markov models, the structured probabilistic models below) are built by asserting such conditional independencies to make a joint tractable.

Expectation, variance, covariance

The expectation of a function under a distribution is its probability-weighted average, the single most-used operation in the subject, since every loss is an expectation and every gradient an expectation of a gradient.

Variance measures spread around the mean; covariance measures how two variables move together; correlation is covariance normalized to $[-1,1]$.

For a random vector $\mathbf{x}\in {\mathbb{R}}^d$ the pairwise covariances collect into the covariance matrix ${\Sigma }_{ij}=\mathrm{Cov}({\mathrm{x}}_i,{\mathrm{x}}_j)$, with the variances on its diagonal. $\Sigma$ is symmetric and positive semi-definite.

A worked covariance

Let $(\mathrm{x},\mathrm{y})$ take four equally likely values: $(1,2)$, $(2,1)$, $(3,5)$, $(4,4)$. Compute the moments by the definitions above.

The sign is positive, so $\mathrm{x}$ and $\mathrm{y}$ tend to rise together. To read it on the bounded scale, normalize: with $\mathrm{Var}(\mathrm{x})=\frac{1+4+9+16}{4}-2.5^2=1.25$ and $\mathrm{Var}(\mathrm{y})=\frac{4+1+25+16}{4}-3^2=2.5$,

Common distributions

A short catalogue carries most of deep learning. Each row is a model for the output of some layer or the noise in some assumption.

The Gaussian is ubiquitous for two reasons: the central limit theorem makes it the default model of summed noise, and among all distributions with a given variance it is the maximum-entropy choice — the least-committal assumption you can make.³ Its bell shape is fixed by two numbers, the mean $\mu$ (location) and the variance ${\sigma }^2$ (spread).

The empirical distribution is the most important row for practice: it is the training set itself, viewed as a sum of Dirac spikes, one at each data point. Every expectation under it is a plain average over the data, ${\mathbb{E}}_{\hat{p}}[f]=\frac{1}{n}{\sum }_{i}f(x^{(i)})$ — the empirical risk a network minimizes.

A Gaussian mixture stacks several bells with mixing weights ${\pi }_k$ summing to one. With enough components it approximates any smooth density, and its multimodality is what a single Gaussian cannot capture.

Bayes' rule

Bayes' rule inverts a conditional: it turns $P(x\mid y)$, which you can model, into $P(y\mid x)$, which you want to infer. It is a one-line consequence of the chain rule.

The four pieces have standard names: $P(y)$ the prior, $P(x\mid y)$ the likelihood, $P(x)$ the evidence (a normalizer), and $P(y\mid x)$ the posterior.

A worked example

A disease afflicts $1\%$ of a population. A test detects it with $99\%$ sensitivity and has a $5\%$ false-positive rate. A patient tests positive; what is the probability they are sick? Let $\mathrm{y}=1$ denote disease and $\mathrm{x}=1$ a positive test.

The evidence is the total positive rate, sick and healthy paths combined:

A positive test leaves the patient only $\approx 17\%$ likely to be sick: the base rate is so low that false positives swamp true ones. The tree makes the arithmetic visible — of every $10,000$ people, only $99$ true positives compete against $495$ false positives, so a positive result is wrong five times out of six.

This base-rate neglect is the most common probabilistic error; Bayes' rule corrects it.⁴

Information theory

Information theory quantifies surprise. The guiding intuition: a likely event carries little information, an unlikely event carries a lot, and the information of independent events adds. The unique function (up to base) satisfying these is the logarithm.⁵

Self-information of a single outcome, averaged over the distribution, is the Shannon entropy: the expected surprise, or the irreducible number of bits needed to encode draws from $P$.

For a single binary variable with $P(\mathrm{x}=1)=p$, entropy is the binary entropy function $H(p)=-p\log p-(1-p)\log (1-p)$. It peaks where uncertainty is greatest — a fair coin — and vanishes at the certain ends.

KL divergence

To compare two distributions $P$ and $Q$ over the same variable, measure the extra cost of encoding $P$'s data with a code built for $Q$. That is the Kullback–Leibler divergence.

For example, take $P=(0.9,0.1)$ and $Q=(0.5,0.5)$ over two outcomes, in nats:

The two numbers differ, so KL is not a metric. Which direction to minimize is a modeling choice with visible consequences, spelled out below.

Two properties make it the right tool, and one caveat makes it tricky.

The KL divergence is not symmetric: in general $D_{KL}(P\Vert Q)\ne D_{KL}(Q\Vert P)$, so it is not a distance. The asymmetry matters in practice. $D_{KL}(P\Vert Q)$ is weighted by $P$, so it penalizes a $Q$ that puts low mass where $P$ is high. Fitting $Q$ to $P$ this way makes $Q$ mass-covering (spread out to cover all of $P$'s support). The reverse $D_{KL}(Q\Vert P)$ is weighted by $Q$ and makes $Q$ mode-seeking (it picks one mode of $P$ and ignores the rest), the form variational inference minimizes.⁶

Cross-entropy and the loss

Cross-entropy is the entropy of $P$ plus the divergence from $P$ to $Q$, the total bits to encode $P$'s data using $Q$'s code. The decomposition is immediate from the definitions.

This is the bridge from information theory to the training loop. Set $P=\hat{p}$, the empirical distribution of the labels, and $Q=p_{\theta }$, the network's predictive distribution. Cross-entropy then equals the negative log-likelihood, the standard classification loss:

For a one-hot label $y$ and softmax output $\hat{y}=p_{\theta }(\cdot \mid x)$ over $K$ classes, the per-example cross-entropy collapses to a single term, the negative log-probability the model assigned to the true class:

The shifted-logit form computes $-\log {\hat{y}}_c$ without ever forming the tiny probability ${\hat{y}}_c$ explicitly, the same log-sum-exp trick the numerical computation lesson develops in full.

Graphical models: distributions as factorizations

A joint over many variables is astronomically large — $K^n$ entries for $n$$K$-valued variables. Structured probabilistic models (graphical models) address this by writing the joint as a product of factors over small subsets of variables, encoding the conditional independencies that make most of those entries redundant.

The two families are two ways to draw the same idea (a joint is the product of local factors, one per edge group of a graph), and the chain rule is the directed case made fully explicit (every variable conditioned on all predecessors).

We develop inference and learning in these models (belief propagation, sampling, the partition function's intractability) in the structured probabilistic models chapter. For now the takeaway is structural: a graph is a set of conditional independence assumptions, and without them a joint over millions of pixels or tokens would not be learnable at all.

Which KL you minimize

The information-theoretic quantities here are the objectives of whole model families, and the direction of the KL is the design choice that separates them.

Which KL you minimize is which model you get. The variational autoencoder (Kingma and Welling, 2013) minimizes the forward, mass-covering $D_{KL}(q\Vert p)$ inside its evidence lower bound, which is why VAE samples tend to be blurry averages that cover the data's support. A GAN's discriminator drives the generator toward a more mode-seeking objective, which is why GAN samples are sharp but can collapse to a few modes. The mass-covering / mode-seeking split this lesson draws from a bimodal toy is, at scale, the difference between the two dominant families of generative models.

Making the expectation differentiable. Training these models needs the gradient of an expectation ${\mathbb{E}}_{z\sim q_{\theta }}[f(z)]$ where the distribution depends on $\theta$. The reparameterization trick (Kingma and Welling, 2013) rewrites a Gaussian draw as $z={\mu }_{\theta }+{\sigma }_{\theta }\odot ϵ$ with $ϵ\sim \mathcal{N}(0,I)$ fixed, moving $\theta$ out of the sampling step so back-propagation can flow through it — a direct application of the change-of-variable view of a distribution.

Cross-entropy, softened. The cross-entropy loss derived above assumes a one-hot target, which pushes the model toward infinite-confidence logits. Label smoothing (Szegedy et al., 2016) replaces the one-hot target with a slightly spread distribution, so the objective is cross-entropy against a target of nonzero entropy — a calibrated, better-generalizing classifier that the KL decomposition $H(P,Q)=H(P)+D_{KL}(P\Vert Q)$ explains exactly: raising $H(P)$ lifts the loss floor and discourages overconfidence.⁷⁸

Takeaways

• A network outputs a distribution; PMFs (discrete, values in $[0,1]$) and PDFs (continuous, densities that may exceed $1$) describe it. Marginalize to drop a variable, condition to fix one.
• The chain rule $p(x_{1:n})={\prod }_{i}p(x_i\mid x_{<i})$ factors any joint and underlies autoregressive models. Conditional independence is the weaker, more useful variant of independence.
• Expectation is the universal average; covariance measures linear co-movement (zero covariance $\ne$ independence). The worked example gave $\mathrm{Cov}=1.25$, $\rho \approx 0.707$.
• A short distribution catalogue — Bernoulli, categorical, Gaussian (and multivariate), exponential, Laplace, Dirac/empirical, Gaussian mixture — models every output head and noise assumption in the field.
• Bayes' rule falls out of the chain rule and inverts a conditional; the base-rate example ($17\%$ sick after a positive test) is the canonical lesson in not ignoring the prior.
• Information theory ties it together: self-information $I(x)=-\log P(x)$, entropy $H(P)$ (peaking at the uniform/fair-coin case), the non-negative, asymmetric $D_{KL}(P\Vert Q)$, and cross-entropy $H(P,Q)=H(P)+D_{KL}(P\Vert Q)$.
• Minimizing cross-entropy in $Q$ minimizes $D_{KL}(\hat{p}\Vert p_{\theta })$ — which is the negative-log-likelihood cross-entropy loss every classifier trains on.