---
title: Bayesian & Ensemble Methods
module: Theory & Frontiers
moduleNumber: 6
lessonNumber: 4
order: 604
summary: >
  A trained network returns a single point prediction and, with the softmax,
  a confidence, but that confidence is usually miscalibrated, collapsing to near-
  certainty even on inputs the model has never seen. This lesson covers uncertainty
  estimation for networks: the two kinds of uncertainty, the Bayesian
  posterior over weights and its tractable stand-ins (MC dropout, deep ensembles),
  and how to check whether a model's reported confidences match observed frequencies.
topics: [Theory & Frontiers]
sources:
  - book: Goodfellow
    ref: "§5.6 — Bayesian Statistics"
  - book: Goodfellow
    ref: "§7.11 Bagging and Other Ensemble Methods; §7.12 Dropout"
---

A standard classifier ends in a softmax, so it always returns a probability
vector, and it is tempting to read the largest entry as the model's confidence.
That reading is wrong in a specific, dangerous way. A network trained to minimize
cross-entropy is pushed toward _saturated_ outputs on the training set, and
nothing in the objective constrains what it reports off that set. Feed it an input
from a class it has never seen and it will still report $99\%$ confidence;
the softmax has no way to indicate that the input lies outside everything the
model was trained on.

> **Definition (Point estimate).** A single best-fit parameter vector
> $\hat\theta$ (typically the maximum-likelihood or MAP estimate) and the single
> prediction $f_{\hat\theta}(x)$ it produces, with no representation of how much
> that estimate could have varied had the data been different.

The fix is to treat the network's output as a _distribution_ rather than a single
number. This lesson develops three routes to that distribution (the
fully Bayesian posterior over weights, its cheap approximation by dropout left on
at test time, and the brute-force ensemble) and closes with the question that
decides whether any of it worked: is the reported confidence **calibrated**?

## Two kinds of uncertainty

Not all uncertainty has the same cause, and the distinction governs what you can
do about it. Predicting a coin flip, you are uncertain no matter how much data you
collect; the noise is in the world. Predicting in a region where you have _no_
data, you are uncertain only because you have not looked; more data would resolve
it. These are the two irreducible categories.

> **Definition (Aleatoric uncertainty).** Uncertainty from inherent noise in the
> data-generating process: sensor noise, label disagreement, genuine stochasticity.
> It is a property of the _task_, not the model, and **cannot** be reduced by
> collecting more data. In regression it is the variance of $y$ given $x$.

> **Definition (Epistemic uncertainty).** Uncertainty about the model itself —
> which parameters, or which function, is correct given finite data. It is largest
> where data is scarce or out-of-distribution and **shrinks to zero** as data
> grows. It is what a Bayesian posterior over weights captures.

| Type | Source | Reducible with more data? | How to estimate |
| --- | --- | --- | --- |
| Aleatoric | noise in the data / task itself | no, irreducible | predict a variance head $\sigma^2(x)$; heteroscedastic loss |
| Epistemic | ignorance about the model / weights | yes, vanishes as $n \to \infty$ | posterior over weights; ensemble spread; MC dropout |

The total predictive variance decomposes additively into the two, and the
decomposition is the whole point: it tells you whether to collect more data
(epistemic) or accept the floor (aleatoric):

$$
\underbrace{\Var(y \mid x)}_{\text{total}}
=
\underbrace{\mathbb{E}_{\theta}\!\brackets{\sigma^2_\theta(x)}}_{\text{aleatoric}}
+
\underbrace{\Var_{\theta}\!\brackets{\mu_\theta(x)}}_{\text{epistemic}},
$$

where the expectation and variance are taken over the posterior on $\theta$. The
first term is the average noise each model expects; the second is the disagreement
_between_ models. The signature of epistemic uncertainty is a predictive band that
hugs the data where it is dense and fans open where it is absent.

$$
% caption: Aleatoric vs. epistemic uncertainty: the predictive band is narrow near the data
% and fans out away from it, where no observations pin the function down.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{acclo}{HTML}{A7B5FB}
  % axes
  \draw[->, thick] (-0.3,0) -- (9.6,0) node[right, font=\footnotesize] {input $x$};
  \draw[->, thick] (0,-2.4) -- (0,2.6) node[above, font=\footnotesize] {output $y$};
  % data region marker (dense in the middle band)
  \draw[black, fill=black!8] (2.6,-2.3) rectangle (6.4,2.5);
  \node[black, font=\footnotesize, anchor=south] at (4.5,-2.3) {data is dense here};
  % predictive band: light fill, but SOLID accent boundary curves
  \fill[acclo!35]
    plot[domain=0.3:9.2, samples=60] (\x, {0.45*sin(\x r) + 0.55 + 0.035*(\x-4.5)*(\x-4.5)})
    -- plot[domain=9.2:0.3, samples=60] (\x, {0.45*sin(\x r) - 0.55 - 0.035*(\x-4.5)*(\x-4.5)})
    -- cycle;
  \draw[acc, thick] plot[domain=0.3:9.2, samples=60] (\x, {0.45*sin(\x r) + 0.55 + 0.035*(\x-4.5)*(\x-4.5)});
  \draw[acc, thick] plot[domain=0.3:9.2, samples=60] (\x, {0.45*sin(\x r) - 0.55 - 0.035*(\x-4.5)*(\x-4.5)});
  % mean curve
  \draw[acc, very thick] plot[domain=0.3:9.2, samples=80] (\x, {0.45*sin(\x r)});
  % data points clustered in the dense band
  \foreach \x in {2.8,3.3,3.7,4.2,4.6,5.0,5.5,5.9,6.2}
    \fill[black] (\x, {0.45*sin(\x r) + 0.13*sin(40*\x)}) circle (1.6pt);
  \node[acc, font=\footnotesize, anchor=south east] at (9.2,1.95) {band fans out (epistemic)};
\end{tikzpicture}
$$

To estimate aleatoric uncertainty directly, give the network a second output head
that predicts the noise variance $\sigma^2(x)$ and train with the negative
log-likelihood of a Gaussian, which trades fit against predicted variance:

$$
\mathcal{L}(\theta)
= \frac{1}{n}\sum_{i=1}^n
\brackets{
\frac{\parens{y_i - \mu_\theta(x_i)}^2}{2\,\sigma^2_\theta(x_i)}
+ \tfrac12 \log \sigma^2_\theta(x_i)
}.
$$

The $\log\sigma^2$ term is a penalty that stops the model from declaring infinite
noise everywhere to zero out the first term; together they let the network _learn_
where the data is noisy. Epistemic uncertainty needs a different mechanism: a
distribution over the weights themselves.

## The Bayesian view

Maximum likelihood collapses everything into one $\hat\theta$. The Bayesian
treatment instead carries the _whole distribution_ of weights
consistent with the data. Bayes' rule turns a prior over weights into a posterior
once the data $\mathcal{D}$ is observed.[^gf-bayes]

> **Definition (Weight posterior).** Given a prior $p(\theta)$ and a likelihood
> $p(\mathcal{D}\mid\theta)$, the posterior over weights is
> $p(\theta \mid \mathcal{D}) \propto p(\mathcal{D}\mid\theta)\,p(\theta)$. It places
> mass on _every_ setting of the weights, weighted by how well that setting explains
> the data and by its prior plausibility.

A point estimate is the single tallest peak of this posterior; the Bayesian keeps
the entire distribution, and its width _is_ the epistemic
uncertainty.

$$
% caption: A point estimate keeps only the mode (left); the Bayesian posterior keeps the whole distribution over weights (right), whose spread is the epistemic uncertainty.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % ---- left: point estimate ----
  \begin{scope}
    \draw[->, thick] (-0.3,0) -- (4.2,0) node[right, font=\footnotesize] {weight};
    \draw[->, thick] (0,-0.2) -- (0,2.4);
    % single spike
    \draw[green, very thick] (2.0,0) -- (2.0,2.0);
    \fill[green] (2.0,2.0) circle (2.2pt);
    \node[green, font=\footnotesize, anchor=south] at (2.0,2.05) {point estimate};
    \node[font=\footnotesize, anchor=north] at (2.0,-0.05) {best weight};
    \node[font=\footnotesize, anchor=south] at (1.95,-1.15) {one value};
  \end{scope}
  % ---- right: posterior ----
  \begin{scope}[xshift=6cm]
    \draw[->, thick] (-0.3,0) -- (4.2,0) node[right, font=\footnotesize] {weight};
    \draw[->, thick] (0,-0.2) -- (0,2.4);
    % bell curve over the same location
    \draw[acc, very thick] plot[domain=0.2:3.8, samples=60]
      (\x, {2.0*exp(-(\x-2.0)*(\x-2.0)/0.42)});
    \fill[acc] (2.0,2.0) circle (2.0pt);
    \node[acc, font=\footnotesize, anchor=south] at (2.0,2.05) {posterior};
    % spread arrows
    \draw[<->, black, thick] (1.35,0.55) -- (2.65,0.55);
    \node[black, font=\footnotesize, anchor=south] at (2.0,-1.15) {a distribution};
  \end{scope}
\end{tikzpicture}
$$

### The predictive distribution

The point of a posterior is not the weights themselves but the predictions they
imply. To predict $y$ at a new $x$, you do not pick one $\theta$ — you average the
prediction over _all_ of them, each weighted by its posterior probability. This is
the **posterior predictive distribution**:

$$
p(y \mid x, \mathcal{D})
= \int p(y \mid x, \theta)\, p(\theta \mid \mathcal{D})\,\mathrm{d}\theta
= \mathbb{E}_{\theta \sim p(\theta\mid\mathcal{D})}\brackets{\,p(y \mid x, \theta)\,}.
$$

This integral is the entire Bayesian deep-learning programme, and it is
**intractable**.[^gf-mle] The posterior $p(\theta\mid\mathcal{D})$ over millions of weights
has no closed form, and the integral runs over that whole space. Every practical
method is a way to _approximate_ this expectation. Three families dominate, trading
fidelity for cost.

> **Definition (Variational inference).** Replace the true posterior by a simple
> family $q_\phi(\theta)$ (e.g. a diagonal Gaussian over weights) and fit $\phi$ by
> minimizing $\mathrm{KL}\!\parens{q_\phi \,\|\, p(\theta\mid\mathcal{D})}$,
> equivalently maximizing the evidence lower bound (ELBO). **Bayes-by-backprop**
> does exactly this, learning a mean and variance per weight via the
> reparameterization trick.

> **Definition (Laplace approximation).** Fit a point estimate $\hat\theta$, then
> approximate the posterior by the Gaussian centered there whose covariance is the
> inverse Hessian of the loss, $p(\theta\mid\mathcal{D}) \approx
> \mathcal{N}(\hat\theta, H^{-1})$. A second-order curvature read-off around the
> existing optimum: cheap, but only a local picture.

The third route is the one that costs almost nothing to implement, because the
machinery is already in the network.

## MC dropout as approximate inference

[Dropout](/deep-learning/regularization/dropout-and-data-augmentation) randomly
zeroes units during training and is normally switched _off_ at test time.
**Monte Carlo dropout** instead leaves it _on_ at test time: each forward
pass then samples a different sub-network, and the spread of their outputs
approximates the posterior predictive. Gal and Ghahramani showed this procedure
coincides with variational inference under a particular Bernoulli $q_\phi$: dropout was a Bayesian
approximation all along.[^gf-dropout][^postdate-mc]

> **Definition (MC dropout).** At test time, keep dropout active and run $T$
> stochastic forward passes on the same input $x$, each with an independently
> sampled dropout mask $\hat\theta_t$. The set $\{f_{\hat\theta_t}(x)\}_{t=1}^T$ is
> a sample from an approximate posterior predictive; its mean and variance estimate
> the prediction and its epistemic uncertainty.

The predictive mean is the Monte-Carlo average over the $T$ masked passes, a direct
estimate of the intractable integral by sampling:

$$
\widehat{\mathbb{E}}[y \mid x]
= \frac{1}{T}\sum_{t=1}^{T} f_{\hat\theta_t}(x)
\;\approx\;
\int f_\theta(x)\, q_\phi(\theta)\,\mathrm{d}\theta.
$$

The predictive variance is the sample variance of those same passes — and for a
heteroscedastic network it splits cleanly into the two uncertainty types, the
average per-pass noise (aleatoric) plus the spread of the means (epistemic):

$$
\widehat{\Var}[y \mid x]
=
\underbrace{\frac{1}{T}\sum_{t=1}^{T} \sigma^2_{\hat\theta_t}(x)}_{\text{aleatoric}}
\;+\;
\underbrace{\frac{1}{T}\sum_{t=1}^{T}\parens{f_{\hat\theta_t}(x) - \bar f(x)}^2}_{\text{epistemic}},
\qquad
\bar f(x) = \frac{1}{T}\sum_{t=1}^{T} f_{\hat\theta_t}(x).
$$

$$
% caption: MC dropout: $T$ stochastic forward passes of one input, each dropping a different subset of units, yield a spread of outputs whose mean and variance give prediction and uncertainty.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  net/.style={draw, minimum width=20mm, minimum height=8mm, align=center},
  io/.style={circle, draw, minimum size=8mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  % input
  \node[io] (x) at (0,0) {$x$};
  % three sampled sub-networks
  \node[net] (n1) at (3.0,1.7)  {pass 1\\(mask 1)};
  \node[net] (n2) at (3.0,0.0)  {pass 2\\(mask 2)};
  \node[net] (n3) at (3.0,-1.7) {pass 3\\(mask 3)};
  \draw[->, acc, thick] (x) -- (n1);
  \draw[->, acc, thick] (x) -- (n2);
  \draw[->, acc, thick] (x) -- (n3);
  % outputs
  \node (y1) at (6.0,1.7)  {output 1};
  \node (y2) at (6.0,0.0)  {output 2};
  \node (y3) at (6.0,-1.7) {output 3};
  \draw[->, thick] (n1) -- (y1);
  \draw[->, thick] (n2) -- (y2);
  \draw[->, thick] (n3) -- (y3);
  % aggregate
  \node[net, draw=acc, text=acc, thick] (agg) at (9.0,0) {mean\\+ variance};
  \draw[->, acc, thick] (y1) -- (agg);
  \draw[->, acc, thick] (y2) -- (agg);
  \draw[->, acc, thick] (y3) -- (agg);
\end{tikzpicture}
$$

The mechanism is visible on a regression: each stochastic pass draws a slightly
different curve, the curves agree where the data constrains them and scatter where
it does not, and the band between them is the epistemic estimate.

$$
% caption: MC-dropout forward passes on a 1-D regression: the sampled curves pile together
% near the data and scatter where it is absent, their spread being the predictive variance.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, thick] (-0.3,0) -- (9.6,0) node[right, font=\footnotesize] {input $x$};
  \draw[->, thick] (0,-2.2) -- (0,2.4) node[above, font=\footnotesize] {output};
  % data cluster band
  \draw[black, fill=black!8] (3.0,-2.1) rectangle (6.0,2.3);
  % several sampled curves: distinct SOLID colors, scatter away from center
  \draw[black, thick] plot[domain=0.3:9.2, samples=70]
    (\x, {0.6*sin(\x r) + 0.05*(\x-4.5)});
  \draw[red!75, thick] plot[domain=0.3:9.2, samples=70]
    (\x, {0.6*sin(\x r) - 0.05*(\x-4.5)});
  \draw[green, thick] plot[domain=0.3:9.2, samples=70]
    (\x, {0.6*sin(\x r) + 0.10*(\x-4.5) - 0.012*(\x-4.5)*(\x-4.5)});
  \draw[black, thick] plot[domain=0.3:9.2, samples=70]
    (\x, {0.6*sin(\x r) - 0.10*(\x-4.5) + 0.012*(\x-4.5)*(\x-4.5)});
  % the mean curve (dominant accent)
  \draw[acc, very thick] plot[domain=0.3:9.2, samples=80] (\x, {0.6*sin(\x r)});
  % data points
  \foreach \x in {3.2,3.7,4.1,4.5,5.0,5.4,5.8}
    \fill[black] (\x, {0.6*sin(\x r)}) circle (1.7pt);
  \node[acc, font=\footnotesize, anchor=south] at (4.5,2.3) {mean (\texttt{curves} agree on data)};
  \node[black, font=\footnotesize, anchor=south] at (8.4,1.1) {scatter o\/f\/f-data};
\end{tikzpicture}
$$

MC dropout is nearly free (no extra parameters, no retraining, just $T$ forward
passes), which is why it is the default first choice for uncertainty. Its weakness
is that the dropout rate fixes the approximate posterior in advance, so the bands
can be poorly calibrated. The method that consistently does better is also the
simplest.

## Deep ensembles

Train $M$ networks independently (same architecture, different random
initializations and a different data-shuffling order) and average their
predictions.[^gf-bagging] Because the loss surface is riddled with many distinct minima, the
$M$ networks settle into _different_ functions that nonetheless all fit the
training data, and their disagreement off the data is a good estimate of
epistemic uncertainty.

> **Definition (Deep ensemble).** A collection of $M$ networks
> $\{f_{\theta^{(m)}}\}_{m=1}^M$ trained from independent random initializations.
> The ensemble prediction averages the members,
> $\bar f(x) = \tfrac{1}{M}\sum_m f_{\theta^{(m)}}(x)$, and their disagreement
> $\tfrac1M\sum_m (f_{\theta^{(m)}}(x) - \bar f(x))^2$ estimates epistemic
> uncertainty.

The ensemble is itself a crude posterior: each independently trained network is one
sample from the (multi-modal) distribution of good solutions, so averaging the $M$
members is a Monte-Carlo estimate of the same predictive integral, but with samples
drawn from genuinely _different_ modes rather than around a single one.

$$
p(y \mid x, \mathcal{D})
\;\approx\;
\frac{1}{M}\sum_{m=1}^{M} p\parens{y \mid x, \theta^{(m)}},
\qquad
\theta^{(m)} \text{ from independent training runs.}
$$

This is why ensembles often **beat** single Bayesian nets in practice: variational
methods and the Laplace approximation hug _one_ mode of the posterior, while the
ensemble's members land in different basins and so capture the genuine multi-modal
disagreement that epistemic uncertainty is made of. The cost is linear ($M$ full
training runs and $M$ forward passes), but the members are independent, so they
train and serve in parallel.

$$
% caption: Ensemble disagreement: members from different initializations agree where the data
% constrains them and diverge off-data, their spread being the epistemic uncertainty.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, thick] (-0.3,0) -- (9.6,0) node[right, font=\footnotesize] {input $x$};
  \draw[->, thick] (0,-2.4) -- (0,2.6) node[above, font=\footnotesize] {output};
  % data region
  \draw[black, fill=black!8] (3.2,-2.3) rectangle (5.8,2.5);
  \node[black, font=\footnotesize, anchor=north] at (4.5,2.5) {agree on data};
  % four member curves: distinct SOLID colors, identical near center, diverge at the ends
  \draw[acc, thick]    plot[domain=0.3:9.2, samples=70] (\x, {0.5*sin(\x r) + 0.09*(\x-4.5)});
  \draw[red!75, thick] plot[domain=0.3:9.2, samples=70] (\x, {0.5*sin(\x r) - 0.11*(\x-4.5)});
  \draw[green, thick]  plot[domain=0.3:9.2, samples=70] (\x, {0.5*sin(\x r) + 0.02*(\x-4.5)*(\x-4.5)*0.06});
  \draw[black, thick]  plot[domain=0.3:9.2, samples=70] (\x, {0.5*sin(\x r) - 0.02*(\x-4.5)*(\x-4.5)*0.06});
  % data
  \foreach \x in {3.4,3.8,4.2,4.6,5.0,5.4}
    \fill[black] (\x, {0.5*sin(\x r)}) circle (1.7pt);
  \node[black, font=\footnotesize, anchor=south] at (8.3,1.4) {\texttt{diverge} o\/f\/f-data};
  \node[black, font=\footnotesize, anchor=north] at (1.2,-1.2) {= \texttt{uncertainty}};
\end{tikzpicture}
$$

The four methods occupy a clean cost-quality frontier: pick by how much compute and
implementation effort the uncertainty is worth.

| Method | Extra cost | Posterior captured | Quality |
| --- | --- | --- | --- |
| Deep ensemble | $M\times$ training, $M\times$ inference | multi-modal (different basins) | best, but expensive |
| MC dropout | none to train, $T\times$ inference | single mode, fixed by dropout rate | cheap, often under-confident |
| Variational inference (Bayes-by-backprop) | $2\times$ parameters, harder to train | one mode, learned mean+variance | moderate; sensitive to priors |
| Laplace approximation | one Hessian/curvature pass | one mode, local Gaussian | cheap post-hoc; only local |

A single network's confidence may still be useful if it matches reality, and that
match has a precise, measurable definition.

## Calibration

A model is **calibrated** if its confidence matches its accuracy: of all the
predictions it makes with $80\%$ confidence, $80\%$ should be correct. Modern deep
networks are systematically **over-confident**: high accuracy, but confidence that
runs well ahead of it.[^chollet-cal] Calibration is checked, not assumed.

> **Definition (Calibration).** A classifier is perfectly calibrated if, for every
> confidence level $p$, the fraction of predictions made at confidence $p$ that are
> correct equals $p$: $\;\mathbb{P}\parens{\hat y = y \mid \hat p = p} = p$ for all
> $p \in [0,1]$.

The diagnostic is a **reliability diagram**: bin predictions by confidence, and for
each bin plot mean confidence against measured accuracy. Perfect calibration is the
diagonal; an over-confident model bows _below_ it (confidence exceeds accuracy).

$$
% caption: A reliability diagram: the diagonal is perfect calibration, and a curve bowing below it marks an over-confident model whose accuracy lags its reported confidence.
\begin{tikzpicture}[>=stealth, font=\small, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  % axes (unit square)
  \draw[->, thick] (0,0) -- (5.4,0) node[right, font=\footnotesize] {con\/f\/idence};
  \draw[->, thick] (0,0) -- (0,5.4) node[above, font=\footnotesize] {accuracy};
  \foreach \t in {1,2,3,4,5} {
    \draw[black] (\t,0) -- (\t,5);
    \draw[black] (0,\t) -- (5,\t);
  }
  \node[font=\footnotesize, anchor=north east] at (5,-0.05) {1};
  \node[font=\footnotesize, anchor=east] at (-0.05,5) {1};
  % perfect-calibration diagonal
  \draw[green, very thick, dashed] (0,0) -- (5,5);
  \node[font=\footnotesize, text=green, anchor=east] at (2.35,3.5) {perfect};
  % over-confident curve, below the diagonal
  \draw[red, very thick] plot[domain=0:5, samples=40] (\x, {\x - 0.16*\x*(5-\x)});
  \node[red, font=\footnotesize, anchor=north west] at (2.7,1.75) {over-con\/f\/ident};
  % gap arrow at x=3
  \draw[<->, black, thick] (3,3) -- (3,2.04);
  \node[black, font=\footnotesize, anchor=west] at (3.08,2.55) {gap};
\end{tikzpicture}
$$

The gap is summarized by one number. Partition the $n$ predictions into $B$
confidence bins $\{S_b\}$; the **Expected Calibration Error** is the average
bin-level gap between accuracy and confidence, weighted by bin population:

$$
\mathrm{ECE}
= \sum_{b=1}^{B} \frac{|S_b|}{n}\,
\abs{\acc(S_b) - \conf(S_b)},
$$

where $\acc(S_b)$ is the fraction correct in bin $b$ and
$\conf(S_b)$ is the mean confidence there. ECE $=0$ is perfect
calibration; a typical over-confident network sits at several percent.

For example, take $n = 100$ predictions sorted into three confidence bins. Bin
$b_1$ holds $20$ predictions at mean confidence $0.70$ of which $13$ are correct
(accuracy $0.65$); bin $b_2$ holds $30$ at mean confidence $0.85$ with $24$ correct
(accuracy $0.80$); bin $b_3$ holds $50$ at mean confidence $0.95$ with $43$ correct
(accuracy $0.86$). Each bin's gap is confidence minus accuracy, weighted by how many
predictions it holds:

| bin | $\abs{S_b}$ | conf | acc | gap $\abs{\text{acc} - \text{conf}}$ | weight $\tfrac{\abs{S_b}}{n}$ | contribution |
| --- | --- | --- | --- | --- | --- | --- |
| $b_1$ | $20$ | $0.70$ | $0.65$ | $0.05$ | $0.20$ | $0.010$ |
| $b_2$ | $30$ | $0.85$ | $0.80$ | $0.05$ | $0.30$ | $0.015$ |
| $b_3$ | $50$ | $0.95$ | $0.86$ | $0.09$ | $0.50$ | $0.045$ |

Summing the contributions, $\mathrm{ECE} = 0.010 + 0.015 + 0.045 = 0.070$: the model
is over-confident by $7$ percentage points on average, and the populous
high-confidence bin $b_3$ — where the gap is widest _and_ the weight is largest —
supplies most of it. This is the usual signature of a modern network: it is most
miscalibrated exactly where it is most sure, so the largest bin dominates the error.

### Temperature scaling

The cheapest fix is post-hoc: divide the pre-softmax logits $z$ by a single scalar
$T > 0$ before the softmax, and tune $T$ on a held-out set to minimize negative
log-likelihood. The argmax (and hence the accuracy) is unchanged, since dividing
all logits by the same $T$ never reorders them; only the _sharpness_ of the
distribution moves.

$$
\softmax(z/T)_k = \frac{\exp(z_k / T)}{\sum_j \exp(z_j / T)},
\qquad
\begin{cases}
T > 1 & \text{softer — tempers over-confidence,}\\
T = 1 & \text{unchanged,}\\
T < 1 & \text{sharper.}
\end{cases}
$$

> **Definition (Temperature scaling).** A single-parameter recalibration that
> replaces the logits $z$ with $z/T$ before the softmax, with $T$ fit on validation
> data. It is the simplest calibration map: it preserves accuracy exactly and, for
> an over-confident model, $T>1$ pulls confidence back down toward accuracy.

$$
% caption: Temperature on the same logits. $T=1$ (left) is spiky and over-confident; $T>1$ (right) softens the distribution toward the true accuracy while the tallest bar, and thus the prediction, is unchanged.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % --- left: T = 1, sharp ---
  \begin{scope}
    \draw[->, black] (0,0) -- (4.4,0) node[right, font=\scriptsize]{class};
    \draw[->, black] (0,0) -- (0,3.0) node[above, font=\scriptsize]{prob};
    \foreach \c/\h in {0.6/2.6, 1.5/0.35, 2.4/0.55, 3.3/0.2} {
      \draw[red, very thick, fill=red!20] (\c,0) rectangle (\c+0.55,\h);
    }
    \node[red, font=\scriptsize, anchor=north] at (2.1,-0.15) {$T=1$: spiky};
  \end{scope}
  % --- right: T > 1, softer ---
  \begin{scope}[xshift=6.0cm]
    \draw[->, black] (0,0) -- (4.4,0) node[right, font=\scriptsize]{class};
    \draw[->, black] (0,0) -- (0,3.0) node[above, font=\scriptsize]{prob};
    \foreach \c/\h in {0.6/1.5, 1.5/0.8, 2.4/1.05, 3.3/0.65} {
      \draw[acc, very thick, fill=acc!20] (\c,0) rectangle (\c+0.55,\h);
    }
    \node[acc, font=\scriptsize, anchor=north] at (2.1,-0.15) {$T>1$: softened};
  \end{scope}
  \draw[->, black, thick] (4.6,1.3) -- (5.8,1.3) node[midway, above, font=\scriptsize] {raise $T$};
\end{tikzpicture}
$$

```algorithm
caption: $\textsc{Calibrate}(z_{1:n}, y_{1:n})$ — fit temperature on a held-out set
$T \gets 1$
repeat
  $p_i \gets \softmax(z_i / T)$ for each validation example $i$
  $L \gets -\frac{1}{n}\sum_i \log p_i[y_i]$ // validation negative log-likelihood
  $T \gets T - \eta\,\dfrac{\partial L}{\partial T}$ // one scalar to tune
until $L$ converged
return $T$
```

> **Theorem (Accuracy invariance).** Temperature scaling with any $T > 0$ leaves
> every prediction's argmax (and therefore the classifier's accuracy) unchanged.

> **Proof.** The predicted class is $\arg\max_k \softmax(z/T)_k$.
> Softmax is strictly monotone in its logits, so $\arg\max_k \softmax(z/T)_k
> = \arg\max_k (z_k/T)$. Since $T > 0$, dividing every logit by $T$ preserves their
> order: $z_k/T > z_j/T \iff z_k > z_j$. Hence $\arg\max_k (z_k/T) = \arg\max_k z_k$,
> the original prediction, for every input. The set of correct predictions is
> unchanged, so accuracy is invariant. $\qed$

## The modern uncertainty toolbox

Goodfellow's §5.6 covers the Bayesian machinery but names none of the practical
methods here, all of which postdate the 2016 text; the public papers are the
citation of record. Three results built the modern default stack.

**MC dropout** (Gal & Ghahramani, 2016) supplied the theoretical bridge — the proof
that leaving dropout on at test time _is_ variational inference under a Bernoulli
posterior — which is what turned a training regularizer into an uncertainty
estimator. **Deep ensembles** (Lakshminarayanan et al., 2017) then showed
that simply training a handful of networks
from different seeds and averaging them produces better-calibrated uncertainty than
most single-model approximate posteriors, because the members land in genuinely
different loss basins. **Temperature scaling** (Guo et al., 2017) closed the loop on
calibration: the paper that documented modern networks are systematically
over-confident and that a single scalar, fit post-hoc, fixes most of it without
touching accuracy.

Later work has pushed the cost-quality frontier from both ends. **SWAG**
(Maddox et al., 2019) fits a Gaussian posterior cheaply by tracking the running mean
and covariance of the SGD iterates themselves — a Bayesian read-off of the training
trajectory rather than a separate inference run. The **Laplace approximation** has
been revived as a fast, scalable post-hoc method (Daxberger et al., 2021) that fits
the curvature around an already-trained model. And **evidential deep learning**
(Sensoy et al., 2018) trains a network to output the parameters of a distribution
_over_ the softmax probabilities, so a single forward pass reports both the
prediction and its epistemic uncertainty without any sampling at all. As in the
cost-quality table above, every method trades how much of the posterior it captures
against how much compute it spends, and deep
ensembles remain the quality baseline the cheaper methods are measured against.

## Takeaways

- A softmax confidence is **not** an uncertainty estimate — networks stay
  near-certain even [out-of-distribution](/deep-learning/theory/adversarial-robustness),
  which is the root of their [opacity](/deep-learning/foundations/what-is-deep-learning).
- Uncertainty splits into **aleatoric** (data noise, irreducible) and **epistemic**
  (model ignorance, vanishing with data); the total variance is their sum, and the
  split tells you whether more data would help.
- The Bayesian object is the **posterior over weights**
  $p(\theta\mid\mathcal{D}) \propto p(\mathcal{D}\mid\theta)p(\theta)$ and the
  **predictive integral** $p(y\mid x,\mathcal{D}) = \int p(y\mid x,\theta)
  p(\theta\mid\mathcal{D})\,\mathrm{d}\theta$, which is intractable and must be
  approximated.
- **MC [dropout](/deep-learning/regularization/dropout-and-data-augmentation)**
  leaves dropout on at test time; $T$ stochastic passes give a predictive mean and a
  variance that decomposes into aleatoric and epistemic parts — nearly free.
- **Deep ensembles** train $M$ nets from different inits; their disagreement
  estimates epistemic uncertainty and, by sampling distinct loss basins, they often
  outperform single Bayesian approximations.
- **Calibration** requires that confidence match accuracy; read it off a **reliability
  diagram**, summarize it with **ECE**, and fix over-confidence cheaply with
  **temperature scaling**, which rescales the logits without touching accuracy.

[^gf-bayes]: **Goodfellow**, _Deep Learning_, §5.6 — Bayesian Statistics: carrying the full posterior $p(\theta\mid\mathcal{D})\propto p(\mathcal{D}\mid\theta)p(\theta)$ rather than collapsing to a point estimate.
[^gf-mle]: **Goodfellow**, _Deep Learning_, §5.5 — Maximum Likelihood Estimation: the point estimate the Bayesian predictive integral generalizes, and why that integral is intractable for deep nets.
[^gf-dropout]: **Goodfellow**, _Deep Learning_, §7.12 — Dropout: random unit-masking as an implicit ensemble, the mechanism MC dropout repurposes at test time.
[^gf-bagging]: **Goodfellow**, _Deep Learning_, §7.11 — Bagging and Other Ensemble Methods: averaging independently trained models to reduce variance — the deep-ensemble template.
[^chollet-cal]: **Chollet**, _Deep Learning with Python_, Ch. 4 — softmax outputs are not calibrated probabilities; high-confidence errors are routine and must be measured, not assumed away.
[^postdate-mc]: MC dropout as variational inference (Gal & Ghahramani, 2016), deep ensembles for predictive uncertainty (Lakshminarayanan et al., 2017), and temperature-scaling calibration (Guo et al., 2017) all postdate **Goodfellow** (2016), which covers the underlying dropout (§7.12) and Bayesian (§5.6) machinery but not these uncertainty methods by name.