---
title: Loss Functions & Output Units
module: Neural Networks
moduleNumber: 2
lessonNumber: 5
order: 205
summary: >
  The last layer is where a network's hidden representation meets the task. Choosing
  an output unit and a loss is not two independent choices; maximum likelihood
  fixes the pair. We derive the standard couplings (linear/MSE, sigmoid/BCE,
  softmax/cross-entropy), show why softmax and cross-entropy were built to cancel into
  the residual $\hat y - y$, and prove why squared error is the wrong loss for a
  saturating classifier.
topics: [Neural Networks]
sources:
  - book: Goodfellow
    ref: "§6.2 — Gradient-Based Learning: Cost Functions and Output Units"
  - book: Goodfellow
    ref: "§6.2.2 — Output Units; §3.13 Information Theory"
  - book: Chollet
    ref: "§4.5 — The Universal Workflow; Ch. 3 — last-layer activation and loss"
---

A network's hidden layers learn a representation $h = f_\theta(x)$; the **output
unit** is the thin final transform that turns $h$ into a prediction, and the
**loss** scores that prediction against the target. These two are not chosen
independently. Fix the conditional distribution $p_\theta(y \mid x)$ you intend the
network to model, and maximum likelihood determines the output activation _and_ the
loss as a matched pair. The whole lesson is one table and the derivations behind
its five rows.

| task | output unit | output activation | loss | $p_\theta(y\mid x)$ |
| --- | --- | --- | --- | --- |
| regression | 1 linear | identity | mean squared error | Gaussian |
| binary | 1 logit | sigmoid | binary cross-entropy | Bernoulli |
| multiclass | $K$ logits | softmax | categorical cross-entropy | Categorical |
| multi-label | $K$ logits | sigmoid (per class) | sum of $K$ binary cross-entropies | $K$ independent Bernoullis |
| count | 1 logit | softplus | Poisson negative log-likelihood | Poisson |

The pattern is rigid: pick the distribution that matches the target's _type_ —
real, binary, one-of-$K$, several-of-$K$, a nonnegative integer — let its natural
parameter be the network's pre-activation, and the loss is the negative
log-likelihood of that distribution. The rest of the lesson derives each
row.[^chollet-lastlayer]

## Maximum likelihood is the loss

Every row above is one instance of a single principle. The network outputs the
parameters of a conditional distribution $p_\theta(y \mid x)$, and training
maximizes the likelihood of the observed targets, equivalently minimizes the
**negative log-likelihood** (NLL).

> **Definition (Negative log-likelihood loss).** Let the network map $x$ to the
> parameters of a distribution $p_\theta(y \mid x)$. The per-example loss is
> $\ell = -\log p_\theta(y \mid x)$, and the training objective is the empirical
> risk $\mathcal{L}(\theta) = -\tfrac1n\sum_{i=1}^n \log p_\theta(y_i \mid x_i)$.

This single choice removes the guesswork from loss design: the loss is read off
as $-\log p_\theta(y\mid x)$ for whichever $p_\theta$
suits the target. Two consequences recur throughout.[^gf-cost]

- The output activation is whatever **maps an unbounded score into a valid
  parameter** of $p_\theta$ — a probability needs $(0,1)$, so a sigmoid; a rate
  needs $(0,\infty)$, so a softplus.
- The $\log$ in the NLL **undoes the $\exp$** in every exponential-family output
  unit, and that cancellation is why the gradients below collapse so cleanly. A loss
  without that $\log$ — squared error on a sigmoid — leaves the $\exp$ intact and
  the gradient saturates.

> **Theorem (MLE = cross-entropy minimization).** Minimizing the NLL is equivalent
> to minimizing the cross-entropy $H(p_{\text{data}}, p_\theta)$ between the
> empirical data distribution and the model, and hence to minimizing
> $\mathrm{KL}(p_{\text{data}} \,\|\, p_\theta)$ up to a constant in $\theta$.

> **Proof.** $H(p_{\text{data}}, p_\theta) = -\mathbb{E}_{p_{\text{data}}}[\log
> p_\theta] = \tfrac1n\sum_i -\log p_\theta(y_i\mid x_i)$, which is exactly
> $\mathcal{L}(\theta)$. And $\mathrm{KL}(p_{\text{data}}\,\|\,p_\theta) =
> H(p_{\text{data}}, p_\theta) - H(p_{\text{data}})$, whose second term does not
> depend on $\theta$. So the three objectives share a minimizer. $\qed$

The second consequence is worth making precise, because it is the mechanism behind
every clean gradient in this lesson. An exponential-family output writes the model
probability as $p_\theta = e^{a(z)} / Z(z)$ for some score $z$ and normalizer $Z$.
Taking the negative log turns the ratio into a difference,

$$
-\log p_\theta = -a(z) + \log Z(z),
$$

so the $\exp$ that built the probability is gone and the loss is _linear_ in the
score plus a smooth log-normalizer. Differentiating a linear term gives a constant,
and differentiating $\log Z$ returns the model's own predicted probabilities. The
two pieces combine into $\hat y - y$ with no leftover $\exp$ and no activation
derivative multiplying the signal. Drop the $\log$ — square the residual of a
sigmoid instead — and the $\exp$ survives inside the gradient as the factor
$\hat y(1-\hat y)$, which vanishes exactly where the model is most wrong. The last
section of this lesson shows that failure numerically.

## Regression: linear output, squared error

For a real-valued target take $p_\theta(y\mid x) = \mathcal{N}\!\parens{y;\,
\hat y, \sigma^2}$ with the mean predicted by an **identity** output,
$\hat y = w^\top h + b$, and $\sigma^2$ fixed. The NLL is

$$
\ell = -\log \mathcal{N}(y;\hat y,\sigma^2)
= \frac{1}{2\sigma^2}\,(\hat y - y)^2 + \tfrac12\log(2\pi\sigma^2).
$$

Drop the additive constant and the scale, and the loss is the **mean squared
error** $\tfrac12(\hat y - y)^2$. Squared error is not an arbitrary penalty but
the log-likelihood of a Gaussian with fixed variance. Its gradient with respect to
the score is the residual, with no saturating factor:

$$
\frac{\partial \ell}{\partial \hat y} = \hat y - y.
$$

Because the output is the identity, this residual passes straight to the weights as
$(\hat y - y)\,h$. The lesson's recurring theme — _gradient equals
prediction minus target_ — starts here, in its simplest form.

## Softmax: the multiclass output unit

For a one-of-$K$ label, the network emits a vector of $K$ scores ("logits")
$z \in \mathbb{R}^K$, and the **softmax** turns them into a categorical
distribution over the classes.

> **Definition (Softmax).** The softmax map
> $\softmax : \mathbb{R}^K \to \Delta^{K-1}$ sends a logit vector $z$
> to the probability simplex by
> $\softmax(z)_i = e^{z_i} \big/ \sum_{j=1}^{K} e^{z_j}$, so each
> $\hat y_i \in (0,1)$ and $\sum_i \hat y_i = 1$.

The figure traces the three stages (score, exponentiate, normalize) that carry a
logit vector to a probability distribution.

$$
% caption: The softmax output unit. A logit vector $z$ is exponentiated elementwise,
% then normalized by the sum, yielding a probability vector $\hat y$ on the simplex.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  cell/.style={draw, minimum width=8mm, minimum height=8mm, inner sep=1pt},
  op/.style={draw, minimum width=18mm, minimum height=9mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  % logits column
  \node[font=\footnotesize] at (0,2.0) {logits $z$};
  \node[cell] (z1) at (0,1.2) {$2$};
  \node[cell] (z2) at (0,0.4) {$1$};
  \node[cell] (z3) at (0,-0.4) {$0$};
  % exponentiate
  \node[op] (exp) at (3.0,0.4) {exponentiate\\$e^{z_i}$};
  \draw[->, acc, thick] (z1.east) -- (exp.west);
  \draw[->, acc, thick] (z2.east) -- (exp.west);
  \draw[->, acc, thick] (z3.east) -- (exp.west);
  % normalize (no \sum in node text; formula lives in the caption)
  \node[op] (nrm) at (6.4,0.4) {normalize\\to sum $= 1$};
  \draw[->, acc, thick] (exp.east) -- (nrm.west);
  % probability bars
  \node[font=\footnotesize] at (10.5,2.0) {output $\hat y$};
  \draw[acc, very thick, fill=acc!15] (9.5,0.0) rectangle ++(0.55,1.30); \node[font=\scriptsize] at (9.775,-0.30) {high};
  \draw[acc, very thick, fill=acc!15] (10.3,0.0) rectangle ++(0.55,0.48); \node[font=\scriptsize] at (10.575,-0.30) {mid};
  \draw[acc, very thick, fill=acc!15] (11.1,0.0) rectangle ++(0.55,0.20); \node[font=\scriptsize] at (11.375,-0.30) {low};
  \draw[black] (9.35,0.0) -- (11.8,0.0);
  \draw[->, acc, thick] (nrm.east) -- (9.2,0.4);
\end{tikzpicture}
$$

Softmax inherits two structural facts. It is **shift-invariant**: adding a constant
$c$ to every logit multiplies numerator and denominator by $e^{c}$ and leaves
$\hat y$ unchanged. And the raw form _overflows_: $e^{z_i}$ explodes for large
logits. Both are fixed at once by subtracting the maximum logit before
exponentiating, which is the form every library actually computes.

$$
\softmax(z)_i
= \frac{e^{\,z_i - \max_k z_k}}{\sum_{j=1}^{K} e^{\,z_j - \max_k z_k}}.
$$

> **Remark (Numerical stability).** The shift $z \mapsto z - \max_k z_k$ makes the
> largest exponent $0$, so every $e^{(\cdot)} \in (0, 1]$ — no overflow, and the
> denominator is at least $1$. Shift-invariance guarantees the result is
> identical to the naive formula in exact arithmetic.

A **temperature** $T$ rescales the logits before the softmax, $\operatorname{
softmax}(z/T)$, interpolating between a one-hot argmax ($T \to 0$) and the uniform
distribution ($T \to \infty$). The bar chart shows the same logits sharpened and
flattened.

$$
% caption: Temperature reshapes a softmax distribution. Low $T$ sharpens toward the
% argmax; high $T$ flattens toward uniform; the logits are unchanged.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  % --- low T (sharp) ---
  \node[font=\footnotesize] at (1.1,2.55) {\texttt{low T (sharp)}};
  \draw[black] (0,0) -- (2.2,0);
  \draw[acc, very thick, fill=acc!15] (0.05,0) rectangle ++(0.45,2.10);
  \draw[acc, very thick, fill=acc!15] (0.60,0) rectangle ++(0.45,0.55);
  \draw[acc, very thick, fill=acc!15] (1.15,0) rectangle ++(0.45,0.22);
  \draw[acc, very thick, fill=acc!15] (1.70,0) rectangle ++(0.45,0.10);
  % --- medium T ---
  \node[font=\footnotesize] at (4.4,2.55) {\texttt{medium T}};
  \draw[black] (3.3,0) -- (5.5,0);
  \draw[acc, very thick, fill=acc!15] (3.35,0) rectangle ++(0.45,1.30);
  \draw[acc, very thick, fill=acc!15] (3.90,0) rectangle ++(0.45,0.85);
  \draw[acc, very thick, fill=acc!15] (4.45,0) rectangle ++(0.45,0.55);
  \draw[acc, very thick, fill=acc!15] (5.00,0) rectangle ++(0.45,0.35);
  % --- high T (flat) ---
  \node[font=\footnotesize] at (7.7,2.55) {\texttt{high T (flat)}};
  \draw[black] (6.6,0) -- (8.8,0);
  \draw[acc, very thick, fill=acc!15] (6.65,0) rectangle ++(0.45,0.78);
  \draw[acc, very thick, fill=acc!15] (7.20,0) rectangle ++(0.45,0.68);
  \draw[acc, very thick, fill=acc!15] (7.75,0) rectangle ++(0.45,0.60);
  \draw[acc, very thick, fill=acc!15] (8.30,0) rectangle ++(0.45,0.52);
\end{tikzpicture}
$$

## The softmax + cross-entropy gradient

The matched loss for a softmax output is the **categorical cross-entropy**. With a
one-hot target $y$ (so $y_c = 1$ for the true class $c$ and $0$ otherwise),

$$
\ell = -\sum_{i=1}^{K} y_i \log \hat y_i = -\log \hat y_c,
\qquad
\hat y_i = \frac{e^{z_i}}{\sum_j e^{z_j}}.
$$

The result that makes this pairing canonical is that the gradient with respect to
the logits collapses to the residual $\hat y - y$. Derive it by differentiating
$\ell = -\log\hat y_c$ through the softmax. First the log-sum-exp:

$$
\frac{\partial}{\partial z_k}\parens{-\log \hat y_c}
= \frac{\partial}{\partial z_k}\parens{-z_c + \log\textstyle\sum_j e^{z_j}}
= -\,[\,k=c\,] + \frac{e^{z_k}}{\sum_j e^{z_j}}.
$$

The first term is the indicator $[k=c]$, which is precisely $y_k$ for a one-hot
target; the second term is $\hat y_k$ by the softmax definition. Therefore, for
every coordinate,

$$
\frac{\partial \ell}{\partial z_k} = \hat y_k - y_k
\qquad\Longrightarrow\qquad
\frac{\partial \ell}{\partial z} = \hat y - y.
$$

> **Theorem (Softmax cross-entropy gradient).** For categorical cross-entropy with
> a softmax output and one-hot target, the gradient with respect to the logits is
> $\partial\ell/\partial z = \hat y - y$. The vanishing of the softmax Jacobian and
> the $1/\hat y_c$ from the log cancel exactly, leaving no factor that shrinks as
> the model grows confident.

> **Proof.** The Jacobian of the softmax is $\partial \hat y_i/\partial z_k =
> \hat y_i([\,i=k\,] - \hat y_k)$. The log-loss derivative is
> $\partial\ell/\partial\hat y_i = -y_i/\hat y_i$. Chain them:
> $\partial\ell/\partial z_k = \sum_i (-y_i/\hat y_i)\,\hat y_i([\,i=k\,] -
> \hat y_k) = -\sum_i y_i([\,i=k\,] - \hat y_k) = -y_k + \hat y_k \sum_i y_i$. Since
> $\sum_i y_i = 1$, this is $\hat y_k - y_k$. $\qed$

This is the central fact of classification training. The error signal passed
back to the network is just _predicted distribution minus true distribution_, with
no activation derivative attenuating it.[^gf-output]

### A worked pass through the numbers

For example, take $K = 3$ classes with
true class $c = 1$ (so the one-hot target is $y = (1, 0, 0)$), and logits
$z = (2, 1, 0) \in \mathbb{R}^3$. The stable softmax subtracts $\max_k z_k = 2$:

$$
e^{z - 2} = (e^{0},\, e^{-1},\, e^{-2}) = (1.000,\, 0.368,\, 0.135),
\qquad
\textstyle\sum = 1.503.
$$

Dividing gives the predicted distribution
$\hat y = (0.665,\, 0.245,\, 0.090)$, which sums to $1$ and is identical to what
the naive formula returns. The per-example loss is
$\ell = -\log \hat y_1 = -\log 0.665 = 0.408$ nats. The gradient with respect to
the logits is one subtraction, $\hat y - y$:

$$
\frac{\partial \ell}{\partial z}
= \hat y - y
= (0.665, 0.245, 0.090) - (1, 0, 0)
= (-0.335,\, 0.245,\, 0.090).
$$

The true-class logit gets a negative
gradient, so gradient descent _raises_ $z_1$; the two wrong-class logits get
positive gradients, so descent _lowers_ them. The magnitudes sum to zero
($-0.335 + 0.245 + 0.090 = 0$), a direct consequence of $\sum_i(\hat y_i - y_i) =
\sum_i \hat y_i - \sum_i y_i = 1 - 1 = 0$: softmax cross-entropy only ever
_redistributes_ probability mass, never creates or destroys it. In a minibatch of
$B$ examples the logits are a matrix $Z \in \mathbb{R}^{B \times K}$, the softmax
acts row-wise, and the gradient $\hat Y - Y \in \mathbb{R}^{B\times K}$ has the same
shape as the logits, ready to seed the backward pass.

$$
% caption: The softmax cross-entropy gradient on a 3-class example. Predicted
% $\hat y = (0.67, 0.24, 0.09)$ against one-hot target for class 1; the gradient
% $\hat y - y$ pulls the true logit up and the wrong logits down, summing to zero.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % baseline
  \draw[black] (-0.4,0) -- (7.4,0);
  \node[font=\footnotesize, anchor=east] at (-0.4,0.35) {\texttt{grad}};
  % three bars: g = (-0.335, +0.245, +0.090), scale *4
  % class 1: negative (raise logit) -> acc bar downward
  \draw[acc, very thick, fill=acc!15] (0.6,0) rectangle ++(1.2,-1.34);
  \node[font=\footnotesize, anchor=north] at (1.2,-1.5) {\texttt{class 1}};
  \node[acc, font=\footnotesize, anchor=north east] at (1.75,-0.05) {\texttt{-0.34}};
  % class 2: positive (lower logit) -> red bar upward
  \draw[red, very thick, fill=red!12] (2.9,0) rectangle ++(1.2,0.98);
  \node[font=\footnotesize, anchor=south] at (3.5,1.05) {\texttt{class 2}};
  \node[red, font=\footnotesize, anchor=south west] at (2.95,0.05) {\texttt{+0.24}};
  % class 3: positive smaller
  \draw[red, very thick, fill=red!12] (5.2,0) rectangle ++(1.2,0.36);
  \node[font=\footnotesize, anchor=south] at (5.8,1.05) {\texttt{class 3}};
  \node[red, font=\footnotesize, anchor=south west] at (5.25,0.05) {\texttt{+0.09}};
\end{tikzpicture}
$$

## Binary: sigmoid + binary cross-entropy

The two-class case is softmax with $K = 2$ collapsed to a single logit. Model
$p_\theta(y=1\mid x) = \hat y = \sigma(z)$ with the **logistic sigmoid**
$\sigma(z) = 1/(1+e^{-z})$, take $y\in\{0,1\}$, and the Bernoulli NLL is the
**binary cross-entropy**

$$
\ell = -\brackets{\,y\log\hat y + (1-y)\log(1-\hat y)\,}.
$$

Using $\sigma'(z) = \sigma(z)(1-\sigma(z)) = \hat y(1-\hat y)$, the awkward
$1/[\hat y(1-\hat y)]$ from the log derivative is cancelled by the sigmoid's own
derivative, leaving the same residual:

$$
\frac{\partial \ell}{\partial z}
= \underbrace{\frac{\hat y - y}{\hat y(1-\hat y)}}_{\partial\ell/\partial\hat y}
\cdot \underbrace{\hat y(1-\hat y)}_{\partial\hat y/\partial z}
= \hat y - y.
$$

This is the binary specialization of the softmax result, and it is why both output
units share one backward implementation: combine the activation and the loss into
a single op whose gradient is $\hat y - y$, never materializing the unstable
intermediate $1/[\hat y(1-\hat y)]$.

## Why squared error is wrong for a classifier

A tempting mistake is to keep MSE but put it on a sigmoid output,
$\ell_{\text{MSE}} = \tfrac12(\hat y - y)^2$ with $\hat y = \sigma(z)$. The
problem is the gradient with respect to the score. By the chain rule it picks up
the sigmoid's derivative as a multiplicative factor:

$$
\frac{\partial \ell_{\text{MSE}}}{\partial z}
= (\hat y - y)\,\sigma'(z)
= (\hat y - y)\,\hat y(1-\hat y),
\qquad\text{versus}\qquad
\frac{\partial \ell_{\text{BCE}}}{\partial z} = \hat y - y.
$$

The factor $\hat y(1-\hat y)$ causes saturation. When the model is
**confidently wrong** — say $y=1$ but $z \ll 0$, so $\hat y \approx 0$ — the
residual $\hat y - y \approx -1$ is large, yet $\hat y(1-\hat y)\approx 0$ drives
the MSE gradient to nearly zero. The example that most needs correcting produces
almost no learning signal. Cross-entropy has no such factor, so its gradient stays
order $1$ exactly where it matters. The table below quantifies the failure.[^gf-saturate]

| state | $z$ | $\hat y$ | $y$ | $\partial\ell_{\text{BCE}}/\partial z$ | $\partial\ell_{\text{MSE}}/\partial z$ |
| --- | --- | --- | --- | --- | --- |
| confidently wrong | $-5$ | $0.007$ | $1$ | $-0.993$ | $-0.0066$ |
| uncertain | $0$ | $0.500$ | $1$ | $-0.500$ | $-0.125$ |
| confidently right | $+5$ | $0.993$ | $1$ | $-0.007$ | $-0.00005$ |

The first row is the key case: a $150\times$ weaker gradient on the worst
prediction. Two figures plot the comparison: first the per-example loss as a function
of the predicted probability of the true class, then the gradient magnitude across
the score.

$$
% caption: Loss versus predicted probability of the true class. Cross-entropy
% $-\log p$ diverges as $p\to0$ (strong pull when wrong); squared error $(1-p)^2$ stays
% bounded and flat.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, thick] (0,0) -- (5.3,0) node[right, font=\footnotesize] {\texttt{predicted prob.}};
  \draw[->, thick] (0,0) -- (0,3.6) node[above, font=\footnotesize] {loss};
  \node[font=\scriptsize, anchor=north] at (0,0) {$0$};
  \node[font=\scriptsize, anchor=north] at (5.0,0) {$1$};
  % cross-entropy: -log p, scaled x in [0.05,1]->[0,5], y clipped at 3.4
  \draw[acc, very thick] plot[domain=0.032:1.0, samples=80] (\x*5, {-ln(\x)});
  \node[acc, font=\footnotesize, anchor=west] at (1.1,2.95) {\texttt{cross-entropy}};
  % squared error: (1-p)^2, max 1 at p=0
  \draw[red, very thick] plot[domain=0.0:1.0, samples=60] (\x*5, {(1-\x)*(1-\x)});
  \node[red, font=\footnotesize, anchor=south west] at (2.55,1.35) {\texttt{squared error}};
\end{tikzpicture}
$$

$$
% caption: Gradient magnitude $\abs{\partial\ell/\partial z}$ across the score for a
% true label $y=1$. Binary cross-entropy stays near $1$ when wrong; MSE-on-sigmoid
% saturates to $0$ on both ends.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, thick] (-3.4,0) -- (3.6,0) node[right, font=\footnotesize] {score};
  \draw[->, thick] (0,0) -- (0,2.6) node[above, font=\footnotesize] {grad. mag.};
  \draw[black, dashed] (-3.4,2.0) -- (3.2,2.0) node[black, right, font=\scriptsize] {$1$};
  % BCE gradient magnitude: abs(sigmoid(z)-1) = 1-sigmoid(z), scaled to height 2
  \draw[acc, very thick] plot[domain=-3.3:3.3, samples=80] (\x, {2*(1 - 1/(1+exp(-\x)))});
  \node[acc, font=\footnotesize, anchor=west] at (-3.25,1.55) {BCE};
  % MSE-on-sigmoid: (1-s)*s*(1-s) = s(1-s)^2, scaled to height 2 (peak ~0.148 -> *13.5)
  \draw[red, very thick] plot[domain=-3.3:3.3, samples=90]
    (\x, {2*( (1/(1+exp(-\x)))*(1-1/(1+exp(-\x)))*(1-1/(1+exp(-\x))) )/0.148 });
  \node[red, font=\footnotesize, anchor=east] at (3.5,0.72) {MSE (saturates)};
\end{tikzpicture}
$$

The BCE curve approaches its maximum gradient exactly in the region where the
model is most wrong ($z \to -\infty$); the MSE curve collapses to zero at both
extremes, learning slowly precisely when it should learn fastest.

## The output-unit map

Every row of the master table is the same construction applied to a different
target type. The branching diagram fixes the mapping from _what kind of thing $y$ is_ to
the output unit and loss that maximum likelihood forces.

$$
% caption: The output-unit map. The target's type selects the distribution
% $p_\theta(y\mid x)$, which fixes the output activation and the NLL loss.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  root/.style={draw, thick, minimum width=24mm, minimum height=9mm, align=center},
  leaf/.style={draw, minimum width=30mm, minimum height=15mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[root, draw=acc, text=acc] (r) at (0,0) {\texttt{target type}};
  \node[leaf] (reg) at (-6.2,-2.8) {\texttt{real}\\\texttt{identity + MSE}};
  \node[leaf] (bin) at (-3.1,-2.8) {\texttt{binary}\\\texttt{sigmoid + BCE}};
  \node[leaf] (mul) at (0,-2.8)    {\texttt{one-of-K}\\\texttt{softmax + CE}};
  \node[leaf] (lab) at (3.1,-2.8)  {\texttt{several-of-K}\\\texttt{sigmoids + sum BCE}};
  \node[leaf] (cnt) at (6.2,-2.8)  {\texttt{count}\\\texttt{softplus + Poisson}};
  \draw[->, acc, thick] (r) -- (reg);
  \draw[->, acc, thick] (r) -- (bin);
  \draw[->, acc, thick] (r) -- (mul);
  \draw[->, acc, thick] (r) -- (lab);
  \draw[->, acc, thick] (r) -- (cnt);
\end{tikzpicture}
$$

Two rows of the table extend the binary and count cases.

**Multi-label.** When an example may carry several labels at once (an image tagged
both "beach" and "sunset"), the classes are not mutually exclusive, so softmax is
wrong: it forces $\sum_i \hat y_i = 1$. Instead apply an **independent sigmoid per
class** and sum $K$ binary cross-entropies,

$$
\ell = \sum_{i=1}^{K} -\brackets{\,y_i\log\hat y_i + (1-y_i)\log(1-\hat y_i)\,},
\qquad \hat y_i = \sigma(z_i),
$$

which is the NLL of $K$ independent Bernoullis. Each logit's gradient is its own
residual $\hat y_i - y_i$.

**Count.** For a nonnegative integer target (events per interval) model
$p_\theta(y\mid x) = \text{Poisson}(\lambda)$ with rate $\lambda = \operatorname{
softplus}(z) = \log(1+e^{z}) > 0$. The NLL drops the data-only $\log y!$ term:

$$
\ell = -\log\frac{\lambda^{y} e^{-\lambda}}{y!}
= \lambda - y\log\lambda + \log y!
\;\;\propto\;\; \softplus(z) - y\log\softplus(z).
$$

Softplus keeps the rate strictly positive while staying smooth, where a bare
$e^{z}$ would overflow and a ReLU would kill the gradient at $z<0$.

## When the network predicts its own uncertainty

The regression row fixed $\sigma^2$ and threw it away. Nothing forces that. If the
target's noise varies across inputs (quiet in one region of $x$, noisy in another),
the network can predict the variance too. This is a **heteroscedastic** Gaussian
output: two heads, $\hat y(x) = w_\mu^\top h + b_\mu$ for the mean and
$\sigma^2(x) = \softplus(w_\sigma^\top h + b_\sigma) > 0$ for the
variance. The NLL keeps both terms this time:

$$
\ell = \frac{(\hat y - y)^2}{2\sigma^2(x)} + \tfrac12\log \sigma^2(x) + \text{const}.
$$

The first term is a squared error _weighted by the network's own confidence_:
where the model predicts small $\sigma^2$ it is penalized heavily for being wrong,
and where it predicts large $\sigma^2$ the residual is discounted. The second term,
$\tfrac12\log\sigma^2$, penalizes that discount, and it blocks the trivial
solution of setting $\sigma^2 \to \infty$ to zero out the first term. The two
terms balance at the true noise level, so the network learns a calibrated variance
alongside the mean.

A **mixture-density network** (MDN) goes further when the target is genuinely
multimodal, one $x$ that maps to several plausible $y$ where any single Gaussian
would average them into an implausible midpoint. The output parameterizes a mixture
of $M$ Gaussians, emitting mixing weights $\pi_m(x)$ through a softmax over $M$
components, plus a mean $\mu_m(x)$ and variance $\sigma_m^2(x)$ per component:

$$
p_\theta(y\mid x) = \sum_{m=1}^{M} \pi_m(x)\,\mathcal{N}\!\parens{y;\, \mu_m(x),\,
\sigma_m^2(x)},
\qquad
\ell = -\log \sum_{m=1}^{M} \pi_m\,\mathcal{N}(y;\mu_m,\sigma_m^2).
$$

The loss is still a negative log-likelihood, but now of a mixture, so the log sits
outside the sum and does not collapse into a clean residual. The principle is
unchanged though: pick the conditional distribution that matches the target, and
the loss is $-\log p_\theta(y\mid x)$.[^gf-output]

## Label smoothing and class imbalance

Two practical edits to the target keep cross-entropy well-behaved. Both modify $y$,
not the loss.

**Label smoothing.** A hard one-hot target requires $\hat y_c = 1$,
which is reachable only as $z_c \to \infty$ — the logits grow without bound and the
model becomes overconfident. **Label smoothing** replaces the one-hot target with a
softened version that mixes in the uniform distribution:

$$
y_i^{\text{LS}} = (1-\varepsilon)\,[\,i = c\,] + \frac{\varepsilon}{K},
$$

for a small $\varepsilon$ (commonly $0.1$). The cross-entropy minimum now sits at
finite logits, regularizing confidence and improving calibration.[^gf-info]

> **Definition (Label smoothing).** Replacing a one-hot target $y$ with
> $y^{\text{LS}} = (1-\varepsilon)\,y + \varepsilon\,u$, where $u$ is uniform over
> the $K$ classes. The cross-entropy target becomes a soft distribution, bounding
> the optimal logit gap and discouraging overconfident predictions.

**Class imbalance.** When one class dominates the data, the majority dominates the
averaged loss and the minority's gradient is negligible. The standard fix
**reweights** each class's contribution by a factor $\alpha_y$ inversely related to
its frequency:

$$
\mathcal{L} = -\frac1n\sum_{i=1}^n \alpha_{y_i}\,\log p_\theta(y_i\mid x_i),
\qquad
\alpha_k \propto \frac{1}{n_k},
$$

with $n_k$ the count of class $k$. This restores per-class influence without
touching the architecture; the gradient is simply scaled per example by $\alpha_y$.

## Losses beyond maximum likelihood

Maximum likelihood is the base construction, but three widely used losses extend
it in ways Goodfellow predates or treats only in passing.

**Focal loss down-weights the easy examples.** Class reweighting scales by label
frequency, but in extreme imbalance (dense object detection, where background
"boxes" outnumber objects thousands to one) even the correctly classified easy
negatives dominate the summed gradient. Lin et al.'s _focal loss_ (2017) multiplies
the cross-entropy of each example by $(1 - \hat y_c)^\gamma$, so a confident correct
prediction ($\hat y_c \to 1$) contributes almost nothing and the loss concentrates
on the hard, still-misclassified cases. It is cross-entropy with a per-example
confidence gate, and it is the modern default wherever imbalance is severe.

**Calibration is a separate axis from accuracy.** The clean $\hat y - y$ gradient
trains a classifier to be _accurate_, but a network can be accurate and still badly
_calibrated_ — its softmax probabilities systematically over- or under-state the
true likelihood. Guo et al. (2017) documented that modern deep nets are overconfident
and that a one-parameter fix, **temperature scaling** — dividing the logits by a
scalar $T$ tuned on held-out data, exactly the temperature of the softmax section —
recovers calibration without touching accuracy. Label smoothing above is the
training-time cousin of the same concern. The lesson from the literature is that the
loss you minimize and the probabilities you can trust are related but not identical.

**Contrastive losses are structurally cross-entropy.** Self-supervised learning
trains without labels by a loss that is structurally the softmax cross-entropy of
this lesson. The **InfoNCE** objective (van den Oord et al., 2018; SimCLR, Chen et
al., 2020) treats one positive pair against $N$ negatives as an $(N{+}1)$-way
classification problem: the logits are similarities, the target is "the positive is
class $0$," and the loss is $-\log$ of the softmax over similarities. The residual
gradient $\hat y - y$ derived here drives it too — the same output unit doing a
different job.[^beyond-loss]

## The shared backward op

Across four of the five rows the backward pass is identical: the gradient at the
score is the residual $\hat y - y$. That is the practical payoff of letting
maximum likelihood choose the pairing: the forward unit and the loss are designed
together so their derivatives cancel into one subtraction.

```algorithm
caption: $\textsc{OutputBackward}(z, y)$ — fused activation + NLL gradient
if task is regression then
  $\hat y \gets z$ // identity output
else if task is binary or multi-label then
  $\hat y \gets \text{sigmoid}(z)$ // per-logit Bernoulli
else if task is multiclass then
  $\hat y \gets \text{softmax}(z - \max z)$ // stable softmax
$g \gets \hat y - y$ // one residual, no activation-derivative factor
return $g$ // gradient w.r.t. the logits, ready for backprop
```

The lesson reduces to one instruction: choose the output distribution that matches
the target's type, and the activation, the loss, and this clean residual gradient
all follow. The matched pair then plugs straight into
[backpropagation](/deep-learning/neural-networks/backpropagation) as the seed of
the backward pass, and into the
[optimizer](/deep-learning/optimization/gradient-descent-and-sgd) as the gradient
it descends.

[^gf-cost]: **Goodfellow**, _Deep Learning_, §6.2.1 — Learning Conditional Distributions with Maximum Likelihood: the loss is the negative log-likelihood $-\log p_\theta(y\mid x)$, not an arbitrary penalty.
[^gf-output]: **Goodfellow**, _Deep Learning_, §6.2.2 — Output Units: linear/Gaussian, sigmoid/Bernoulli, and softmax/Categorical units, and why each pairs with the matching NLL so the gradient collapses to $\hat y - y$.
[^gf-saturate]: **Goodfellow**, _Deep Learning_, §6.2.2.2 / §6.2.1 — why mean squared error on a sigmoid saturates: the $\hat y(1-\hat y)$ factor crushes the gradient exactly when the model is confidently wrong, which the $\log$ of cross-entropy removes.
[^gf-info]: **Goodfellow**, _Deep Learning_, §3.13 — Information Theory: cross-entropy, KL divergence, and entropy — the substrate in which NLL minimization equals minimizing $\mathrm{KL}(p_{\text{data}} \,\|\, p_\theta)$, and label smoothing softens the target distribution.
[^chollet-lastlayer]: **Chollet**, _Deep Learning with Python_, Ch. 4–6 — the last-layer-activation/loss cheat sheet: sigmoid+`binary_crossentropy`, softmax+`categorical_crossentropy`, linear+`mse`, the practical face of the maximum-likelihood pairing.
[^beyond-loss]: Primary sources: Lin et al., "Focal Loss for Dense Object Detection" (2017) for the $(1-\hat y_c)^\gamma$ down-weighting; Guo et al., "On Calibration of Modern Neural Networks" (2017) for temperature scaling; van den Oord et al., "Representation Learning with Contrastive Predictive Coding" (2018) and Chen et al., "SimCLR" (2020) for InfoNCE as a softmax cross-entropy over positives and negatives.