---
title: Normalization
module: Regularization
moduleNumber: 4
lessonNumber: 4
order: 404
summary: >
  Normalization layers standardize activations to zero mean and unit variance
  inside the network, then hand the model a learnable scale and shift to undo the
  constraint when it pays to. Batch normalization does this across the batch and
  must keep separate train-time and test-time statistics; layer, instance, and
  group norm change only the axes they average over. The result is faster,
  better-conditioned optimization and a free dose of regularizing batch noise.
topics: [Regularization]
sources:
  - book: Goodfellow
    ref: "§8.7.1 — Batch Normalization"
  - book: Goodfellow
    ref: "§8.7 — Optimization Strategies and Meta-Algorithms"
  - book: Chollet
    ref: "§7.4.2 — Batch Normalization"
---

A deep stack has a moving-target problem. Every layer is trained assuming the
distribution of its inputs is fixed, but those inputs are the outputs of the
layers below, whose weights are changing on every step. The signal that arrives
at layer $k$ drifts in mean and scale throughout training, so each layer must
continually re-adapt to a drifting distribution. **Normalization**
attacks this directly: rescale the activations to a fixed first and second moment
_inside_ the forward pass, then let the network learn back any scale it actually
needs. The standardization is a differentiable layer, not a preprocessing step on
the data; its statistics flow through
[backpropagation](/deep-learning/neural-networks/backpropagation) like everything
else.

The one decision that separates the whole family is _which axes you average
over_. Standardizing is always the same three operations — subtract a mean,
divide by a standard deviation, apply a learnable affine map. What changes from
batch norm to layer norm to group norm is only the set of tensor entries pooled
into each mean and variance. This lesson derives batch normalization in full
(forward pass, running statistics, the backward pass through the batch moments),
then walks the family by re-choosing that axis set, and closes on why the whole
construction speeds up and regularizes training.

## Batch normalization

Fix one feature (one coordinate of the activation vector, or one channel of a
convolutional map). Over a minibatch $\mathcal{B} = \{x_1, \dots, x_m\}$ of that
feature's values, **batch normalization** computes the batch mean and variance,
standardizes, then applies a learnable affine map:

$$
\mu_\mathcal{B} = \frac{1}{m}\sum_{i=1}^m x_i,
\qquad
\sigma_\mathcal{B}^2 = \frac{1}{m}\sum_{i=1}^m (x_i - \mu_\mathcal{B})^2,
$$

$$
\hat{x}_i = \frac{x_i - \mu_\mathcal{B}}{\sqrt{\sigma_\mathcal{B}^2 + \epsilon}},
\qquad
y_i = \gamma\,\hat{x}_i + \beta.
$$

The small constant $\epsilon$ (typically $10^{-5}$) keeps the denominator away
from zero. The pair $(\gamma, \beta)$ is learned per feature by gradient descent;
they are the layer's only parameters.[^gf-batchnorm]

Read the four equations in order. The first two summarize the batch: $\mu_\mathcal{B}$
is where this feature's values sit on average, $\sigma_\mathcal{B}^2$ is how spread
out they are. The third centers and rescales every value so the batch has mean $0$
and variance $1$; after this step the feature is on a fixed, canonical scale no
matter what the layers below produced. The fourth returns two learnable parameters
to the network: $\gamma$ stretches the standardized feature and $\beta$ shifts it,
so the network can place the final distribution anywhere.

**A concrete pass.** Take a minibatch of $m = 4$ values of one feature,
$x = (2, 4, 6, 8)$. Then $\mu_\mathcal{B} = 5$ and
$\sigma_\mathcal{B}^2 = \tfrac{1}{4}\big((-3)^2 + (-1)^2 + 1^2 + 3^2\big) = 5$, so
(ignoring $\epsilon$) the standardized values are
$\hat{x} = (-3,-1,1,3)/\sqrt{5} \approx (-1.34, -0.45, 0.45, 1.34)$ — mean $0$,
variance $1$, as promised. With learned $\gamma = 2,\ \beta = 1$ the outputs are
$y \approx (-1.68, 0.11, 1.89, 3.68)$: recentered on $\beta = 1$ and spread by
$\gamma = 2$. Change $x$ to $(12, 14, 16, 18)$ and $\hat{x}$ is unchanged: the
standardization is invariant to any shift of the whole batch, which is the point.

**Where the parameters live.** Normalization runs per feature, so the statistics
and parameters are vectors indexed by the channel axis. For a convolutional
activation of shape $(N, C, H, W)$ — batch $N$, channels $C$, height $H$, width $W$
— batch norm computes one $\mu_\mathcal{B}$ and one $\sigma_\mathcal{B}^2$ per
channel by pooling the $N \cdot H \cdot W$ entries of that channel, and it holds one
$\gamma$ and one $\beta$ per channel. The parameter count is therefore $2C$,
independent of batch size or spatial resolution.

> **Definition (Batch normalization).** A layer that, for each feature, subtracts
> the minibatch mean $\mu_\mathcal{B}$ and divides by the minibatch standard
> deviation $\sqrt{\sigma_\mathcal{B}^2 + \epsilon}$, producing a standardized
> $\hat{x}$ with zero mean and unit variance _across the batch_, then returns
> $y = \gamma\hat{x} + \beta$ with learnable scale $\gamma$ and shift $\beta$.

The affine map is what keeps the layer from _losing_ expressive power. Forcing
unit variance is a constraint; the network can cancel it whenever the constraint
hurts.

> **Theorem (Recovery of the identity).** For any feature, setting
> $\gamma = \sqrt{\sigma_\mathcal{B}^2 + \epsilon}$ and $\beta = \mu_\mathcal{B}$
> makes the batch-norm layer reproduce its input exactly: $y_i = x_i$.

> **Proof.** Substituting into $y_i = \gamma\hat{x}_i + \beta$,
> $$
> y_i = \sqrt{\sigma_\mathcal{B}^2 + \epsilon}\cdot
>   \frac{x_i - \mu_\mathcal{B}}{\sqrt{\sigma_\mathcal{B}^2 + \epsilon}}
>   + \mu_\mathcal{B}
> = (x_i - \mu_\mathcal{B}) + \mu_\mathcal{B} = x_i.
> $$
> So the identity transformation lies in the layer's hypothesis class; whatever
> representational power standardization removes, the affine parameters can
> restore. $\qed$

The standardization moves a feature's distribution to a canonical position and
the affine map relocates it — now under the
optimizer's explicit control through two parameters, rather than as a byproduct of
the layers below.

$$
% caption: Batch norm in three moves on one feature: a wide activation is
% recentered to zero mean and unit variance, then relocated by a learnable shift.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  % --- panel 1: raw, shifted + wide ---
  \draw[->, black] (-0.3,0) -- (3.4,0) node[right, font=\footnotesize, black] {\texttt{value}};
  \draw[black, dashed] (1.0,0) -- (1.0,2.0);
  \node[black, font=\scriptsize, anchor=north] at (1.0,-0.05) {0};
  \draw[acc, very thick] plot[domain=1.4:4.0, samples=60]
    (\x, {1.7*exp(-(\x-2.7)*(\x-2.7)/0.55)});
  \node[align=center, font=\footnotesize] at (1.7,-0.75) {\texttt{raw}\\\texttt{(shifted, wide)}};
  % --- panel 2: standardized ---
  \begin{scope}[xshift=4.6cm]
    \draw[->, black] (-0.3,0) -- (3.4,0) node[right, font=\footnotesize, black] {\texttt{value}};
    \draw[black, dashed] (1.5,0) -- (1.5,2.0);
    \node[black, font=\scriptsize, anchor=north] at (1.5,-0.05) {0};
    \draw[acc, very thick] plot[domain=0.1:2.9, samples=60]
      (\x, {1.7*exp(-(\x-1.5)*(\x-1.5)/0.32)});
    \node[align=center, font=\footnotesize] at (1.5,-0.75) {\texttt{standardized}\\\texttt{(mean 0, var 1)}};
  \end{scope}
  % --- panel 3: rescaled by gamma, beta ---
  \begin{scope}[xshift=9.2cm]
    \draw[->, black] (-0.3,0) -- (3.4,0) node[right, font=\footnotesize, black] {\texttt{value}};
    \draw[black, dashed] (0.6,0) -- (0.6,2.0);
    \node[black, font=\scriptsize, anchor=north] at (0.6,-0.05) {0};
    \draw[acc, very thick] plot[domain=0.4:3.4, samples=60]
      (\x, {1.5*exp(-(\x-2.0)*(\x-2.0)/0.42)});
    \node[align=center, font=\footnotesize] at (1.7,-0.75) {\texttt{scale + shift}\\\texttt{(learnable)}};
  \end{scope}
  % arrows between panels
  \draw[->, black, thick] (3.55,1.1) -- (4.25,1.1);
  \draw[->, black, thick] (8.15,1.1) -- (8.85,1.1);
\end{tikzpicture}
$$

### The forward pass as an algorithm

Collecting the four equations gives the training-time forward pass for a single
feature; in practice the same operations run vectorized over every feature at once.

```algorithm
caption: $\textsc{BatchNorm}(\{x_i\}_{i=1}^m;\ \gamma, \beta)$ — train-time forward pass over one feature
$\mu \gets \frac{1}{m}\sum_{i} x_i$ // batch mean
$v \gets \frac{1}{m}\sum_{i} (x_i - \mu)^2$ // batch variance
for $i \gets 1$ to $m$ do
  $\hat{x}_i \gets (x_i - \mu) / \sqrt{v + \epsilon}$ // standardize
  $y_i \gets \gamma \cdot \hat{x}_i + \beta$ // scale and shift
$r_\mu \gets (1 - \alpha)\,r_\mu + \alpha\,\mu$ // accumulate running mean
$r_v \gets (1 - \alpha)\,r_v + \alpha\,v$ // accumulate running variance
return $\{y_i\}$
```

The layer is differentiable end to end. Backprop must route gradients not only
through the affine $y_i = \gamma\hat{x}_i + \beta$ but also through $\mu_\mathcal{B}$
and $\sigma_\mathcal{B}^2$, since both depend on _every_ $x_j$ in the batch — so the
gradient with respect to one input mixes in contributions from all the others.

The parameter gradients are the easy half: $\gamma$ and $\beta$ appear only in the
final affine, so they just sum the upstream gradient over the batch, weighted by
$\hat{x}_i$ for the scale and unweighted for the shift:

$$
\frac{\partial L}{\partial \gamma} = \sum_i \frac{\partial L}{\partial y_i}\,\hat{x}_i,
\qquad
\frac{\partial L}{\partial \beta} = \sum_i \frac{\partial L}{\partial y_i}.
$$

The input gradient is the interesting half. Writing $g_i = \partial L/\partial\hat{x}_i
= \gamma\,\partial L/\partial y_i$ for the gradient after the affine, the chain rule
carries $g_i$ back through the three paths by which $x_i$ reaches the loss: directly
through its own $\hat{x}_i$, through the shared mean $\mu_\mathcal{B}$, and through
the shared variance $\sigma_\mathcal{B}^2$. Collecting the three paths gives

$$
\frac{\partial L}{\partial x_i}
= \frac{1}{\sqrt{\sigma_\mathcal{B}^2 + \epsilon}}
\parens{
  g_i
  - \frac{1}{m}\sum_j g_j
  - \frac{\hat{x}_i}{m}\sum_j g_j\,\hat{x}_j
},
$$

with $g_i = \gamma\,\partial L/\partial y_i$. The first term is the naive gradient
you would get if $\mu_\mathcal{B}$ and $\sigma_\mathcal{B}^2$ were constants. The
second subtracts the batch mean of the upstream gradient — the correction from
$x_i$'s influence on $\mu_\mathcal{B}$. The third subtracts an $\hat{x}$-weighted
mean — the correction from its influence on $\sigma_\mathcal{B}^2$. Together the two
corrections project the incoming gradient onto the subspace orthogonal to constant
shifts and to rescalings of $\hat{x}$; that projection is what leaves the layer's
output invariant to the mean and scale of its input, and it is the reason an
example's gradient depends on which other examples share its batch.[^gf-bn-backprop]

The dataflow makes the coupling visible: every input feeds the two batch
statistics, which in turn feed the normalization of _every_ input, before the
per-feature affine finishes the map.

$$
% caption: Batch-norm dataflow for one feature. The whole batch $x_1,\dots,x_m$
% feeds the shared mean $\mu_\mathcal{B}$ and variance $\sigma_\mathcal{B}^2$; each
% standardized $\hat{x}_i$ then passes through the per-feature affine
% $y_i = \gamma\,\hat{x}_i + \beta$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  bx/.style={draw, minimum width=13mm, minimum height=7mm, align=center, font=\scriptsize},
  st/.style={draw, minimum width=15mm, minimum height=7mm, align=center, font=\scriptsize},
  op/.style={draw=acc, text=acc, thick, minimum width=15mm, minimum height=7mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  % inputs column
  \node[bx] (x1) at (0,2.2) {\texttt{x1}};
  \node[bx] (x2) at (0,1.1) {\texttt{x2}};
  \node[font=\scriptsize] (xd) at (0,0.35) {$\vdots$};
  \node[bx] (xm) at (0,-0.4) {\texttt{xm}};
  % batch stats
  \node[st] (mu) at (3.0,1.5) {\texttt{mean}};
  \node[st] (sg) at (3.0,0.2) {\texttt{var}};
  % normalize
  \node[op] (nm) at (6.0,0.85) {\texttt{normalize}};
  % affine
  \node[op] (af) at (9.0,0.85) {\texttt{scale, shift}};
  \node[bx] (y) at (11.6,0.85) {\texttt{y}};
  % edges: all inputs -> stats
  \foreach \n in {x1,x2,xm} {
    \draw[->, black] (\n.east) -- (mu.west);
    \draw[->, black] (\n.east) -- (sg.west);
  }
  % inputs also flow to normalize
  \draw[->, thick] (x1.east) .. controls (1.6,2.2) and (4.2,1.9) .. (nm.north);
  % stats -> normalize
  \draw[->, thick] (mu.east) -- (nm.west);
  \draw[->, thick] (sg.east) -- (nm.west);
  % normalize -> affine -> y
  \draw[->, thick] (nm.east) -- (af.west);
  \draw[->, thick] (af.east) -- (y.west);
  % labels
  \node[black, font=\footnotesize, anchor=north] at (1.5,-0.75) {\texttt{whole batch}};
  \node[acc, font=\footnotesize, anchor=north] at (9.0,0.15) {\texttt{per feature}};
\end{tikzpicture}
$$

## Train versus inference

The batch statistics $\mu_\mathcal{B}, \sigma_\mathcal{B}^2$ are a property of the
batch, not the example — they are unavailable at test time, where inputs may arrive
one at a time and predictions must not depend on which other examples happen to be
present. The fix is to accumulate **running estimates** during training (lines 6–7
of the algorithm, an exponential moving average with momentum $\alpha$) and freeze
them for inference:

$$
\text{train:}\quad \hat{x} = \frac{x - \mu_\mathcal{B}}{\sqrt{\sigma_\mathcal{B}^2 + \epsilon}},
\qquad\qquad
\text{test:}\quad \hat{x} = \frac{x - r_\mu}{\sqrt{r_v + \epsilon}}.
$$

The running statistics $r_\mu, r_v$ are buffers, not parameters: updated by a
formula, never by a gradient. Each training step folds the current batch moments
into the running estimate with momentum $\alpha$ (typically $\alpha \approx 0.1$,
or its complement $0.9$ depending on the convention),

$$
r_\mu \gets (1-\alpha)\,r_\mu + \alpha\,\mu_\mathcal{B},
\qquad
r_v \gets (1-\alpha)\,r_v + \alpha\,\sigma_\mathcal{B}^2,
$$

so that after enough steps $r_\mu, r_v$ track the population mean and variance of
the activation while the batch statistics still do the normalizing during training.
At test time the layer becomes a fixed affine map of its input,

$$
y = \frac{\gamma}{\sqrt{r_v + \epsilon}}\,x
  + \parens{\beta - \frac{\gamma\,r_\mu}{\sqrt{r_v + \epsilon}}},
$$

a single scale-and-shift with no batch and no per-example coupling, which is why
it can be folded into the preceding linear layer at inference for free.[^chollet-bn]

$$
% caption: Noisy per-step batch statistics (red) jitter between minibatches; the
% running average (blue) smooths them into the value used at inference.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, thick] (0,0) -- (8.2,0) node[right, font=\footnotesize] {\texttt{training step}};
  \draw[->, thick] (0,0) -- (0,3.2) node[above, font=\footnotesize] {\texttt{batch mean}};
  % noisy per-step batch stats (red, jagged)
  \draw[red, thick]
    (0.4,2.4) -- (0.9,1.5) -- (1.4,2.7) -- (1.9,1.7) -- (2.4,2.5)
    -- (2.9,1.6) -- (3.4,2.3) -- (3.9,1.9) -- (4.4,2.4) -- (4.9,1.8)
    -- (5.4,2.2) -- (5.9,1.9) -- (6.4,2.25) -- (6.9,2.0) -- (7.4,2.15);
  \node[red, font=\footnotesize, anchor=south west] at (2.7,2.7) {\texttt{per-step batch stats}};
  % running average (dark, smooth, converging)
  \draw[acc, very thick]
    (0.4,2.4) .. controls (2.0,1.95) and (4.0,2.05) .. (7.4,2.08);
  \node[acc, font=\footnotesize, anchor=north east] at (7.4,1.85) {\texttt{running average}};
  % inference level
  \draw[black, dashed] (0,2.08) -- (7.8,2.08);
  \node[black, font=\footnotesize, anchor=west] at (7.5,2.4) {\texttt{used at test}};
\end{tikzpicture}
$$

> **Definition (Running statistics).** Exponential moving averages $r_\mu, r_v$ of
> the per-batch mean and variance, maintained during training and substituted for
> the batch statistics at inference, so that test-time predictions are deterministic
> and independent of batch composition.

The train/test asymmetry is the single richest source of batch-norm bugs. Confuse
the two modes and the network silently misbehaves:

| Bug | Cause | Symptom |
| --- | --- | --- |
| Test accuracy collapses | model left in train mode at eval (`model.train()` not `model.eval()`) | predictions depend on batch order; single-example inference garbage |
| Train fine, deploy broken | running stats never warmed up (too few steps, or frozen from init) | $r_\mu, r_v$ stale or zero, so test-time standardization is wrong |
| Batch size 1 | variance over one example is $0$ (or undefined) | division by $\sqrt{\epsilon}$ blows activations up |
| Fine-tuning drift | BN stats inherited from a different data distribution | frozen $r_\mu, r_v$ mismatch the new domain |

The first row is the canonical one: with the layer in training mode at test time,
each prediction normalizes by the statistics of whatever batch it arrives in, so the
_same input_ yields different outputs depending on its neighbors.

## Why it helps

Batch norm reliably lets you train deeper networks at higher learning rates, but
_why_ has been contested. Two explanations compete.

The original account was **internal covariate shift**: because lower layers keep
changing, the distribution feeding each layer drifts, forcing upper layers to
perpetually re-adapt. Fixing the mean and variance of every layer's input is meant
to halt this drift, so each layer sees a stationary distribution and can learn
faster.[^gf-covariate]

> **Definition (Internal covariate shift).** The change in the distribution of a
> layer's inputs during training, caused by updates to the parameters of all
> preceding layers. Batch norm was originally motivated as a way to reduce it by
> pinning the first two moments of each layer's input.

The modern account is that the covariate-shift explanation is at best incomplete:
networks can be trained with deliberately _injected_ post-norm distribution shift
and still retain batch norm's benefits. The better-supported explanation is that
normalization **conditions the optimization** — it makes the loss and its
gradient smoother (smaller, more predictable curvature along the update direction),
so larger, more reliable steps are safe. The decoupling of each weight's effective
scale from the loss is a second mechanism: scaling a weight by $c$ leaves a
normalized layer's output unchanged, which reduces the loss surface's sensitivity to
weight magnitude.[^gf-conditioning]

$$
% caption: Loss along an update direction: jagged and sharply curved without
% normalization, smoother and better-conditioned with it.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  % axes
  \draw[->, thick] (0,0) -- (8.2,0) node[right, font=\footnotesize] {\texttt{step along direction}};
  \draw[->, thick] (0,0) -- (0,3.4) node[above, font=\footnotesize] {\texttt{loss}};
  % jagged, no-norm (red)
  \draw[red, thick]
    (0.4,2.9) -- (0.9,1.6) -- (1.3,2.7) -- (1.8,1.1) -- (2.3,2.2)
    -- (2.8,0.9) -- (3.4,1.9) -- (4.0,0.7) -- (4.7,1.5) -- (5.5,0.6)
    -- (6.3,1.2) -- (7.2,0.55);
  \node[red, font=\footnotesize, anchor=west] at (5.6,2.0) {\texttt{no norm (jagged)}};
  % smooth, norm (green)
  \draw[green, very thick]
    (0.4,2.4) .. controls (2.5,0.2) and (5.0,0.2) .. (7.4,0.55);
  \node[green, font=\footnotesize, anchor=north] at (4.6,0.4) {\texttt{with norm (smooth)}};
\end{tikzpicture}
$$

> **Remark (Honest caveat).** The two explanations are not settled. Internal
> covariate shift gave batch norm its name and intuition, but controlled
> experiments cast doubt on it as the operative mechanism; the loss-smoothing /
> conditioning account is currently better supported, though batch norm's full
> effect is still an area of active study. The technique works regardless of which
> explanation is correct.

## The family: which axes are normalized

Batch norm is one choice of _which axes to average over_. For a 4-D activation
tensor of shape $(N, C, H, W)$ — batch $N$, channels $C$, spatial $H \times W$ — the
variants differ only in the set of axes contributing to each $(\mu, \sigma^2)$.
**Layer norm** averages over the features of a single example (so it is batch-
independent); **instance norm** normalizes each channel of each example on its own;
**group norm** splits channels into groups and normalizes within a group, a middle
ground that, like layer/instance norm, never touches the batch axis.

The cleanest way to see the whole family is as different shadings of the same
tensor. Lay the activation out as an $N \times C$ grid of spatial maps (each cell is
one $H \times W$ slice). Every scheme computes its statistics over the shaded cells,
and only the shading changes.

$$
% caption: One $(N, C, H, W)$ activation, shown as an $N \times C$ grid of
% $H \times W$ maps. Each scheme pools the shaded cells into one $(\mu, \sigma^2)$:
% batch norm a channel across the batch, layer norm all channels of one example,
% instance norm a single map, group norm a channel-group of one example.
\begin{tikzpicture}[>=stealth, font=\scriptsize, scale=0.9]
  \definecolor{acc}{HTML}{2348F2}
  \def\cell{0.5}
  % ---------- panel BatchNorm ----------
  \begin{scope}[xshift=0cm]
    \foreach \r in {0,1,2} \foreach \c in {0,1,2,3} {
      \draw[black] (\c*\cell, \r*\cell) rectangle (\c*\cell+\cell, \r*\cell+\cell);
    }
    \foreach \r in {0,1,2} {
      \draw[acc, very thick, fill=acc!18] (1*\cell, \r*\cell) rectangle (1*\cell+\cell, \r*\cell+\cell);
    }
    \node[font=\footnotesize, anchor=north] at (2*\cell, -0.7) {\texttt{Batch}};
    \draw[->, black] (-0.35,-0.15) -- (-0.35,3*\cell+0.15);
    \node[black, font=\footnotesize, anchor=east, rotate=90] at (-0.55, 1.5*\cell) {\texttt{N}};
    \draw[->, black] (-0.15,-0.35) -- (4*\cell+0.15,-0.35);
    \node[black, font=\footnotesize, anchor=north] at (2*\cell, -0.45) {\texttt{C}};
  \end{scope}
  % ---------- panel LayerNorm ----------
  \begin{scope}[xshift=3.0cm]
    \foreach \r in {0,1,2} \foreach \c in {0,1,2,3} {
      \draw[black] (\c*\cell, \r*\cell) rectangle (\c*\cell+\cell, \r*\cell+\cell);
    }
    \foreach \c in {0,1,2,3} {
      \draw[acc, very thick, fill=acc!18] (\c*\cell, 1*\cell) rectangle (\c*\cell+\cell, 1*\cell+\cell);
    }
    \node[font=\footnotesize, anchor=north] at (2*\cell, -0.7) {\texttt{Layer}};
  \end{scope}
  % ---------- panel InstanceNorm ----------
  \begin{scope}[xshift=6.0cm]
    \foreach \r in {0,1,2} \foreach \c in {0,1,2,3} {
      \draw[black] (\c*\cell, \r*\cell) rectangle (\c*\cell+\cell, \r*\cell+\cell);
    }
    \draw[acc, very thick, fill=acc!18] (1*\cell, 1*\cell) rectangle (1*\cell+\cell, 1*\cell+\cell);
    \node[font=\footnotesize, anchor=north] at (2*\cell, -0.7) {\texttt{Instance}};
  \end{scope}
  % ---------- panel GroupNorm ----------
  \begin{scope}[xshift=9.0cm]
    \foreach \r in {0,1,2} \foreach \c in {0,1,2,3} {
      \draw[black] (\c*\cell, \r*\cell) rectangle (\c*\cell+\cell, \r*\cell+\cell);
    }
    \foreach \c in {0,1} {
      \draw[acc, very thick, fill=acc!18] (\c*\cell, 1*\cell) rectangle (\c*\cell+\cell, 1*\cell+\cell);
    }
    \node[font=\footnotesize, anchor=north] at (2*\cell, -0.7) {\texttt{Group}};
  \end{scope}
  \node[black, font=\footnotesize] at (5.5,-1.25) {\texttt{cell = one H x W map,\ \ shaded = pooled into one mean and variance}};
\end{tikzpicture}
$$

$$
% caption: The four normalization schemes over an $(N, C, H, W)$ tensor; shaded
% cells are pooled into one mean and variance.
\begin{tikzpicture}[>=stealth, font=\scriptsize, scale=0.95]
  \definecolor{acc}{HTML}{2348F2}
  % a 3x4 grid: rows = batch (N=3), cols = channel (C=4); each cell = one spatial map
  % we draw four copies labelled by the scheme and shade the pooled set.
  \def\cell{0.52}
  % helper macro emulated by explicit loops per panel
  % ---------- panel BatchNorm ----------
  \begin{scope}[xshift=0cm]
    \foreach \r in {0,1,2} \foreach \c in {0,1,2,3} {
      \draw[black] (\c*\cell, \r*\cell) rectangle (\c*\cell+\cell, \r*\cell+\cell);
    }
    % shade one channel column across all batch rows (BN pools N and spatial, per channel)
    \foreach \r in {0,1,2} {
      \draw[acc, very thick, fill=acc!18] (1*\cell, \r*\cell) rectangle (1*\cell+\cell, \r*\cell+\cell);
    }
    \node[font=\footnotesize, anchor=north] at (2*\cell, -0.25) {\texttt{Batch}};
    \node[black, font=\footnotesize, anchor=south] at (2*\cell, 3*\cell+0.1) {\texttt{pool batch}};
  \end{scope}
  % ---------- panel LayerNorm ----------
  \begin{scope}[xshift=3.1cm]
    \foreach \r in {0,1,2} \foreach \c in {0,1,2,3} {
      \draw[black] (\c*\cell, \r*\cell) rectangle (\c*\cell+\cell, \r*\cell+\cell);
    }
    % shade one batch row across all channels (LN pools C and spatial, per example)
    \foreach \c in {0,1,2,3} {
      \draw[acc, very thick, fill=acc!18] (\c*\cell, 1*\cell) rectangle (\c*\cell+\cell, 1*\cell+\cell);
    }
    \node[font=\footnotesize, anchor=north] at (2*\cell, -0.25) {\texttt{Layer}};
    \node[black, font=\footnotesize, anchor=south] at (2*\cell, 3*\cell+0.1) {\texttt{pool channels}};
  \end{scope}
  % ---------- panel InstanceNorm ----------
  \begin{scope}[xshift=6.2cm]
    \foreach \r in {0,1,2} \foreach \c in {0,1,2,3} {
      \draw[black] (\c*\cell, \r*\cell) rectangle (\c*\cell+\cell, \r*\cell+\cell);
    }
    % shade a single cell (IN pools only spatial, per example per channel)
    \draw[acc, very thick, fill=acc!18] (1*\cell, 1*\cell) rectangle (1*\cell+\cell, 1*\cell+\cell);
    \node[font=\footnotesize, anchor=north] at (2*\cell, -0.25) {\texttt{Instance}};
    \node[black, font=\footnotesize, anchor=south] at (2*\cell, 3*\cell+0.1) {\texttt{pool spatial}};
  \end{scope}
  % ---------- panel GroupNorm ----------
  \begin{scope}[xshift=9.3cm]
    \foreach \r in {0,1,2} \foreach \c in {0,1,2,3} {
      \draw[black] (\c*\cell, \r*\cell) rectangle (\c*\cell+\cell, \r*\cell+\cell);
    }
    % shade two adjacent channels of one example (GN pools a channel-group, per example)
    \foreach \c in {0,1} {
      \draw[acc, very thick, fill=acc!18] (\c*\cell, 1*\cell) rectangle (\c*\cell+\cell, 1*\cell+\cell);
    }
    \node[font=\footnotesize, anchor=north] at (2*\cell, -0.25) {\texttt{Group}};
    \node[black, font=\footnotesize, anchor=south] at (2*\cell, 3*\cell+0.1) {\texttt{pool a group}};
  \end{scope}
  % shared axis labels
  \node[black, font=\footnotesize, anchor=east, rotate=90] at (-0.35, 1.5*\cell) {\texttt{batch}};
  \node[black, font=\footnotesize] at (5.0,-0.7) {\texttt{columns = channels,\ \ rows = batch,\ \ each cell = one spatial map}};
\end{tikzpicture}
$$

The table below makes the trade-offs explicit. The decisive column is _batch-
dependent?_ — the one property that determines whether a layer behaves identically
at train and test, and whether it survives a batch of size one.

| Norm | Normalized over | Batch-dependent? | Typical use |
| --- | --- | --- | --- |
| Batch norm | batch $N$ and spatial $H,W$, per channel | yes | CNN classifiers (large batch) |
| Layer norm | channels $C$ and spatial, per example | no | RNN, Transformer |
| Instance norm | spatial $H,W$, per example per channel | no | style transfer, generation |
| Group norm | a channel group and spatial, per example | no | CNN with small batch (detection) |
| RMS norm | features, per example (no centering) | no | large language models |

> **Definition (Layer normalization).** Normalization over all features of a single
> example, independent of the batch, with one $(\gamma, \beta)$ per feature. Because
> it uses no batch statistics, train and test are identical and the layer is
> unaffected by batch size — the property that makes it the default in
> [Transformers](/deep-learning/architectures/attention-and-transformers) and
> [recurrent networks](/deep-learning/architectures/recurrent-networks).

> **Definition (Group normalization).** Normalization within fixed groups of
> channels of a single example. With one group it is layer norm; with one channel
> per group it is instance norm — group norm interpolates between them and matches
> batch norm's accuracy on vision tasks when the batch is too small for reliable
> batch statistics.

**Layer norm, written out.** For one example with feature vector
$x \in \mathbb{R}^{d}$, layer norm pools all $d$ features into a single mean and
variance:

$$
\mu = \frac{1}{d}\sum_{k=1}^d x_k,
\qquad
\sigma^2 = \frac{1}{d}\sum_{k=1}^d (x_k - \mu)^2,
\qquad
y_k = \gamma_k\,\frac{x_k - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta_k.
$$

The formulas are batch norm's with the sum running over features instead of over
the batch. Because no other example enters, the computation is identical at train
and test time, and a batch of size one is handled exactly like a batch of a
thousand — the property that makes it the default in Transformers, where the batch
axis carries variable-length sequences, and in recurrent networks, where the same
statistics must apply at every timestep.

**RMS norm** drops the mean-centering: it divides by the root-mean-square of the
features and keeps only a scale, no shift.

$$
y_k = \gamma_k\,\frac{x_k}{\sqrt{\tfrac{1}{d}\sum_{j} x_j^2 + \epsilon}}.
$$

Skipping the subtraction of $\mu$ costs almost nothing in accuracy on large
Transformers while removing one reduction and one buffer per layer, which is why
recent large language models tend to use it in place of layer norm.

Batch norm's accuracy degrades when the batch is small because $\mu_\mathcal{B},
\sigma_\mathcal{B}^2$ become noisy estimates of the population moments; the
batch-independent variants sidestep this entirely, which is why detection and
segmentation models (forced into tiny batches by memory) use group norm,
and why sequence models, whose batch axis carries variable-length examples, use
layer norm.[^gf-normfamily]

## Placement

Where the layer sits relative to the linear map and the
[activation](/deep-learning/neural-networks/activation-functions) is a genuine
design choice. The original prescription was linear $\to$ norm $\to$ activation, so
the normalized pre-activation feeds the nonlinearity; a common alternative
normalizes the activation output instead. In residual architectures the analogous
question is **pre-norm versus post-norm**: whether normalization sits _inside_ the
residual branch (before the sublayer) or _after_ the residual addition.

$$
% caption: Left: a layer as linear $\to$ norm $\to$ activation. Right: the
% Transformer post-norm versus pre-norm placement debate.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=18mm, minimum height=8mm, align=center, font=\scriptsize},
  nb/.style={draw=acc, text=acc, thick, minimum width=18mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  % ---- left: in-layer placement ----
  \node[box] (li) at (0,0)   {\texttt{input}};
  \node[box] (ll) at (0,1.2) {\texttt{linear}};
  \node[nb]  (ln) at (0,2.4) {\texttt{norm}};
  \node[box] (la) at (0,3.6) {\texttt{activation}};
  \draw[->, thick] (li) -- (ll);
  \draw[->, thick] (ll) -- (ln);
  \draw[->, thick] (ln) -- (la);
  \node[black, font=\footnotesize, anchor=north] at (0,-0.7) {\texttt{in-layer}};
  % ---- middle: post-norm ----
  \begin{scope}[xshift=4.2cm]
    \node[box] (pi) at (0,0)   {\texttt{input}};
    \node[box] (ps) at (0,1.3) {\texttt{sublayer}};
    \node[circle, draw, inner sep=1pt, font=\scriptsize] (pa) at (0,2.6) {$+$};
    \node[nb]  (pn) at (0,3.7) {\texttt{norm}};
    \draw[->, thick] (pi) -- (ps);
    \draw[->, thick] (ps) -- (pa);
    \draw[->, thick] (pa) -- (pn);
    % residual skip around the sublayer into the add
    \draw[->, black, thick] (pi.east) -- ++(1.0,0) |- (pa.east);
    \node[black, font=\footnotesize, anchor=north] at (0,-0.7) {\texttt{post-norm}};
  \end{scope}
  % ---- right: pre-norm ----
  \begin{scope}[xshift=8.8cm]
    \node[box] (ri) at (0,0)   {\texttt{input}};
    \node[nb]  (rn) at (0,1.3) {\texttt{norm}};
    \node[box] (rs) at (0,2.6) {\texttt{sublayer}};
    \node[circle, draw, inner sep=1pt, font=\scriptsize] (ra) at (0,3.7) {$+$};
    \draw[->, thick] (ri) -- (rn);
    \draw[->, thick] (rn) -- (rs);
    \draw[->, thick] (rs) -- (ra);
    % residual skip from input straight to the add (clean path)
    \draw[->, black, thick] (ri.east) -- ++(1.2,0) |- (ra.east);
    \node[black, font=\footnotesize, anchor=north] at (0,-0.7) {\texttt{pre-norm}};
  \end{scope}
\end{tikzpicture}
$$

Pre-norm leaves an unnormalized identity path from input to output of each block,
so the gradient reaches early layers without passing through a normalization at
every step — the reason very deep Transformers train more stably with pre-norm,
often without learning-rate warmup. Post-norm can reach slightly better final
quality when it trains at all, but is more fragile at depth.

| Placement | Norm position | Gradient path | Trains deep nets |
| --- | --- | --- | --- |
| Post-norm | after residual add | through norm each block | fragile, needs warmup |
| Pre-norm | inside branch, before sublayer | clean identity skip | stable to great depth |

## Normalization, revisited

Goodfellow §8.7.1 introduces batch normalization (Ioffe & Szegedy, 2015) and gives
the internal-covariate-shift motivation; the public literature since has both
challenged that motivation and multiplied the variants. The covariate-shift explanation
was directly tested by **Santurkar et al. (2018)**, who trained batch-norm networks
with deliberately _injected_ post-normalization distribution shift and found the
training speedup intact — evidence that the operative mechanism is the loss-surface
smoothing described above, not the drift the name suggests. The technique works; the
name is a historical accident.

The batch-independent variants each arose from a specific need. **Layer norm** (Ba
et al., 2016) freed normalization from the batch axis for recurrent and sequence
models; **instance norm** (Ulyanov et al., 2016) suited style transfer; **group
norm** (Wu & He, 2018) filled the small-batch gap for detection and segmentation; and
**RMS norm** (Zhang & Sennrich, 2019) dropped the mean-centering for a cheaper layer.
The decisive outcome is architectural: modern large language models are built almost
entirely on **pre-norm** layer or RMS normalization, because the clean identity path
through each residual block (shown above) is what lets very deep transformers train
stably, often without the learning-rate warmup post-norm requires. Batch norm remains
the default for convolutional image classifiers with large batches, but the sequence
world it never fit well moved to the batch-independent family and stayed there.

## The regularizing side effect

Batch norm is filed under regularization for a concrete reason: it injects noise.
Each example's normalized value $\hat{x}_i$ depends on the _random_ minibatch it was
sampled with, through $\mu_\mathcal{B}$ and $\sigma_\mathcal{B}^2$. Two identical
inputs in different batches get slightly different activations — a stochastic
perturbation closely analogous to
[dropout](/deep-learning/regularization/dropout-and-data-augmentation), which is
why models with batch norm often need less of it.[^gf-bn-noise]

> **Remark (Batch noise as regularizer).** Because $\mu_\mathcal{B},
> \sigma_\mathcal{B}^2$ are estimated from a random batch, batch norm perturbs each
> activation with batch-dependent noise of order $1/\sqrt{m}$. This discourages the
> network from relying on any single precise activation value and contributes a
> mild regularizing effect — one that _shrinks_ as the batch grows, since larger
> batches estimate the moments more precisely.

The effect cuts both ways: the noise that regularizes also destabilizes
small batches and forces the train/test statistic split. Normalization
trades a constraint on the activations for better conditioning and a mild dose of
noise, at the bookkeeping cost of maintaining running statistics.
It composes with the other tools in this module:
[parameter penalties](/deep-learning/regularization/regularization-overview),
[dropout and augmentation](/deep-learning/regularization/dropout-and-data-augmentation),
and [early stopping](/deep-learning/regularization/early-stopping-and-parameter-sharing)
all attack overfitting from different angles, and normalization is the one that
also reshapes the loss surface the
[optimizer](/deep-learning/optimization/gradient-descent-and-sgd) has to descend.

[^gf-batchnorm]: **Goodfellow**, _Deep Learning_, §8.7.1 — Batch Normalization: standardizing each feature over the minibatch, then the learnable affine $\gamma\hat x+\beta$ that can recover the identity so the constraint never costs expressive power.
[^gf-bn-backprop]: **Goodfellow**, _Deep Learning_, §8.7.1 — Batch norm backprop: because $\mu_\mathcal{B}$ and $\sigma_\mathcal{B}^2$ depend on every example, the gradient through one input mixes in contributions from all the others in its batch.
[^gf-covariate]: **Goodfellow**, _Deep Learning_, §8.7.1 — Internal covariate shift: the original motivation, pinning the first two moments of each layer's input to halt the distributional drift caused by updates to lower layers.
[^gf-conditioning]: **Goodfellow**, _Deep Learning_, §8.7 — Optimization Strategies: normalization smooths and reconditions the loss surface and decouples a weight's scale from the loss, the better-supported account of why it speeds training.
[^gf-normfamily]: **Goodfellow**, _Deep Learning_, §8.7.1 — Normalization variants: batch norm pools the batch axis (and so needs a batch), while layer, instance, and group norm change only which axes the statistics average over, staying batch-independent.
[^gf-bn-noise]: **Goodfellow**, _Deep Learning_, §8.7.1 — Batch noise as regularizer: estimating $\mu_\mathcal{B},\sigma_\mathcal{B}^2$ from a random batch perturbs each activation by noise of order $1/\sqrt{m}$, a dropout-like effect that fades as the batch grows.
[^chollet-bn]: **Chollet**, _Deep Learning with Python_, §7.4.2 — Batch Normalization: the practical layer, its running-statistics buffers for inference, and the train/eval mode switch that is the canonical source of batch-norm bugs.