---
title: Weight Initialization
module: Optimization
moduleNumber: 3
lessonNumber: 3
order: 303
summary: >
  The initial weights determine whether training can succeed before the first
  gradient step. Initialize every weight equal and all hidden units compute
  the same function forever; initialize too small or too large and the signal
  vanishes or explodes as it crosses depth. A single variance condition,
  $n_{\text{in}}\mathrm{Var}(W)=1$, fixes both, and reading it off the forward
  and backward passes yields Xavier and He initialization directly.
topics: [Optimization]
sources:
  - book: Goodfellow
    ref: "§8.4 — Parameter Initialization Strategies"
  - book: Goodfellow
    ref: "§6.2, §8.2 — Gradient flow, ill-conditioning"
  - book: Stevens
    ref: "Ch. 6 — Initialization in practice (torch.nn.init)"
---

[Gradient descent](/deep-learning/optimization/gradient-descent-and-sgd) finds a
direction to step; it cannot choose where to start. That choice is **weight
initialization**, and for a deep network it matters: it sets
whether the very first forward pass carries usable signal to the output and
whether the very first backward pass carries usable gradient to the input. Two
distinct failures follow from a careless choice: a _symmetry_ that no gradient can
break, and a _scale_ that makes the signal vanish or explode with depth. This
lesson derives the single variance condition that prevents both.[^gf-init]

We work with a plain feedforward net. Layer $l$ takes input $a^{(l-1)}\in\mathbb R^{n_{l-1}}$, computes the pre-activation $z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}$, and applies an elementwise nonlinearity $a^{(l)}=\phi(z^{(l)})$. We write $n_{\text{in}}=n_{l-1}$ for the fan-in and $n_{\text{out}}=n_l$ for the fan-out of layer $l$.

## The symmetry-breaking problem

The most obvious initialization — set every weight and bias to zero — fails
before training begins. Suppose $W^{(l)}=0$ and $b^{(l)}=0$ for all
$l$. Then every pre-activation in a layer is identical, so every unit computes
the same activation, and backpropagation sends every unit the same
gradient. Identical units that receive identical updates stay identical forever.

> **Definition (Symmetry breaking).** The requirement that distinct hidden units
> begin with distinct parameters, so that they receive distinct gradients and can
> specialize to compute different features. Initialization is the only stage that
> can supply it; nothing in the forward or backward pass distinguishes two units
> that start equal.

The argument is not limited to zero; it holds for _any_ constant. Take two units
$i,j$ in layer $l$ with equal incoming weight rows, $W^{(l)}_{i,:}=W^{(l)}_{j,:}$,
and equal biases. Then for every input their pre-activations agree, $z^{(l)}_i=z^{(l)}_j$, so $a^{(l)}_i=a^{(l)}_j$.

$$
% caption: Constant init collapses a layer: equal weight rows give two units the
% same pre-activation, so backprop hands them identical gradients forever.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  u/.style={circle, draw, minimum size=9mm, inner sep=0pt},
  twin/.style={circle, draw=red, thick, minimum size=9mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[u] (x1) at (0,1.2)  {$x_1$};
  \node[u] (x2) at (0,-1.2) {$x_2$};
  \node[twin, text=red] (h1) at (3,1.4)  {$h_1$};
  \node[twin, text=red] (h2) at (3,0)    {$h_2$};
  \node[twin, text=red] (h3) at (3,-1.4) {$h_3$};
  \node[u] (y) at (6,0) {$y$};
  \foreach \h in {h1,h2,h3} {
    \draw[->, red, thick] (x1) -- (\h);
    \draw[->, red, thick] (x2) -- (\h);
    \draw[->, black, thick] (\h) -- (y);
  }
  \node[red, fill=white, inner sep=1pt] at (3,0.7)  {$=$};
  \node[red, fill=white, inner sep=1pt] at (3,-0.7) {$=$};
  \node[red, align=center, font=\footnotesize] at (3,-2.4) {\texttt{equal rows give equal units}};
\end{tikzpicture}
$$

The gradient inherits the same equality. Writing $\delta^{(l)}_k=\partial \mathcal L/\partial z^{(l)}_k$ for the error signal, backpropagation gives $\partial\mathcal L/\partial W^{(l)}_{k,:}=\delta^{(l)}_k\,(a^{(l-1)})^{\top}$, and because units $i,j$ share both their outputs and (by the same collapse one layer up) their downstream weights, $\delta^{(l)}_i=\delta^{(l)}_j$. The update is therefore identical:

$$
W^{(l)}_{i,:}\gets W^{(l)}_{i,:}-\eta\,\delta^{(l)}_i (a^{(l-1)})^{\top}
=W^{(l)}_{j,:}-\eta\,\delta^{(l)}_j (a^{(l-1)})^{\top}\gets W^{(l)}_{j,:}.
$$

> **Theorem (Constant init never breaks symmetry).** If every weight in a layer
> is initialized to the same value and the activation is deterministic, then the
> units of that layer compute identical functions and receive identical gradients
> for the entire course of training, regardless of the data.

> **Proof.** By induction on training steps. At step $0$ the weight rows are equal
> by hypothesis, so $z^{(l)}_i=z^{(l)}_j$ and $a^{(l)}_i=a^{(l)}_j$ for every input.
> Assume the rows are equal at step $t$. Backprop computes
> $\delta^{(l)}_i=\phi'(z^{(l)}_i)\sum_k W^{(l+1)}_{k,i}\delta^{(l+1)}_k$; since
> $z^{(l)}_i=z^{(l)}_j$ and the column $W^{(l+1)}_{:,i}=W^{(l+1)}_{:,j}$ (equal by
> the inductive hypothesis applied to layer $l+1$), we get $\delta^{(l)}_i=\delta^{(l)}_j$.
> The gradient descent step subtracts the same quantity from each row, so the rows
> remain equal at step $t+1$. The bias term follows identically. $\qed$

The fix is randomness: sample each weight independently from a zero-mean
distribution. Independent draws break the row equality at step $0$, and from
there the units diverge under their distinct gradients. Biases _can_ safely start
at zero, since they do not enter the symmetry argument, because the weights already
distinguish the units. This leaves only the question of _scale_: from how wide a
distribution should we draw?[^gf-symmetry]

## Variance propagation: the forward pass

Scale is not free to choose. Each layer is a linear map followed by a
nonlinearity, and a linear map rescales variance. Track how the variance of the
activations changes from layer to layer and a hard constraint appears: to keep
the signal at a fixed scale across depth, the per-layer rescaling must be exactly $1$.

Assume at initialization that the weights $W^{(l)}_{ij}$ are i.i.d. zero-mean with
variance $\mathrm{Var}(W)$, independent of the inputs, and (for this first pass)
take the activation to be linear near zero, $\phi(z)\approx z$ (true for $\tanh$
about the origin). The bias starts at zero. A single pre-activation is a sum over
the fan-in,

$$
z^{(l)}_i=\sum_{j=1}^{n_{\text{in}}} W^{(l)}_{ij}\,a^{(l-1)}_j.
$$

Because the $W^{(l)}_{ij}$ are zero-mean and independent of the $a^{(l-1)}_j$,
the variance of a sum of independent products factors. Each weight is
independent, so cross terms vanish and the variances add:

$$
\mathrm{Var}(z^{(l)}_i)
=\sum_{j=1}^{n_{\text{in}}}\mathrm{Var}\!\parens{W^{(l)}_{ij}\,a^{(l-1)}_j}
=\sum_{j=1}^{n_{\text{in}}}\mathrm{Var}(W^{(l)}_{ij})\,\mathbb E\brackets{(a^{(l-1)}_j)^2}.
$$

The middle step uses $\mathrm{Var}(WA)=\mathrm{Var}(W)\,\mathbb E[A^2]$ for
independent zero-mean $W$. Assuming the activations of the previous layer are
identically distributed with variance $\mathrm{Var}(a^{(l-1)})$ and zero mean (so
$\mathbb E[A^2]=\mathrm{Var}(a^{(l-1)})$), the sum is $n_{\text{in}}$ copies of
the same term:

$$
\;\mathrm{Var}(z^{(l)})=n_{\text{in}}\,\mathrm{Var}(W)\,\mathrm{Var}(a^{(l-1)}).\;
$$

This is the **variance recursion**. The factor $n_{\text{in}}\,\mathrm{Var}(W)$ is
the gain by which each layer multiplies the signal's variance. Iterating across
$L$ layers compounds it:

$$
\mathrm{Var}(z^{(L)})=\parens{n_{\text{in}}\,\mathrm{Var}(W)}^{L}\,\mathrm{Var}(a^{(0)}).
$$

The factor compounds geometrically with depth. If the gain exceeds $1$
the variance _explodes_; if it falls below $1$ it _vanishes_; only a
gain of exactly $1$ keeps the signal at a fixed scale across arbitrary depth.

> **Theorem (Forward variance-preservation condition).** For a deep linear-regime
> network with i.i.d. zero-mean weights, the activation variance is preserved
> across layers (neither vanishing nor exploding with depth) if and only if
> $$
> n_{\text{in}}\,\mathrm{Var}(W)=1,
> \qquad\text{i.e.}\qquad
> \mathrm{Var}(W)=\frac{1}{n_{\text{in}}}.
> $$

> **Proof.** From the recursion $\mathrm{Var}(z^{(l)})=g\,\mathrm{Var}(a^{(l-1)})$
> with gain $g=n_{\text{in}}\,\mathrm{Var}(W)$, depth-$L$ propagation multiplies
> the variance by $g^{L}$. The sequence $g^{L}$ is bounded away from both $0$ and
> $\infty$ as $L\to\infty$ exactly when $g=1$; any $g<1$ sends $g^{L}\to 0$ and
> any $g>1$ sends $g^{L}\to\infty$. Solving $g=1$ for the weight variance gives
> $\mathrm{Var}(W)=1/n_{\text{in}}$. $\qed$

### A worked example: variance through five layers

For example, take a network of width
$n_{\text{in}}=100$ at every layer, unit-variance input $\mathrm{Var}(a^{(0)})=1$,
and the linear regime $\phi(z)\approx z$, and compare three weight scales. Sampling
$W$ from a Gaussian with the stated standard deviation $\sigma$ fixes
$\mathrm{Var}(W)=\sigma^2$, and the gain is $g=n_{\text{in}}\,\mathrm{Var}(W)=100\,\sigma^2$.

- **Too small**, $\sigma=0.05$: $\mathrm{Var}(W)=0.0025$, gain $g=0.25$. The
  variance after five layers is $0.25^5\approx 9.8\times10^{-4}$ — the signal has
  shrunk by three orders of magnitude.
- **Preserving**, $\sigma=\sqrt{1/100}=0.1$: $\mathrm{Var}(W)=0.01$, gain $g=1$.
  The variance stays at $1$ no matter how deep the stack.
- **Too large**, $\sigma=0.2$: $\mathrm{Var}(W)=0.04$, gain $g=4$. The variance
  after five layers is $4^5=1024$ — a thousandfold blow-up, and each additional
  layer multiplies it again.

The table traces $\mathrm{Var}(z^{(l)})$ layer by layer.

| Layer $l$ | $g=0.25$ (too small) | $g=1$ (preserving) | $g=4$ (too large) |
| --- | --- | --- | --- |
| $0$ | $1$ | $1$ | $1$ |
| $1$ | $0.25$ | $1$ | $4$ |
| $2$ | $0.0625$ | $1$ | $16$ |
| $3$ | $0.0156$ | $1$ | $64$ |
| $4$ | $0.0039$ | $1$ | $256$ |
| $5$ | $0.00098$ | $1$ | $1024$ |

Only the middle column stays stable with depth. The figure plots the same three trajectories
on a log scale, where the geometric decay and growth appear as straight lines fanning
away from the flat preserving line.

$$
% caption: Activation variance $\mathrm{Var}(z^{(l)})$ through five width-$100$
% layers on a $\log_{10}$ axis. The preserving gain $g=1$ holds a flat line; the
% gains $g=0.25$ and $g=4$ fall and rise as straight lines, since $\log g^{l}$ is
% linear in depth $l$.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % axes: x = layer 0..5 mapped to 0..6cm; y = log10 var from -4..4 mapped to 0..4cm (0.5cm per unit, origin at log=0 -> y=2)
  \draw[->, thick] (0,0) -- (6.6,0) node[right] {\texttt{layer} $l$};
  \draw[->, thick] (0,-0.2) -- (0,4.2) node[above] {$\log_{10}\mathrm{Var}(z)$};
  % gridline at log=0
  \draw[black, dashed] (0,2) -- (6,2);
  \node[font=\scriptsize, anchor=east] at (-0.05,2) {$0$};
  \node[font=\scriptsize, anchor=east] at (-0.05,3.5) {$3$};
  \node[font=\footnotesize, anchor=east] at (-0.05,0.5) {\texttt{-3}};
  % x ticks
  \foreach \l in {0,1,2,3,4,5} { \draw (\l*1.2,0.05) -- (\l*1.2,-0.05) node[below, font=\scriptsize] {$\l$}; }
  % preserving: log = 0 always -> y=2
  \draw[green, very thick] (0,2) -- (6,2);
  \node[green, font=\footnotesize, anchor=south] at (4.3,2.05) {\texttt{preserving} (\texttt{g=1})};
  % too large: log10(4^l) = l*0.602 ; y = 2 + 0.5*l*0.602 = 2 + 0.301 l ; at l=5 -> 3.505
  \draw[red, very thick] (0,2) -- (6,3.505);
  \node[red, font=\footnotesize, anchor=south east] at (5.9,3.55) {\texttt{too large} (\texttt{g=4})};
  % too small: log10(0.25^l) = -l*0.602 ; y = 2 - 0.301 l ; at l=5 -> 0.495
  \draw[acc, very thick] (0,2) -- (6,0.495);
  \node[acc, font=\footnotesize, anchor=north east] at (5.9,0.45) {\texttt{too small} (\texttt{g=0.25})};
\end{tikzpicture}
$$

## Variance propagation: the backward pass

The same calculation run on the gradient gives a _second_ condition, and the two
do not in general agree. Backpropagation pushes the error signal $\delta^{(l)}=\partial\mathcal L/\partial z^{(l)}$ through the _transpose_ of the weight matrix:

$$
\delta^{(l-1)}_j=\phi'(z^{(l-1)}_j)\sum_{i=1}^{n_{\text{out}}} W^{(l)}_{ij}\,\delta^{(l)}_i.
$$

The two passes traverse the same weight matrix in opposite directions and sum over
opposite fans. Forward, a unit collects $n_{\text{in}}$ incoming terms; backward, it
collects $n_{\text{out}}$ outgoing terms. That asymmetry is what splits the two
variance conditions.

$$
% caption: The same layer $W^{(l)}\in\mathbb R^{n_{\text{out}}\times n_{\text{in}}}$
% seen both ways. Forward (blue) sums $n_{\text{in}}$ terms into each output;
% backward (red) sums $n_{\text{out}}$ terms into each input. The gain is
% $n_{\text{in}}\mathrm{Var}(W)$ one way and $n_{\text{out}}\mathrm{Var}(W)$ the other.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  u/.style={circle, draw, minimum size=7mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % left column: 3 units (fan-in side)
  \node[u] (a1) at (0,1.6)  {};
  \node[u] (a2) at (0,0.4)  {};
  \node[u] (a3) at (0,-0.8) {};
  \node[font=\footnotesize, anchor=east] at (-0.5,0.4) {\texttt{prev layer}};
  \node[font=\footnotesize, anchor=north, align=center] at (0,-1.5) {$n_{\text{in}}$ \texttt{units}};
  % right column: 2 units (fan-out side)
  \node[u] (b1) at (4,1.0)  {};
  \node[u] (b2) at (4,-0.2) {};
  \node[font=\scriptsize, anchor=west] at (4.5,0.4) {$z^{(l)}$};
  \node[font=\footnotesize, anchor=north, align=center] at (4,-1.5) {$n_{\text{out}}$ \texttt{units}};
  % forward edges (blue)
  \foreach \a in {a1,a2,a3} \foreach \b in {b1,b2} { \draw[acc] (\a) -- (\b); }
  % forward arrow on top
  \draw[->, acc, very thick] (0.6,2.4) -- (3.4,2.4) node[midway, above, font=\footnotesize] {\texttt{forward: sum over} $n_{\text{in}}$};
  % backward arrow below
  \draw[->, red, very thick] (3.4,-2.4) -- (0.6,-2.4) node[midway, below, font=\footnotesize] {\texttt{backward: sum over} $n_{\text{out}}$};
\end{tikzpicture}
$$

In the linear regime $\phi'\approx 1$. The sum now ranges over the _fan-out_
$n_{\text{out}}$ (the number of units the signal flows back from), so the
identical variance argument yields

$$
\mathrm{Var}(\delta^{(l-1)})=n_{\text{out}}\,\mathrm{Var}(W)\,\mathrm{Var}(\delta^{(l)}),
$$

and across depth the gradient variance compounds by $\parens{n_{\text{out}}\,\mathrm{Var}(W)}^{L}$. Preserving _gradient_ scale therefore demands

$$
n_{\text{out}}\,\mathrm{Var}(W)=1,
\qquad\text{i.e.}\qquad
\mathrm{Var}(W)=\frac{1}{n_{\text{out}}}.
$$

The forward condition gives $1/n_{\text{in}}$; the backward condition gives $1/n_{\text{out}}$.
Both can be satisfied exactly only when $n_{\text{in}}=n_{\text{out}}$. The two
conditions sit side by side:

| Pass | Quantity tracked | Recursion gain | Variance condition |
| --- | --- | --- | --- |
| Forward | activation $\mathrm{Var}(z^{(l)})$ | $n_{\text{in}}\,\mathrm{Var}(W)$ | $\mathrm{Var}(W)=1/n_{\text{in}}$ |
| Backward | gradient $\mathrm{Var}(\delta^{(l)})$ | $n_{\text{out}}\,\mathrm{Var}(W)$ | $\mathrm{Var}(W)=1/n_{\text{out}}$ |

## Xavier / Glorot initialization

Glorot and Bengio resolved the conflict with a compromise: take the harmonic-style
average of the two targets so that _neither_ pass is badly served. Set the variance
to the reciprocal of the _mean_ fan, which gives

$$
\;\mathrm{Var}(W)=\frac{2}{n_{\text{in}}+n_{\text{out}}}.\;
$$

This is **Xavier** (or **Glorot**) initialization.[^gf-xavier] When $n_{\text{in}}=n_{\text{out}}$
it reduces to the exact $1/n$ both conditions require; otherwise it splits the
difference, keeping forward and backward gains both near $1$. Drawn from a uniform
distribution $U(-a,a)$, which has variance $a^2/3$, matching the target variance
fixes the bound:

$$
\frac{a^2}{3}=\frac{2}{n_{\text{in}}+n_{\text{out}}}
\;\Longrightarrow\;
a=\sqrt{\frac{6}{n_{\text{in}}+n_{\text{out}}}},
\qquad
W\sim U\!\parens{-\sqrt{\tfrac{6}{n_{\text{in}}+n_{\text{out}}}},\ \sqrt{\tfrac{6}{n_{\text{in}}+n_{\text{out}}}}}.
$$

> **Definition (Xavier/Glorot init).** Sample weights i.i.d. zero-mean with
> variance $2/(n_{\text{in}}+n_{\text{out}})$ — normal with that variance, or
> uniform on $\pm\sqrt{6/(n_{\text{in}}+n_{\text{out}})}$. Designed for activations
> that are linear near the origin and symmetric, such as $\tanh$ and the logistic
> sigmoid.

Xavier rests on $\phi'(0)\approx 1$, which holds for $\tanh$ but _fails badly_ for
the rectifier, and the rectifier is what modern networks use.

## He initialization: correcting for ReLU

[ReLU](/deep-learning/neural-networks/activation-functions) breaks the linear-regime
assumption: it zeros every negative pre-activation, discarding half the signal. The
fan-in derivation must be redone with that loss accounted for. For a symmetric
zero-mean pre-activation $z$, the rectifier $a=\max(0,z)$ keeps the positive half
and flattens the negative half to zero.

$$
% caption: ReLU halves the variance. A symmetric zero-mean input distribution
% (blue) loses its entire negative half to $\max(0,z)$ (shaded), leaving
% $\mathbb E[\max(0,z)^2]=\tfrac12\mathrm{Var}(z)$.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % axis
  \draw[->, thick] (-3.6,0) -- (3.8,0) node[right, font=\footnotesize] {\texttt{input} $z$};
  \draw[black, dashed] (0,-0.15) -- (0,2.3);
  % the symmetric bell (gaussian), height 2
  \draw[acc, very thick] plot[domain=-3.3:3.3, samples=90] (\x, {2*exp(-\x*\x/2)});
  % shade the kept (positive) half
  \fill[green!22] plot[domain=0:3.3, samples=60] (\x, {2*exp(-\x*\x/2)}) -- (3.3,0) -- (0,0) -- cycle;
  % mark the killed half
  \node[red, font=\footnotesize, align=center] at (-1.7,0.95) {\texttt{killed}\\\texttt{half}};
  \node[green, font=\footnotesize, align=center] at (1.55,0.95) {\texttt{kept}\\\texttt{half}};
  \node[font=\footnotesize, anchor=north] at (-1.7,-0.05) {$z<0$ \texttt{zeroed}};
  \node[font=\footnotesize, anchor=north] at (1.55,-0.05) {$z>0$ \texttt{kept}};
\end{tikzpicture}
$$

The factor of $\tfrac12$ is exact, not heuristic. For $z$ symmetric about zero,
the second moment of the rectified output splits into the negative half (which
contributes nothing) and the positive half (which by symmetry carries exactly half
the mass):

$$
\mathbb E\brackets{\max(0,z)^2}
=\int_{0}^{\infty} z^2\,p(z)\,dz
=\frac12\int_{-\infty}^{\infty} z^2\,p(z)\,dz
=\frac12\,\mathrm{Var}(z),
$$

using $\mathbb E[z]=0$ in the last equality. So $\mathrm{Var}(a^{(l)})=\tfrac12\mathrm{Var}(z^{(l)})$,
and the forward recursion picks up that factor:

$$
\mathrm{Var}(z^{(l)})
=n_{\text{in}}\,\mathrm{Var}(W)\cdot\underbrace{\tfrac12\,\mathrm{Var}(z^{(l-1)})}_{\mathrm{Var}(a^{(l-1)})}
=\frac{n_{\text{in}}\,\mathrm{Var}(W)}{2}\,\mathrm{Var}(z^{(l-1)}).
$$

Setting the gain $\tfrac12 n_{\text{in}}\mathrm{Var}(W)=1$ gives **He** (Kaiming)
initialization, the same derivation as Xavier with one extra factor of $2$ to
compensate for the variance ReLU removes:[^gf-he]

$$
\;\mathrm{Var}(W)=\frac{2}{n_{\text{in}}}.\;
$$

> **Definition (He/Kaiming init).** Sample weights i.i.d. zero-mean with variance
> $2/n_{\text{in}}$ — normal with standard deviation $\sqrt{2/n_{\text{in}}}$, or
> uniform on $\pm\sqrt{6/n_{\text{in}}}$. The factor $2$ compensates for ReLU
> zeroing half the pre-activations; it is the default for rectifier networks.

> **Remark (Where the 2 comes from).** Xavier's $2/(n_{\text{in}}+n_{\text{out}})$
> and He's $2/n_{\text{in}}$ both carry a $2$, but for unrelated reasons: Xavier's
> averages two fans; He's compensates for the half of the variance ReLU removes. Conflating them
> is a common error: using Xavier with ReLU shrinks activations by $\sqrt{1/2}$ per
> layer and they decay geometrically with depth.

## The catalogue of schemes

The same skeleton (zero mean, variance set by a fan count) generates every
standard scheme. They differ only in which fan they use and whether they correct
for a nonlinearity.

| Scheme | $\mathrm{Var}(W)$ | Activation | Normal form $\sigma$ | Uniform bound $a$ |
| --- | --- | --- | --- | --- |
| LeCun | $1/n_{\text{in}}$ | SELU, linear | $\sqrt{1/n_{\text{in}}}$ | $\sqrt{3/n_{\text{in}}}$ |
| Xavier / Glorot | $2/(n_{\text{in}}{+}n_{\text{out}})$ | $\tanh$, sigmoid | $\sqrt{2/(n_{\text{in}}{+}n_{\text{out}})}$ | $\sqrt{6/(n_{\text{in}}{+}n_{\text{out}})}$ |
| He / Kaiming | $2/n_{\text{in}}$ | ReLU, leaky ReLU | $\sqrt{2/n_{\text{in}}}$ | $\sqrt{6/n_{\text{in}}}$ |
| Orthogonal | (norm-preserving) | deep / recurrent | — | — |

The uniform bound is $a=\sqrt{3}\,\sigma$ throughout, since $U(-a,a)$ has variance
$a^2/3$. For a leaky rectifier with negative slope $\alpha$ the surviving fraction
is $(1+\alpha^2)/2$, so He generalizes to $\mathrm{Var}(W)=2/\parens{(1+\alpha^2)\,n_{\text{in}}}$.

For a square layer of width $256$ the three closed-form schemes place
$\mathrm{Var}(W)$ at $1/256\approx 0.0039$ (LeCun), $2/512\approx 0.0039$ (Xavier,
which coincides with LeCun here because $n_{\text{in}}=n_{\text{out}}$), and
$2/256\approx 0.0078$ (He). He is exactly twice the others, reflecting the
factor that compensates for ReLU's halving.

$$
% caption: $\mathrm{Var}(W)$ for each scheme on a square layer with
% $n_{\text{in}}=n_{\text{out}}=256$, in units of $10^{-3}$. LeCun and Xavier
% coincide at $1/256$; He doubles it to $2/256$ to offset ReLU's halving.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % y axis: value in units of 1e-3, 0..8 mapped to 0..3.6cm (0.45cm per unit)
  \draw[->, thick] (0,0) -- (0,3.9) node[above, font=\footnotesize] {\texttt{Var(W), units of 0.001}};
  \draw[thick] (0,0) -- (6.4,0);
  \foreach \v in {2,4,6,8} { \draw[black] (0,\v*0.45) -- (6.2,\v*0.45); \node[font=\scriptsize, anchor=east] at (-0.08,\v*0.45) {$\v$}; }
  % LeCun 3.9 -> 3.9*0.45=1.755
  \fill[acc!70] (0.5,0) rectangle (1.7,1.755);
  \node[font=\footnotesize, anchor=north] at (1.1,-0.1) {\texttt{LeCun}};
  \node[font=\footnotesize, anchor=south] at (1.1,1.8) {\texttt{3.9}};
  % Xavier 3.9
  \fill[acc!70] (2.3,0) rectangle (3.5,1.755);
  \node[font=\footnotesize, anchor=north] at (2.9,-0.1) {\texttt{Xavier}};
  \node[font=\footnotesize, anchor=south] at (2.9,1.8) {\texttt{3.9}};
  % He 7.8 -> 3.51
  \fill[green!75] (4.1,0) rectangle (5.3,3.51);
  \node[font=\footnotesize, anchor=north] at (4.7,-0.1) {\texttt{He}};
  \node[font=\footnotesize, anchor=south] at (4.7,3.55) {\texttt{7.8}};
\end{tikzpicture}
$$

## The failure mode: vanishing and exploding signal

When the per-layer gain departs from $1$, the consequence is visible at the output
as activation magnitude that decays toward zero or grows without bound across
depth, and the backward pass mirrors it exactly.

$$
% caption: Activation magnitude across depth. Gain below $1$ decays geometrically,
% gain above $1$ explodes, and the variance-preserving gain holds a flat scale.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, thick] (0,0) -- (7.2,0) node[right, font=\footnotesize] {\texttt{layer depth}};
  \draw[->, thick] (0,0) -- (0,3.5) node[above, font=\footnotesize] {\texttt{activation scale}};
  \draw[black, dashed] (0,1.6) -- (6.6,1.6);
  \node[font=\scriptsize, anchor=east] at (-0.1,1.6) {$1$};
  % good: flat near 1
  \draw[green, very thick] plot[domain=0:6.6, samples=40] (\x, {1.6 + 0.12*sin(\x*120)});
  \node[green, font=\footnotesize, anchor=south] at (5.2,1.78) {\texttt{good (gain 1)}};
  % too small: decays
  \draw[acc, very thick] plot[domain=0:6.6, samples=60] (\x, {1.6*exp(-0.42*\x)});
  \node[acc, font=\footnotesize, anchor=north] at (4.9,0.12) {\texttt{too small (decays)}};
  % too large: explodes (clip at 3.3)
  \draw[red, very thick] plot[domain=0:4.0, samples=60] (\x, {1.6*exp(0.34*\x)});
  \node[red, font=\footnotesize, anchor=south east] at (3.9,3.0) {\texttt{too large (explodes)}};
\end{tikzpicture}
$$

The backward pass behaves identically: a sub-unit gain shrinks the gradient
toward zero, so early layers stop learning (the **vanishing gradient**), while a
super-unit gain blows it up, so updates diverge (the **exploding gradient**).

$$
% caption: Gradient magnitude on the backward pass, plotted from output back to
% input. Bad init makes the gradient vanish (early layers learn nothing) or
% explode (updates diverge); good init keeps it flat.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, thick] (0,0) -- (7.2,0) node[right, font=\footnotesize] {\texttt{depth (output to input)}};
  \draw[->, thick] (0,0) -- (0,3.5) node[above, font=\footnotesize] {\texttt{gradient scale}};
  \draw[black, dashed] (0,1.6) -- (6.6,1.6);
  \node[font=\scriptsize, anchor=east] at (-0.1,1.6) {$1$};
  % good
  \draw[green, very thick] plot[domain=0:6.6, samples=40] (\x, {1.6 + 0.12*cos(\x*120)});
  \node[green, font=\footnotesize, anchor=south] at (5.2,1.78) {\texttt{good}};
  % vanishing
  \draw[acc, very thick] plot[domain=0:6.6, samples=60] (\x, {1.6*exp(-0.44*\x)});
  \node[acc, font=\footnotesize, anchor=north] at (4.7,0.78) {\texttt{vanishing gradient}};
  % exploding
  \draw[red, very thick] plot[domain=0:4.0, samples=60] (\x, {1.6*exp(0.34*\x)});
  \node[red, font=\footnotesize, anchor=south east] at (3.9,3.0) {\texttt{exploding gradient}};
\end{tikzpicture}
$$

The same instability has a distributional reading: the pre-activation _spread_ at
each layer should hold near unit variance. Good init keeps the histogram a fixed
width; bad init collapses it to a spike (dead units, no gradient) or fans it out
until the nonlinearity saturates (zero gradient at the tails).

$$
% caption: Pre-activation spread by layer. Good init holds unit variance
% (center); too-small init collapses the distribution toward zero (left);
% too-large init fans it out until activations saturate (right).
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % three groups of vertical spread bars at increasing depth
  % LEFT: collapsing (blue), shrinking bars
  \foreach \x/\h in {0/1.3, 0.55/0.8, 1.1/0.45, 1.65/0.22} {
    \draw[acc, very thick] (\x,{1.8-\h}) -- (\x,{1.8+\h});
    \draw[acc] (\x,{1.8-\h}) -- ++(0.12,0) (\x,{1.8+\h}) -- ++(0.12,0);
  }
  \node[acc, font=\footnotesize, align=center] at (0.8,-0.1) {\texttt{too small}\\\texttt{(collapses)}};
  % CENTER: stable (green), equal bars
  \foreach \x in {3.3,3.85,4.4,4.95} {
    \draw[green, very thick] (\x,{1.8-1.1}) -- (\x,{1.8+1.1});
    \draw[green] (\x,{1.8-1.1}) -- ++(0.12,0) (\x,{1.8+1.1}) -- ++(0.12,0);
  }
  \node[green, font=\footnotesize, align=center] at (4.1,-0.1) {\texttt{good}\\\texttt{(unit variance)}};
  % RIGHT: exploding (red), growing bars
  \foreach \x/\h in {6.6/0.4, 7.15/0.75, 7.7/1.2, 8.25/1.75} {
    \draw[red, very thick] (\x,{1.8-\h}) -- (\x,{1.8+\h});
    \draw[red] (\x,{1.8-\h}) -- ++(0.12,0) (\x,{1.8+\h}) -- ++(0.12,0);
  }
  \node[red, font=\footnotesize, align=center] at (7.4,-0.1) {\texttt{too large}\\\texttt{(saturates)}};
  % depth arrows under each group
  \draw[->, black, thin] (-0.1,3.4) -- (1.85,3.4) node[midway, above, font=\footnotesize, text=black] {\texttt{depth}};
  \draw[->, black, thin] (3.2,3.4) -- (5.15,3.4) node[midway, above, font=\footnotesize, text=black] {\texttt{depth}};
  \draw[->, black, thin] (6.5,3.4) -- (8.45,3.4) node[midway, above, font=\footnotesize, text=black] {\texttt{depth}};
\end{tikzpicture}
$$

## Beyond i.i.d. draws

The variance analysis fixes the _scale_ of independent draws but says nothing about
their _geometry_. Two refinements address the residual instability that remains in
very deep and recurrent nets.

> **Definition (Orthogonal init).** Initialize each weight matrix to a (scaled)
> orthogonal matrix $W$ with $W^{\top}W=I$, obtained from the QR factorization of a
> random Gaussian matrix. An orthogonal map preserves the norm of every vector it
> acts on (all its singular values equal $1$), so it preserves both activation and
> gradient norm _exactly_, not merely in expectation. Especially valuable for
> recurrent networks, where the same matrix is applied at every time step and any
> singular value off $1$ compounds over the sequence.

> **Definition (LSUV).** Layer-Sequential Unit-Variance initialization: start from
> an orthogonal init, then run one data minibatch forward and, layer by layer,
> rescale each weight matrix so that the measured output variance equals $1$. It
> replaces the analytic variance estimate with a _measured_ one, correcting for
> whatever the real activation statistics turn out to be.

| Method | Controls | Guarantee | Cost |
| --- | --- | --- | --- |
| Xavier / He | per-element variance | scale preserved in expectation | closed form |
| Orthogonal | singular values | norm preserved exactly | one QR per matrix |
| LSUV | measured layer variance | unit variance on a real batch | one forward pass |

## Initialization and normalization together

Good initialization sets the signal scale at step $0$; it cannot keep it set as
the weights move during training. The modern practice pairs a variance-preserving
init with a [normalization layer](/deep-learning/regularization/normalization)
(batch, layer, or group normalization) that _re-centers and re-scales_
activations at every step, not just the first. Normalization makes the network far
less sensitive to the exact init, but it does not remove the need for it: the very first forward
pass, before any normalization statistics have stabilized, still relies on a
sensible starting scale, and residual connections still need He-style init to keep
their skip paths near unit gain.[^gf-flow]

> **Remark (The combination, not the substitute).** He init plus a normalization
> layer is the default for deep convolutional and transformer networks. Init sets
> the scale of the first pass; normalization holds it for every pass after. Together they
> are what let networks of hundreds of layers train at all — and they are why the
> careful per-layer tuning of the pre-2015 era is no longer necessary.

## Initialization and training without normalization

The two closed-form schemes are named for two public papers, and the line of work
that follows them is what lets modern networks skip normalization entirely.

- **The two source papers.** The $2/(n_{\text{in}}+n_{\text{out}})$ rule is
  Glorot and Bengio's, derived from exactly the forward/backward variance argument
  above; the extra factor of $2$ for rectifiers is He et al.'s, and it was the
  change that first let plain (unnormalized) $30$-layer ReLU nets train.[^glorot][^he-paper]
- **Fixup and training without normalization.** Zhang, Dauphin, and Ma showed
  that a residual network can be trained to full accuracy with _no_ normalization
  layer at all, purely by rescaling the initialization of the residual branches
  ($1/\sqrt{L}$-style shrinkage across $L$ blocks). The signal-preservation goal
  of this lesson, pushed far enough, removes the need for the normalization layer
  the previous section paired with it.[^fixup]
- **Dynamical isometry.** Pennington, Schoenholz, and Ganguli made the
  orthogonal-init argument precise: controlling the entire _spectrum_ of the
  input-output Jacobian (not just its mean gain) — "dynamical isometry" — lets
  networks of $10{,}000$ layers train, far past what a variance-only argument can
  guarantee. It is the rigorous version of the "preserve the norm exactly" claim
  behind orthogonal init.[^isometry]
- **Init for transformers.** The warmup this module keeps invoking is partly an
  initialization patch: T-Fixup (Huang et al.) shows that scaling the initial
  weights of a transformer removes the need for a learning-rate warmup entirely,
  tying the two modules together — bad init and mandatory warmup are the same
  problem seen twice.[^tfixup]

In summary: break symmetry with independent zero-mean draws, set
the variance to $2/n_{\text{in}}$ for ReLU (or $2/(n_{\text{in}}+n_{\text{out}})$
for $\tanh$), use orthogonal init in recurrent stacks, and use
[normalization](/deep-learning/regularization/normalization) to hold the scale
steady as training proceeds.[^stevens-init] The next lesson studies the
[optimization landscape](/deep-learning/optimization/the-optimization-landscape)
that this well-scaled signal must now descend.

[^gf-init]: **Goodfellow**, _Deep Learning_, §8.4 — Parameter Initialization Strategies: why initialization sets whether deep training succeeds, and the heuristics that govern scale.
[^gf-symmetry]: **Goodfellow**, _Deep Learning_, §8.4 — the symmetry-breaking requirement: identical units receiving identical gradients never specialize, so random draws are mandatory.
[^gf-xavier]: **Goodfellow**, _Deep Learning_, §8.4 — the Glorot/Xavier normalized initialization $\mathrm{Var}(W)=2/(n_{\text{in}}+n_{\text{out}})$ as a compromise between forward and backward variance preservation.
[^gf-he]: **Goodfellow**, _Deep Learning_, §8.4 — rectifier-aware scaling: the extra factor of $2$ (He init) that refills the variance ReLU discards.
[^gf-flow]: **Goodfellow**, _Deep Learning_, §6.2, §8.2 — gradient flow and ill-conditioning: how per-layer gain off $1$ makes signal and gradient vanish or explode with depth, and why normalization complements init.
[^stevens-init]: **Stevens**, _Deep Learning with PyTorch_, Ch. 6 — initialization in practice with `torch.nn.init`: Xavier/Kaiming helpers and pairing them with normalization layers.
[^glorot]: **Glorot & Bengio**, _Understanding the Difficulty of Training Deep Feedforward Neural Networks_, AISTATS 2010 — the forward/backward variance argument and the $2/(n_{\text{in}}+n_{\text{out}})$ initialization.
[^he-paper]: **He, Zhang, Ren & Sun**, _Delving Deep into Rectifiers_, ICCV 2015 — the ReLU-aware factor of $2$ (He/Kaiming init) that enabled very deep rectifier networks.
[^fixup]: **Zhang, Dauphin & Ma**, _Fixup Initialization: Residual Learning Without Normalization_, ICLR 2019 — training deep residual nets to full accuracy by initialization alone, with no normalization layer.
[^isometry]: **Pennington, Schoenholz & Ganguli**, _Resurrecting the Sigmoid in Deep Learning through Dynamical Isometry_, NeurIPS 2017 — controlling the full Jacobian spectrum to train networks thousands of layers deep.
[^tfixup]: **Huang et al.**, _Improving Transformer Optimization Through Better Initialization_, ICML 2020 — T-Fixup: rescaling transformer init to remove the learning-rate warmup requirement.