Weight Initialization

Gradient descent finds a direction to step; it cannot choose where to start. That choice is weight initialization, and for a deep network it matters: it sets whether the very first forward pass carries usable signal to the output and whether the very first backward pass carries usable gradient to the input. Two distinct failures follow from a careless choice: a symmetry that no gradient can break, and a scale that makes the signal vanish or explode with depth. This lesson derives the single variance condition that prevents both.¹

We work with a plain feedforward net. Layer $l$ takes input $a^{(l - 1)} \in R^{n_{l - 1}}$ , computes the pre-activation $z^{(l)} = W^{(l)} a^{(l - 1)} + b^{(l)}$ , and applies an elementwise nonlinearity $a^{(l)} = ϕ (z^{(l)})$ . We write $n_{in} = n_{l - 1}$ for the fan-in and $n_{out} = n_{l}$ for the fan-out of layer $l$ .

The symmetry-breaking problem

The most obvious initialization — set every weight and bias to zero — fails before training begins. Suppose $W^{(l)} = 0$ and $b^{(l)} = 0$ for all $l$ . Then every pre-activation in a layer is identical, so every unit computes the same activation, and backpropagation sends every unit the same gradient. Identical units that receive identical updates stay identical forever.

The argument is not limited to zero; it holds for any constant. Take two units $i, j$ in layer $l$ with equal incoming weight rows, $W_{i, :}^{(l)} = W_{j, :}^{(l)}$ , and equal biases. Then for every input their pre-activations agree, $z_{i}^{(l)} = z_{j}^{(l)}$ , so $a_{i}^{(l)} = a_{j}^{(l)}$ .

Constant init collapses a layer: equal weight rows give two units the same pre-activation, so backprop hands them identical gradients forever.

The gradient inherits the same equality. Writing $δ_{k}^{(l)} = \partial L / \partial z_{k}^{(l)}$ for the error signal, backpropagation gives $\partial L / \partial W_{k, :}^{(l)} = δ_{k}^{(l)} (a^{(l - 1)})^{⊤}$ , and because units $i, j$ share both their outputs and (by the same collapse one layer up) their downstream weights, $δ_{i}^{(l)} = δ_{j}^{(l)}$ . The update is therefore identical:

W_{i, :}^{(l)} \leftarrow W_{i, :}^{(l)} - η δ_{i}^{(l)} (a^{(l - 1)})^{⊤} = W_{j, :}^{(l)} - η δ_{j}^{(l)} (a^{(l - 1)})^{⊤} \leftarrow W_{j, :}^{(l)} .

The fix is randomness: sample each weight independently from a zero-mean distribution. Independent draws break the row equality at step $0$ , and from there the units diverge under their distinct gradients. Biases can safely start at zero, since they do not enter the symmetry argument, because the weights already distinguish the units. This leaves only the question of scale: from how wide a distribution should we draw?²

Variance propagation: the forward pass

Scale is not free to choose. Each layer is a linear map followed by a nonlinearity, and a linear map rescales variance. Track how the variance of the activations changes from layer to layer and a hard constraint appears: to keep the signal at a fixed scale across depth, the per-layer rescaling must be exactly $1$ .

Assume at initialization that the weights $W_{ij}^{(l)}$ are i.i.d. zero-mean with variance $Var (W)$ , independent of the inputs, and (for this first pass) take the activation to be linear near zero, $ϕ (z) \approx z$ (true for $tanh$ about the origin). The bias starts at zero. A single pre-activation is a sum over the fan-in,

z_{i}^{(l)} = j = 1 \sum n_{in} W_{ij}^{(l)} a_{j}^{(l - 1)} .

Because the $W_{ij}^{(l)}$ are zero-mean and independent of the $a_{j}^{(l - 1)}$ , the variance of a sum of independent products factors. Each weight is independent, so cross terms vanish and the variances add:

Var (z_{i}^{(l)}) = j = 1 \sum n_{in} Var (W_{ij}^{(l)} a_{j}^{(l - 1)}) = j = 1 \sum n_{in} Var (W_{ij}^{(l)}) E [(a_{j}^{(l - 1)})^{2}] .

The middle step uses $Var (W A) = Var (W) E [A^{2}]$ for independent zero-mean $W$ . Assuming the activations of the previous layer are identically distributed with variance $Var (a^{(l - 1)})$ and zero mean (so $E [A^{2}] = Var (a^{(l - 1)})$ ), the sum is $n_{in}$ copies of the same term:

Var (z^{(l)}) = n_{in} Var (W) Var (a^{(l - 1)}) .

This is the variance recursion. The factor $n_{in} Var (W)$ is the gain by which each layer multiplies the signal's variance. Iterating across $L$ layers compounds it:

Var (z^{(L)}) = (n_{in} Var (W))^{L} Var (a^{(0)}) .

The factor compounds geometrically with depth. If the gain exceeds $1$ the variance explodes; if it falls below $1$ it vanishes; only a gain of exactly $1$ keeps the signal at a fixed scale across arbitrary depth.

A worked example: variance through five layers

For example, take a network of width $n_{in} = 100$ at every layer, unit-variance input $Var (a^{(0)}) = 1$ , and the linear regime $ϕ (z) \approx z$ , and compare three weight scales. Sampling $W$ from a Gaussian with the stated standard deviation $σ$ fixes $Var (W) = σ^{2}$ , and the gain is $g = n_{in} Var (W) = 100 σ^{2}$ .

Too small, $σ = 0.05$ : $Var (W) = 0.0025$ , gain $g = 0.25$ . The variance after five layers is $0.2 5^{5} \approx 9.8 \times 1 0^{- 4}$ — the signal has shrunk by three orders of magnitude.
Preserving, $σ = 1/100 = 0.1$ : $Var (W) = 0.01$ , gain $g = 1$ . The variance stays at $1$ no matter how deep the stack.
Too large, $σ = 0.2$ : $Var (W) = 0.04$ , gain $g = 4$ . The variance after five layers is $4^{5} = 1024$ — a thousandfold blow-up, and each additional layer multiplies it again.

The table traces $Var (z^{(l)})$ layer by layer.

Layer $l$	$g = 0.25$ (too small)	$g = 1$ (preserving)	$g = 4$ (too large)
$0$	$1$	$1$	$1$
$1$	$0.25$	$1$	$4$
$2$	$0.0625$	$1$	$16$
$3$	$0.0156$	$1$	$64$
$4$	$0.0039$	$1$	$256$
$5$	$0.00098$	$1$	$1024$

Only the middle column stays stable with depth. The figure plots the same three trajectories on a log scale, where the geometric decay and growth appear as straight lines fanning away from the flat preserving line.

Activation variance

Var (z^{(l)})

through five width-

100

layers on a

log_{10}

axis. The preserving gain

g = 1

holds a flat line; the gains

g = 0.25

and

g = 4

fall and rise as straight lines, since

log g^{l}

is linear in depth

l

Variance propagation: the backward pass

The same calculation run on the gradient gives a second condition, and the two do not in general agree. Backpropagation pushes the error signal $δ^{(l)} = \partial L / \partial z^{(l)}$ through the transpose of the weight matrix:

δ_{j}^{(l - 1)} = ϕ^{'} (z_{j}^{(l - 1)}) i = 1 \sum n_{out} W_{ij}^{(l)} δ_{i}^{(l)} .

The two passes traverse the same weight matrix in opposite directions and sum over opposite fans. Forward, a unit collects $n_{in}$ incoming terms; backward, it collects $n_{out}$ outgoing terms. That asymmetry is what splits the two variance conditions.

The same layer

W^{(l)} \in R^{n_{out} \times n_{in}}

seen both ways. Forward (blue) sums

n_{in}

terms into each output; backward (red) sums

n_{out}

terms into each input. The gain is

n_{in} Var (W)

one way and

n_{out} Var (W)

the other.

In the linear regime $ϕ^{'} \approx 1$ . The sum now ranges over the fan-out $n_{out}$ (the number of units the signal flows back from), so the identical variance argument yields

Var (δ^{(l - 1)}) = n_{out} Var (W) Var (δ^{(l)}),

and across depth the gradient variance compounds by $(n_{out} Var (W))^{L}$ . Preserving gradient scale therefore demands

n_{out} Var (W) = 1, i.e. Var (W) = \frac{1}{n _{out}} .

The forward condition gives $1/ n_{in}$ ; the backward condition gives $1/ n_{out}$ . Both can be satisfied exactly only when $n_{in} = n_{out}$ . The two conditions sit side by side:

Pass	Quantity tracked	Recursion gain	Variance condition
Forward	activation $Var (z^{(l)})$	$n_{in} Var (W)$	$Var (W) = 1/ n_{in}$
Backward	gradient $Var (δ^{(l)})$	$n_{out} Var (W)$	$Var (W) = 1/ n_{out}$

Xavier / Glorot initialization

Glorot and Bengio resolved the conflict with a compromise: take the harmonic-style average of the two targets so that neither pass is badly served. Set the variance to the reciprocal of the mean fan, which gives

Var (W) = \frac{2}{n _{in} + n _{out}} .

This is Xavier (or Glorot) initialization.³ When $n_{in} = n_{out}$ it reduces to the exact $1/ n$ both conditions require; otherwise it splits the difference, keeping forward and backward gains both near $1$ . Drawn from a uniform distribution $U (- a, a)$ , which has variance $a^{2} /3$ , matching the target variance fixes the bound:

\frac{a ^{2}}{3} = \frac{2}{n _{in} + n _{out}} ⟹ a = \frac{6}{n _{in} + n _{out}}, W \sim U (- \frac{6}{n _{in} + n _{out}}, \frac{6}{n _{in} + n _{out}}) .

Xavier rests on $ϕ^{'} (0) \approx 1$ , which holds for $tanh$ but fails badly for the rectifier, and the rectifier is what modern networks use.

He initialization: correcting for ReLU

ReLU breaks the linear-regime assumption: it zeros every negative pre-activation, discarding half the signal. The fan-in derivation must be redone with that loss accounted for. For a symmetric zero-mean pre-activation $z$ , the rectifier $a = max (0, z)$ keeps the positive half and flattens the negative half to zero.

ReLU halves the variance. A symmetric zero-mean input distribution (blue) loses its entire negative half to

max (0, z)

(shaded), leaving

E [max (0, z)^{2}] = \frac{1}{2} Var (z)

The factor of $\frac{1}{2}$ is exact, not heuristic. For $z$ symmetric about zero, the second moment of the rectified output splits into the negative half (which contributes nothing) and the positive half (which by symmetry carries exactly half the mass):

E [max (0, z)^{2}] = 0 \int \infty z^{2} p (z) d z = \frac{1}{2} - \infty \int \infty z^{2} p (z) d z = \frac{1}{2} Var (z),

using $E [z] = 0$ in the last equality. So $Var (a^{(l)}) = \frac{1}{2} Var (z^{(l)})$ , and the forward recursion picks up that factor:

Var (z^{(l)}) = n_{in} Var (W) \cdot Var (a^{(l - 1)}) \frac{1}{2} Var (z^{(l - 1)}) = \frac{n _{in} Var ( W )}{2} Var (z^{(l - 1)}) .

Setting the gain $\frac{1}{2} n_{in} Var (W) = 1$ gives He (Kaiming) initialization, the same derivation as Xavier with one extra factor of $2$ to compensate for the variance ReLU removes:⁴

Var (W) = \frac{2}{n _{in}} .

The catalogue of schemes

The same skeleton (zero mean, variance set by a fan count) generates every standard scheme. They differ only in which fan they use and whether they correct for a nonlinearity.

Scheme	$Var (W)$	Activation	Normal form $σ$	Uniform bound $a$
LeCun	$1/ n_{in}$	SELU, linear	$1/ n_{in}$	$3/ n_{in}$
Xavier / Glorot	$2/ (n_{in} + n_{out})$	$tanh$ , sigmoid	$2/ (n_{in} + n_{out})$	$6/ (n_{in} + n_{out})$
He / Kaiming	$2/ n_{in}$	ReLU, leaky ReLU	$2/ n_{in}$	$6/ n_{in}$
Orthogonal	(norm-preserving)	deep / recurrent	—	—

The uniform bound is $a = 3 σ$ throughout, since $U (- a, a)$ has variance $a^{2} /3$ . For a leaky rectifier with negative slope $α$ the surviving fraction is $(1 + α^{2}) /2$ , so He generalizes to $Var (W) = 2/ ((1 + α^{2}) n_{in})$ .

For a square layer of width $256$ the three closed-form schemes place $Var (W)$ at $1/256 \approx 0.0039$ (LeCun), $2/512 \approx 0.0039$ (Xavier, which coincides with LeCun here because $n_{in} = n_{out}$ ), and $2/256 \approx 0.0078$ (He). He is exactly twice the others, reflecting the factor that compensates for ReLU's halving.

Var (W)

for each scheme on a square layer with

n_{in} = n_{out} = 256

, in units of

1 0^{- 3}

. LeCun and Xavier coincide at

1/256

; He doubles it to

2/256

to offset ReLU's halving.

The failure mode: vanishing and exploding signal

When the per-layer gain departs from $1$ , the consequence is visible at the output as activation magnitude that decays toward zero or grows without bound across depth, and the backward pass mirrors it exactly.

Activation magnitude across depth. Gain below

1

decays geometrically, gain above

1

explodes, and the variance-preserving gain holds a flat scale.

The backward pass behaves identically: a sub-unit gain shrinks the gradient toward zero, so early layers stop learning (the vanishing gradient), while a super-unit gain blows it up, so updates diverge (the exploding gradient).

Gradient magnitude on the backward pass, plotted from output back to input. Bad init makes the gradient vanish (early layers learn nothing) or explode (updates diverge); good init keeps it flat.

The same instability has a distributional reading: the pre-activation spread at each layer should hold near unit variance. Good init keeps the histogram a fixed width; bad init collapses it to a spike (dead units, no gradient) or fans it out until the nonlinearity saturates (zero gradient at the tails).

Pre-activation spread by layer. Good init holds unit variance (center); too-small init collapses the distribution toward zero (left); too-large init fans it out until activations saturate (right).

Beyond i.i.d. draws

The variance analysis fixes the scale of independent draws but says nothing about their geometry. Two refinements address the residual instability that remains in very deep and recurrent nets.

Method	Controls	Guarantee	Cost
Xavier / He	per-element variance	scale preserved in expectation	closed form
Orthogonal	singular values	norm preserved exactly	one QR per matrix
LSUV	measured layer variance	unit variance on a real batch	one forward pass

Initialization and normalization together

Good initialization sets the signal scale at step $0$ ; it cannot keep it set as the weights move during training. The modern practice pairs a variance-preserving init with a normalization layer (batch, layer, or group normalization) that re-centers and re-scales activations at every step, not just the first. Normalization makes the network far less sensitive to the exact init, but it does not remove the need for it: the very first forward pass, before any normalization statistics have stabilized, still relies on a sensible starting scale, and residual connections still need He-style init to keep their skip paths near unit gain.⁵

Initialization and training without normalization

The two closed-form schemes are named for two public papers, and the line of work that follows them is what lets modern networks skip normalization entirely.

The two source papers. The $2/ (n_{in} + n_{out})$ rule is Glorot and Bengio's, derived from exactly the forward/backward variance argument above; the extra factor of $2$ for rectifiers is He et al.'s, and it was the change that first let plain (unnormalized) $30$ -layer ReLU nets train.⁶⁷
Fixup and training without normalization. Zhang, Dauphin, and Ma showed that a residual network can be trained to full accuracy with no normalization layer at all, purely by rescaling the initialization of the residual branches ( $1/ L$ -style shrinkage across $L$ blocks). The signal-preservation goal of this lesson, pushed far enough, removes the need for the normalization layer the previous section paired with it.⁸
Dynamical isometry. Pennington, Schoenholz, and Ganguli made the orthogonal-init argument precise: controlling the entire spectrum of the input-output Jacobian (not just its mean gain) — dynamical isometry — lets networks of $10, 000$ layers train, far past what a variance-only argument can guarantee. It is the rigorous version of the preserve the norm exactly claim behind orthogonal init.⁹
Init for transformers. The warmup this module keeps invoking is partly an initialization patch: T-Fixup (Huang et al.) shows that scaling the initial weights of a transformer removes the need for a learning-rate warmup entirely, tying the two modules together — bad init and mandatory warmup are the same problem seen twice.¹⁰

In summary: break symmetry with independent zero-mean draws, set the variance to $2/ n_{in}$ for ReLU (or $2/ (n_{in} + n_{out})$ for $tanh$ ), use orthogonal init in recurrent stacks, and use normalization to hold the scale steady as training proceeds.¹¹ The next lesson studies the optimization landscape that this well-scaled signal must now descend.

Goodfellow, Deep Learning, §8.4 — Parameter Initialization Strategies: why initialization sets whether deep training succeeds, and the heuristics that govern scale. ↩
Goodfellow, Deep Learning, §8.4 — the symmetry-breaking requirement: identical units receiving identical gradients never specialize, so random draws are mandatory. ↩
Goodfellow, Deep Learning, §8.4 — the Glorot/Xavier normalized initialization $Var (W) = 2/ (n_{in} + n_{out})$ as a compromise between forward and backward variance preservation. ↩
Goodfellow, Deep Learning, §8.4 — rectifier-aware scaling: the extra factor of $2$ (He init) that refills the variance ReLU discards. ↩
Goodfellow, Deep Learning, §6.2, §8.2 — gradient flow and ill-conditioning: how per-layer gain off $1$ makes signal and gradient vanish or explode with depth, and why normalization complements init. ↩
Glorot & Bengio, Understanding the Difficulty of Training Deep Feedforward Neural Networks, AISTATS 2010 — the forward/backward variance argument and the $2/ (n_{in} + n_{out})$ initialization. ↩
He, Zhang, Ren & Sun, Delving Deep into Rectifiers, ICCV 2015 — the ReLU-aware factor of $2$ (He/Kaiming init) that enabled very deep rectifier networks. ↩
Zhang, Dauphin & Ma, Fixup Initialization: Residual Learning Without Normalization, ICLR 2019 — training deep residual nets to full accuracy by initialization alone, with no normalization layer. ↩
Pennington, Schoenholz & Ganguli, Resurrecting the Sigmoid in Deep Learning through Dynamical Isometry, NeurIPS 2017 — controlling the full Jacobian spectrum to train networks thousands of layers deep. ↩
Huang et al., Improving Transformer Optimization Through Better Initialization, ICML 2020 — T-Fixup: rescaling transformer init to remove the learning-rate warmup requirement. ↩
Stevens, Deep Learning with PyTorch, Ch. 6 — initialization in practice with torch.nn.init: Xavier/Kaiming helpers and pairing them with normalization layers. ↩