Backpropagation

The multilayer perceptron gave us a forward pass: inputs flow up through affine maps and nonlinearities to a scalar loss. Training needs the reverse: how the loss responds to every weight, ${\nabla }_{\theta }\mathcal{L}$. Backpropagation computes that gradient exactly, in a single backward sweep, by applying the chain rule to the training loop's forward computation read as a graph. It is not a learning algorithm; it is the differentiation algorithm that feeds the optimizer.¹

The computational graph

Every model is a composition of primitive operations, and that composition has a shape: a directed acyclic graph whose nodes compute and whose edges carry values.

Nodes are operations; edges are tensors. Take the single-example squared error of a linear unit, $\ell =(wx+b-y{)}^2$. Its graph threads $w,x,b,y$ through a multiply, two adds, and a square — and once forward values are pinned to the nodes, backprop annotates each edge with a gradient flowing the other way.

The forward pass labels every edge with a value; the backward pass labels every edge with the loss's sensitivity to that value. The two passes share the same graph and traverse it in opposite topological directions.

The chain rule for vector functions

A node rarely carries a scalar; it carries a tensor. So the local derivative is not a number but a Jacobian, and the chain rule multiplies Jacobians.

For a deep net the output of interest is the scalar loss $\ell$, so the quantity we propagate is the gradient row vector $\partial \ell /\partial y$. Pushing it through a node $z=f(y)$ requires not the full Jacobian $J$ but the product of the incoming gradient with $J$ — the vector-Jacobian product.²

The VJP is the core operation: each operation supplies a rule for turning its output gradient into its input gradient, without ever materializing the (possibly huge) Jacobian $J$. For a matrix multiply $z=Wy$, $J=W$ and the VJP is just $\bar{y}=W^{\top }\bar{z}$, a second matrix multiply, transposed.

For a single node the rule is: multiply the upstream gradient by the local derivative to get the downstream gradient. Backprop repeats this step at every node.

The four backprop equations

Now specialize to an MLP. Layers $l=1,\dots ,L$ compute pre-activations and activations

with a per-example loss $\ell \left(h^{(L)},y\right)$ (the output $\hat{y}=h^{(L)}$, or $z^{(L)}$ for a linear final layer). Define the error of layer $l$ as the gradient of the loss with respect to that layer's pre-activation,

Everything reduces to computing the ${\delta }^{(l)}$. The four equations below do it recursively, then read the parameter gradients off the errors.

Two facts make these equations the entire algorithm. First, the recursion BP2 is the VJP of the affine-plus-activation block, reused at every layer — the same two matrix operations, transposed. Second, BP4 needs the cached activation $h^{(l-1)}$ from the forward pass, which is why the forward pass must store its intermediates.³

Every shape, checked

The four equations are easy to write and easy to get wrong by a transpose. Pin down every shape and the algebra becomes mechanical. Let layer $l$ have $n_l$ units and $n_{l-1}$ inputs, so $W^{(l)}\in {\mathbb{R}}^{n_l\times n_{l-1}}$ and $b^{(l)}\in {\mathbb{R}}^{n_l}$. Every activation, pre-activation, and error at layer $l$ is a vector of length $n_l$; every weight gradient matches its weight.

BP2 reads left to right as a shape contraction: the $(n_l\times n_{l+1})$ transpose $W^{(l+1)\top }$ hits the $(n_{l+1}\times 1)$ upstream error ${\delta }^{(l+1)}$ to give an $(n_l\times 1)$ vector, which the Hadamard product with $g^{\prime }(z^{(l)})\in {\mathbb{R}}^{n_l}$ leaves at $n_l\times 1$ — exactly the shape of ${\delta }^{(l)}$. BP4 is an outer product: the $(n_l\times 1)$ column ${\delta }^{(l)}$ times the $(1\times n_{l-1})$ row $h^{(l-1)\top }$ yields the $(n_l\times n_{l-1})$ matrix that matches $W^{(l)}$ entry for entry. A single rule catches most sign-and- transpose bugs: every gradient carries the shape of the thing it differentiates, $\partial \ell /\partial W^{(l)}$ is shaped like $W^{(l)}$, ${\delta }^{(l)}$ is shaped like $z^{(l)}$.

The batched form

Training runs on a minibatch of $B$ examples at once, not one vector. Stack the $B$ activations of layer $l$ as columns of a matrix $H^{(l)}\in {\mathbb{R}}^{n_l\times B}$ (some frameworks use rows; the transposes flip but the content is the same). The forward pass becomes a matrix product with a broadcast bias, and the backward pass sums the per-example gradients — the reason a minibatch gradient is an average over examples.

with $X\in {\mathbb{R}}^{n_0\times B}$ and $1\in {\mathbb{R}}^B$ the all-ones vector that broadcasts the bias across columns. The error matrix ${\Delta }^{(l)}\in {\mathbb{R}}^{n_l\times B}$ holds one $\delta$ per column, and the batched equations are the single-example ones with the outer product replaced by a matrix product that sums over the batch axis:

The weight gradient ${\Delta }^{(l)}{H}^{(l-1)\top }$ contracts the $(n_l\times B)$ error against the $(B\times n_{l-1})$ transposed activations, and the shared batch axis $B$ vanishes — the matrix product is the sum ${\sum }_{i=1}^B{\delta }_i^{(l)}{h}_i^{(l-1)\top }$ over examples. The bias gradient ${\Delta }^{(l)}1$ is that same sum over the batch axis. Dividing by $B$ (or folding $1/B$ into $\mathcal{L}$) turns the summed gradient into the mean gradient the optimizer expects.

The dimensions flow one direction on the forward pass and the mirror direction on the backward pass, with the batch axis $B$ carried along both ways.

The error ${\delta }^{(l)}$ flows right to left across the layers, the mirror of the forward signal. Each layer turns the incoming error into the outgoing one by a transposed weight multiply and a gate by $g^{\prime }$.

Forward and backward passes

The two passes are one algorithm in two directions. The forward pass evaluates the graph in topological order and caches every pre-activation $z^{(l)}$ and activation $h^{(l)}$; the backward pass walks the layers in reverse, applying BP1–BP4.

In modern frameworks both passes are automatic. As Stevens describes, PyTorch's autograd records each operation into a graph as the forward pass runs; calling .backward() on the scalar loss replays that tape in reverse, evaluating one VJP per recorded op. The four equations are what autograd executes: you never hand-derive them, yet they are the rules autograd applies underneath.

A worked example

Take a concrete two-layer net: input $x=(1,0.5)$, hidden layer of two sigmoid units, a single linear output unit, target $y=0$, and squared loss $\ell =\frac{1}{2}(\hat{y}-y{)}^2$. Parameters:

Forward pass. Compute $z^{(1)}=W^{(1)}x+b^{(1)}$, gate through $\sigma (z)=1/(1+e^{-z})$, then the linear readout $\hat{y}=z^{(2)}=W^{(2)}{h}^{(1)}+b^{(2)}$.

Backward pass. The output is linear, so $g^{\prime }(z^{(2)})=1$ and BP1 gives ${\delta }^{(2)}=(\hat{y}-y)\cdot 1=0.2083$. BP4/BP3 read off the layer-2 gradients as ${\delta }^{(2)}{h}^{(1)\top }$ and ${\delta }^{(2)}$. BP2 then backs the error into the hidden layer using ${\sigma }^{\prime }(z)=h(1-h)$, and BP4/BP3 again yield the layer-1 gradients.

Every gradient is now explicit, and a single descent step $\theta \leftarrow \theta -\eta {\nabla }_{\theta }\ell$ would nudge all six tensors. Notice the hidden-layer gradients are an order of magnitude smaller than the output's — the ${\sigma }^{\prime }$ factor, capped at $0.25$, already shrinks the signal. Stack many sigmoid layers and those factors multiply toward zero: the vanishing gradient that motivates ReLU and careful initialization.

The backward pass is a chain of three multiplications seeded at the loss. The scalar ${\delta }^{(2)}=0.2083$ enters the hidden layer through the transposed weight, $W^{(2)\top }{\delta }^{(2)}$, then the sigmoid gate ${\sigma }^{\prime }(z^{(1)})$ scales each component, producing ${\delta }^{(1)}$. The figure traces those numbers end to end.

Gradient checking

A hand-derived or hand-coded backward pass is easy to get wrong by a transpose or a missing $g^{\prime }$ factor. Gradient checking verifies it against the definition of the derivative: perturb one parameter by a tiny $ϵ$, remeasure the loss, and compare the finite difference to the analytic gradient.

The centered difference is not optional. The one-sided form $(\ell (\theta +ϵe_i)-\ell (\theta ))/ϵ$ carries $O(ϵ)$ error; the centered form cancels the first-order term and drops to $O({ϵ}^2)$, gaining several digits of agreement for the same $ϵ$. Check on the worked net above: at $W_1^{(2)}=0.50$, bumping to $0.50\pm 10^{-4}$ and remeasuring $\ell$ gives a finite difference of $0.1222$, matching the analytic $\partial \ell /\partial W_1^{(2)}=0.1222$ from BP4 to four digits.

Gradient checking is a debugging tool, not a training step — one finite difference per parameter costs a full forward pass, so it runs on a small net, on a handful of parameters, once. Practical cautions carry over from Goodfellow: use double precision, avoid $ϵ$ so small that rounding dominates ($10^{-4}$ to $10^{-6}$ is the usable band), and check a few parameters rather than all of them. Turn it off before training; it is the unit test for the backward pass, not part of it.⁴

Reverse mode versus forward mode

Backprop is one of two ways to evaluate a chain of Jacobians. Forward mode multiplies them input-to-output, propagating a directional derivative; reverse mode multiplies output-to-input, propagating a gradient. For a composition ${\mathbb{R}}^n\to \cdots \to {\mathbb{R}}^m$ the cost is set by which end you start from.

A neural network has $n$ in the millions (parameters) and $m=1$ (the scalar loss). Reverse mode delivers the entire gradient in a single backward pass; forward mode would need one pass per parameter. That asymmetry is the whole reason backpropagation — not forward-mode differentiation — trains deep nets.⁵

Computational cost

Backprop is cheap in time and expensive in memory, a trade every deep-learning system must manage.

The forward pass writes the cache; the backward pass consumes it. The timeline below shows the schedule, with the cache as the shared store.

When memory is the binding constraint, gradient checkpointing trades it back for time: store only a sparse subset of activations on the forward pass and recompute the rest during the backward pass. Caching every $\sqrt{L}$-th layer cuts activation memory from $O(L)$ to $O(\sqrt{L})$ at the cost of one extra forward segment — the standard lever for training networks too deep to cache whole.⁶

Chain rule as a product of Jacobians

Zoom out and the whole network is one long composition, and its gradient is the product of per-layer Jacobians, read right to left, which is reverse mode.

Associativity lets us choose the multiplication order, and that choice is the difference between forward and reverse mode. Multiplying left-to-right keeps a running row vector (a gradient) — cheap, because every partial product stays a vector. Multiplying right-to-left would build full matrices. Reverse mode is simply the left-to-right grouping of this product.

Autodiff as a compiler

Backprop predates deep learning, and modern implementations treat it as a compiler problem rather than a hand derivation. Three developments extend Goodfellow's chapter.

The algorithm predates the 1986 paper. Reverse-mode automatic differentiation was described by Linnainmaa in 1970 as the reverse accumulation of rounding errors, and Werbos applied it to networks in his 1974 thesis. What made it famous was Rumelhart, Hinton & Williams's 1986 Nature paper, which showed backprop learning useful internal representations in a working network — the demonstration, not the derivative, was the contribution. The four equations here are the same reverse sweep those authors ran by hand.

Autodiff is now define-by-run. Modern frameworks split into two styles. The tape-based / define-by-run style (PyTorch autograd) records each operation as the forward pass executes and replays the tape backward, which the worked example above mirrors. The traced / transform style (JAX) treats differentiation as a program transformation: grad rewrites a function into the function that computes its gradient, and composable transforms like vmap (auto-batching) and jit (fusion) stack on top. Both compute exactly the VJPs of this lesson; they differ only in when the graph is built. In practice no one writes BP1–BP4 by hand anymore, but the framework runs precisely these equations, which is what makes gradient bugs readable.

Memory, not arithmetic, is the modern bottleneck. The $O({\sum }_l{n}_l)$ activation cache is the binding constraint at scale, and the checkpointing lever of this lesson (Chen et al., 2016, Training Deep Nets with Sublinear Memory Cost) generalized into a small industry: reversible layers (Gomez et al., 2017) reconstruct each layer's input from its output so no activations need storing at all, and fused kernels like FlashAttention (Dao et al., 2022) recompute the attention matrix in the backward pass rather than caching it. Each is the same trade this lesson names — spend a little recompute to save a lot of memory — pushed to where a network otherwise would not fit on the hardware.⁷

Takeaways

• A computational graph has operations for nodes and tensors for edges; the forward pass evaluates it in topological order, the backward pass differentiates it in reverse.
• The chain rule for vector maps multiplies Jacobians; backprop never forms them, propagating vector-Jacobian products instead — for a matmul $z=Wy$ the VJP is just $\bar{y}=W^{\top }\bar{z}$.
• The four equations are the entire MLP algorithm: ${\delta }^{(L)}={\nabla }_{\hat{y}}\ell \odot g^{\prime }(z^{(L)})$ (BP1), the recursion ${\delta }^{(l)}=(W^{(l+1)\top }{\delta }^{(l+1)})\odot g^{\prime }(z^{(l)})$ (BP2), and the parameter gradients $\partial \ell /\partial b^{(l)}={\delta }^{(l)}$ (BP3), $\partial \ell /\partial W^{(l)}={\delta }^{(l)}{h}^{(l-1)\top }$ (BP4).
• Track shapes: every gradient carries the shape of what it differentiates, so $\partial \ell /\partial W^{(l)}$ is $n_l\times n_{l-1}$ and ${\delta }^{(l)}$ is $n_l\times 1$. Batched over $B$ examples, the outer product becomes a matrix product ${\Delta }^{(l)}{H}^{(l-1)\top }$ that sums per-example gradients over the batch axis.
• Reverse mode wins for deep nets because the loss is scalar ($m=1$): one backward pass yields the full gradient, versus one forward pass per parameter.
• Gradient checking validates a backward pass against centered finite differences ($O({ϵ}^2)$ error, relative agreement below $\sim 10^{-7}$); it is the unit test for the derivatives, run once on a small net, never during training.
• Backprop costs $\approx 2\times$ the forward pass in time and $O({\sum }_l{n}_l)$ in memory for the activation cache; checkpointing buys memory back by recomputing activations, the lever that fits very deep nets on a fixed budget.
• The next step is to use this gradient well: the optimizer decides how far and in what direction to step, given the ${\nabla }_{\theta }\ell$ backprop hands it.