---
title: Backpropagation
module: Neural Networks
moduleNumber: 2
lessonNumber: 4
order: 204
summary: >
  Backpropagation is the chain rule run backward over a computational graph. We
  formalize the graph, derive the four backprop equations for an MLP, present the
  forward and backward passes as algorithms, and work a tiny two-layer net by hand
  with explicit numbers. The result: one scalar loss, reverse-mode autodiff, and
  a gradient for every parameter at twice the cost of a forward pass.
topics: [Neural Networks]
sources:
  - book: Goodfellow
    ref: "§6.5 — Back-Propagation and Other Differentiation Algorithms"
  - book: Stevens
    ref: "Ch. 5 — The Mechanics of Learning; autograd and the backward pass"
---

The [multilayer perceptron](/deep-learning/neural-networks/the-multilayer-perceptron)
gave us a forward pass: inputs flow up through affine maps and nonlinearities to a
scalar loss. Training needs the reverse: how the loss responds to every weight,
$\nabla_\theta \mathcal{L}$. **Backpropagation** computes that gradient exactly, in
a single backward sweep, by applying the chain rule to the
[training loop](/deep-learning/foundations/what-is-deep-learning)'s forward
computation read as a graph. It is not a learning algorithm; it is the
_differentiation_ algorithm that feeds the optimizer.[^gf-backprop]

## The computational graph

Every model is a composition of primitive operations, and that composition has a
shape: a directed acyclic graph whose nodes compute and whose edges carry values.

> **Definition (Computational graph).** A directed acyclic graph $G = (V, E)$ in
> which each node $v \in V$ is either an _input_/_parameter_ or an _operation_
> $v = f_v(\text{pa}(v))$ applied to its parents $\text{pa}(v)$, and each edge
> $(u, v) \in E$ carries the tensor value of $u$ into $v$. A topological order of
> $G$ is a valid evaluation order: the **forward pass**.

Nodes are operations; edges are tensors. Take the single-example squared error of a
linear unit, $\ell = (wx + b - y)^2$. Its graph threads $w, x, b, y$ through a
multiply, two adds, and a square — and once forward values are pinned to the nodes,
backprop annotates each edge with a gradient flowing the other way.

$$
% caption: Computational graph of $\ell=(wx+b-y)^2$. Black edges carry forward values;
% blue edges carry the gradient $\partial\ell/\partial(\cdot)$ written right to left.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  op/.style={circle, draw, thick, minimum size=8mm, inner sep=0pt},
  val/.style={draw, minimum size=6mm, inner sep=2pt}]
  \definecolor{acc}{HTML}{2348F2}
  % leaves (left column, feed in)
  \node[val] (w) at (0,1.7) {$w$};
  \node[val] (x) at (0,0.7) {$x$};
  \node[val] (b) at (1.9,-1.6) {$b$};
  \node[val] (y) at (5.0,-1.6) {$y$};
  % ops on a single horizontal line at y=0.7
  \node[op] (mul) at (2.6,0.7) {mul};
  \node[op] (add) at (5.0,0.7) {add};
  \node[op] (sub) at (7.4,0.7) {sub};
  \node[op] (sq)  at (9.8,0.7) {sq};
  \node[val] (L)  at (11.8,0.7) {loss};
  % forward edges with values (above the line)
  \draw[->, thick] (w) -- (mul);
  \draw[->, thick] (x) -- (mul);
  \draw[->, thick] (mul) -- (add) node[pos=0.5, above, font=\scriptsize] {$u=6$};
  \draw[->, thick] (b) -- (add);
  \draw[->, thick] (add) -- (sub) node[pos=0.5, above, font=\scriptsize] {$v=7$};
  \draw[->, thick] (y) -- (sub);
  \draw[->, thick] (sub) -- (sq) node[pos=0.5, above, font=\scriptsize] {$r=2$};
  \draw[->, thick] (sq) -- (L);
  % backward annotations (blue, below the line)
  \node[acc, font=\footnotesize] at (3.8,1.25) {\texttt{grad u = 4}};
  \node[acc, font=\footnotesize] at (6.2,1.25) {\texttt{grad v = 4}};
  \node[acc, font=\footnotesize] at (8.6,0.05) {\texttt{grad r = 4}};
  \node[acc, font=\footnotesize, anchor=south east] at (-0.3,1.55) {\texttt{grad w = 12}};
  \node[acc, font=\footnotesize, anchor=east] at (1.5,-1.6) {\texttt{grad b = 4}};
  \draw[->, acc, thick, dashed] (11.4,-2.6) -- (0.6,-2.6)
    node[pos=0.5, above, font=\footnotesize, text=acc] {\texttt{backward sweep}};
\end{tikzpicture}
$$

The forward pass labels every edge with a value; the backward pass labels every
edge with the loss's sensitivity to that value. The two passes share the same graph
and traverse it in opposite topological directions.

## The chain rule for vector functions

A node rarely carries a scalar; it carries a tensor. So the local derivative is
not a number but a **Jacobian**, and the chain rule multiplies Jacobians.

> **Definition (Jacobian).** For a map $f : \mathbb{R}^n \to \mathbb{R}^m$, the
> Jacobian $J = \partial f / \partial x \in \mathbb{R}^{m \times n}$ has entries
> $J_{ij} = \partial f_i / \partial x_j$. For a composition $z = f(y)$, $y = g(x)$,
> the chain rule is the matrix product $\partial z / \partial x = (\partial z /
> \partial y)\,(\partial y / \partial x)$.

For a deep net the output of interest is the _scalar_ loss $\ell$, so the quantity
we propagate is the gradient row vector $\partial \ell / \partial y$. Pushing it
through a node $z = f(y)$ requires not the full Jacobian $J$ but the product of the
incoming gradient with $J$ — the **vector-Jacobian product**.[^gf-graph]

> **Definition (Vector-Jacobian product, VJP).** Given an upstream gradient
> $\bar z = (\partial \ell / \partial z)^\top$ and a node $z = f(y)$ with Jacobian
> $J = \partial z / \partial y$, the downstream gradient is the VJP
> $\bar y = J^\top \bar z$. Reverse-mode autodiff chains these VJPs, one per node,
> evaluated in reverse topological order.

The VJP is the core operation: each operation supplies a rule for turning its output
gradient into its input gradient, _without ever materializing_ the (possibly huge)
Jacobian $J$. For a matrix multiply $z = Wy$, $J = W$ and the VJP is just
$\bar y = W^\top \bar z$, a second matrix multiply, transposed.

| node $z = f(y)$ | Jacobian $J = \partial z/\partial y$ | VJP $\bar y = J^\top \bar z$ |
| --- | --- | --- |
| matrix multiply $z = Wy$ | $W$ | $W^\top \bar z$ |
| add $z = y + b$ | $I$ | $\bar z$ (and $\bar b = \bar z$) |
| elementwise $z = g(y)$ | $\diag\!\parens{g'(y)}$ | $g'(y) \odot \bar z$ |
| sum $z = \mathbf 1^\top y$ | $\mathbf 1^\top$ | $\bar z\,\mathbf 1$ |
| scale $z = c\,y$ | $cI$ | $c\,\bar z$ |

For a single node the rule is: multiply the upstream gradient by the local
derivative to get the downstream gradient. Backprop repeats this step at every
node.

$$
% caption: The local-gradient gate. A node multiplies the upstream gradient by its
% local Jacobian transpose to emit the downstream gradient.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  op/.style={circle, draw, thick, minimum size=13mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \node[op] (f) at (0,0) {$z=f(y)$};
  % forward (black, top)
  \draw[->, thick] (-3.4,0.55) -- (f.west |- 0,0.55) node[pos=0.1, above, font=\scriptsize] {$y$};
  \draw[->, thick] (f.east |- 0,0.55) -- (3.4,0.55) node[pos=0.9, above, font=\scriptsize] {$z$};
  % backward (blue, bottom, right-to-left)
  \draw[<-, acc, thick] (-3.4,-0.55) -- (f.west |- 0,-0.55);
  \draw[<-, acc, thick] (f.east |- 0,-0.55) -- (3.4,-0.55);
  \node[acc, font=\footnotesize, anchor=west] at (1.4,-0.95) {\texttt{upstream grad z}};
  \node[acc, font=\footnotesize, anchor=east] at (-1.4,-0.95) {\texttt{downstream grad y}};
  \node[font=\footnotesize] at (0,-2.1) {\texttt{local rule: grad y = Jacobian\^{}T times grad z}};
  \node[font=\footnotesize, anchor=north] at (0,1.6) {\texttt{forward}};
  \node[acc, font=\footnotesize, anchor=south] at (0,-1.55) {\texttt{backward}};
\end{tikzpicture}
$$

## The four backprop equations

Now specialize to an MLP. Layers $l = 1, \dots, L$ compute pre-activations and
activations

$$
z^{(l)} = W^{(l)} h^{(l-1)} + b^{(l)}, \qquad h^{(l)} = g\parens{z^{(l)}},
\qquad h^{(0)} = x,
$$

with a per-example loss $\ell\parens{h^{(L)}, y}$ (the output $\hat y = h^{(L)}$,
or $z^{(L)}$ for a linear final layer). Define the **error** of layer $l$ as the
gradient of the loss with respect to that layer's pre-activation,

$$
\delta^{(l)} \;:=\; \frac{\partial \ell}{\partial z^{(l)}} \;\in\; \mathbb{R}^{n_l}.
$$

Everything reduces to computing the $\delta^{(l)}$. The four equations below do it
recursively, then read the parameter gradients off the errors.

> **Theorem (Backpropagation, BP1–BP4).** For the MLP above with elementwise
> activation $g$ and $\odot$ the Hadamard product, the layer errors and parameter
> gradients satisfy
> $$
> \begin{aligned}
> \textbf{(BP1)}\quad & \delta^{(L)} = \nabla_{\hat y}\,\ell \;\odot\; g'\parens{z^{(L)}}, \\
> \textbf{(BP2)}\quad & \delta^{(l)} = \parens{W^{(l+1)\top}\delta^{(l+1)}} \;\odot\; g'\parens{z^{(l)}}, \\
> \textbf{(BP3)}\quad & \frac{\partial \ell}{\partial b^{(l)}} = \delta^{(l)}, \\
> \textbf{(BP4)}\quad & \frac{\partial \ell}{\partial W^{(l)}} = \delta^{(l)}\, h^{(l-1)\top}.
> \end{aligned}
> $$

> **Proof.** _BP1 (output layer)._ The loss depends on $z^{(L)}$ only through
> $\hat y = g(z^{(L)})$, applied elementwise. By the chain rule, component $j$ is
> $\delta^{(L)}_j = \sum_k \frac{\partial \ell}{\partial \hat y_k}\frac{\partial
> \hat y_k}{\partial z^{(L)}_j}$. Because $g$ acts elementwise, $\partial \hat y_k
> / \partial z^{(L)}_j = g'(z^{(L)}_j)\,[k{=}j]$, collapsing the sum to
> $\delta^{(L)}_j = \frac{\partial \ell}{\partial \hat y_j} g'(z^{(L)}_j)$, i.e.
> $\delta^{(L)} = \nabla_{\hat y}\ell \odot g'(z^{(L)})$.
>
> _BP2 (recursion)._ The pre-activation $z^{(l)}$ influences $\ell$ only through
> $z^{(l+1)} = W^{(l+1)} g(z^{(l)}) + b^{(l+1)}$. Chain rule:
> $\delta^{(l)} = \parens{\partial z^{(l+1)} / \partial z^{(l)}}^\top \delta^{(l+1)}$.
> The Jacobian factors as the affine map's $W^{(l+1)}$ times the elementwise
> $\diag(g'(z^{(l)}))$, so $\partial z^{(l+1)}/\partial z^{(l)} =
> W^{(l+1)}\diag(g'(z^{(l)}))$. Transposing and applying to
> $\delta^{(l+1)}$ gives $\delta^{(l)} = \diag(g'(z^{(l)}))\,
> W^{(l+1)\top}\delta^{(l+1)} = (W^{(l+1)\top}\delta^{(l+1)}) \odot g'(z^{(l)})$.
>
> _BP3, BP4 (parameters)._ Since $z^{(l)} = W^{(l)} h^{(l-1)} + b^{(l)}$, we have
> $\partial z^{(l)}/\partial b^{(l)} = I$, so $\partial \ell/\partial b^{(l)} =
> \delta^{(l)}$. For the weights, $z^{(l)}_i = \sum_k W^{(l)}_{ik} h^{(l-1)}_k +
> b^{(l)}_i$ gives $\partial z^{(l)}_i / \partial W^{(l)}_{ik} = h^{(l-1)}_k$, hence
> $\partial \ell / \partial W^{(l)}_{ik} = \delta^{(l)}_i h^{(l-1)}_k$, which in
> matrix form is the outer product $\delta^{(l)} h^{(l-1)\top}$. $\qed$

Two facts make these equations the entire algorithm. First, the recursion BP2 is
the VJP of the affine-plus-activation block, reused at every layer — _the same two
matrix operations, transposed._ Second, BP4 needs the cached activation
$h^{(l-1)}$ from the forward pass, which is why the forward pass must store its
intermediates.[^gf-bpeqs]

### Every shape, checked

The four equations are easy to write and easy to get wrong by a transpose. Pin
down every shape and the algebra becomes mechanical. Let layer $l$ have $n_l$ units
and $n_{l-1}$ inputs, so $W^{(l)} \in \mathbb{R}^{n_l \times n_{l-1}}$ and
$b^{(l)} \in \mathbb{R}^{n_l}$. Every activation, pre-activation, and error at layer
$l$ is a vector of length $n_l$; every weight gradient matches its weight.

| object | expression | shape |
| --- | --- | --- |
| activation | $h^{(l-1)}$ | $n_{l-1} \times 1$ |
| weight | $W^{(l)}$ | $n_l \times n_{l-1}$ |
| pre-activation | $z^{(l)} = W^{(l)} h^{(l-1)} + b^{(l)}$ | $n_l \times 1$ |
| activation derivative | $g'\parens{z^{(l)}}$ | $n_l \times 1$ |
| error | $\delta^{(l)} = \partial\ell/\partial z^{(l)}$ | $n_l \times 1$ |
| transposed weight | $W^{(l+1)\top}$ | $n_l \times n_{l+1}$ |
| back-propagated error | $W^{(l+1)\top}\delta^{(l+1)}$ | $n_l \times 1$ |
| weight gradient | $\delta^{(l)} h^{(l-1)\top}$ | $n_l \times n_{l-1}$ |
| bias gradient | $\delta^{(l)}$ | $n_l \times 1$ |

BP2 reads left to right as a shape contraction: the $(n_l \times n_{l+1})$ transpose
$W^{(l+1)\top}$ hits the $(n_{l+1} \times 1)$ upstream error $\delta^{(l+1)}$ to
give an $(n_l \times 1)$ vector, which the Hadamard product with
$g'(z^{(l)}) \in \mathbb{R}^{n_l}$ leaves at $n_l \times 1$ — exactly the shape of
$\delta^{(l)}$. BP4 is an outer product: the $(n_l \times 1)$ column $\delta^{(l)}$
times the $(1 \times n_{l-1})$ row $h^{(l-1)\top}$ yields the $(n_l \times n_{l-1})$
matrix that matches $W^{(l)}$ entry for entry. A single rule catches most sign-and-
transpose bugs: **every gradient carries the shape of the thing it differentiates**,
$\partial\ell/\partial W^{(l)}$ is shaped like $W^{(l)}$, $\delta^{(l)}$ is shaped
like $z^{(l)}$.

### The batched form

Training runs on a minibatch of $B$ examples at once, not one vector. Stack the
$B$ activations of layer $l$ as columns of a matrix $H^{(l)} \in \mathbb{R}^{n_l
\times B}$ (some frameworks use rows; the transposes flip but the content is the
same). The forward pass becomes a matrix product with a broadcast bias, and the
backward pass sums the per-example gradients — the reason a minibatch gradient is
an _average_ over examples.

$$
Z^{(l)} = W^{(l)} H^{(l-1)} + b^{(l)}\mathbf 1^\top, \qquad
H^{(l)} = g\parens{Z^{(l)}}, \qquad H^{(0)} = X,
$$

with $X \in \mathbb{R}^{n_0 \times B}$ and $\mathbf 1 \in \mathbb{R}^{B}$ the
all-ones vector that broadcasts the bias across columns. The error matrix
$\Delta^{(l)} \in \mathbb{R}^{n_l \times B}$ holds one $\delta$ per column, and the
batched equations are the single-example ones with the outer product replaced by a
matrix product that sums over the batch axis:

$$
\begin{aligned}
\Delta^{(L)} &= \nabla_{\hat Y}\,\mathcal{L} \odot g'\parens{Z^{(L)}}
  && (n_L \times B), \\
\Delta^{(l)} &= \parens{W^{(l+1)\top}\Delta^{(l+1)}} \odot g'\parens{Z^{(l)}}
  && (n_l \times B), \\
\frac{\partial \mathcal{L}}{\partial W^{(l)}} &= \Delta^{(l)}\,H^{(l-1)\top}
  && (n_l \times n_{l-1}), \\
\frac{\partial \mathcal{L}}{\partial b^{(l)}} &= \Delta^{(l)}\mathbf 1
  && (n_l \times 1).
\end{aligned}
$$

The weight gradient $\Delta^{(l)} H^{(l-1)\top}$ contracts the $(n_l \times B)$ error
against the $(B \times n_{l-1})$ transposed activations, and the shared batch axis
$B$ vanishes — the matrix product _is_ the sum $\sum_{i=1}^{B} \delta^{(l)}_i
h^{(l-1)\top}_i$ over examples. The bias gradient $\Delta^{(l)}\mathbf 1$ is that same
sum over the batch axis. Dividing by $B$ (or folding $1/B$ into $\mathcal{L}$) turns
the summed gradient into the mean gradient the optimizer expects.

The dimensions flow one direction on the forward pass and the mirror direction on
the backward pass, with the batch axis $B$ carried along both ways.

$$
% caption: Delta recursion with tensor shapes. Forward (black) carries $H^{(l)}$ of
% shape $n_l\times B$; backward (blue) carries $\Delta^{(l)}$ of the same shape, gated by $g'(Z^{(l)})$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  lay/.style={draw, thick, minimum width=20mm, minimum height=15mm, inner sep=2pt, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[lay] (l1) at (0,0) {\texttt{W(1) h}};
  \node[lay] (l2) at (4.0,0) {\texttt{W(2) h}};
  \node[lay] (l3) at (8.0,0) {\texttt{W(3) h}};
  \node (loss) at (11.0,0) {\texttt{loss}};
  % forward (top)
  \draw[->, thick] (-2.2,0.75) -- (l1.west |- 0,0.75)
    node[pos=0.4, above, font=\footnotesize] {\texttt{n0 x B}};
  \draw[->, thick] (l1.east |- 0,0.75) -- (l2.west |- 0,0.75)
    node[pos=0.5, above, font=\footnotesize] {\texttt{n1 x B}};
  \draw[->, thick] (l2.east |- 0,0.75) -- (l3.west |- 0,0.75)
    node[pos=0.5, above, font=\footnotesize] {\texttt{n2 x B}};
  \draw[->, thick] (l3.east |- 0,0.75) -- (loss.west |- 0,0.75)
    node[pos=0.5, above, font=\footnotesize] {\texttt{n3 x B}};
  \node[font=\footnotesize, anchor=east] at (-2.2,0.75) {\texttt{forward}};
  % backward (bottom)
  \draw[<-, acc, thick] (l1.west |- 0,-0.75) -- (-2.2,-0.75);
  \draw[<-, acc, thick] (l1.east |- 0,-0.75) -- (l2.west |- 0,-0.75)
    node[pos=0.5, below, font=\footnotesize, text=acc] {\texttt{n1 x B}};
  \draw[<-, acc, thick] (l2.east |- 0,-0.75) -- (l3.west |- 0,-0.75)
    node[pos=0.5, below, font=\footnotesize, text=acc] {\texttt{n2 x B}};
  \draw[<-, acc, thick] (l3.east |- 0,-0.75) -- (loss.west |- 0,-0.75)
    node[pos=0.5, below, font=\footnotesize, text=acc] {\texttt{n3 x B}};
  \node[acc, font=\footnotesize, anchor=east] at (-2.2,-0.75) {\texttt{backward}};
  \node[font=\footnotesize, anchor=north] at (4.0,-1.5) {\texttt{each layer: transposed matmul W(l+1)\^{}T, then gate by g'(Z(l))}};
\end{tikzpicture}
$$

The error $\delta^{(l)}$ flows right to left across the layers, the mirror of the
forward signal. Each layer turns the incoming error into the outgoing one by a
transposed weight multiply and a gate by $g'$.

$$
% caption: Backward flow through an MLP. The error $\delta^{(l)}$ propagates right to
% left through each weight transpose and activation gate; weight gradients fall out as outer products.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  lay/.style={draw, thick, minimum width=15mm, minimum height=22mm, inner sep=2pt, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[lay] (l1) at (0,0) {layer 1};
  \node[lay] (l2) at (3.2,0) {layer 2};
  \node[lay, draw=acc, text=acc] (l3) at (6.4,0) {layer 3};
  \node (loss) at (9.2,0) {loss};
  % forward (top, black)
  \draw[->, thick] (-2.0,0.7) -- (l1.west |- 0,0.7) node[pos=0.0, above, font=\scriptsize] {$x$};
  \draw[->, thick] (l1.east |- 0,0.7) -- (l2.west |- 0,0.7);
  \draw[->, thick] (l2.east |- 0,0.7) -- (l3.west |- 0,0.7);
  \draw[->, thick] (l3.east |- 0,0.7) -- (loss.west |- 0,0.7);
  \node[font=\footnotesize, anchor=south] at (-1.4,0.85) {\texttt{forward}};
  % backward (bottom, blue, right-to-left)
  \draw[<-, acc, thick] (l1.west |- 0,-0.7) -- (-2.0,-0.7);
  \draw[<-, acc, thick] (l1.east |- 0,-0.7) -- (l2.west |- 0,-0.7)
    node[pos=0.5, below, font=\footnotesize, text=acc] {\texttt{grad 1}};
  \draw[<-, acc, thick] (l2.east |- 0,-0.7) -- (l3.west |- 0,-0.7)
    node[pos=0.5, below, font=\footnotesize, text=acc] {\texttt{grad 2}};
  \draw[<-, acc, thick] (l3.east |- 0,-0.7) -- (loss.west |- 0,-0.7)
    node[pos=0.5, below, font=\footnotesize, text=acc] {\texttt{grad 3}};
  \node[acc, font=\footnotesize, anchor=north] at (-1.4,-0.85) {\texttt{backward}};
\end{tikzpicture}
$$

## Forward and backward passes

The two passes are one algorithm in two directions. The forward pass evaluates the
graph in topological order and **caches** every pre-activation $z^{(l)}$ and
activation $h^{(l)}$; the backward pass walks the layers in reverse, applying
BP1–BP4.

```algorithm
caption: $\textsc{Forward}(x, \{W^{(l)}, b^{(l)}\})$ — evaluate and cache activations
$h^{(0)} \gets x$
for $l \gets 1$ to $L$ do
  $z^{(l)} \gets W^{(l)} h^{(l-1)} + b^{(l)}$ // cache $z^{(l)}$
  $h^{(l)} \gets g\parens{z^{(l)}}$ // cache $h^{(l)}$
$\hat y \gets h^{(L)}$
return $\hat y$ and the cache $\{z^{(l)}, h^{(l)}\}$
```

```algorithm
caption: $\textsc{Backward}(\hat y, y, \text{cache})$ — gradients via BP1–BP4
$\delta \gets \nabla_{\hat y}\,\ell(\hat y, y) \odot g'\parens{z^{(L)}}$ // BP1
for $l \gets L$ down to $1$ do
  $\partial \ell / \partial W^{(l)} \gets \delta \, h^{(l-1)\top}$ // BP4
  $\partial \ell / \partial b^{(l)} \gets \delta$ // BP3
  if $l > 1$ then
    $\delta \gets \parens{W^{(l)\top}\delta} \odot g'\parens{z^{(l-1)}}$ // BP2
return $\{\partial \ell / \partial W^{(l)}, \partial \ell / \partial b^{(l)}\}$
```

In modern frameworks both passes are automatic. As [Stevens](/deep-learning/neural-networks/the-multilayer-perceptron)
describes, PyTorch's **autograd** records each operation into a graph as the
forward pass runs; calling `.backward()` on the scalar loss replays that tape in
reverse, evaluating one VJP per recorded op. The four equations are what autograd
executes: you never hand-derive them, yet they are the rules autograd applies
underneath.

## A worked example

Take a concrete two-layer net: input $x = (1,\, 0.5)$, hidden layer of two
**sigmoid** units, a single **linear** output unit, target $y = 0$, and squared
loss $\ell = \tfrac12(\hat y - y)^2$. Parameters:

$$
W^{(1)} = \begin{pmatrix} 0.10 & 0.30 \\ -0.20 & 0.40 \end{pmatrix},\quad
b^{(1)} = \begin{pmatrix} 0.10 \\ -0.10 \end{pmatrix},\quad
W^{(2)} = \begin{pmatrix} 0.50 & -0.60 \end{pmatrix},\quad
b^{(2)} = 0.20.
$$

**Forward pass.** Compute $z^{(1)} = W^{(1)}x + b^{(1)}$, gate through
$\sigma(z) = 1/(1+e^{-z})$, then the linear readout $\hat y = z^{(2)} = W^{(2)}h^{(1)} + b^{(2)}$.

| quantity | formula | value |
| --- | --- | --- |
| $z^{(1)}_1$ | $0.10\cdot 1 + 0.30\cdot 0.5 + 0.10$ | $0.3500$ |
| $z^{(1)}_2$ | $-0.20\cdot 1 + 0.40\cdot 0.5 - 0.10$ | $-0.1000$ |
| $h^{(1)}_1$ | $\sigma(0.3500)$ | $0.5866$ |
| $h^{(1)}_2$ | $\sigma(-0.1000)$ | $0.4750$ |
| $z^{(2)} = \hat y$ | $0.50\cdot 0.5866 - 0.60\cdot 0.4750 + 0.20$ | $0.2083$ |
| $\ell$ | $\tfrac12(0.2083 - 0)^2$ | $0.0217$ |

**Backward pass.** The output is linear, so $g'(z^{(2)}) = 1$ and BP1 gives
$\delta^{(2)} = (\hat y - y)\cdot 1 = 0.2083$. BP4/BP3 read off the layer-2
gradients as $\delta^{(2)} h^{(1)\top}$ and $\delta^{(2)}$. BP2 then backs the
error into the hidden layer using $\sigma'(z) = h(1-h)$, and BP4/BP3 again yield
the layer-1 gradients.

| quantity | equation | value |
| --- | --- | --- |
| $\delta^{(2)}$ | $(\hat y - y)\cdot 1$ | $0.2083$ |
| $\partial\ell/\partial W^{(2)}$ | $\delta^{(2)}\,h^{(1)\top}$ | $(0.1222,\ 0.0989)$ |
| $\partial\ell/\partial b^{(2)}$ | $\delta^{(2)}$ | $0.2083$ |
| $W^{(2)\top}\delta^{(2)}$ | $(0.50,\,-0.60)^\top\cdot 0.2083$ | $(0.1041,\ -0.1250)$ |
| $\sigma'(z^{(1)})$ | $h^{(1)}\odot(1 - h^{(1)})$ | $(0.2425,\ 0.2494)$ |
| $\delta^{(1)}$ | $(W^{(2)\top}\delta^{(2)}) \odot \sigma'(z^{(1)})$ | $(0.0253,\ -0.0312)$ |
| $\partial\ell/\partial W^{(1)}$ | $\delta^{(1)}\,x^\top$ | $\begin{smallmatrix}0.0253 & 0.0126\\ -0.0312 & -0.0156\end{smallmatrix}$ |
| $\partial\ell/\partial b^{(1)}$ | $\delta^{(1)}$ | $(0.0253,\ -0.0312)$ |

Every gradient is now explicit, and a single descent step $\theta \gets \theta -
\eta\,\nabla_\theta\ell$ would nudge all six tensors. Notice the hidden-layer
gradients are an order of magnitude smaller than the output's — the $\sigma'$
factor, capped at $0.25$, already shrinks the signal. Stack many sigmoid layers and
those factors multiply toward zero: the **vanishing gradient** that motivates ReLU
and careful [initialization](/deep-learning/optimization/initialization).

The backward pass is a chain of three multiplications seeded at the loss. The scalar
$\delta^{(2)} = 0.2083$ enters the hidden layer through the transposed weight,
$W^{(2)\top}\delta^{(2)}$, then the sigmoid gate $\sigma'(z^{(1)})$ scales each
component, producing $\delta^{(1)}$. The figure traces those numbers end to end.

$$
% caption: Numeric backward pass for the worked two-layer net. The output error
% $\delta^{(2)}{=}0.2083$ enters through $W^{(2)\top}$, is gated by $\sigma'(z^{(1)})$, and yields $\delta^{(1)}$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  bx/.style={draw, thick, minimum width=24mm, minimum height=9mm, inner sep=2pt, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[bx, draw=acc, text=acc] (d2) at (0,0) {\texttt{delta2 = 0.2083}};
  \node[bx] (wt) at (4.2,0) {\texttt{W(2)\^{}T times delta2}};
  \node[bx, draw=acc, text=acc] (d1) at (10.4,0) {\texttt{delta1}};
  % arrows
  \draw[->, acc, thick] (d2) -- (wt);
  \draw[->, acc, thick] (wt) -- (d1)
    node[pos=0.5, above, font=\footnotesize] {\texttt{gate by sig'(z1)}}
    node[pos=0.5, below, font=\footnotesize] {\texttt{= (0.2425, 0.2494)}};
  % value annotations
  \node[font=\footnotesize, anchor=north] at (4.2,-0.75) {\texttt{= (0.1041, -0.1250)}};
  \node[acc, font=\footnotesize, anchor=north] at (10.4,-0.75) {\texttt{= (0.0253, -0.0312)}};
  % weight gradients branching down
  \node[font=\footnotesize, anchor=south] at (0,0.75) {\texttt{grad W2 = (0.1222,\ 0.0989)}};
  \node[acc, font=\footnotesize, anchor=south] at (10.4,0.75) {\texttt{grad W1 = delta1 times x\^{}T}};
\end{tikzpicture}
$$

## Gradient checking

A hand-derived or hand-coded backward pass is easy to get wrong by a transpose or a
missing $g'$ factor. **Gradient checking** verifies it against the definition of the
derivative: perturb one parameter by a tiny $\epsilon$, remeasure the loss, and
compare the finite difference to the analytic gradient.

> **Definition (Numerical gradient check).** For a scalar parameter $\theta_i$, the
> centered finite difference
> $$
> \widetilde{\partial_i \ell} = \frac{\ell(\theta + \epsilon e_i) - \ell(\theta - \epsilon e_i)}{2\epsilon}
> $$
> approximates $\partial\ell/\partial\theta_i$ with error $O(\epsilon^2)$. The backward
> pass is correct when the relative error
> $|\widetilde{\partial_i \ell} - \partial_i\ell| / (|\widetilde{\partial_i \ell}| + |\partial_i\ell|)$
> is below roughly $10^{-7}$ (double precision).

The **centered** difference is not optional. The one-sided form $(\ell(\theta +
\epsilon e_i) - \ell(\theta))/\epsilon$ carries $O(\epsilon)$ error; the centered form
cancels the first-order term and drops to $O(\epsilon^2)$, gaining several digits of
agreement for the same $\epsilon$. Check on the worked net above: at $W^{(2)}_1 =
0.50$, bumping to $0.50 \pm 10^{-4}$ and remeasuring $\ell$ gives a finite difference
of $0.1222$, matching the analytic $\partial\ell/\partial W^{(2)}_1 = 0.1222$ from
BP4 to four digits.

Gradient checking is a debugging tool, not a training step — one finite difference
per parameter costs a full forward pass, so it runs on a small net, on a handful of
parameters, once. Practical cautions carry over from Goodfellow: use double
precision, avoid $\epsilon$ so small that rounding dominates ($10^{-4}$ to $10^{-6}$
is the usable band), and check a few parameters rather than all of them. Turn it off
before training; it is the unit test for the backward pass, not part of it.[^gf-gradcheck]

## Reverse mode versus forward mode

Backprop is one of two ways to evaluate a chain of Jacobians. **Forward mode**
multiplies them input-to-output, propagating a directional derivative; **reverse
mode** multiplies output-to-input, propagating a gradient. For a composition
$\mathbb{R}^n \to \cdots \to \mathbb{R}^m$ the cost is set by which end you start
from.

| | forward mode | reverse mode (backprop) |
| --- | --- | --- |
| propagates | Jacobian-vector product $Jv$ | vector-Jacobian product $J^\top u$ |
| sweeps | input $\to$ output | output $\to$ input |
| one pass yields | one column of $J$ (one input's effect) | one row of $J$ (one output's gradient) |
| passes for full $J$ | $n$ (number of inputs) | $m$ (number of outputs) |
| cheap when | inputs few: $n \ll m$ | outputs few: $m \ll n$ |
| memory | $O(1)$ extra (no tape) | $O(\text{graph})$ (cache activations) |

A neural network has $n$ in the millions (parameters) and $m = 1$ (the scalar
loss). Reverse mode delivers the entire gradient in a _single_ backward pass; forward
mode would need one pass per parameter. That asymmetry is the whole reason
backpropagation — not forward-mode differentiation — trains deep nets.[^gf-reverse]

> **Definition (Reverse-mode automatic differentiation).** The algorithm that, given
> a computational graph for a scalar output, computes the gradient with respect to
> all inputs by evaluating one vector-Jacobian product per node in reverse
> topological order. Backpropagation is reverse-mode autodiff specialized to the
> layered structure of a neural network.

## Computational cost

Backprop is cheap in time and expensive in memory, a trade every deep-learning
system must manage.

> **Theorem (Cost of backprop).** Computing $\nabla_\theta \ell$ by reverse-mode
> autodiff costs $O(1)$ forward passes in time — empirically about $2\times$ — and
> requires storing every intermediate activation, $O(\sum_l n_l)$ memory.

> **Proof sketch.** Each node's VJP costs a constant multiple of its forward
> evaluation (e.g. a matmul $z = Wy$ costs one matmul forward and one transposed
> matmul, $\bar y = W^\top\bar z$, backward; the weight gradient $\bar z\, y^\top$
> is a third of the same order). Summing the constant over all nodes gives total
> backward work within a small constant of the forward work — the standard estimate
> is $\approx 2\times$. The activation $h^{(l-1)}$ is needed by BP4 at layer $l$, so
> every layer's output must survive until its backward step, forcing $O(\sum_l n_l)$
> live memory. $\qed$

| pass | time | memory | needs |
| --- | --- | --- | --- |
| forward | $1\times$ | activations cached | inputs, parameters |
| backward | $\approx 2\times$ | gradients (same shape as params) | cached activations |
| forward + backward | $\approx 3\times$ forward | activations + gradients | full cache |

The forward pass writes the cache; the backward pass consumes it. The timeline
below shows the schedule, with the cache as the shared store.

$$
% caption: The two-pass schedule. The forward pass caches activations left-to-right;
% the backward pass reads them right-to-left, so peak memory holds the whole cache.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  store/.style={draw, dashed, minimum width=58mm, minimum height=8mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  % forward timeline (top)
  \node[font=\footnotesize] at (-2.3,2.0) {\texttt{forward}};
  \foreach \i/\xx in {0/0, 1/2.0, 2/4.0, 3/6.0} \node[circle, draw, minimum size=6mm, inner sep=0pt] (f\i) at (\xx,2.0) {};
  \foreach \i/\j in {0/1, 1/2, 2/3} \draw[->, thick] (f\i) -- (f\j);
  \node[anchor=south, font=\scriptsize] at (0,2.3) {$x$};
  \node[anchor=south, font=\scriptsize] at (6,2.3) {loss};
  % cache band (middle)
  \node[store] (cache) at (3.0,0.6) {\texttt{activation cache}};
  \draw[->, thick] (f1) -- (1.0,1.05);
  \draw[->, thick] (f2) -- (3.0,1.05);
  \draw[->, thick] (f3) -- (5.0,1.05);
  % backward timeline (bottom)
  \node[acc, font=\footnotesize] at (-2.3,-0.8) {\texttt{backward}};
  \foreach \i/\xx in {0/0, 1/2.0, 2/4.0, 3/6.0} \node[circle, draw=acc, thick, minimum size=6mm, inner sep=0pt] (b\i) at (\xx,-0.8) {};
  \foreach \i/\j in {1/0, 2/1, 3/2} \draw[->, acc, thick] (b\i) -- (b\j);
  \draw[<-, acc, thick] (1.0,0.15) -- (b1);
  \draw[<-, acc, thick] (3.0,0.15) -- (b2);
  \draw[<-, acc, thick] (5.0,0.15) -- (b3);
  \node[acc, anchor=north, font=\scriptsize] at (6,-1.1) {grad $L$};
  \node[acc, anchor=north, font=\scriptsize] at (0,-1.1) {grad $x$};
\end{tikzpicture}
$$

When memory is the binding constraint, **gradient checkpointing** trades it back
for time: store only a sparse subset of activations on the forward pass and
_recompute_ the rest during the backward pass. Caching every $\sqrt{L}$-th layer
cuts activation memory from $O(L)$ to $O(\sqrt{L})$ at the cost of one extra
forward segment — the standard lever for training networks too deep to cache whole.[^stevens-autograd]

| strategy | activation memory | extra compute |
| --- | --- | --- |
| cache all (default) | $O(L)$ | none |
| $\sqrt{L}$-checkpointing | $O(\sqrt{L})$ | $\approx 1$ extra forward |
| recompute everything | $O(1)$ | $\approx L$ extra forwards |

## Chain rule as a product of Jacobians

Zoom out and the whole network is one long composition, and its gradient is the
product of per-layer Jacobians, read right to left, which is reverse mode.

$$
\frac{\partial \ell}{\partial \theta^{(1)}}
= \underbrace{\frac{\partial \ell}{\partial z^{(L)}}}_{1 \times n_L}
\;\underbrace{\frac{\partial z^{(L)}}{\partial z^{(L-1)}}}_{n_L \times n_{L-1}}
\cdots
\underbrace{\frac{\partial z^{(2)}}{\partial z^{(1)}}}_{n_2 \times n_1}
\;\underbrace{\frac{\partial z^{(1)}}{\partial \theta^{(1)}}}_{n_1 \times \dim\theta^{(1)}}.
$$

Associativity lets us choose the multiplication order, and that choice _is_ the
difference between forward and reverse mode. Multiplying left-to-right keeps a
running row vector (a gradient) — cheap, because every partial product stays a
vector. Multiplying right-to-left would build full matrices. Reverse mode is simply
the left-to-right grouping of this product.

$$
% caption: The gradient is a product of per-layer Jacobians. Reverse mode carries a
% row vector through each $J^{(l)}$ as a vector-Jacobian product, never forming a full matrix.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  jac/.style={draw, thick, minimum width=12mm, minimum height=10mm, inner sep=1pt, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[jac] (jL) at (0,0) {$J^{(L)}$};
  \node[jac] (j3) at (2.4,0) {$J^{(3)}$};
  \node[jac] (j2) at (4.8,0) {$J^{(2)}$};
  \node[jac] (j1) at (7.2,0) {$J^{(1)}$};
  \node (th) at (9.6,0) {grad weights};
  % loss seed on the left
  \node[acc, anchor=east, font=\scriptsize] at (-1.1,0) {loss grad};
  \draw[->, acc, thick] (-1.1,0.0) -- (jL.west);
  % carry the running row vector right-to-left across the product
  \draw[->, acc, thick] (jL) -- (j3) node[pos=0.5, above, font=\scriptsize, text=acc] {VJP};
  \draw[->, acc, thick] (j3) -- (j2) node[pos=0.5, above, font=\scriptsize, text=acc] {VJP};
  \draw[->, acc, thick] (j2) -- (j1) node[pos=0.5, above, font=\scriptsize, text=acc] {VJP};
  \draw[->, acc, thick] (j1) -- (th);
  \node[font=\footnotesize, anchor=north] at (3.6,-0.9) {\texttt{each step: new row vector = (row vector) times J\textasciicircum(l), stays a vector}};
\end{tikzpicture}
$$

## Autodiff as a compiler

Backprop predates deep learning, and modern implementations treat it as a
compiler problem rather than a hand derivation. Three developments extend
Goodfellow's chapter.

**The algorithm predates the 1986 paper.** Reverse-mode automatic differentiation was
described by Linnainmaa in 1970 as the "reverse accumulation" of rounding errors,
and Werbos applied it to networks in his 1974 thesis. What made it famous was
Rumelhart, Hinton & Williams's 1986 _Nature_ paper, which showed backprop learning
useful internal representations in a working network — the demonstration, not the
derivative, was the contribution. The four equations here are the same reverse
sweep those authors ran by hand.

**Autodiff is now define-by-run.** Modern frameworks split into two styles. The
tape-based / _define-by-run_ style (PyTorch autograd) records each operation as the
forward pass executes and replays the tape backward, which the worked example above
mirrors. The _traced / transform_ style (JAX) treats differentiation as a program
transformation: `grad` rewrites a function into the function that computes its
gradient, and composable transforms like `vmap` (auto-batching) and `jit` (fusion)
stack on top. Both compute exactly the VJPs of this lesson; they differ only in
_when_ the graph is built. In practice no one writes
BP1–BP4 by hand anymore, but the framework runs precisely these equations, which is
what makes gradient bugs readable.

**Memory, not arithmetic, is the modern bottleneck.** The $O(\sum_l n_l)$
activation cache is the binding constraint at scale, and the checkpointing lever of
this lesson (Chen et al., 2016, "Training Deep Nets with Sublinear Memory Cost")
generalized into a small industry: _reversible_ layers (Gomez et al., 2017)
reconstruct each layer's input from its output so no activations need storing at
all, and fused kernels like FlashAttention (Dao et al., 2022) recompute the
attention matrix in the backward pass rather than caching it. Each is the same
trade this lesson names — spend a little recompute to save a lot of memory — pushed
to where a network otherwise would not fit on the hardware.[^beyond-bp]

## Takeaways

- A **computational graph** has operations for nodes and tensors for edges; the
  forward pass evaluates it in topological order, the backward pass differentiates
  it in reverse.
- The chain rule for vector maps multiplies **Jacobians**; backprop never forms
  them, propagating **vector-Jacobian products** instead — for a matmul $z = Wy$
  the VJP is just $\bar y = W^\top\bar z$.
- The **four equations** are the entire MLP algorithm: $\delta^{(L)} =
  \nabla_{\hat y}\ell \odot g'(z^{(L)})$ (BP1), the recursion $\delta^{(l)} =
  (W^{(l+1)\top}\delta^{(l+1)}) \odot g'(z^{(l)})$ (BP2), and the parameter
  gradients $\partial\ell/\partial b^{(l)} = \delta^{(l)}$ (BP3),
  $\partial\ell/\partial W^{(l)} = \delta^{(l)} h^{(l-1)\top}$ (BP4).
- Track **shapes**: every gradient carries the shape of what it differentiates, so
  $\partial\ell/\partial W^{(l)}$ is $n_l \times n_{l-1}$ and $\delta^{(l)}$ is
  $n_l \times 1$. Batched over $B$ examples, the outer product becomes a matrix
  product $\Delta^{(l)} H^{(l-1)\top}$ that sums per-example gradients over the batch axis.
- **Reverse mode wins** for deep nets because the loss is scalar ($m = 1$): one
  backward pass yields the full gradient, versus one forward pass per parameter.
- **Gradient checking** validates a backward pass against centered finite
  differences ($O(\epsilon^2)$ error, relative agreement below $\sim 10^{-7}$); it is
  the unit test for the derivatives, run once on a small net, never during training.
- Backprop costs $\approx 2\times$ the forward pass in **time** and $O(\sum_l n_l)$
  in **memory** for the activation cache; **checkpointing** buys memory back by
  recomputing activations, the lever that fits very deep nets on a fixed budget.
- The next step is to use this gradient well: the
  [optimizer](/deep-learning/optimization/gradient-descent-and-sgd) decides how
  far and in what direction to step, given the $\nabla_\theta\ell$ backprop hands it.

[^gf-backprop]: **Goodfellow**, _Deep Learning_, §6.5 — Back-Propagation: reverse-mode differentiation over the computational graph, distinguished from the learning rule (gradient descent) it feeds.
[^gf-graph]: **Goodfellow**, _Deep Learning_, §6.5.1–6.5.2 — the computational graph and the chain rule of calculus: each node propagates a vector-Jacobian product, never materializing the full Jacobian.
[^gf-bpeqs]: **Goodfellow**, _Deep Learning_, §6.5.4 — Back-Propagation in fully connected MLPs: the recurrence for the layer error $\delta^{(l)}$ and the outer-product weight gradient $\delta^{(l)} h^{(l-1)\top}$.
[^gf-reverse]: **Goodfellow**, _Deep Learning_, §6.5.6–6.5.9 — general reverse-mode autodiff: cheap when outputs are few ($m=1$ scalar loss, parameters in the millions), the exact regime of neural-network training.
[^stevens-autograd]: **Stevens**, _Deep Learning with PyTorch_, Ch. 5 — autograd records the forward graph and replays it backward via `.backward()`; the cached-activation cost is what gradient checkpointing trades against recompute.
[^gf-gradcheck]: **Goodfellow**, _Deep Learning_, §6.5.10 — verifying an implementation with centered finite differences; the $O(\epsilon^2)$ centered estimate and the double-precision relative-error threshold for a correct backward pass.
[^beyond-bp]: Primary sources: Linnainmaa (1970) for reverse-mode accumulation and Rumelhart, Hinton & Williams, "Learning Representations by Back-Propagating Errors" (Nature, 1986) for the network demonstration; Bradbury et al., "JAX: composable transformations of Python+NumPy programs" (2018) for the transform-style autodiff; Chen et al., "Training Deep Nets with Sublinear Memory Cost" (2016) for checkpointing, Gomez et al., "The Reversible Residual Network" (2017), and Dao et al., "FlashAttention" (2022) for recompute-over-cache memory strategies.