---
title: Calculus
module: Mathematical Background
moduleNumber: 0
lessonNumber: 4
order: 4
summary: >
  This lesson assembles the differential
  calculus used in training networks: the gradient and directional derivative, the
  Jacobian and Hessian, and the chain rule in scalar, vector, and matrix form.
  From the chain rule it derives back-propagation as a single sweep over the
  computational graph, tabulates the matrix-calculus identities that recur in
  layer gradients, reads optimization off a second-order Taylor expansion, and
  ends with why reverse-mode automatic differentiation is the algorithm every
  framework runs.
topics: [Mathematical Background]
sources:
  - book: Goodfellow
    ref: "Ch. 4 — Numerical Computation (§4.3 Gradient-Based Optimization)"
  - book: Goodfellow
    ref: "Ch. 6 — Deep Feedforward Networks (§6.5 Back-Propagation)"
---

A network learns by descending a loss, and descent needs a direction: the
gradient. Every weight update a model ever makes is a derivative of the loss with
respect to a parameter, and the algorithm that computes all of those derivatives
at once, back-propagation, is the chain rule applied with care to a graph of
operations. This lesson is the differential calculus that machinery rests on, kept
to the objects a network differentiates and the one rule that ties them
together.[^gf-grad]

## Derivatives, gradients, and directions

For a scalar function of a scalar, $f : \mathbb{R} \to \mathbb{R}$, the derivative
$f'(x) = \mathrm{d}f/\mathrm{d}x$ is the local rate of change: scale a small step
$\epsilon$ and the output moves by about $f'(x)\,\epsilon$. Deep learning almost
never has one input, though. A loss is a function of millions of parameters at
once, $f : \mathbb{R}^{n} \to \mathbb{R}$, so the single derivative becomes a
vector of **partial derivatives**, each holding all coordinates fixed but one.

> **Definition (Gradient).** For $f : \mathbb{R}^{n} \to \mathbb{R}$, the gradient
> is the vector of partial derivatives,
> $\nabla f(x) = \big(\tfrac{\partial f}{\partial x_1}, \dots, \tfrac{\partial f}{\partial x_n}\big)^\top$.
> It points in the direction of steepest _increase_, and its negative is the
> direction gradient descent steps along.

The "steepest" claim follows from the **directional derivative**,
the rate of change of $f$ as you move along a unit vector $u$. It equals
$\nabla f(x)^\top u$, and since $\nabla f(x)^\top u = \lVert \nabla f(x)\rVert \cos\theta$,
it is largest when $u$ aligns with $\nabla f$ and most negative when $u$ points the
opposite way.[^gf-grad] That is the whole justification for moving along
$-\nabla f$: among all directions of a fixed length, it decreases $f$ fastest.

For example, take $f(x_1, x_2) = x_1^2 + 3x_2^2$, a stretched
bowl, evaluated at $x = (1, 1)$. The partials are $\partial f/\partial x_1 = 2x_1$
and $\partial f/\partial x_2 = 6x_2$, so

$$
\nabla f(1, 1) = (2, 6)^\top,
\qquad
\lVert \nabla f \rVert = \sqrt{4 + 36} = \sqrt{40} \approx 6.32.
$$

Now compare three unit directions and read the directional derivative off each.
Along the steepest direction $u^\star = \nabla f / \lVert \nabla f\rVert =
(2, 6)/\sqrt{40}$, the slope is $\nabla f^\top u^\star = \lVert \nabla f\rVert
\approx 6.32$ — the maximum. Along a horizontal step $u = (1, 0)$ it is
$2 \cdot 1 + 6 \cdot 0 = 2$, and along $u = (0, 1)$ it is $6$: the bowl climbs
three times faster in the $x_2$ direction, exactly because that axis has the
steeper coefficient. Every one of these is $\lVert \nabla f\rVert \cos\theta$ with
the corresponding angle, and the largest, $6.32$, belongs to $\cos\theta = 1$.
Descent flips the sign: stepping along $-u^\star$ drops $f$ at rate $6.32$, faster
than any other unit step.

## The Jacobian and the Hessian

When the output is itself a vector, as it is at every hidden layer
($f : \mathbb{R}^{n} \to \mathbb{R}^{m}$), the first derivative is a matrix.

> **Definition (Jacobian).** For $f : \mathbb{R}^{n} \to \mathbb{R}^{m}$, the
> Jacobian $J \in \mathbb{R}^{m \times n}$ collects every first-order partial,
> $J_{i,j} = \tfrac{\partial f_i}{\partial x_j}$. Row $i$ is the gradient of the
> $i$-th output; a small input step $\epsilon$ moves the output by about
> $J\epsilon$.

For example, the softmax layer
$f(x)_i = e^{x_i}/\sum_j e^{x_j}$ maps $\mathbb{R}^2 \to \mathbb{R}^2$; its
Jacobian entries are $\partial f_i/\partial x_j = f_i(\delta_{ij} - f_j)$. At
$x = (0, 0)$ the outputs are $f = (\tfrac12, \tfrac12)$, so

$$
J = \begin{bmatrix}
f_1(1 - f_1) & -f_1 f_2 \\
-f_2 f_1 & f_2(1 - f_2)
\end{bmatrix}
= \begin{bmatrix} \tfrac14 & -\tfrac14 \\ -\tfrac14 & \tfrac14 \end{bmatrix}.
$$

Each row sums to zero — a nudge that raises one logit lowers the other output by
the same amount, since softmax outputs must keep summing to one. That structural
fact, visible directly in the Jacobian, is why softmax gradients redistribute
probability rather than create it.

Second derivatives measure _curvature_. For a scalar loss the matrix of second
partials is the **Hessian** $H$, with $H_{i,j} = \tfrac{\partial^2 f}{\partial x_i \partial x_j}$;
because differentiation order does not matter for the smooth losses we meet, $H$ is
symmetric. Its eigenvalues are the curvatures along the principal directions, and
their spread — the condition number — decides how hard the loss surface is to
descend: a direction of large curvature forces a small step, while a direction
of small curvature then crawls.[^gf-hessian] For the bowl $f = x_1^2 + 3x_2^2$ the
Hessian is $H = \diag(2, 6)$, already diagonal, so its eigenvalues are
$2$ and $6$: the $x_2$ axis curves three times as sharply. The condition number is
$\kappa = 6/2 = 3$, a mild stretch here, but real losses reach $\kappa$ in the
thousands, and that ratio is precisely how many more steps gradient descent needs.
Curvature is what separates first-order methods (gradient descent, which sees only
$\nabla f$) from second-order ones (Newton's method, which divides by $H$).

## The chain rule

Composition is the one operation a network does over and over, since a layer feeds
the next, so the rule for differentiating a composition is the rule that matters most.
For scalars, if $y = g(x)$ and $z = f(y)$, then

$$
\frac{\mathrm{d}z}{\mathrm{d}x} = \frac{\mathrm{d}z}{\mathrm{d}y}\,\frac{\mathrm{d}y}{\mathrm{d}x}.
$$

Local rates multiply. The vector form is the same statement with Jacobians in place
of derivatives, and order now matters because matrix products do not commute: for
$x \in \mathbb{R}^{n}$, $y = g(x) \in \mathbb{R}^{m}$, and a scalar $z = f(y)$,

$$
\nabla_{x} z = \left(\frac{\partial y}{\partial x}\right)^{\!\top} \nabla_{y} z,
$$

where $\partial y/\partial x$ is the $m \times n$ Jacobian of $g$.[^gf-chain] Read
right to left, this says: take the gradient with respect to the later quantity $y$,
then pull it back through the layer's Jacobian to get the gradient with respect to
the earlier quantity $x$. Stack many layers and you multiply many Jacobians — and
the _order_ in which you multiply them is the difference between a tractable
algorithm and an intractable one.

## Back-propagation: the chain rule on a graph

Write the network as a **computational graph**: a directed acyclic graph whose
nodes are intermediate values and whose edges are the operations that produce them.
The forward pass evaluates nodes in topological order to reach the loss. The
backward pass then visits the nodes in _reverse_ order, and at each one multiplies
the incoming gradient by that node's local derivative — the chain rule, applied one
edge at a time.

$$
% caption: Back-propagation on a computational graph. The forward pass (black)
%          evaluates the loss left to right; the backward pass (blue) carries the
%          gradient right to left, each node multiplying the incoming gradient by
%          its own local derivative. Here $g_v=\partial L/\partial v$ and
%          $g_z=g_a\,h'(z)$, $g_x=g_z\,w$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  nd/.style={draw, thick, circle, minimum size=8mm}]
  \definecolor{acc}{HTML}{2348F2}
  \node[nd] (x) at (0,0)   {$x$};
  \node[nd] (z) at (2.6,0) {$z$};
  \node[nd] (a) at (5.2,0) {$a$};
  \node[nd] (L) at (7.8,0) {$L$};
  \draw[->, thick] (x) -- node[above, font=\scriptsize] {$z = wx+b$} (z);
  \draw[->, thick] (z) -- node[above, font=\scriptsize] {$a = h(z)$} (a);
  \draw[->, thick] (a) -- node[above, font=\scriptsize] {$L(a)$} (L);
  \draw[->, thick, acc] (L) to[bend left=26] node[below, font=\scriptsize, text=acc] {$g_a$} (a);
  \draw[->, thick, acc] (a) to[bend left=26] node[below, font=\scriptsize, text=acc] {$g_z = g_a \, h_z$} (z);
  \draw[->, thick, acc] (z) to[bend left=26] node[below, font=\scriptsize, text=acc] {$g_x = g_z \, w$} (x);
  \node[font=\footnotesize] at (0.2,1.15) {\texttt{forward}};
  \node[font=\footnotesize, acc] at (0.2,-1.6) {\texttt{backward}};
\end{tikzpicture}
$$

The local factors read off the operations directly: $g_a = \partial L/\partial a$
comes from the loss, $h_z = h'(z)$ is the activation's slope, and $w$ is the
linear layer's weight. Each blue arrow multiplies the gradient arriving from its
right by one local derivative, which is all back-propagation ever does.

Writing $g_v = \partial L / \partial v$ for the gradient of the loss with respect
to a node $v$, the backward sweep is the recurrence $g_v = \sum_{u} g_u \,
(\partial u / \partial v)$, summed over the children $u$ that $v$ feeds. The key
property is the cost: one forward and one backward pass compute the gradient with respect to
_every_ parameter in time proportional to the forward pass itself, independent of
how many parameters there are.[^gf-backprop] A full gradient for roughly the price
of one evaluation is what makes training networks with millions of weights feasible
at all.

### A backward pass by hand

For example, take the single-neuron graph above with
$w = 2$, $b = 1$, input $x = 3$, a squared-error loss against target $t = 4$, and a
$\tanh$ activation. The **forward pass** fills in each node:

$$
z = wx + b = 2\cdot 3 + 1 = 7,
\qquad
a = \tanh(7) \approx 0.99999,
\qquad
L = \tfrac12 (a - t)^2 \approx \tfrac12 (0.99999 - 4)^2 \approx 4.5.
$$

The **backward pass** starts at the output with $g_L = 1$ and multiplies one local
derivative per edge, right to left:

$$
g_a = \frac{\partial L}{\partial a} = a - t \approx -3.0,
\qquad
g_z = g_a \, h'(z) = g_a\,(1 - a^2) \approx -3.0 \cdot (1 - 0.99998) \approx -6\times10^{-5}.
$$

$$
g_w = g_z \, \frac{\partial z}{\partial w} = g_z \, x \approx -1.8\times10^{-4},
\qquad
g_b = g_z \, \frac{\partial z}{\partial b} = g_z \approx -6\times10^{-5}.
$$

The weight gradient is tiny even though the loss is large: $\tanh$ has saturated at
$z = 7$, so $h'(z) = 1 - a^2 \approx 0$ throttles every gradient flowing back
through it. This is the **vanishing-gradient** mechanism in miniature — a saturated
activation multiplies the backward signal by nearly zero, and stacking many such
factors is why deep sigmoid/tanh networks were once hard to train.

### When a value feeds two places

The single-neuron chain hid a subtlety: what happens when a node feeds _more than
one_ child? The recurrence $g_v = \sum_u g_u\,(\partial u/\partial v)$ says the
gradients **add**. Consider $v$ used twice, once as $a = v^2$ and once as
$b = 3v$, with the loss $L = a + b$. The forward graph forks at $v$ and rejoins at
$L$; the backward pass sends a gradient down each branch and sums them at $v$:

$$
% caption: A value $v$ that feeds two children $a=v^2$ and $b=3v$. The backward
% pass sends a gradient down each branch (blue) and sums them at $v$:
% $g_v = g_a\,2v + g_b\,3$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  nd/.style={draw, thick, circle, minimum size=8mm}]
  \definecolor{acc}{HTML}{2348F2}
  \node[nd] (v) at (0,0)    {$v$};
  \node[nd] (a) at (3,1.1)  {$a$};
  \node[nd] (b) at (3,-1.1) {$b$};
  \node[nd] (L) at (6,0)    {$L$};
  \draw[->, thick] (v) -- node[pos=0.62, above left, font=\scriptsize] {$a = v^2$} (a);
  \draw[->, thick] (v) -- node[pos=0.62, below left, font=\scriptsize] {$b = 3v$} (b);
  \draw[->, thick] (a) -- node[above right, font=\scriptsize] {$+$} (L);
  \draw[->, thick] (b) -- node[below right, font=\scriptsize] {$+$} (L);
  \draw[->, thick, acc] (a) to[bend right=40] node[pos=0.72, left, font=\scriptsize, text=acc] {$g_a\,2v$} (v);
  \draw[->, thick, acc] (b) to[bend left=40] node[pos=0.72, left, font=\scriptsize, text=acc] {$g_b\,3$} (v);
\end{tikzpicture}
$$

With $L = a + b$ both $g_a = g_b = 1$, so $g_v = 1 \cdot 2v + 1 \cdot 3 = 2v + 3$
— which is exactly $\mathrm{d}L/\mathrm{d}v$ for $L = v^2 + 3v$, checked directly.
The **multivariate chain rule** is this summation: a shared value accumulates one
contribution per path it influences the loss through. In a real network the shared
value is a layer's activation feeding every unit of the next layer, and summing
those contributions is what the matrix-transpose in $\nabla_x z = J^\top \nabla_y z$
performs at once.

## Matrix calculus: the identities that recur

Layer gradients are the same handful of expressions over and over, so it pays to
know them by shape rather than rederiving each one. With $x, b$ vectors and $A, W$
matrices of compatible shape:

| Expression | Gradient | Where it shows up |
| --- | --- | --- |
| $b^\top x$ | $\nabla_x = b$ | a linear readout |
| $x^\top x$ | $\nabla_x = 2x$ | an $L^2$ penalty |
| $x^\top A x$ | $\nabla_x = (A + A^\top)x$ | a quadratic form / curvature term |
| $\lVert Wx - y\rVert^2$ | $\nabla_W = 2(Wx-y)x^\top$ | a linear layer's weight gradient |
| $\lVert W\rVert_F^2$ | $\nabla_W = 2W$ | weight decay |

None of these need memorizing as facts; each falls out of the scalar rules applied
componentwise. Take the quadratic form $x^\top A x = \sum_{i,j} A_{ij} x_i x_j$.
Differentiating with respect to $x_k$, the terms that survive are those containing
$x_k$: one from $i = k$ (giving $\sum_j A_{kj} x_j$, the $k$-th entry of $Ax$) and
one from $j = k$ (giving $\sum_i A_{ik} x_i$, the $k$-th entry of $A^\top x$). Their
sum is the $k$-th entry of $(A + A^\top)x$, which is the whole gradient. When $A$ is
symmetric this collapses to the familiar $2Ax$, the multivariate analogue of
$\tfrac{\mathrm d}{\mathrm dx}(ax^2) = 2ax$.

The outer-product shape of the linear layer's gradient, $g_W = g_z\, x^\top$,
is worth memorizing: the weight gradient is the upstream gradient times the layer's
input, transposed — exactly the local rule the graph above applies at a matrix
multiply. One small case makes the shapes concrete. Let a layer have input
$x = (2, -1)^\top$, and suppose the backward pass has delivered an upstream gradient
$g_z = (0.5, 3, -1)^\top$ at the layer's three outputs. Then

$$
g_W = g_z\, x^\top
= \begin{bmatrix} 0.5 \\ 3 \\ -1 \end{bmatrix}
  \begin{bmatrix} 2 & -1 \end{bmatrix}
= \begin{bmatrix} 1 & -0.5 \\ 6 & -3 \\ -2 & 1 \end{bmatrix},
\qquad
g_x = W^\top g_z,
$$

a $3 \times 2$ matrix matching $W$ exactly, while the gradient _passed further back_
to the input is $g_x = W^\top g_z$. Row $i$, column $j$ of $g_W$ is
$g_{z,i}\,x_j$: output $i$'s error scaled by the input $j$ that fed the weight
connecting them. The shapes line up as an outer product, an $(m,1)$ column times a $(1,n)$
row giving the $(m,n)$ gradient that matches $W$ exactly:

$$
% caption: The weight gradient as an outer product: upstream gradient $g_z$
% (shape $m\times 1$) times input $x^\top$ (shape $1\times n$) gives $g_W$
% with the same $m\times n$ shape as $W$.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \draw[acc, fill=acc!10] (0,0) rectangle (0.5,2.0);
  \node[acc, anchor=north] at (0.25,-0.1) {$g_z$};
  \node[anchor=east, font=\scriptsize] at (-0.1,1.0) {$m$};
  \node[font=\scriptsize] at (1.05,1.0) {times};
  \draw[acc, fill=acc!10] (1.6,1.5) rectangle (4.1,2.0);
  \node[acc, anchor=south] at (2.85,2.05) {$x^{T}$};
  \node[anchor=south, font=\scriptsize] at (2.85,2.35) {$n$};
  \node at (4.6,1.0) {$=$};
  \draw[acc, fill=acc!18] (5.2,0) rectangle (7.7,2.0);
  \node[acc] at (6.45,1.0) {$g_W$};
  \node[anchor=west, font=\scriptsize] at (7.8,1.0) {shape (m, n), same as W};
\end{tikzpicture}
$$

## Taylor expansion and what curvature buys

Truncating the Taylor series turns a messy loss into a polynomial we can optimize in
closed form, which is where the optimization algorithms come from. Around a point
$x$, a step $d$ changes the loss by

$$
f(x + d) \approx f(x) + \nabla f(x)^\top d + \tfrac{1}{2}\,d^\top H\, d.
$$

Keep only the linear term and minimizing pushes $d$ along $-\nabla f$: that is
**gradient descent**, and the step size must stay small enough for the linear
approximation to hold. Keep the quadratic term and minimizing over $d$ gives
$d = -H^{-1}\nabla f$: **Newton's method**, which rescales the gradient by the
inverse curvature so every direction descends at a matched rate.[^gf-taylor]

The one-step property is easiest to see on the very bowl from earlier,
$f(x) = x_1^2 + 3x_2^2$, starting from $x = (1, 1)$. Its gradient is
$\nabla f = (2x_1, 6x_2) = (2, 6)$ and its Hessian is the constant
$H = \diag(2, 6)$, so the Newton step is

$$
d = -H^{-1}\nabla f
= -\begin{bmatrix} \tfrac12 & 0 \\ 0 & \tfrac16 \end{bmatrix}
   \begin{bmatrix} 2 \\ 6 \end{bmatrix}
= (-1, -1),
$$

landing at $x + d = (0, 0)$ — the exact minimum, in a single step, from a starting
point where plain gradient descent would zig-zag because the $x_2$ axis has three
times the curvature of $x_1$. Newton divides each coordinate by its own curvature
($2$ and $6$), undoing the conditioning that gradient descent's single step size
cannot. On a quadratic this is exact; on a real loss it holds only locally, and the
Hessian is far too large to invert. Newton
is too expensive to run on a full network (the Hessian $H$ is
parameter-by-parameter), but the
expansion explains why poorly conditioned losses are slow for plain gradient
descent and why so much of optimization is, in effect, cheap approximations to
$H^{-1}$.

## Automatic differentiation

Back-propagation is one instance of **automatic differentiation** (autodiff):
mechanically applying the chain rule across a program's operations, each of which
ships with a known local derivative. Two orderings are possible. _Forward mode_
pushes derivatives along with values during the forward pass, which is efficient
when there are few inputs and many outputs. _Reverse mode_ records the forward
computation, then walks it backward, efficient when there are many inputs and one
output. A loss maps many inputs to a single output, so reverse-mode autodiff _is_
back-propagation, and it is what every framework's automatic-gradient engine
runs.[^gf-autodiff] In practice you write only the forward computation; the engine
records each operation on a tape and replays it in reverse to accumulate every
gradient, so a model's `backward` step is autodiff, not a derivative you ever code
by hand.[^stevens-autograd]

The choice of mode is a cost argument about matrix-product association. Composing
$k$ layers, the chain rule is a product of Jacobians $J_k J_{k-1} \cdots J_1$.
Forward mode multiplies right to left, carrying an $n \times 1$ tangent forward;
reverse mode multiplies left to right, carrying a $1 \times m$ gradient backward.
For $n$ inputs and $1$ output, reverse mode touches each intermediate once and
wins decisively.

For example, a network with $n = 10^6$ parameters and a single scalar loss
would, under forward mode, need one sweep _per input direction_ to fill in all
partials — roughly $10^6$ forward passes for one gradient. Reverse mode records a
single forward pass, then a single backward pass recovers all $10^6$ partials at
once, because the $1 \times m$ gradient it carries backward already has one slot per
parameter. The asymmetry is entirely about which end has one number: differentiate
a $\mathbb{R}^1 \to \mathbb{R}^{10^6}$ map instead and forward mode would win by the
same margin. Losses are always the many-in, one-out shape, so every training
framework defaults to reverse mode.

$$
% caption: Forward mode (top) propagates derivatives with values, one sweep per
% input; reverse mode (bottom) records the forward values then sweeps gradients
% back, one sweep for all inputs. A loss (many inputs, one output) picks reverse.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  nd/.style={draw, circle, minimum size=6.5mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % forward mode row
  \node[nd] (fx) at (0,1.4) {$x$};
  \node[nd] (f1) at (2,1.4) {$v_1$};
  \node[nd] (f2) at (4,1.4) {$v_2$};
  \node[nd] (fy) at (6,1.4) {$L$};
  \draw[->, green, thick] (fx) -- (f1);
  \draw[->, green, thick] (f1) -- (f2);
  \draw[->, green, thick] (f2) -- (fy);
  \node[green, anchor=west] at (6.4,1.4) {\texttt{forward} mode};
  \node[anchor=east, font=\scriptsize] at (-0.35,1.4) {value + deriv};
  % reverse mode row
  \node[nd] (rx) at (0,-0.2) {$x$};
  \node[nd] (r1) at (2,-0.2) {$v_1$};
  \node[nd] (r2) at (4,-0.2) {$v_2$};
  \node[nd] (ry) at (6,-0.2) {$L$};
  \draw[->, black, thick] (rx) -- (r1);
  \draw[->, black, thick] (r1) -- (r2);
  \draw[->, black, thick] (r2) -- (ry);
  \draw[->, acc, thick] (ry) to[bend left=32] (r2);
  \draw[->, acc, thick] (r2) to[bend left=32] (r1);
  \draw[->, acc, thick] (r1) to[bend left=32] (rx);
  \node[acc, anchor=west] at (6.4,-0.2) {reverse mode};
  \node[anchor=east, font=\scriptsize] at (-0.35,-0.2) {value then grad};
\end{tikzpicture}
$$

## Differentiation as a programmable object

The chain rule Goodfellow presents as a fixed backward sweep has, since 2016,
become a **programmable** object. Three developments matter for practice.

**Composable transforms.** The JAX system (Bradbury et al., 2018) exposes
reverse-mode (`grad`) and forward-mode (`jvp`) as functions that take a function
and return a function, so they compose freely. A **Hessian-vector product** $Hv$,
needed by second-order and curvature-based methods, is then just forward-over-reverse
differentiation, $\nabla(\nabla f \cdot v)$, computed in the cost of two passes and
_without ever forming_ the $n \times n$ Hessian. That trick is what makes
Newton-flavored ideas usable on networks the naive $O(n^3)$ inverse rules out.

**Trading memory for compute.** Reverse mode must store every intermediate from the
forward pass to reuse on the way back, and for a deep network that tape is the
memory bottleneck. **Gradient checkpointing** (Chen et al., 2016) keeps only a
sparse set of activations and _recomputes_ the rest during the backward pass,
cutting memory from $O(\text{depth})$ to $O(\sqrt{\text{depth}})$ at the price of
one extra forward pass — the standard way very deep or very wide models fit in
device memory.

**Differentiating through everything.** Once autodiff is a general program
transformation, the "operations" it differentiates need not be layers: physics
simulators, ODE solvers (the neural-ODE line of Chen et al., 2018, which
back-propagates through an adaptive integrator via the adjoint method), and even
optimization solvers become differentiable subroutines. The single idea of this
lesson — local derivatives multiplied along a graph in reverse — is what all of
them run underneath.[^jax][^checkpoint]

## Takeaways

- The **gradient** $\nabla f$ points in the direction of steepest increase; descent
  moves along $-\nabla f$ because it decreases the loss fastest among all directions.
- The **Jacobian** is the first derivative of a vector-valued map; the **Hessian** is
  the second derivative of a scalar loss, and its eigenvalue spread (conditioning)
  governs how hard the surface is to descend.
- The **chain rule** multiplies local rates; in vector form it pulls a gradient back
  through a layer's Jacobian, and the multiplication order is what makes the
  algorithm cheap or expensive.
- **Back-propagation** is the chain rule run in reverse topological order over the
  computational graph, computing every parameter's gradient for about the cost of one
  forward pass.
- A second-order **Taylor expansion** reads off gradient descent (linear term) and
  Newton's method (quadratic term), and explains why conditioning controls
  convergence.
- **Reverse-mode automatic differentiation** is back-propagation generalized to
  arbitrary programs, and it is the algorithm the framework runs when you call
  `backward`.

[^gf-grad]: **Goodfellow**, §4.3 — Gradient-Based Optimization: the gradient as the direction of steepest ascent and the directional-derivative argument for descending along its negative.
[^gf-hessian]: **Goodfellow**, §4.3.1 — the Hessian's eigenvalues as directional curvatures, and the condition number as the cause of slow, zig-zagging descent on poorly scaled problems.
[^gf-chain]: **Goodfellow**, §6.5.2 — the chain rule of calculus in scalar and vector (Jacobian) form, the basis for propagating gradients through composed functions.
[^gf-backprop]: **Goodfellow**, §6.5 — Back-Propagation: computing the gradient of the loss with respect to all parameters in a single reverse sweep at the cost of the forward pass.
[^gf-taylor]: **Goodfellow**, §4.3 — the second-order Taylor expansion, the gradient-descent step from its linear term, and Newton's $-H^{-1}\nabla f$ step from the quadratic term.
[^gf-autodiff]: **Goodfellow**, §6.5.9 — back-propagation as a special case of automatic differentiation, and the reverse-mode ordering for the many-inputs, single-output case.
[^stevens-autograd]: **Stevens**, _Deep Learning with PyTorch_, Ch. 5 — autograd records operations on the forward pass and replays them in reverse, so `backward` accumulates gradients without hand-coded derivatives.
[^jax]: Bradbury, Frostig, Hawkins, Johnson, Leary, Maclaurin, Necula, Paszke, VanderPlas, Wanderman-Milne, Zhang (2018), _JAX: composable transformations of Python+NumPy programs_ — `grad`/`jvp`/`vjp` as composable function transforms, and Hessian-vector products via forward-over-reverse differentiation without materializing the Hessian.
[^checkpoint]: Chen, Xu, Zhang, Guestrin (2016), _Training Deep Nets with Sublinear Memory Cost_, arXiv:1604.06174 — gradient checkpointing recomputes activations in the backward pass to reduce memory from $O(n)$ to $O(\sqrt{n})$; see also Chen, Rubanova, Bettencourt, Duvenaud (2018), _Neural Ordinary Differential Equations_, NeurIPS, for the adjoint method that differentiates through an ODE solver.