Calculus

A network learns by descending a loss, and descent needs a direction: the gradient. Every weight update a model ever makes is a derivative of the loss with respect to a parameter, and the algorithm that computes all of those derivatives at once, back-propagation, is the chain rule applied with care to a graph of operations. This lesson is the differential calculus that machinery rests on, kept to the objects a network differentiates and the one rule that ties them together.¹

Derivatives, gradients, and directions

For a scalar function of a scalar, $f:\mathbb{R}\to \mathbb{R}$, the derivative $f^{\prime }(x)=\mathrm{d}f/\mathrm{d}x$ is the local rate of change: scale a small step $ϵ$ and the output moves by about $f^{\prime }(x)ϵ$. Deep learning almost never has one input, though. A loss is a function of millions of parameters at once, $f:{\mathbb{R}}^n\to \mathbb{R}$, so the single derivative becomes a vector of partial derivatives, each holding all coordinates fixed but one.

The steepest claim follows from the directional derivative, the rate of change of $f$ as you move along a unit vector $u$. It equals $\nabla f(x{)}^{\top }u$, and since $\nabla f(x{)}^{\top }u=\Vert \nabla f(x)\Vert \cos \theta$, it is largest when $u$ aligns with $\nabla f$ and most negative when $u$ points the opposite way.¹ That is the whole justification for moving along $-\nabla f$: among all directions of a fixed length, it decreases $f$ fastest.

For example, take $f(x_1,x_2)=x_1^2+3x_2^2$, a stretched bowl, evaluated at $x=(1,1)$. The partials are $\partial f/\partial x_1=2x_1$ and $\partial f/\partial x_2=6x_2$, so

Now compare three unit directions and read the directional derivative off each. Along the steepest direction $u^{\star }=\nabla f/\Vert \nabla f\Vert =(2,6)/\sqrt{40}$, the slope is $\nabla f^{\top }{u}^{\star }=\Vert \nabla f\Vert \approx 6.32$ — the maximum. Along a horizontal step $u=(1,0)$ it is $2\cdot 1+6\cdot 0=2$, and along $u=(0,1)$ it is $6$: the bowl climbs three times faster in the $x_2$ direction, exactly because that axis has the steeper coefficient. Every one of these is $\Vert \nabla f\Vert \cos \theta$ with the corresponding angle, and the largest, $6.32$, belongs to $\cos \theta =1$. Descent flips the sign: stepping along $-u^{\star }$ drops $f$ at rate $6.32$, faster than any other unit step.

The Jacobian and the Hessian

When the output is itself a vector, as it is at every hidden layer ($f:{\mathbb{R}}^n\to {\mathbb{R}}^m$), the first derivative is a matrix.

For example, the softmax layer $f(x{)}_i=e^{x_i}/{\sum }_j{e}^{x_j}$ maps ${\mathbb{R}}^2\to {\mathbb{R}}^2$; its Jacobian entries are $\partial f_i/\partial x_j=f_i({\delta }_{ij}-f_j)$. At $x=(0,0)$ the outputs are $f=(\frac{1}{2},\frac{1}{2})$, so

Each row sums to zero — a nudge that raises one logit lowers the other output by the same amount, since softmax outputs must keep summing to one. That structural fact, visible directly in the Jacobian, is why softmax gradients redistribute probability rather than create it.

Second derivatives measure curvature. For a scalar loss the matrix of second partials is the Hessian $H$, with $H_{i,j}=\frac{{\partial }^{2}f}{\partial x_i\partial x_j}$; because differentiation order does not matter for the smooth losses we meet, $H$ is symmetric. Its eigenvalues are the curvatures along the principal directions, and their spread — the condition number — decides how hard the loss surface is to descend: a direction of large curvature forces a small step, while a direction of small curvature then crawls.² For the bowl $f=x_1^2+3x_2^2$ the Hessian is $H=\mathrm{diag}(2,6)$, already diagonal, so its eigenvalues are $2$ and $6$: the $x_2$ axis curves three times as sharply. The condition number is $\kappa =6/2=3$, a mild stretch here, but real losses reach $\kappa$ in the thousands, and that ratio is precisely how many more steps gradient descent needs. Curvature is what separates first-order methods (gradient descent, which sees only $\nabla f$) from second-order ones (Newton's method, which divides by $H$).

The chain rule

Composition is the one operation a network does over and over, since a layer feeds the next, so the rule for differentiating a composition is the rule that matters most. For scalars, if $y=g(x)$ and $z=f(y)$, then

Local rates multiply. The vector form is the same statement with Jacobians in place of derivatives, and order now matters because matrix products do not commute: for $x\in {\mathbb{R}}^n$, $y=g(x)\in {\mathbb{R}}^m$, and a scalar $z=f(y)$,

where $\partial y/\partial x$ is the $m\times n$ Jacobian of $g$.³ Read right to left, this says: take the gradient with respect to the later quantity $y$, then pull it back through the layer's Jacobian to get the gradient with respect to the earlier quantity $x$. Stack many layers and you multiply many Jacobians — and the order in which you multiply them is the difference between a tractable algorithm and an intractable one.

Back-propagation: the chain rule on a graph

Write the network as a computational graph: a directed acyclic graph whose nodes are intermediate values and whose edges are the operations that produce them. The forward pass evaluates nodes in topological order to reach the loss. The backward pass then visits the nodes in reverse order, and at each one multiplies the incoming gradient by that node's local derivative — the chain rule, applied one edge at a time.

The local factors read off the operations directly: $g_a=\partial L/\partial a$ comes from the loss, $h_z=h^{\prime }(z)$ is the activation's slope, and $w$ is the linear layer's weight. Each blue arrow multiplies the gradient arriving from its right by one local derivative, which is all back-propagation ever does.

Writing $g_v=\partial L/\partial v$ for the gradient of the loss with respect to a node $v$, the backward sweep is the recurrence $g_v={\sum }_u{g}_u(\partial u/\partial v)$, summed over the children $u$ that $v$ feeds. The key property is the cost: one forward and one backward pass compute the gradient with respect to every parameter in time proportional to the forward pass itself, independent of how many parameters there are.⁴ A full gradient for roughly the price of one evaluation is what makes training networks with millions of weights feasible at all.

A backward pass by hand

For example, take the single-neuron graph above with $w=2$, $b=1$, input $x=3$, a squared-error loss against target $t=4$, and a $\tanh$ activation. The forward pass fills in each node:

The backward pass starts at the output with $g_L=1$ and multiplies one local derivative per edge, right to left:

The weight gradient is tiny even though the loss is large: $\tanh$ has saturated at $z=7$, so $h^{\prime }(z)=1-a^2\approx 0$ throttles every gradient flowing back through it. This is the vanishing-gradient mechanism in miniature — a saturated activation multiplies the backward signal by nearly zero, and stacking many such factors is why deep sigmoid/tanh networks were once hard to train.

When a value feeds two places

The single-neuron chain hid a subtlety: what happens when a node feeds more than one child? The recurrence $g_v={\sum }_u{g}_u(\partial u/\partial v)$ says the gradients add. Consider $v$ used twice, once as $a=v^2$ and once as $b=3v$, with the loss $L=a+b$. The forward graph forks at $v$ and rejoins at $L$; the backward pass sends a gradient down each branch and sums them at $v$:

With $L=a+b$ both $g_a=g_b=1$, so $g_v=1\cdot 2v+1\cdot 3=2v+3$ — which is exactly $\mathrm{d}L/\mathrm{d}v$ for $L=v^2+3v$, checked directly. The multivariate chain rule is this summation: a shared value accumulates one contribution per path it influences the loss through. In a real network the shared value is a layer's activation feeding every unit of the next layer, and summing those contributions is what the matrix-transpose in ${\nabla }_{x}z=J^{\top }{\nabla }_{y}z$ performs at once.

Matrix calculus: the identities that recur

Layer gradients are the same handful of expressions over and over, so it pays to know them by shape rather than rederiving each one. With $x,b$ vectors and $A,W$ matrices of compatible shape:

None of these need memorizing as facts; each falls out of the scalar rules applied componentwise. Take the quadratic form $x^{\top }Ax={\sum }_{i,j}{A}_{ij}{x}_i{x}_j$. Differentiating with respect to $x_k$, the terms that survive are those containing $x_k$: one from $i=k$ (giving ${\sum }_j{A}_{kj}{x}_j$, the $k$-th entry of $Ax$) and one from $j=k$ (giving ${\sum }_i{A}_{ik}{x}_i$, the $k$-th entry of $A^{\top }x$). Their sum is the $k$-th entry of $(A+A^{\top })x$, which is the whole gradient. When $A$ is symmetric this collapses to the familiar $2Ax$, the multivariate analogue of $\frac{\mathrm{d}}{\mathrm{d}x}(a{x}^2)=2ax$.

The outer-product shape of the linear layer's gradient, $g_W=g_z{x}^{\top }$, is worth memorizing: the weight gradient is the upstream gradient times the layer's input, transposed — exactly the local rule the graph above applies at a matrix multiply. One small case makes the shapes concrete. Let a layer have input $x=(2,-1{)}^{\top }$, and suppose the backward pass has delivered an upstream gradient $g_z=(0.5,3,-1{)}^{\top }$ at the layer's three outputs. Then

a $3\times 2$ matrix matching $W$ exactly, while the gradient passed further back to the input is $g_x=W^{\top }{g}_z$. Row $i$, column $j$ of $g_W$ is $g_{z,i}{x}_j$: output $i$'s error scaled by the input $j$ that fed the weight connecting them. The shapes line up as an outer product, an $(m,1)$ column times a $(1,n)$ row giving the $(m,n)$ gradient that matches $W$ exactly:

Taylor expansion and what curvature buys

Truncating the Taylor series turns a messy loss into a polynomial we can optimize in closed form, which is where the optimization algorithms come from. Around a point $x$, a step $d$ changes the loss by

Keep only the linear term and minimizing pushes $d$ along $-\nabla f$: that is gradient descent, and the step size must stay small enough for the linear approximation to hold. Keep the quadratic term and minimizing over $d$ gives $d=-H^{-1}\nabla f$: Newton's method, which rescales the gradient by the inverse curvature so every direction descends at a matched rate.⁵

The one-step property is easiest to see on the very bowl from earlier, $f(x)=x_1^2+3x_2^2$, starting from $x=(1,1)$. Its gradient is $\nabla f=(2x_1,6x_2)=(2,6)$ and its Hessian is the constant $H=\mathrm{diag}(2,6)$, so the Newton step is

landing at $x+d=(0,0)$ — the exact minimum, in a single step, from a starting point where plain gradient descent would zig-zag because the $x_2$ axis has three times the curvature of $x_1$. Newton divides each coordinate by its own curvature ($2$ and $6$), undoing the conditioning that gradient descent's single step size cannot. On a quadratic this is exact; on a real loss it holds only locally, and the Hessian is far too large to invert. Newton is too expensive to run on a full network (the Hessian $H$ is parameter-by-parameter), but the expansion explains why poorly conditioned losses are slow for plain gradient descent and why so much of optimization is, in effect, cheap approximations to $H^{-1}$.

Automatic differentiation

Back-propagation is one instance of automatic differentiation (autodiff): mechanically applying the chain rule across a program's operations, each of which ships with a known local derivative. Two orderings are possible. Forward mode pushes derivatives along with values during the forward pass, which is efficient when there are few inputs and many outputs. Reverse mode records the forward computation, then walks it backward, efficient when there are many inputs and one output. A loss maps many inputs to a single output, so reverse-mode autodiff is back-propagation, and it is what every framework's automatic-gradient engine runs.⁶ In practice you write only the forward computation; the engine records each operation on a tape and replays it in reverse to accumulate every gradient, so a model's backward step is autodiff, not a derivative you ever code by hand.⁷

The choice of mode is a cost argument about matrix-product association. Composing $k$ layers, the chain rule is a product of Jacobians $J_k{J}_{k-1}\cdots J_1$. Forward mode multiplies right to left, carrying an $n\times 1$ tangent forward; reverse mode multiplies left to right, carrying a $1\times m$ gradient backward. For $n$ inputs and $1$ output, reverse mode touches each intermediate once and wins decisively.

For example, a network with $n=10^6$ parameters and a single scalar loss would, under forward mode, need one sweep per input direction to fill in all partials — roughly $10^6$ forward passes for one gradient. Reverse mode records a single forward pass, then a single backward pass recovers all $10^6$ partials at once, because the $1\times m$ gradient it carries backward already has one slot per parameter. The asymmetry is entirely about which end has one number: differentiate a ${\mathbb{R}}^1\to {\mathbb{R}}^{10^6}$ map instead and forward mode would win by the same margin. Losses are always the many-in, one-out shape, so every training framework defaults to reverse mode.

Differentiation as a programmable object

The chain rule Goodfellow presents as a fixed backward sweep has, since 2016, become a programmable object. Three developments matter for practice.

Composable transforms. The JAX system (Bradbury et al., 2018) exposes reverse-mode (grad) and forward-mode (jvp) as functions that take a function and return a function, so they compose freely. A Hessian-vector product $Hv$, needed by second-order and curvature-based methods, is then just forward-over-reverse differentiation, $\nabla (\nabla f\cdot v)$, computed in the cost of two passes and without ever forming the $n\times n$ Hessian. That trick is what makes Newton-flavored ideas usable on networks the naive $O(n^3)$ inverse rules out.

Trading memory for compute. Reverse mode must store every intermediate from the forward pass to reuse on the way back, and for a deep network that tape is the memory bottleneck. Gradient checkpointing (Chen et al., 2016) keeps only a sparse set of activations and recomputes the rest during the backward pass, cutting memory from $O(\text{depth})$ to $O(\sqrt{\text{depth}})$ at the price of one extra forward pass — the standard way very deep or very wide models fit in device memory.

Differentiating through everything. Once autodiff is a general program transformation, the operations it differentiates need not be layers: physics simulators, ODE solvers (the neural-ODE line of Chen et al., 2018, which back-propagates through an adaptive integrator via the adjoint method), and even optimization solvers become differentiable subroutines. The single idea of this lesson — local derivatives multiplied along a graph in reverse — is what all of them run underneath.⁸⁹

Takeaways

• The gradient $\nabla f$ points in the direction of steepest increase; descent moves along $-\nabla f$ because it decreases the loss fastest among all directions.
• The Jacobian is the first derivative of a vector-valued map; the Hessian is the second derivative of a scalar loss, and its eigenvalue spread (conditioning) governs how hard the surface is to descend.
• The chain rule multiplies local rates; in vector form it pulls a gradient back through a layer's Jacobian, and the multiplication order is what makes the algorithm cheap or expensive.
• Back-propagation is the chain rule run in reverse topological order over the computational graph, computing every parameter's gradient for about the cost of one forward pass.
• A second-order Taylor expansion reads off gradient descent (linear term) and Newton's method (quadratic term), and explains why conditioning controls convergence.
• Reverse-mode automatic differentiation is back-propagation generalized to arbitrary programs, and it is the algorithm the framework runs when you call backward.