Numerical Computation

A learning algorithm is a continuous-mathematics object (gradients, integrals, matrix inverses) executed on a machine that stores real numbers in a finite number of bits. The translation is lossy. Every real is rounded to the nearest representable float, and the gap between adjacent floats grows with magnitude, so arithmetic accumulates rounding error that a careless formula can amplify without bound. Two failures dominate, and each has a standard remedy.¹

Overflow and underflow

A 64-bit float represents a number as a sign, a mantissa, and an exponent. The exponent is bounded, so the representable magnitudes occupy a finite window: below it numbers collapse to $0$, above it they saturate to $\pm \infty$.

For IEEE-754 double precision the boundaries are concrete: the largest finite value is about $1.8\times 10^{308}$ and the smallest positive normal value about $2.2\times 10^{-308}$. The function $\exp$ reaches these limits at tiny arguments ($e^x$ overflows for $x\gtrsim 709$ and underflows for $x\lesssim -745$), and that fragility makes softmax, a tower of exponentials, the canonical place where numerical care is mandatory.²

The softmax stability trick

The softmax turns a vector of scores into a probability distribution,

Two ways this breaks: if every $x_i$ is large and positive, each $e^{x_i}$ overflows; if every $x_i$ is large and negative, every $e^{x_i}$ underflows to $0$ and the denominator becomes $0/0$. The fix exploits a shift-invariance. For any constant $c$,

Subtracting a constant from every logit leaves the softmax unchanged. Choose $c={\max }_j{x}_j$:

The log-sum-exp trick

A second hazard hides downstream. Cross-entropy needs $\log \text{softmax}(x{)}_i$, and computing the softmax first then taking its log underflows: a correctly computed but tiny probability rounds to $0$, and $\log 0=-\infty$. Fold the log into the exponentials instead. Starting from the definition and pulling the shift through the logarithm,

The recurring quantity $\log {\sum }_j{e}^{x_j}$ is the log-sum-exp function, and the same shift makes it safe:

The shifted form also reveals $\mathrm{LSE}$ as a smooth maximum: ${\max }_j{x}_j\le \mathrm{LSE}(x)\le {\max }_j{x}_j+\log n$, so it never exceeds the true max by more than $\log n$, and equals it when one logit dominates. Library softmax/cross-entropy implementations fuse all of this into a single numerically stable kernel; never compute softmax and its log as two separate steps.³

For example, take logits $x=(1000,1001,1002)$. The naive denominator asks for $e^{1000}+e^{1001}+e^{1002}$, and each term overflows to $+\infty$ in double precision (recall $e^x$ saturates near $x\approx 709$), so the ratio returns $\infty /\infty =\text{NaN}$. Subtracting $c=1002$ first,

which is exact and never leaves the safe window. The shift changed nothing mathematically — only which intermediate values the machine had to hold.

Poor conditioning

Even arithmetic that never overflows can be untrustworthy if the underlying problem amplifies small input changes into large output changes. The amplifier is the condition number.

When $\kappa (A)$ is large the matrix is ill-conditioned, and the solution $x=A^{-1}b$ is highly sensitive to perturbations in $b$ (and to the rounding error already present in $A$). Decompose $A=Q\Lambda Q^{\top }$; the inverse stretches the component of $b$ along eigenvector $v_i$ by $1/{\lambda }_i$, so the direction of smallest eigenvalue is magnified most:

A perturbation $\delta b$ aligned with the small-eigenvalue direction is amplified by $1/{\lambda }_{\min }$ while $b$ itself may only be of size $1/{\lambda }_{\max }$, giving a worst-case relative error blowup of exactly $\kappa (A)$:

Gradient-based optimization

Most deep-learning algorithms reduce to optimization: choosing $x$ to minimize (or maximize) an objective function $f(x)$. When the goal is minimization we call $f$ the cost, loss, or error function, and write the target as $x^{\star }=\arg {\min }_{x}f(x)$.

In one dimension the derivative $f^{\prime }(x)$ gives the slope, and the update $x\leftarrow x-\eta f^{\prime }(x)$ steps downhill: where the slope is positive we decrease $x$, where negative we increase it. The generalization to many variables replaces the derivative with the gradient $\nabla f(x)$, the vector of partial derivatives ${\left[\partial f/\partial x_1,\dots ,\partial f/\partial x_n\right]}^{\top }$.

To descend fastest we pick the direction $u$ that minimizes this slope:

minimized at $\cos \theta =-1$, i.e. $u$ points opposite the gradient. This is the method of steepest descent: step against the gradient,

with $\eta >0$ the learning rate (step size).⁵

The full procedure is the base every later optimizer refines.

The Jacobian and the Hessian

When $f$ is vector-valued, $f:{\mathbb{R}}^m\to {\mathbb{R}}^n$, its first derivatives form a matrix.

For a scalar objective $f:{\mathbb{R}}^n\to \mathbb{R}$, the gradient's own derivative (the matrix of second partials) is the Hessian, and it encodes curvature.

The Hessian sharpens the critical-point test. The second-derivative test reads curvature off the eigenvalues of $H$ at a critical point:

Curvature sets the step size

Approximate $f$ near the current point $x$ by its second-order Taylor expansion in the gradient direction. Writing $g=\nabla f(x)$ and $H$ the Hessian at $x$, a step of size $\eta$ along $-g$ gives

The middle term is the gain we want; the last term is the curvature penalty. If $g^{\top }Hg>0$ the quadratic in $\eta$ is minimized by setting its derivative to zero, $-g^{\top }g+\eta g^{\top }Hg=0$, giving the optimal step size

High curvature ($g^{\top }Hg$ large) shrinks ${\eta }^{\star }$: a steep, narrow valley demands tiny steps; low curvature permits large ones.

The condition number of the Hessian

When the Hessian's eigenvalues differ wildly, a single scalar learning rate cannot serve every direction at once.

A step large enough to make progress along the gentle (small-$\lambda$) direction overshoots and oscillates along the steep (large-$\lambda$) direction; a step small enough to be stable on the steep direction barely moves on the gentle one. Gradient descent is forced to compromise and zig-zags down the valley, taking many steps to cross what is geometrically a short distance.

Newton's method

Second-order methods address this by using $H$ to rescale the step per-direction. Expand $f$ to second order around $x_0$ and minimize the approximation exactly. The quadratic model is

Setting the gradient of the right-hand side to zero,

yields the Newton step

The power comes at a cost: forming and inverting $H$ is $O(n^3)$ in the parameter count $n$, intractable for deep networks with millions of parameters, which is why practice relies on cheaper approximations. A second problem is subtler.

Constrained optimization

Often we minimize $f$ not over all of ${\mathbb{R}}^n$ but over a feasible set$\mathbb{S}$ carved out by constraints. The standard form has equality and inequality constraints,

The method of Lagrange multipliers converts this constrained problem into an unconstrained one by folding each constraint into the objective with its own multiplier.

At an equality-constrained optimum the objective's gradient must be parallel to the constraint's, $\nabla f=-\lambda \nabla g$; otherwise a move along the constraint surface would still decrease $f$. The general first-order optimality conditions are the Karush–Kuhn–Tucker (KKT) conditions.

Complementary slackness encodes a clean dichotomy: each inequality is either inactive ($h_j<0$, and its multiplier ${\alpha }_j=0$ — the constraint does not bind) or active ($h_j=0$, and ${\alpha }_j\ge 0$ — the constraint pins the solution to the boundary).⁷ The figure shows the active case, where the optimum leaves the unconstrained minimum and rests against the boundary.

Mixed precision and low-bit training

The overflow/underflow window this lesson treats for double precision has become a front-line engineering concern, because modern training deliberately uses fewer bits.

Mixed precision and the two 16-bit formats. Training in 16-bit floats halves memory and roughly doubles throughput on modern accelerators, but the choice of format comes down to an overflow/underflow tradeoff. IEEE fp16 has a 10-bit mantissa and only 5 exponent bits, so its representable window stops around $6.5\times 10^4$ and $6\times 10^{-8}$ — small gradients underflow to zero. bfloat16 keeps 8 exponent bits (the same dynamic range as fp32, up to $\sim 3\times 10^{38}$) at the cost of a 7-bit mantissa, trading precision for range precisely because underflow, not rounding, is what breaks training. This is the window diagram of this lesson made into a hardware decision (Micikevicius et al., 2017).

Loss scaling. When fp16 is unavoidable, loss scaling multiplies the loss by a large constant $S$ before back-propagation, shifting every gradient up into the representable window, then divides by $S$ before the weight update. It is the softmax shift trick in reverse: rescale to dodge the boundary, then undo the rescale, changing nothing mathematically while keeping every intermediate in the safe range.

Cheap curvature. The $O(n^3)$ Newton step this lesson rules out for large $n$ has practical stand-ins. K-FAC (Martens and Grosse, 2015) approximates the Hessian (really the Fisher information) as a Kronecker product of two small per-layer factors, each cheap to invert, capturing the conditioning that plain gradient descent ignores without ever forming the full matrix — the standard way second-order ideas reach real networks.⁸⁹

Takeaways

• Underflow (nonzero $\to 0$, deadly in denominators and logs) and overflow ($\to \pm \infty \to \text{NaN}$) are the two finite-precision failures; $\exp$ hits both at modest arguments.
• Softmax is stabilized by subtracting ${\max }_j{x}_j$ from every logit (leaving the result unchanged); the log-sum-exp identity $\mathrm{LSE}(x)=c+\log {\sum }_j{e}^{x_j-c}$ keeps $\log \text{softmax}$ finite; always fuse them, never compute softmax then log.
• The condition number $\kappa ={\lambda }_{\max }/{\lambda }_{\min }$ measures how much a problem amplifies error; ill-conditioned Hessians make gradient descent zig-zag.
• Steepest descent steps against the gradient; critical points classify by the Hessian eigenvalues (the second-derivative test), and curvature $g^{\top }Hg$ sets the optimal step ${\eta }^{\star }=g^{\top }g/g^{\top }Hg$.
• Newton's method $x\leftarrow x-H^{-1}\nabla f$ jumps to the minimum of a quadratic in one step but costs $O(n^3)$ and is attracted to saddle points.
• Constrained problems fold constraints into the Lagrangian; the KKT conditions (stationarity, feasibility, dual feasibility, complementary slackness) characterize the optimum, which an active inequality pins to the feasible boundary.