The Optimization Landscape

Every optimizer in this chapter (SGD, momentum and the adaptive methods) is a rule for walking downhill on the same object: the empirical risk $\mathcal{L}(\theta )$ as a function of the parameter vector $\theta \in {\mathbb{R}}^d$, with $d$ in the millions or billions. This lesson studies the shape of that surface. The shape is what makes optimization hard, and it is also why optimization works at all.¹

Non-convexity

A function is convex if the chord between any two points lies on or above its graph; equivalently its Hessian $H={\nabla }^2\mathcal{L}$ is positive semidefinite everywhere. Convexity is the property that makes linear and logistic regression easy: a convex function has one basin, so any point where the gradient vanishes is a global minimum, and gradient descent is guaranteed to find it.

Deep networks lose this property once they have a hidden layer. The reason is weight-space symmetry: permuting the $m$ hidden units of a layer, or flipping the sign convention across a $\tanh$ unit, produces a different $\theta$ computing the same function with the same loss. Any minimum therefore comes with an exponential orbit of equivalent minima (at least $m!$ of them per layer), and a function with many separated global minima cannot be convex.²

There is no algorithm guaranteed to find a global minimum of a general non-convex function in polynomial time. By the symmetry argument the global minimum is anyway not unique, so the goal shifts: we do not need the minimum, only a $\theta$ whose loss is low enough. Empirically, plain local search reliably finds such a point; the rest of this lesson explains why the obstacles that could trap it mostly do not.

Critical points and the Hessian

Local search stalls wherever the gradient vanishes. Such a point is a critical point; its type is decided by the local curvature, which is the Hessian $H={\nabla }^2\mathcal{L}$, the symmetric matrix of second partials $H_{ij}={\partial }^2\mathcal{L}/\partial {\theta }_i\partial {\theta }_j$.

The claim that curvature alone fixes the type follows from the second-order Taylor expansion around any point ${\theta }_0$, displaced by a step $d$:

At a critical point the linear term $g^{\top }d$ vanishes because $g=0$, so the leading behaviour is the quadratic form $\frac{1}{2}{d}^{\top }Hd$ alone: the surface looks, to second order, like a pure paraboloid centred at ${\theta }^{\star }$. Everything about whether that paraboloid opens upward, downward, or both is encoded in $H$.

Diagonalize $H=Q\Lambda Q^{\top }$ (possible because $H$ is symmetric, so its eigenvectors are orthonormal and its eigenvalues real) and change coordinates to the eigenbasis, $u=Q^{\top }v$ with $v=\theta -{\theta }^{\star }$. Since $Q$ is orthogonal it preserves lengths, and the quadratic term decouples into a sum of $d$ independent parabolas, one per eigendirection:

Along eigendirection $i$ the loss curves up if ${\lambda }_i>0$ and down if ${\lambda }_i<0$, with steepness set by $|{\lambda }_i|$: the eigenvalue is literally the second derivative of the loss along its eigenvector. The sign pattern across all $d$ eigenvalues classifies the point completely.

With two eigendirections the whole taxonomy fits on one strip: each panel draws the loss profile along $u_1$ and along $u_2$, and the pair of curvature signs names the point. Both cupped up is a minimum; both cupped down is a maximum; one of each is a saddle.

The three non-degenerate types are easiest to see as contour insets. A minimum has closed elliptical contours; a maximum has the same closed contours but with the loss decreasing outward; a saddle has the open, hyperbolic contours of $u_1^2-u_2^2$.

The saddle is the important case, because it dominates high-dimensional loss surfaces. Its descent and escape directions are orthogonal: the gradient pulls the iterate into the saddle along the up-curving axis, where it stalls; the only way out is along the down-curving axis, where the gradient gives no pull until the iterate is displaced.

Why saddles, not local minima, are the obstacle

The classical worry, inherited from low-dimensional intuition, is local minima: basins of bad loss that trap the optimizer below the global value. In high dimension this worry is misplaced. The reason is a counting argument on the eigenvalue signs.

At a random critical point, model each eigenvalue's sign as a near-independent coin flip with $P({\lambda }_i>0)=p$. A point is a local minimum only if every one of its $d$ eigenvalues is positive:

For any $p$ short of certainty, $p^d$ collapses to zero as $d$ grows. With even a modest $d=10^6$, the probability that all signs agree is astronomically small, so critical points are overwhelmingly mixed: saddles. The same vanishing applies to maxima. Local minima do exist, but the counting says they sit only at the very bottom: random-matrix theory makes the relationship sharp.

The Hessian's eigenvalues fill a semicircular band whose whole position drifts rightward as the loss drops. At high loss the band straddles zero with most of its mass on the negative side, so the point has a majority of down-directions and is a highly unstable saddle. As the loss falls the band slides right; the count of negative eigenvalues shrinks. Only in the lowest sliver does the entire band clear zero, and only there is the point a true local minimum.

The practical consequence: a high-loss critical point almost never traps the optimizer, because it almost always has a downhill escape direction. The escape can be slow, though. Near a saddle the gradient is small in every direction (it must vanish at the center), so plain gradient descent makes little progress there. The surrounding lessons supply the remedies.

Why noise escapes is again the per-eigencoordinate recursion. Along an escape direction with ${\lambda }_i<0$, one gradient step scales the displacement by $1-\eta {\lambda }_i>1$, so any nonzero component $u_i$ grows each step:

If the iterate lands exactly on the saddle's stable manifold — the subspace spanned by the positive-eigenvalue directions, where $u_i^{(0)}=0$ along every negative axis — it stalls, because there is nothing to amplify. That exact landing has measure zero, and gradient noise from minibatch sampling guarantees a nonzero $u_i^{(0)}$ along the unstable axis. Once seeded, that component blows up geometrically at rate $(1-\eta {\lambda }_i)$ and the iterate slides off the saddle. The stall near the center is real but temporary; the escape time scales with $1/|{\lambda }_{\min }|$, so the flattest saddles (small $|{\lambda }_{\min }|$) hold the iterate longest.

The rest of the terrain

Saddles are the dominant feature, but four further features of the surface shape how a step should be taken.

Plateaus

A plateau is an extended region where the gradient is near zero without a critical point in sight: the eigenvalues are small in magnitude, so the surface is nearly flat. Saturating activations create them: a unit deep in the flat tail of a sigmoid or $\tanh$ has ${\sigma }^{\prime }(z)\approx 0$, so the gradient that reaches its weights is attenuated to almost nothing and progress stalls for many steps.

Cliffs and exploding gradients

The opposite pathology is a cliff: a near-vertical wall where the loss changes by a large amount over a tiny change in $\theta$, so the gradient is enormous. Common in recurrent networks, where repeated multiplication by a recurrent weight matrix makes the gradient magnitude grow geometrically with sequence length. A normal-sized learning rate times a huge gradient yields a huge step that throws the iterate far across the surface, undoing many updates of progress.

The standard remedy is gradient clipping: rescale the gradient to a fixed norm $c$ before stepping, keeping its direction but capping its length,

so the step is bounded by $\eta c$ no matter how steep the wall. The direction (which is correct, pointing along the wall toward lower loss) is preserved; only the magnitude is capped.⁴

Ill-conditioning

Even a perfectly convex bowl can be hard to descend if it is stretched: long and narrow rather than round. The stretch is measured by the condition number of the Hessian, the ratio of its largest to smallest eigenvalue,

The reason a large $\kappa$ hurts is visible in the decoupled coordinates. On a quadratic bowl $\mathcal{L}=\frac{1}{2}{\sum }_i{\lambda }_i{u}_i^2$, gradient descent with rate $\eta$ updates each eigencoordinate independently,

so coordinate $i$ decays geometrically at rate $|1-\eta {\lambda }_i|$. For every coordinate to converge the rate must satisfy $|1-\eta {\lambda }_i|<1$, i.e. $\eta <2/{\lambda }_{\max }$ — the steep axis sets the ceiling. Plug that largest usable rate into the gentle axis and the decay factor is $1-\eta {\lambda }_{\min }\approx 1-2{\lambda }_{\min }/{\lambda }_{\max }=1-2/\kappa$, which is close to $1$ when $\kappa \gg 1$. The steep axis forces a small step; the gentle axis then crawls, needing on the order of $\kappa$ iterations to make the same relative progress. The iterate zig-zags across the narrow valley while inching down its length. This is the structural reason for momentum and for per-coordinate adaptive learning rates, both of which counteract a large $\kappa$.

Sharp versus flat minima

Not all minima generalize equally, and the geometry that distinguishes them is again curvature. A sharp minimum sits at the bottom of a narrow, steep basin (large Hessian eigenvalues); a flat minimum sits in a wide, shallow basin (small eigenvalues). They can have identical training loss yet behave very differently on test data.

The link to generalization is a shift argument. Train and test losses are the same functional shape evaluated on different samples, so the test-loss curve is approximately the train-loss curve shifted by a small displacement $\delta$. At a sharp minimum the steep walls mean a small horizontal shift produces a large vertical gap — the test loss at the trained ${\theta }^{\star }$ is much higher than the train loss. At a flat minimum the same shift barely raises the loss, so train and test nearly coincide.

This is why the implicit bias of SGD toward flat regions (small batches inject more gradient noise, and noise cannot rest in a narrow basin) is widely believed to be part of why deep networks generalize at all.⁵ The formal version of the train-test-gap argument is developed in generalization theory.

The landscape, made quantitative

Every claim in this lesson that Goodfellow states qualitatively has a public paper that made it quantitative.

• Saddles, counted. Dauphin et al. gave the random-matrix argument behind the sliding-spectrum figure and turned it into an algorithm — saddle-free Newton, which uses $|H|$ to escape saddles that both gradient descent and raw Newton get stuck near. Choromanska et al. connected the deep loss surface to the spin-glass models of statistical physics, making high-loss critical points are saddles a theorem about a tractable proxy.⁶⁷
• Flat minima and generalization, measured. Keskar et al. showed empirically that large-batch training lands in sharper minima that generalize worse, and quantified sharpness by the loss increase in a small neighborhood — the shift-argument figure made into a metric. Sharpness-Aware Minimization (Foret et al.) then turned the observation into an optimizer that explicitly seeks flat regions.⁸⁹
• Mode connectivity, made visible. The remark above is Garipov et al.'s and Draxler et al.'s finding: independently trained minima are joined by low-loss curves. Li et al.'s loss-landscape visualizations are the source of the now-standard pictures of wide flat basins versus chaotic sharp ones.¹⁰¹¹

The pattern across these is that high-dimensional geometry is benign in a way low-dimensional intuition does not predict: isolated deep local minima are vanishingly rare, and the real obstacles (saddles, sharp basins, ill-conditioning) all have standard remedies.

Takeaways

• The deep loss surface is non-convex (weight-space symmetries alone force exponentially many equivalent minima), so local search has no global guarantee; the realistic goal is a $\theta$ with low enough loss, which local search reliably finds.
• Critical points ($\nabla \mathcal{L}=0$) are classified by the Hessian eigenvalues: all positive $\Rightarrow$ minimum, all negative $\Rightarrow$ maximum, mixed signs $\Rightarrow$ saddle.
• In high dimension the chance that all $d$ eigenvalues share a sign is $\approx p^d\to 0$, so almost every critical point is a saddle, not a bad local minimum; high-loss critical points have many escape directions, and momentum and SGD noise carry the iterate past them.
• The remaining terrain — plateaus (vanishing gradients), cliffs (exploding gradients, cured by gradient clipping to a fixed norm), and ill-conditioning (large $\kappa (H)={\lambda }_{\max }/{\lambda }_{\min }$) — explains why the plain step needs momentum and adaptive rates.
• Flat minima generalize better than sharp ones: a small train-test shift opens a large gap in a steep basin and a small one in a wide basin, and mode connectivity shows the good minima form one connected low-loss region.