---
title: The Optimization Landscape
module: Optimization
moduleNumber: 3
lessonNumber: 4
order: 304
summary: >
  The loss of a deep network is a non-convex surface in millions of dimensions, so
  local search carries no global guarantee, yet it works. We classify critical
  points by the eigenvalues of the Hessian, show that in high dimension nearly all
  of them are saddle points rather than bad local minima, and read off the practical
  terrain — plateaus, cliffs, ill-conditioning, and the sharp-versus-flat distinction
  that ties the geometry of a minimum to how well it generalizes.
topics: [Optimization]
sources:
  - book: Goodfellow
    ref: "§8.2 — Challenges in Neural Network Optimization"
  - book: Goodfellow
    ref: "§4.3 — Gradient-Based Optimization; §8.2.5 Cliffs and Exploding Gradients"
  - book: Chollet
    ref: "§2.4 — The Engine of Neural Networks: Gradient-Based Optimization"
---

Every optimizer in this chapter ([SGD](/deep-learning/optimization/gradient-descent-and-sgd),
[momentum and the adaptive methods](/deep-learning/optimization/momentum-and-adaptive-methods))
is a rule for walking downhill on the same object: the empirical risk
$\mathcal{L}(\theta)$ as a function of the parameter vector $\theta \in \mathbb{R}^d$,
with $d$ in the millions or billions. This lesson studies the _shape_ of that
surface. The shape is what makes optimization hard, and it is also
why optimization works at all.[^gf-challenges]

## Non-convexity

A function is **convex** if the chord between any two points lies on or above its
graph; equivalently its Hessian $H = \nabla^2 \mathcal{L}$ is positive semidefinite
everywhere. Convexity is the property that makes [linear and logistic
regression](/deep-learning/foundations/linear-models-and-the-perceptron) easy: a
convex function has _one_ basin, so any point where the gradient vanishes is a global
minimum, and gradient descent is guaranteed to find it.

> **Definition (Convex function).** $\mathcal{L}$ is convex if for all
> $\theta_1, \theta_2$ and $t \in [0,1]$,
> $\mathcal{L}(t\theta_1 + (1-t)\theta_2) \le t\,\mathcal{L}(\theta_1) + (1-t)\,\mathcal{L}(\theta_2)$.
> A local minimum of a convex function is global; there are no saddles and no
> spurious basins.

Deep networks lose this property once they have a hidden layer. The
reason is **weight-space symmetry**: permuting the $m$ hidden units of a layer, or
flipping the sign convention across a $\tanh$ unit, produces a _different_ $\theta$
computing the _same_ function with the _same_ loss. Any minimum therefore comes with
an exponential orbit of equivalent minima (at least $m!$ of them per layer), and a
function with many separated global minima cannot be convex.[^gf-symmetry-ls]

$$
% caption: A non-convex loss along a 1D slice: the chord dips below the graph, and
% local search from the left lands in a local, not global, minimum.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, thick] (-0.3,0) -- (8.6,0) node[right, font=\footnotesize] {parameter};
  \draw[->, thick] (0,-0.3) -- (0,3.6) node[above, font=\footnotesize] {loss};
  % a wiggly non-convex curve with two basins
  \draw[acc, very thick] plot[domain=0.3:8.2, samples=120]
    (\x, {1.7 + 1.05*sin(deg(1.15*\x)) - 0.10*\x});
  % chord between two points on the curve dips below the graph
  \draw[black, dashed, thick] (1.05,2.27) -- (6.55,1.18);
  \node[black, anchor=east, font=\footnotesize] at (5.9,2.25) {\texttt{chord below graph}};
  % local minimum (left basin)
  \fill[red] (2.35,0.86) circle (2.6pt);
  \node[red, anchor=east, font=\footnotesize] at (2.15,0.86) {\texttt{local min}};
  % global minimum (right basin)
  \fill[green] (7.78,0.34) circle (2.6pt);
  \node[green, anchor=west, font=\footnotesize] at (7.95,0.34) {\texttt{global min}};
\end{tikzpicture}
$$

There is no algorithm guaranteed to find a global minimum of a general non-convex
function in polynomial time. By the symmetry argument the global minimum is anyway
not _unique_, so the goal shifts: we do not need _the_ minimum, only a
$\theta$ whose loss is _low enough_. Empirically,
plain local search reliably finds such a point; the rest of this lesson
explains why the obstacles that _could_ trap it mostly do not.

## Critical points and the Hessian

Local search stalls wherever the gradient vanishes. Such a point is a **critical
point**; its _type_ is decided by the local curvature, which is the Hessian
$H = \nabla^2 \mathcal{L}$, the symmetric matrix of second partials
$H_{ij} = \partial^2 \mathcal{L}/\partial \theta_i \partial \theta_j$.

> **Definition (Critical point).** A point $\theta^\star$ with
> $\nabla \mathcal{L}(\theta^\star) = 0$. Its character is read from the eigenvalues
> $\lambda_1, \dots, \lambda_d$ of the Hessian $H(\theta^\star)$, since the
> second-order Taylor expansion
> $\mathcal{L}(\theta^\star + v) \approx \mathcal{L}(\theta^\star) + \tfrac12 v^\top H v$
> governs the surface nearby.

The claim that curvature alone fixes the type follows from the second-order Taylor
expansion around any point $\theta_0$, displaced by a step $d$:

$$
\mathcal{L}(\theta_0 + d) \;=\; \mathcal{L}(\theta_0)
\;+\; g^\top d \;+\; \tfrac12\, d^\top H\, d \;+\; O(\lVert d\rVert^3),
\qquad g = \nabla\mathcal{L}(\theta_0),\; H = \nabla^2\mathcal{L}(\theta_0).
$$

At a critical point the linear term $g^\top d$ vanishes because $g = 0$, so the
leading behaviour is the quadratic form $\tfrac12\, d^\top H\, d$ alone: the surface
looks, to second order, like a pure paraboloid centred at $\theta^\star$. Everything
about whether that paraboloid opens upward, downward, or both is encoded in $H$.

Diagonalize $H = Q\Lambda Q^\top$ (possible because $H$ is symmetric, so its
eigenvectors are orthonormal and its eigenvalues real) and change coordinates to the
eigenbasis, $u = Q^\top v$ with $v = \theta - \theta^\star$. Since $Q$ is orthogonal
it preserves lengths, and the quadratic term decouples into a sum of $d$ independent
parabolas, one per eigendirection:

$$
\mathcal{L}(\theta^\star + v) - \mathcal{L}(\theta^\star)
\;\approx\; \tfrac12\, v^\top H v
\;=\; \tfrac12\, u^\top \Lambda u
\;=\; \tfrac12 \sum_{i=1}^{d} \lambda_i\, u_i^2.
$$

Along eigendirection $i$ the loss curves _up_ if $\lambda_i > 0$ and _down_ if
$\lambda_i < 0$, with steepness set by $\lvert\lambda_i\rvert$: the eigenvalue is
literally the second derivative of the loss along its eigenvector. The sign pattern
across all $d$ eigenvalues classifies the point completely.

| Critical-point type | Hessian eigenvalues | Local shape | Behaviour of descent |
| --- | --- | --- | --- |
| Local minimum | all $\lambda_i > 0$ ($H \succ 0$) | bowl, up in every direction | attracts; gradient descent settles |
| Local maximum | all $\lambda_i < 0$ ($H \prec 0$) | dome, down in every direction | repels; never reached by descent |
| Saddle point | mixed signs | up in some axes, down in others | descent stalls, then escapes down a negative direction |
| Degenerate | some $\lambda_i = 0$ | flat in those directions | plateau; second order undecided |

With two eigendirections the whole taxonomy fits on one strip: each panel draws the
loss profile along $u_1$ and along $u_2$, and the pair of curvature signs names the
point. Both cupped up is a minimum; both cupped down is a maximum; one of each is a
saddle.

$$
% caption: Critical points by the sign pair $(\lambda_1,\lambda_2)$ of the two
% Hessian eigenvalues. Each panel shows the loss profile along $u_1$ (blue) and $u_2$
% (red). Two ups: minimum. Two downs: maximum. One up, one down: saddle.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % ---- (+,+) minimum ----
  \begin{scope}[xshift=0cm]
    \draw[acc, thick] plot[domain=-0.9:0.9, samples=30] (\x,{0.55+0.85*\x*\x});
    \draw[red, thick] plot[domain=-0.9:0.9, samples=30] (\x,{0.55+0.85*\x*\x});
    \node[font=\footnotesize, anchor=north] at (0,-0.15) {\texttt{both up}};
    \node[font=\footnotesize, anchor=north] at (0,-0.62) {\texttt{minimum}};
    \node[font=\footnotesize, anchor=south] at (0,1.55) {\texttt{(+,+)}};
  \end{scope}
  % ---- (-,-) maximum ----
  \begin{scope}[xshift=4.0cm]
    \draw[acc, thick] plot[domain=-0.9:0.9, samples=30] (\x,{1.45-0.85*\x*\x});
    \draw[red, thick] plot[domain=-0.9:0.9, samples=30] (\x,{1.45-0.85*\x*\x});
    \node[font=\footnotesize, anchor=north] at (0,-0.15) {\texttt{both down}};
    \node[font=\footnotesize, anchor=north] at (0,-0.62) {\texttt{maximum}};
    \node[font=\footnotesize, anchor=south] at (0,1.55) {\texttt{(-,-)}};
  \end{scope}
  % ---- (+,-) saddle ----
  \begin{scope}[xshift=8.0cm]
    \draw[acc, thick] plot[domain=-0.9:0.9, samples=30] (\x,{0.55+0.85*\x*\x});
    \draw[red, thick] plot[domain=-0.9:0.9, samples=30] (\x,{1.45-0.85*\x*\x});
    \node[font=\footnotesize, anchor=north] at (0,-0.15) {\texttt{one each}};
    \node[font=\footnotesize, anchor=north] at (0,-0.62) {\texttt{saddle}};
    \node[font=\footnotesize, anchor=south] at (0,1.55) {\texttt{(+,-)}};
  \end{scope}
\end{tikzpicture}
$$

> **Definition (Saddle point).** A critical point whose Hessian has both positive
> and negative eigenvalues. The surface curves up along the positive eigendirections
> and down along the negative ones, so the point is a minimum within one subspace
> and a maximum within the complementary subspace.

The three non-degenerate types are easiest to see as contour insets. A minimum has
closed elliptical contours; a maximum has the same closed contours but with the
loss decreasing outward; a saddle has the open, hyperbolic contours of $u_1^2 - u_2^2$.

$$
% caption: The three non-degenerate critical points. Minimum: elliptic contours,
% both $\lambda_i>0$. Maximum: elliptic contours, both $\lambda_i<0$. Saddle:
% hyperbolic contours, $\lambda_1>0>\lambda_2$.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % ---- minimum (left) ----
  \begin{scope}[xshift=0cm]
    \foreach \r in {0.45,0.85,1.25} \draw[acc, thick] (0,0) ellipse ({\r*1.15} and \r);
    \fill[green] (0,0) circle (2.2pt);
    \node[font=\footnotesize, anchor=north] at (0,-1.75) {\texttt{minimum}};
    \node[font=\footnotesize, text=black, anchor=north] at (0,-2.15) {\texttt{all up}};
  \end{scope}
  % ---- maximum (middle) ----
  \begin{scope}[xshift=4.4cm]
    \foreach \r in {0.45,0.85,1.25} \draw[red, thick] (0,0) ellipse ({\r*1.15} and \r);
    \fill[red] (0,0) circle (2.2pt);
    \node[font=\footnotesize, anchor=north] at (0,-1.75) {\texttt{maximum}};
    \node[font=\footnotesize, text=black, anchor=north] at (0,-2.15) {\texttt{all down}};
  \end{scope}
  % ---- saddle (right) ----
  \begin{scope}[xshift=8.8cm]
    % hyperbolae u1^2 - u2^2 = c  (open both ways)
    \foreach \c in {0.35,0.95} {
      \draw[acc, thick] plot[domain=-1.25:1.25, samples=40] ({sqrt(\c + \x*\x)}, {\x});
      \draw[acc, thick] plot[domain=-1.25:1.25, samples=40] ({-sqrt(\c + \x*\x)}, {\x});
      \draw[red, thick] plot[domain=-1.25:1.25, samples=40] ({\x}, {sqrt(\c + \x*\x)});
      \draw[red, thick] plot[domain=-1.25:1.25, samples=40] ({\x}, {-sqrt(\c + \x*\x)});
    }
    \fill[black] (0,0) circle (2.2pt);
    \node[font=\footnotesize, anchor=north] at (0,-1.75) {\texttt{saddle}};
    \node[font=\footnotesize, text=black, anchor=north] at (0,-2.15) {\texttt{mixed}};
  \end{scope}
\end{tikzpicture}
$$

The saddle is the important case, because it dominates
high-dimensional loss surfaces. Its descent and escape directions are
orthogonal: the gradient pulls the iterate _into_ the saddle along the up-curving
axis, where it stalls; the only way out is along the down-curving axis, where the
gradient gives no pull until the iterate is displaced.

$$
% caption: A saddle point. Contours are hyperbolic; the up-curving axis $u_1$
% (descent leads in) and the down-curving axis $u_2$ (the escape direction,
% $\lambda_2<0$) are orthogonal. The gradient vanishes at the center.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % hyperbolic contours of u1^2 - u2^2
  \foreach \c in {0.4,1.1,2.0} {
    \draw[black] plot[domain=-2.1:2.1, samples=60] ({sqrt(\c + \x*\x)}, {\x});
    \draw[black] plot[domain=-2.1:2.1, samples=60] ({-sqrt(\c + \x*\x)}, {\x});
    \draw[black] plot[domain=-2.1:2.1, samples=60] ({\x}, {sqrt(\c + \x*\x)});
    \draw[black] plot[domain=-2.1:2.1, samples=60] ({\x}, {-sqrt(\c + \x*\x)});
  }
  % the saddle point itself
  \fill[black] (0,0) circle (2.6pt);
  \node[anchor=south west, font=\footnotesize] at (0.12,0.10) {\texttt{gradient} $=0$};
  % descent-in axis (horizontal, up-curving): arrows pointing toward center
  \draw[acc, very thick, ->] (-2.7,0) -- (-0.55,0);
  \draw[acc, very thick, ->] (2.7,0) -- (0.55,0);
  \node[acc, anchor=north, font=\footnotesize] at (-1.95,-0.1) {\texttt{descent leads in}};
  % escape axis (vertical, down-curving): arrows pointing away from center
  \draw[red, very thick, ->] (0,0.55) -- (0,2.7);
  \draw[red, very thick, ->] (0,-0.55) -- (0,-2.7);
  \node[red, anchor=south, font=\footnotesize] at (0,2.9) {\texttt{escape direction}};
\end{tikzpicture}
$$

## Why saddles, not local minima, are the obstacle

The classical worry, inherited from low-dimensional intuition, is local minima:
basins of bad loss that trap the optimizer below the global value. In high dimension
this worry is misplaced. The reason is a counting argument on the eigenvalue signs.

At a random critical point, model each eigenvalue's sign as a near-independent coin
flip with $P(\lambda_i > 0) = p$. A point is a local minimum only if _every_ one of
its $d$ eigenvalues is positive:

$$
P(\text{local min}) = P(\text{all } \lambda_i > 0) \approx p^{d}
\;\xrightarrow[d \to \infty]{}\; 0,
\qquad 0 < p < 1.
$$

For any $p$ short of certainty, $p^d$ collapses to zero as $d$ grows. With even a
modest $d = 10^6$, the probability that all signs agree is astronomically small, so
critical points are overwhelmingly _mixed_: saddles. The same vanishing applies to
maxima. Local minima do exist, but the counting says they sit only at the very
bottom: random-matrix theory makes the relationship sharp.

> **Theorem (Saddle prevalence — informal).** On a high-dimensional random loss
> surface, the expected fraction of negative Hessian eigenvalues at a critical point
> _increases_ with the loss at that point. Consequently critical points of high loss
> are saddles with many escape directions, and a critical point can be an approximate
> local minimum (almost all $\lambda_i > 0$) only when its loss is already close to
> the global minimum.[^gf-saddle]

> **Proof sketch.** Model $H$ at a critical point as a random symmetric matrix whose
> spectrum follows the semicircle law, shifted so that the loss level sets the mean
> eigenvalue. High loss shifts the spectrum left, putting most of its mass at
> negative eigenvalues — many down-directions, an unstable saddle. Lowering the loss
> shifts the bulk right; only near the global minimum does the entire spectrum clear
> zero, making all $\lambda_i > 0$. The probability that an $O(d)$-wide random
> spectrum has _no_ eigenvalue below zero is exponentially small in $d$ except in
> that bottom sliver. $\qed$

The Hessian's eigenvalues fill a
semicircular band whose whole position drifts rightward as the loss drops. At high
loss the band straddles zero with most of its mass on the negative side, so the point
has a majority of down-directions and is a highly unstable saddle. As the loss falls
the band slides right; the count of negative eigenvalues shrinks. Only in the
lowest sliver does the entire band clear zero, and only there is the point a true
local minimum.

$$
% caption: The Hessian eigenvalue spectrum slides right as the loss drops. At high
% loss (top) the band straddles zero, so many $\lambda_i<0$ give escape directions —
% a saddle. At low loss (bottom) the whole band clears zero, $\lambda_i>0$ — a
% minimum. The dashed line marks $\lambda=0$.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % zero-eigenvalue reference line
  \draw[black, dashed] (3.4,-0.3) -- (3.4,6.5);
  \node[anchor=south, font=\footnotesize] at (3.4,6.5) {\texttt{zero}};
  % --- high loss (top): band centered left of zero, negative mass ---
  \begin{scope}[yshift=5.0cm]
    \fill[red!22] plot[domain=1.1:3.9, samples=40] (\x,{0.62*sqrt(1-((\x-2.5)/1.4)*((\x-2.5)/1.4))})
      -- (3.9,0) -- (1.1,0) -- cycle;
    \draw[red, thick] plot[domain=1.1:3.9, samples=40] (\x,{0.62*sqrt(1-((\x-2.5)/1.4)*((\x-2.5)/1.4))});
    \draw[black, ->] (1.0,0) -- (6.2,0);
    \node[red, anchor=west, font=\footnotesize] at (5.0,0.55) {\texttt{high loss: saddle}};
  \end{scope}
  % --- mid loss (middle): band straddles zero, few negatives ---
  \begin{scope}[yshift=2.7cm]
    \fill[acc!20] plot[domain=2.5:5.3, samples=40] (\x,{0.62*sqrt(1-((\x-3.9)/1.4)*((\x-3.9)/1.4))})
      -- (5.3,0) -- (2.5,0) -- cycle;
    \draw[acc, thick] plot[domain=2.5:5.3, samples=40] (\x,{0.62*sqrt(1-((\x-3.9)/1.4)*((\x-3.9)/1.4))});
    \draw[black, ->] (1.0,0) -- (6.2,0);
    \node[acc, anchor=west, font=\footnotesize] at (5.4,0.35) {\texttt{fewer down}};
  \end{scope}
  % --- low loss (bottom): band clears zero entirely, all positive ---
  \begin{scope}[yshift=0.4cm]
    \fill[green!22] plot[domain=3.7:6.1, samples=40] (\x,{0.55*sqrt(1-((\x-4.9)/1.2)*((\x-4.9)/1.2))})
      -- (6.1,0) -- (3.7,0) -- cycle;
    \draw[green, thick] plot[domain=3.7:6.1, samples=40] (\x,{0.55*sqrt(1-((\x-4.9)/1.2)*((\x-4.9)/1.2))});
    \draw[black, ->] (1.0,0) -- (6.2,0) node[right, font=\scriptsize] {eigenvalue};
    \node[green, anchor=west, font=\footnotesize] at (5.5,0.5) {\texttt{minimum}};
  \end{scope}
\end{tikzpicture}
$$

The practical consequence: a high-loss critical point almost never traps the
optimizer, because it almost always has a downhill escape direction. The escape
can be _slow_, though. Near a saddle the gradient is small in every direction
(it must vanish at the center), so plain gradient descent makes little progress
there. The surrounding lessons supply the remedies.

| Mechanism | How it escapes a saddle | Lesson |
| --- | --- | --- |
| [Momentum](/deep-learning/optimization/momentum-and-adaptive-methods) | accumulated velocity carries the iterate through the flat region instead of stalling | momentum |
| SGD gradient noise | minibatch sampling perturbs the iterate off the saddle's stable manifold onto a descending direction | [gradient descent and SGD](/deep-learning/optimization/gradient-descent-and-sgd) |
| Adaptive rates (Adam) | per-coordinate scaling enlarges steps along low-curvature escape axes | momentum |

Why noise escapes is again the per-eigencoordinate recursion. Along an escape
direction with $\lambda_i < 0$, one gradient step scales the displacement by
$1 - \eta\lambda_i > 1$, so any nonzero component $u_i$ _grows_ each step:

$$
u_i^{(t)} = (1 - \eta\lambda_i)^t\, u_i^{(0)}, \qquad 1 - \eta\lambda_i > 1
\;\text{ when }\; \lambda_i < 0.
$$

If the iterate lands exactly on the saddle's stable manifold — the subspace spanned
by the positive-eigenvalue directions, where $u_i^{(0)} = 0$ along every negative
axis — it stalls, because there is nothing to amplify. That exact landing has measure
zero, and gradient noise from minibatch sampling guarantees a nonzero $u_i^{(0)}$
along the unstable axis. Once seeded, that component blows up geometrically at rate
$(1 - \eta\lambda_i)$ and the iterate slides off the saddle. The stall near the
center is real but temporary; the escape time scales with $1/\lvert\lambda_{\min}\rvert$,
so the flattest saddles (small $\lvert\lambda_{\min}\rvert$) hold the iterate longest.

> **Definition (Strict saddle).** A saddle with at least one strictly negative
> Hessian eigenvalue, $\lambda_{\min}(H) < 0$. Gradient descent perturbed by noise
> escapes every strict saddle with probability one, because the unstable
> eigendirection grows any perturbation along it geometrically — which is precisely
> why a little noise (SGD) or inertia (momentum) suffices.

## The rest of the terrain

Saddles are the dominant feature, but four further features of the surface shape
how a step should be taken.

### Plateaus

A **plateau** is an extended region where the gradient is near zero without a
critical point in sight: the eigenvalues are small in magnitude, so the surface is
nearly flat. Saturating activations create them: a unit deep in the flat tail of a
sigmoid or $\tanh$ has $\sigma'(z) \approx 0$, so the gradient that reaches its
weights is attenuated to almost nothing and progress stalls for many steps.

$$
% caption: A plateau. Across the flat middle region the gradient
% $\nabla\mathcal{L}\approx 0$, so descent crawls even though no minimum has
% been reached; progress resumes only past the shoulder.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \draw[->, thick] (-0.3,0) -- (9.0,0) node[right, font=\footnotesize] {parameter};
  \draw[->, thick] (0,-0.3) -- (0,3.4) node[above, font=\footnotesize] {loss};
  % high on left, long flat plateau in the middle, drops to a minimum on the right
  \draw[acc, very thick] plot[domain=0.3:8.6, samples=160]
    (\x, {0.55 + 2.45/(1 + exp(2.2*(\x-6.2)))});
  % bracket marking the flat region
  \draw[black, thick] (1.6,2.85) -- (4.9,2.85);
  \draw[black, thick] (1.6,2.78) -- (1.6,2.92);
  \draw[black, thick] (4.9,2.78) -- (4.9,2.92);
  \node[black, anchor=south, font=\footnotesize] at (3.25,2.9) {\texttt{plateau: near-zero gradient}};
  % minimum
  \fill[green] (8.45,0.56) circle (2.6pt);
  \node[green, anchor=west, font=\footnotesize] at (8.0,0.95) {\texttt{minimum}};
\end{tikzpicture}
$$

### Cliffs and exploding gradients

The opposite pathology is a **cliff**: a near-vertical wall where the loss changes
by a large amount over a tiny change in $\theta$, so the gradient is enormous.
Common in [recurrent networks](/deep-learning/architectures/recurrent-networks),
where repeated multiplication by a recurrent weight matrix makes the gradient
magnitude grow geometrically with sequence length. A normal-sized learning rate
times a huge gradient yields a huge step that throws the iterate far across
the surface, undoing many updates of progress.

The standard remedy is **gradient clipping**: rescale the gradient to a fixed norm
$c$ before stepping, keeping its _direction_ but capping its _length_,

$$
g \;\gets\; g \cdot \min\!\parens{1,\; \frac{c}{\norm{g}}},
$$

so the step is bounded by $\eta c$ no matter how steep the wall. The direction
(which is correct, pointing along the wall toward lower loss) is preserved; only the
magnitude is capped.[^gf-cliffs]

$$
% caption: A cliff and gradient clipping. The raw step (red) overshoots past the
% basin; the clipped step (green) keeps the direction but caps the length.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, thick] (-0.3,0) -- (9.0,0) node[right, font=\footnotesize] {parameter};
  \draw[->, thick] (0,-0.3) -- (0,3.7) node[above, font=\footnotesize] {loss};
  % gentle basin on the left, then a steep cliff rising on the right
  \draw[acc, very thick] plot[domain=0.3:4.0, samples=80]
    (\x, {0.55 + 0.14*(\x-2.6)*(\x-2.6)});
  \draw[acc, very thick] plot[domain=4.0:5.05, samples=60]
    (\x, {0.75 + 2.9*(\x-4.0)*(\x-4.0)});
  \node[acc, anchor=south east, font=\footnotesize] at (4.3,2.55) {\texttt{cliff}};
  % current point partway up the cliff
  \fill[black] (4.55,1.45) circle (2.6pt);
  \node[anchor=south east, font=\footnotesize] at (4.5,1.5) {\texttt{here}};
  % raw step: huge, overshoots far to the left and up the far basin (arcs over
  % the landscape rather than plunging through it)
  \draw[red, very thick, ->] (4.55,1.45) to[bend right=16] (0.55,1.7);
  \node[red, anchor=west, font=\footnotesize] at (0.55,2.1) {\texttt{raw step overshoots}};
  % clipped step: lands at the bottom of the near basin
  \draw[green, very thick, ->] (4.55,1.45) .. controls (3.6,0.55) .. (2.65,0.6);
  \node[green, anchor=north, font=\footnotesize] at (3.5,0.35) {\texttt{clipped step}};
  \fill[green] (2.6,0.55) circle (2.2pt);
\end{tikzpicture}
$$

### Ill-conditioning

Even a perfectly convex bowl can be hard to descend if it is _stretched_: long and
narrow rather than round. The stretch is measured by the **condition number** of the
Hessian, the ratio of its largest to smallest eigenvalue,

$$
\kappa(H) = \frac{\lambda_{\max}}{\lambda_{\min}}.
$$

The reason a large $\kappa$ hurts is visible in the decoupled coordinates. On a
quadratic bowl $\mathcal{L} = \tfrac12 \sum_i \lambda_i u_i^2$, gradient descent with
rate $\eta$ updates each eigencoordinate independently,

$$
u_i^{(t+1)} = u_i^{(t)} - \eta\,\lambda_i u_i^{(t)} = (1 - \eta\lambda_i)\, u_i^{(t)},
$$

so coordinate $i$ decays geometrically at rate $\lvert 1 - \eta\lambda_i\rvert$. For
_every_ coordinate to converge the rate must satisfy $\lvert 1 - \eta\lambda_i\rvert <
1$, i.e. $\eta < 2/\lambda_{\max}$ — the steep axis sets the ceiling. Plug that
largest usable rate into the gentle axis and the decay factor is
$1 - \eta\lambda_{\min} \approx 1 - 2\lambda_{\min}/\lambda_{\max} = 1 - 2/\kappa$,
which is close to $1$ when $\kappa \gg 1$. The steep axis forces a small step; the
gentle axis then crawls, needing on the order of $\kappa$ iterations to make the same
relative progress. The iterate zig-zags across the narrow valley while inching down
its length. This is the structural reason for momentum and for per-coordinate
[adaptive learning rates](/deep-learning/optimization/momentum-and-adaptive-methods),
both of which counteract a large $\kappa$.

$$
% caption: An ill-conditioned ravine, $\kappa\gg1$. The elliptical contours are
% stretched; gradient descent (red) zig-zags across the steep short axis while
% creeping along the gentle long axis toward the minimum.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % stretched elliptic contours (wide along x, narrow along y): a ravine
  \foreach \r in {0.35,0.62,0.9,1.2} \draw[acc, thick] (0,0) ellipse ({\r*3.1} and \r);
  \fill[green] (0,0) circle (2.4pt);
  \node[green, anchor=south, font=\footnotesize] at (0,1.4) {\texttt{minimum}};
  % zig-zag descent path from far right, bouncing across the narrow axis
  \draw[red, very thick, ->]
    (3.7,0.72) -- (3.05,-0.66) -- (2.35,0.6) -- (1.75,-0.5)
    -- (1.2,0.42) -- (0.72,-0.34) -- (0.34,0.24) -- (0.06,-0.12);
  \fill[black] (3.7,0.72) circle (2.2pt);
  \node[anchor=west, font=\footnotesize] at (3.75,0.95) {\texttt{start}};
  \node[red, anchor=north, font=\footnotesize] at (2.2,-1.15) {\texttt{zig-zag across the ravine}};
\end{tikzpicture}
$$

| Condition number | Geometry | Gradient descent |
| --- | --- | --- |
| $\kappa \approx 1$ | round bowl | steps straight to the minimum |
| $\kappa \gg 1$ | long narrow ravine | zig-zags across the valley, crawls along it |
| $\kappa \to \infty$ | degenerate (flat direction) | no progress along the flat axis |

### Sharp versus flat minima

Not all minima generalize equally, and the geometry that distinguishes them is
again curvature. A **sharp** minimum sits at the bottom of a narrow, steep basin
(large Hessian eigenvalues); a **flat** minimum sits in a wide, shallow basin (small
eigenvalues). They can have _identical_ training loss yet behave very differently on
test data.

> **Definition (Flat minimum).** A minimum $\theta^\star$ whose surrounding basin is
> wide — the Hessian eigenvalues are small, so the training loss rises only slowly as
> $\theta$ moves away from $\theta^\star$. A sharp minimum is the opposite: small
> Hessian eigenvalues for flat, large for sharp.

The link to generalization is a shift argument. Train and test losses are the same
functional shape evaluated on different samples, so the test-loss curve is
approximately the train-loss curve shifted by a small displacement $\delta$. At a
sharp minimum the steep walls mean a small horizontal shift produces a large vertical
gap — the test loss at the trained $\theta^\star$ is much higher than the train loss.
At a flat minimum the same shift barely raises the loss, so train and test nearly
coincide.

$$
% caption: Sharp versus flat minima. Shifting the train loss (blue) to the test
% loss (red) opens a large gap at the sharp basin but a small one at the flat basin.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, thick] (-0.3,0) -- (9.4,0) node[right, font=\footnotesize] {parameter};
  \draw[->, thick] (0,-0.3) -- (0,3.5) node[above, font=\footnotesize] {loss};
  % --- sharp basin (left): narrow steep parabola, train ---
  \draw[acc, very thick] plot[domain=0.55:3.05, samples=80]
    (\x, {0.45 + 2.4*(\x-1.8)*(\x-1.8)});
  % sharp test = train shifted right by delta
  \draw[red, very thick, dashed] plot[domain=0.85:3.35, samples=80]
    (\x, {0.45 + 2.4*(\x-2.2)*(\x-2.2)});
  \fill[green] (1.8,0.45) circle (2.4pt);
  \node[font=\footnotesize, anchor=north] at (1.8,-0.05) {\texttt{sharp}};
  % gap arrow at the trained minimum location (x=1.8): test value there
  \draw[red, <->] (1.8,0.48) -- (1.8,1.22);
  \node[red, anchor=west, font=\footnotesize] at (3.4,1.1) {\texttt{large gap}};
  % --- flat basin (right): wide shallow parabola, train ---
  \draw[acc, very thick] plot[domain=5.1:8.9, samples=80]
    (\x, {0.45 + 0.28*(\x-7.0)*(\x-7.0)});
  % flat test = train shifted right by same delta
  \draw[red, very thick, dashed] plot[domain=5.5:9.3, samples=80]
    (\x, {0.45 + 0.28*(\x-7.4)*(\x-7.4)});
  \fill[green] (7.0,0.45) circle (2.4pt);
  \node[font=\footnotesize, anchor=north] at (7.0,-0.05) {\texttt{flat}};
  \draw[red, <->] (7.0,0.47) -- (7.0,0.70);
  \node[red, anchor=south, font=\footnotesize] at (7.0,1.15) {\texttt{small gap}};
  \node[acc, anchor=east, font=\footnotesize] at (9.3,2.6) {\texttt{train}};
  \node[red, anchor=east, font=\footnotesize] at (9.3,2.25) {\texttt{test}};
\end{tikzpicture}
$$

This is why the implicit bias of SGD toward flat regions (small batches inject more
gradient noise, and noise cannot rest in a narrow basin) is widely believed to be
part of why deep networks generalize at all.[^chollet-landscape] The formal version of the
train-test-gap argument is developed in
[generalization theory](/deep-learning/theory/generalization-theory).

> **Remark (Mode connectivity).** The exponentially many minima produced by
> weight-space symmetry are not isolated. Empirically, independently trained
> solutions can be joined by paths through parameter space along which the loss stays
> low the whole way — the low-loss region is a single connected manifold rather
> than a collection of separate basins. This _mode connectivity_ reinforces the picture that
> the surface, though non-convex, is benign where it matters: the good basins are
> connected, and local search lands in the connected low-loss set rather than an
> isolated bad minimum.

## The landscape, made quantitative

Every claim in this lesson that Goodfellow states qualitatively has a public paper
that made it quantitative.

- **Saddles, counted.** Dauphin et al. gave the random-matrix argument behind the
  sliding-spectrum figure and turned it into an algorithm — _saddle-free Newton_,
  which uses $\lvert H\rvert$ to escape saddles that both gradient descent and raw
  Newton get stuck near. Choromanska et al. connected the deep loss surface to the
  spin-glass models of statistical physics, making "high-loss critical points are
  saddles" a theorem about a tractable proxy.[^dauphin][^choromanska]
- **Flat minima and generalization, measured.** Keskar et al. showed empirically
  that large-batch training lands in _sharper_ minima that generalize worse, and
  quantified sharpness by the loss increase in a small neighborhood — the
  shift-argument figure made into a metric. Sharpness-Aware Minimization (Foret et
  al.) then turned the observation into an optimizer that explicitly seeks flat
  regions.[^keskar][^sam]
- **Mode connectivity, made visible.** The remark above is Garipov et al.'s and
  Draxler et al.'s finding: independently trained minima are joined by
  low-loss curves. Li et al.'s loss-landscape visualizations are the source of the
  now-standard pictures of wide flat basins versus chaotic sharp ones.[^garipov][^li-viz]

The pattern across these is that high-dimensional geometry is _benign_ in a way
low-dimensional intuition does not predict: isolated deep local minima are
vanishingly rare, and the real obstacles (saddles, sharp basins, ill-conditioning)
all have standard remedies.

## Takeaways

- The deep loss surface is **non-convex** (weight-space symmetries alone force
  exponentially many equivalent minima), so local search has no global guarantee;
  the realistic goal is a $\theta$ with _low enough_ loss, which local search
  reliably finds.
- **Critical points** ($\nabla\mathcal{L} = 0$) are classified by the **Hessian
  eigenvalues**: all positive $\Rightarrow$ minimum, all negative $\Rightarrow$
  maximum, mixed signs $\Rightarrow$ saddle.
- In high dimension the chance that all $d$ eigenvalues share a sign is $\approx p^d
  \to 0$, so almost every critical point is a **saddle**, not a bad local minimum;
  high-loss critical points have many escape directions, and **momentum** and **SGD
  noise** carry the iterate past them.
- The remaining terrain — **plateaus** (vanishing gradients), **cliffs** (exploding
  gradients, cured by **gradient clipping** to a fixed norm), and **ill-conditioning**
  (large $\kappa(H) = \lambda_{\max}/\lambda_{\min}$) — explains why the plain step
  needs momentum and adaptive rates.
- **Flat minima generalize better than sharp ones**: a small train-test shift opens
  a large gap in a steep basin and a small one in a wide basin, and **mode
  connectivity** shows the good minima form one connected low-loss region.

[^gf-challenges]: **Goodfellow**, _Deep Learning_, §8.2 — Challenges in Neural Network Optimization: ill-conditioning, local minima, plateaus, saddles, and cliffs as the obstacles a step must navigate.
[^gf-symmetry-ls]: **Goodfellow**, _Deep Learning_, §8.2.2 — Local Minima: weight-space symmetry (unit permutations, sign flips) forces exponentially many equivalent minima, ruling out convexity.
[^gf-saddle]: **Goodfellow**, _Deep Learning_, §8.2.3 — Plateaus, Saddle Points and Other Flat Regions: in high dimension critical points are overwhelmingly saddles, and high loss correlates with more negative-curvature escape directions.
[^gf-cliffs]: **Goodfellow**, _Deep Learning_, §8.2.4 — Cliffs and Exploding Gradients: steep walls produce giant steps, tamed by norm-based gradient clipping that caps length while preserving direction.
[^chollet-landscape]: **Chollet**, _Deep Learning with Python_, §2.4 — The Engine of Neural Networks: SGD's noise as an implicit regularizer biasing solutions toward flatter, better-generalizing basins.
[^dauphin]: **Dauphin et al.**, _Identifying and Attacking the Saddle Point Problem in High-Dimensional Non-Convex Optimization_, NeurIPS 2014 — the random-matrix saddle argument and the saddle-free Newton method.
[^choromanska]: **Choromanska et al.**, _The Loss Surfaces of Multilayer Networks_, AISTATS 2015 — a spin-glass model of the deep loss surface predicting a band of low-loss minima above a floor of saddles.
[^keskar]: **Keskar et al.**, _On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima_, ICLR 2017 — large batches find sharper minima that generalize worse, with a sharpness metric.
[^sam]: **Foret, Kleiner, Mobahi & Neyshabur**, _Sharpness-Aware Minimization for Efficiently Improving Generalization_, ICLR 2021 — SAM, an optimizer that minimizes the worst-case loss in a neighborhood to seek flat minima.
[^garipov]: **Garipov et al.**, _Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs_, NeurIPS 2018 — low-loss curves connecting independently trained minima.
[^li-viz]: **Li, Xu, Taylor, Studer & Goldstein**, _Visualizing the Loss Landscape of Neural Nets_, NeurIPS 2018 — filter-normalized visualizations of flat versus sharp basins.