---
title: Momentum & Adaptive Methods
module: Optimization
moduleNumber: 3
lessonNumber: 2
order: 302
summary: >
  Plain gradient descent zig-zags across ravines and moves slowly along
  flat valleys, because one global learning rate cannot suit a surface with
  wildly different curvature in different directions. Two fixes address the
  two problems: momentum accumulates a velocity that damps the
  oscillation and accelerates the drift, and adaptive methods give every
  parameter its own learning rate scaled by the history of its gradients. Adam
  fuses both, and is the default optimizer of modern deep learning.
topics: [Optimization]
sources:
  - book: Goodfellow
    ref: "§8.3 — Basic Algorithms (SGD, Momentum, Nesterov)"
  - book: Goodfellow
    ref: "§8.5 — Algorithms with Adaptive Learning Rates (AdaGrad, RMSProp, Adam)"
  - book: Chollet
    ref: "§2.4, §3.6 — Optimizers in practice"
---

Plain [stochastic gradient descent](/deep-learning/optimization/gradient-descent-and-sgd)
takes the same step rule everywhere: $\theta \gets \theta - \eta\,\nabla L$, one
scalar $\eta$, applied identically to every coordinate. Two structural facts of
the loss surface break this. First, curvature differs by direction: a **ravine**
is steep across and shallow along, so a step large enough to make progress down
the valley overshoots across it and the path zig-zags. Second, curvature differs
by _coordinate_, since some weights see huge gradients, others tiny ones, yet a single
$\eta$ serves all of them. This lesson addresses each in turn: **momentum**
for the first, **adaptive learning rates** for the second, and **Adam** combining
them into the modern default.[^gf-basic]

## The pathology: ill-conditioning

Near a minimum the loss is locally quadratic, $L(\theta) \approx \tfrac12
(\theta - \theta^\star)^\top H (\theta - \theta^\star)$, with Hessian $H$. Rotate
to the eigenbasis of $H$ and the coordinates decouple: along eigenvector $i$ with
eigenvalue $\lambda_i$, gradient descent with rate $\eta$ contracts the error by
a fixed factor each step,

$$
e^{(i)}_{t+1} = (1 - \eta\,\lambda_i)\,e^{(i)}_t,
\qquad
e^{(i)}_t = (1 - \eta\,\lambda_i)^t\, e^{(i)}_0.
$$

Stability demands $|1 - \eta\,\lambda_i| < 1$ for _every_ axis, so the largest
eigenvalue $\lambda_{\max}$ caps the step at $\eta < 2/\lambda_{\max}$. But
convergence along the flattest axis proceeds at rate $1 - \eta\,\lambda_{\min}$,
which $\lambda_{\min} \ll \lambda_{\max}$ drives very close to $1$.

> **Definition (Condition number).** The ratio $\kappa = \lambda_{\max} /
> \lambda_{\min}$ of the largest to smallest eigenvalue of the Hessian. It bounds
> the worst-case convergence of gradient descent: the number of steps to a fixed
> accuracy grows linearly in $\kappa$. A surface with $\kappa \gg 1$ is
> **ill-conditioned**: a long, narrow ravine.

The next figure shows the geometry: on
elliptical contours plain SGD oscillates across the steep direction while
making little progress along the shallow one.

$$
% caption: A ravine ($\kappa \gg 1$). SGD (red) zig-zags across the steep axis;
% momentum (blue) cancels the side-to-side motion and accelerates down the valley.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  % elliptical contours: wide along x (shallow), narrow along y (steep)
  \foreach \r in {0.5,1.0,1.5,2.0,2.5}
    \draw[black] (0,0) ellipse ({\r*2.4} and \r);
  \fill[green] (0,0) circle (2.6pt);
  \node[green, anchor=north] at (0,-2.7) {\texttt{minimum}};
  % SGD zig-zag (red): bounces across the steep y-axis while drifting in x
  \draw[red, very thick, ->] (-5.4,1.7) -- (-4.2,-1.35);
  \draw[red, very thick, ->] (-4.2,-1.35) -- (-3.2,1.05);
  \draw[red, very thick, ->] (-3.2,1.05) -- (-2.3,-0.82);
  \draw[red, very thick, ->] (-2.3,-0.82) -- (-1.55,0.6);
  \draw[red, very thick, ->] (-1.55,0.6) -- (-0.95,-0.42);
  \node[red, anchor=south] at (-5.4,1.78) {\texttt{SGD}};
  % momentum (blue): damped, accelerating drift along the valley
  \draw[acc, very thick, ->] (-5.4,-1.7) .. controls (-4.0,-1.05) and (-2.6,-0.55) .. (-1.4,-0.25);
  \draw[acc, very thick, ->] (-1.4,-0.25) .. controls (-0.8,-0.12) and (-0.4,-0.05) .. (-0.08,-0.02);
  \node[acc, anchor=north] at (-5.4,-1.78) {\texttt{momentum}};
\end{tikzpicture}
$$

## Momentum

Momentum addresses the zig-zag with memory. Rather than stepping by the current gradient,
accumulate an exponentially-weighted running sum of gradients (a **velocity**)
and step by that. Oscillating components alternate sign and cancel in the sum;
the consistent down-valley component survives and compounds.

> **Definition (Momentum).** A velocity vector $v$ accumulates past gradients
> with decay $\mu \in [0,1)$, and the parameters move by the velocity:
>
> $$
> v_{t+1} = \mu\,v_t - \eta\,\nabla L(\theta_t),
> \qquad
> \theta_{t+1} = \theta_t + v_{t+1}.
> $$
>
> $\mu$ is the **momentum coefficient** (typically $0.9$); $\eta$ the learning
> rate. Setting $\mu = 0$ recovers plain SGD.

The name is literal. Read $\theta$ as the position of a ball of unit mass,
$-\nabla L$ as the force from the potential $L$, and $\mu$ as a friction (drag)
coefficient: the velocity update is a discrete-time Newtonian dynamics with
viscous damping. The ball does not teleport down the gradient; it _coasts_,
carrying its momentum through small bumps and across narrow ravines.[^gf-momentum]

| Quantity | Physical reading |
| --- | --- |
| $\theta$ | position of the particle |
| $v$ | velocity |
| $-\nabla L(\theta)$ | force ($L$ is the potential energy) |
| $\mu$ | friction / drag (closer to $1$ = less drag) |
| $\eta$ | time step $\times$ inverse mass |

$$
% caption: The "heavy ball" reading. A particle rolls on the loss surface under the
% force $-\nabla L$ with friction; inertia carries it through small ripples that trap plain GD.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % a bumpy potential surface (curve), descending left to right into a basin
  \draw[black, very thick]
    plot[domain=-5:5, samples=160]
    (\x, {0.6*sin(2.4*\x r) * exp(-0.18*\x*\x) + 0.10*\x*\x - 0.9});
  % the heavy ball partway down, on the left slope
  \fill[acc] (-2.55,0.02) circle (5pt);
  % velocity arrow (rolling to the right / downhill)
  \draw[acc, very thick, ->] (-2.15,0.05) -- (-0.85,-0.4);
  \node[acc, anchor=south, font=\footnotesize] at (-1.4,0.08) {\texttt{velocity}};
  % force arrow (downhill tangent)
  \draw[red, thick, ->] (-3.05,0.45) -- (-3.05,-0.5);
  \node[red, anchor=east, font=\footnotesize] at (-3.1,-0.05) {\texttt{force}};
  % basin label
  \node[green, anchor=north, font=\footnotesize] at (0.15,-1.7) {\texttt{basin}};
  \fill[green] (0.1,-1.2) circle (2.4pt);
  \node[black, anchor=south, font=\footnotesize] at (3.4,0.9) {\texttt{small ripples}};
\end{tikzpicture}
$$

### Effective step size

Unrolling the velocity recurrence with a constant gradient $g$ exposes why
momentum _accelerates_. With $v_0 = 0$,

$$
v_t = -\eta\,g\,(1 + \mu + \mu^2 + \cdots + \mu^{t-1})
\;\xrightarrow{t \to \infty}\;
-\frac{\eta}{1 - \mu}\,g.
$$

The terminal velocity is the geometric series $\sum_{k\ge0}\mu^k = 1/(1-\mu)$
times a single SGD step. The effective learning rate along any direction the
gradient persistently points is therefore amplified:

$$
\eta_{\text{eff}} = \frac{\eta}{1 - \mu}.
$$

> **Lemma (Momentum amplification).** Along a direction where the gradient holds
> a constant sign, momentum with coefficient $\mu$ reaches an asymptotic step of
> $\eta/(1-\mu)$: a $10\times$ amplification at $\mu = 0.9$, a $100\times$
> amplification at $\mu = 0.99$. Along a direction where the gradient _flips_
> sign each step, successive contributions cancel and the effective step stays
> near $\eta$.

That asymmetry is the entire mechanism. Down the shallow valley the gradient is
consistent, so momentum compounds it and the path accelerates; across the steep
ravine the gradient reverses every step, so the contributions annihilate and the
oscillation is damped. One coefficient produces both behaviors.

For example, fix $\eta = 0.1$, $\mu =
0.9$, and a constant unit gradient $g = 1$ along a persistent direction. The
velocity magnitude grows step by step,

$$
|v_1| = 0.1,\quad
|v_2| = 0.19,\quad
|v_3| = 0.271,\quad
|v_5| = 0.4095,\quad
|v_{10}| = 0.6513,\quad
|v_\infty| = 1.0,
$$

each term being $0.9$ of the previous velocity plus $0.1$. The steady state
$0.1/(1-0.9) = 1.0$ is ten times a single SGD step of $0.1$. Now flip the sign of
$g$ every step, as it does across a ravine: the velocity oscillates within a
small band around zero instead of growing, because each contribution partly
cancels the last. The same recurrence, fed a consistent signal, integrates it;
fed an alternating signal, filters it out.

$$
% caption: Velocity accumulation under a constant unit gradient ($\eta=0.1$, $\mu=0.9$).
% Each bar is $|v_t|$; the sequence climbs the geometric series toward the terminal step $1/(1-\mu)=10\eta$.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes: x = step, y = |v_t| scaled *3.4 (so v=1.0 -> y=3.4)
  \draw[->, thick] (0,0) -- (8.4,0) node[right, font=\footnotesize] {\texttt{step t}};
  \draw[->, thick] (0,0) -- (0,3.9) node[above, font=\footnotesize] {\texttt{velocity}};
  % terminal asymptote at |v|=1.0 -> y=3.4
  \draw[red, dashed] (0,3.4) -- (8.0,3.4);
  \node[red, anchor=south east, font=\footnotesize] at (8.0,3.42) {\texttt{terminal 10 eta}};
  % bars: |v_t| for t=1..8 : .1,.19,.271,.3439,.4095,.4686,.5217,.5695  (scaled *3.4)
  \foreach \t/\h in {1/0.34,2/0.646,3/0.921,4/1.169,5/1.392,6/1.593,7/1.774,8/1.936}{
    \fill[acc!75] ({\t*0.9-0.25},0) rectangle ({\t*0.9+0.25},\h);
    \node[anchor=north, font=\scriptsize] at (\t*0.9,-0.05) {\t};
  }
\end{tikzpicture}
$$

The dependence on $\mu$ is steep near $1$: the effective step $1/(1-\mu)$ is a
hyperbola, so the last tenth of $\mu$ carries most of the amplification. That is
also why $\mu$ close to $1$ is delicate: it lengthens the memory (a horizon of
$1/(1-\mu)$ steps) and can overshoot on curvature that turns before the velocity
can respond.

$$
% caption: Effective step $\eta_{\text{eff}}/\eta = 1/(1-\mu)$ versus momentum
% coefficient $\mu$. The curve is a hyperbola: $\mu=0.9$ gives $10\times$, $\mu=0.99$ gives $100\times$.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes: x = mu in [0,0.98] mapped to [0,7]; y = 1/(1-mu) capped, mapped /5 to [0,4]
  \draw[->, thick] (0,0) -- (7.8,0) node[right, font=\footnotesize] {\texttt{mu}};
  \draw[->, thick] (0,0) -- (0,4.2) node[above, font=\footnotesize] {\texttt{eff. step}};
  % x ticks at mu = 0, 0.5, 0.9, 0.95  (x = mu/0.14 approx; use explicit map)
  % map mu -> x: x = mu*7.14 ; map factor f=1/(1-mu) -> y = f/12.5 (cap at ~3.8)
  \node[anchor=north, font=\scriptsize] at (0,-0.05) {0};
  \node[anchor=north, font=\scriptsize] at (3.57,-0.05) {0.5};
  \node[anchor=north, font=\scriptsize] at (6.43,-0.05) {0.9};
  % dashed guides at mu=0.9 (x=6.43), factor 10 -> y = 0.8
  \draw[black, dashed] (6.43,0) -- (6.43,0.8);
  \draw[black, dashed] (0,0.8) -- (6.43,0.8);
  \node[black, anchor=east, font=\scriptsize] at (-0.08,0.8) {10};
  % curve 1/(1-mu): sampled points (mu, factor/12.5)
  \draw[acc, very thick] plot coordinates {
    (0,0.08) (1.79,0.107) (3.57,0.16) (4.64,0.222) (5.36,0.32)
    (5.71,0.40) (6.07,0.533) (6.43,0.80) (6.79,1.60) (6.96,2.67) (7.07,3.80)};
  \node[acc, anchor=south east, font=\footnotesize] at (7.0,3.4) {\texttt{1/(1-mu)}};
\end{tikzpicture}
$$

```algorithm
caption: $\textsc{Momentum}(L, \theta, \eta, \mu)$ — SGD with a velocity buffer
initialize $\theta$ randomly,\ $v \gets 0$
repeat
  sample a minibatch $(X, y)$
  $g \gets \nabla_\theta L(\theta; X, y)$ // backward pass
  $v \gets \mu \cdot v - \eta \cdot g$ // accumulate velocity
  $\theta \gets \theta + v$ // step by velocity
until converged
return $\theta$
```

### Nesterov accelerated gradient

Momentum measures the force where the ball _is_; Nesterov's variant measures it
where the ball is _about to be_. Take the momentum step first to the **lookahead**
point $\tilde\theta = \theta + \mu v$, and evaluate the gradient _there_, so the
correction anticipates the surface ahead instead of reacting to the surface
behind.

> **Definition (Nesterov accelerated gradient).** Evaluate the gradient at the
> lookahead point $\theta_t + \mu\,v_t$:
>
> $$
> v_{t+1} = \mu\,v_t - \eta\,\nabla L\parens{\theta_t + \mu\,v_t},
> \qquad
> \theta_{t+1} = \theta_t + v_{t+1}.
> $$
>
> The only change from classical momentum is the argument of $\nabla L$.

The lookahead gives a stiffer, self-correcting response: if the velocity is about
to overshoot, the gradient at the projected point already points back, so NAG
brakes a step earlier than classical momentum. On smooth convex problems this
sharpens the worst-case rate from $O(1/t)$ to $O(1/t^2)$.[^gf-nesterov]

$$
% caption: Classical momentum vs. Nesterov. Momentum (red) evaluates $\nabla L$ at
% $\theta_t$; Nesterov (blue) evaluates it at the lookahead point, anticipating the surface ahead.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.15]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  % current point (bottom-left, generous canvas)
  \fill[black] (0,0) circle (2.6pt);
  \node[anchor=north, font=\footnotesize] at (0,-0.2) {\texttt{current}};
  % momentum carry mu*v (shared, gray dashed) -- runs to the right, label below
  \draw[black, thick, dashed, ->] (0,0) -- (3.4,0);
  \node[black, anchor=north, font=\footnotesize] at (1.7,-0.2) {\texttt{momentum carry}};
  % lookahead point at tip of carry
  \fill[acc] (3.4,0) circle (2.6pt);
  \node[acc, anchor=south, font=\footnotesize] at (3.4,0.18) {\texttt{lookahead}};
  % --- classical momentum: gradient at current, well above the carry line ---
  \draw[red, very thick, ->] (0,0) -- (0.55,1.7);
  \node[red, anchor=south east, font=\footnotesize] at (0.5,1.62) {\texttt{grad at current}};
  \draw[red, very thick, ->] (0,0) -- (3.95,1.7);
  \node[red, anchor=west, font=\footnotesize] at (4.15,1.7) {\texttt{momentum step}};
  % --- nesterov: gradient at lookahead points down, step lands lower ---
  \draw[acc, very thick, ->] (3.4,0) -- (2.95,-1.45);
  \node[acc, anchor=west, font=\footnotesize] at (3.25,-0.85) {\texttt{grad at lookahead}};
  \draw[green, very thick, ->] (0,0) -- (2.95,-1.45);
  \node[green, anchor=north west, font=\footnotesize] at (3.15,-1.7) {\texttt{Nesterov step}};
\end{tikzpicture}
$$

## Adaptive learning rates

Momentum still uses one global $\eta$. The second pathology is that different
_parameters_ need different rates: a weight that consistently sees large
gradients should take small steps, a weight that rarely receives gradient should
take large ones. **Adaptive methods** give each coordinate $i$ its own effective rate,
scaled by the history of $g_i$. The arithmetic is per-coordinate; the figure
below shows the geometric effect: rescaling a stretched ellipse back toward a
circle so a single step heads straight at the minimum.

$$
% caption: Per-parameter rescaling. Raw GD (red) stalls along the flat axis; an
% adaptive method (blue) divides each coordinate by its gradient RMS, reshaping the ellipse toward a circle.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  % --- left: raw gradient on a stretched ellipse ---
  \begin{scope}
    \foreach \r in {0.6,1.2,1.8}
      \draw[black] (0,0) ellipse ({\r*2.0} and \r);
    \fill[green] (0,0) circle (2.4pt);
    % raw step: dominated by steep (vertical) axis, points poorly
    \draw[red, very thick, ->] (-3.2,1.55) -- (-2.5,0.25);
    \node[red, anchor=west, font=\footnotesize] at (-3.15,1.7) {\texttt{raw step}};
    \node[anchor=north, font=\footnotesize] at (0,-2.25) {\texttt{anisotropic curvature}};
  \end{scope}
  % --- right: rescaled space (circle) ---
  \begin{scope}[xshift=9.0cm]
    \foreach \r in {0.6,1.2,1.8}
      \draw[black] (0,0) circle (\r);
    \fill[green] (0,0) circle (2.4pt);
    % adaptive step points straight at the minimum
    \draw[acc, very thick, ->] (-1.65,1.65) -- (-0.15,0.15);
    \node[acc, anchor=west, font=\footnotesize] at (-1.5,2.05) {\texttt{adaptive step}};
    \node[anchor=north, font=\footnotesize] at (0,-2.25) {\texttt{rescaled per axis}};
  \end{scope}
  % arrow between (kept above the axis labels)
  \draw[black, thick, ->] (4.6,0.6) -- (6.6,0.6);
  \node[black, anchor=south, font=\footnotesize] at (5.6,0.7) {\texttt{rescale}};
\end{tikzpicture}
$$

### AdaGrad

AdaGrad accumulates the _sum_ of squared gradients per coordinate and divides the
step by its square root. Let $r$ be the running accumulator (a vector, one entry
per parameter), $\odot$ elementwise product, division elementwise:

$$
r_t = r_{t-1} + g_t \odot g_t,
\qquad
\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{r_t} + \epsilon}\odot g_t.
$$

A coordinate with a long history of large gradients gets a large $r_i$ and so a
small effective rate; a rarely-updated coordinate keeps a large rate. The $\epsilon
\approx 10^{-8}$ only guards the division.[^gf-adagrad]

> **Definition (AdaGrad).** Per-parameter learning rate $\eta / (\sqrt{r_i} +
> \epsilon)$ where $r_i = \sum_{s\le t} g_{i,s}^2$ is the running sum of squared
> gradients. Each coordinate's effective rate decays as $1/\sqrt{\text{cumulative
> gradient energy}}$.

The flaw is structural: $r$ only ever grows, so the effective rate
$\eta/\sqrt{r_i}$ decays monotonically to zero. On a long training run AdaGrad's
steps shrink toward zero before reaching the minimum.

> **Lemma (AdaGrad's vanishing rate).** Because $r_t = \sum_{s\le t} g_s^2$ is
> non-decreasing and unbounded under persistent gradients, the effective learning
> rate $\eta/\sqrt{r_t}$ is monotonically decreasing with $\lim_{t\to\infty}
> \eta/\sqrt{r_t} = 0$. Learning halts even when the loss has not converged.

### RMSProp

RMSProp removes the flaw with one change: replace the cumulative _sum_ by an
**exponential moving average**, so old gradients decay out of the accumulator and
$r$ tracks only the _recent_ gradient magnitude.

$$
r_t = \rho\,r_{t-1} + (1 - \rho)\,g_t \odot g_t,
\qquad
\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{r_t} + \epsilon}\odot g_t.
$$

The decay $\rho$ (typically $0.9$ or $0.99$) gives $r$ a finite effective memory
of about $1/(1-\rho)$ steps. The accumulator no longer grows without bound, so
the effective rate stabilizes instead of vanishing.

> **Definition (RMSProp).** AdaGrad with the cumulative sum replaced by an
> exponential moving average of squared gradients, $r_t = \rho\,r_{t-1} +
> (1-\rho)\,g_t^2$. The window $1/(1-\rho)$ keeps the per-parameter rate
> responsive to recent curvature rather than the entire history.

## Adam

Adam (Adaptive Moment Estimation) is the synthesis: momentum's first moment _and_
RMSProp's second moment, each an exponential moving average, each
bias-corrected. It is the default optimizer of modern deep learning.[^gf-adam]

> **Definition (Adam).** Maintain EMAs of the gradient (first moment $m$) and its
> square (second moment $v$), bias-correct both, and step:
>
> $$
> \begin{aligned}
> m_t &= \beta_1\,m_{t-1} + (1-\beta_1)\,g_t, &
> \hat m_t &= \frac{m_t}{1 - \beta_1^{\,t}}, \\[4pt]
> v_t &= \beta_2\,v_{t-1} + (1-\beta_2)\,g_t^2, &
> \hat v_t &= \frac{v_t}{1 - \beta_2^{\,t}}, \\[4pt]
> \theta_{t+1} &= \theta_t - \eta\,\frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}.
> \end{aligned}
> $$
>
> Defaults: $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$.

The numerator $\hat m_t$ is the momentum direction; the denominator $\sqrt{\hat
v_t}$ is the RMSProp per-parameter scale. Their ratio is a momentum step
normalized by recent gradient magnitude: direction from the first moment,
step-size from the second.

### Why bias correction

The moving averages start at $m_0 = v_0 = 0$, which biases the early estimates
toward zero. The correction undoes this exactly. Unroll the first-moment
recurrence from zero, assuming a stationary gradient with $\mathbb E[g_s] = g$:

$$
m_t = (1-\beta_1)\sum_{s=1}^{t} \beta_1^{\,t-s}\, g_s.
$$

Take expectations and pull out the constant mean:

$$
\mathbb E[m_t] = g\,(1-\beta_1)\sum_{s=1}^{t}\beta_1^{\,t-s}
= g\,(1-\beta_1)\cdot\frac{1 - \beta_1^{\,t}}{1 - \beta_1}
= g\,(1 - \beta_1^{\,t}).
$$

The geometric sum $\sum_{s=1}^t \beta_1^{t-s} = (1-\beta_1^t)/(1-\beta_1)$ leaves
the factor $(1 - \beta_1^t)$, so $m_t$ underestimates $g$ by exactly that factor.
Dividing it out gives an _unbiased_ estimate, $\mathbb E[\hat m_t] = \mathbb
E[m_t]/(1-\beta_1^t) = g$. The identical argument with $\beta_2$ corrects $v_t$.

> **Theorem (Adam bias correction is exact).** Under a stationary gradient,
> $\mathbb E[m_t] = (1-\beta_1^{\,t})\,\mathbb E[g]$, hence $\hat m_t = m_t /
> (1-\beta_1^{\,t})$ satisfies $\mathbb E[\hat m_t] = \mathbb E[g]$. The same holds
> for $\hat v_t$ with $\beta_2$.

> **Proof.** With $m_0 = 0$, induction on $m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$
> gives $m_t = (1-\beta_1)\sum_{s=1}^t \beta_1^{t-s} g_s$. Taking expectations with
> $\mathbb E[g_s] = \mathbb E[g]$ constant and summing the geometric series,
> $\mathbb E[m_t] = (1-\beta_1)\,\mathbb E[g]\,\tfrac{1-\beta_1^t}{1-\beta_1} =
> (1-\beta_1^t)\,\mathbb E[g]$. Dividing by $1-\beta_1^t$ removes the factor. $\qed$

The correction factor $1/(1-\beta_1^t)$ is enormous at $t=1$ and decays to $1$
within a few hundred steps. It is a warm-up baked into the math: early steps,
when the averages are unreliable, are scaled up to compensate.

At the very first step the correction is essential. With
$m_0 = v_0 = 0$ and a single gradient $g_1$,

$$
m_1 = (1-\beta_1)\,g_1, \qquad v_1 = (1-\beta_2)\,g_1^2.
$$

Without correction the update would be $\eta\,(1-\beta_1)\,g_1 / \sqrt{(1-\beta_2)\,g_1^2}$,
which for the defaults $\beta_1 = 0.9$, $\beta_2 = 0.999$ carries a raw scale of
$0.1 / \sqrt{0.001} \approx 3.16$ instead of the intended $|g_1|/|g_1| = 1$ — the
first step would be more than $3\times$ too large. Bias correction restores it:

$$
\hat m_1 = \frac{m_1}{1-\beta_1} = g_1, \qquad
\hat v_1 = \frac{v_1}{1-\beta_2} = g_1^2,
$$

so $\hat m_1 / \sqrt{\hat v_1} = g_1 / |g_1| = \pm 1$ and the first step has
magnitude exactly $\eta$, as intended. The correction and the initialization bias
cancel term for term.

$$
% caption: The bias-correction factor $1/(1-\beta^t)$ versus step $t$. It decays to
% $1$ as a built-in warm-up; $\beta=0.9$ settles in tens of steps, $\beta=0.999$ in thousands.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes: y = correction factor, mapped factor 1 -> y=0.5, factor 3.3 -> y~3.1
  \draw[->, thick] (0,0) -- (7.6,0) node[right, font=\footnotesize] {\texttt{step t}};
  \draw[->, thick] (0,0) -- (0,3.6) node[above, font=\footnotesize] {\texttt{correction factor}};
  % asymptote at factor = 1 (mapped to y = 0.5)
  \draw[black, dashed] (0,0.5) -- (7.4,0.5) node[black, right, font=\scriptsize] {1};
  % Both curves share the shape 1/(1-e^{-s}); difference is only the time-axis scale.
  % Sampled coordinates (precomputed) of the SAME decay, drawn over the full x-range.
  % fast EMA (beta=0.9): reaches ~1 within tens of steps.
  \draw[acc, very thick] plot coordinates {
    (0.30,3.10) (0.55,2.30) (0.85,1.78) (1.20,1.42) (1.65,1.18)
    (2.20,1.02) (2.90,0.86) (3.80,0.72) (5.00,0.62) (7.20,0.55)};
  \node[acc, anchor=south west, font=\footnotesize] at (1.5,1.35) {\texttt{fast EMA}};
  % slow EMA (beta=0.999): same shape, but the horizon is ~100x longer, so over the
  % plotted window it sits well above 1 throughout (decays much more gradually).
  \draw[red, very thick] plot coordinates {
    (0.30,3.30) (0.80,3.05) (1.40,2.78) (2.10,2.50) (2.90,2.22)
    (3.80,1.95) (4.80,1.70) (5.90,1.48) (7.20,1.30)};
  \node[red, anchor=south west, font=\footnotesize] at (3.4,2.35) {\texttt{slow EMA}};
  % tick labels
  \node[anchor=north east, font=\scriptsize] at (0,0) {0};
  \node[anchor=north, font=\footnotesize] at (7.0,-0.05) {\texttt{later}};
\end{tikzpicture}
$$

The whole update is a five-stage pipeline: the gradient feeds two exponential
moving averages, each is bias-corrected, and their ratio becomes a normalized
step. The figure traces one gradient through it.

$$
% caption: The Adam update as dataflow. The gradient $g_t$ feeds the first-moment EMA
% $m_t$ (direction) and second-moment EMA $v_t$ (scale); both are bias-corrected, then the
% normalized ratio $\hat m_t / (\sqrt{\hat v_t}+\epsilon)$ scaled by $\eta$ updates $\theta$.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  \tikzset{
    box/.style={draw, minimum width=1.9cm, minimum height=0.85cm, align=center, font=\scriptsize},
    fbox/.style={draw=acc, fill=acc!8, minimum width=1.9cm, minimum height=0.85cm, align=center, font=\scriptsize},
    vbox/.style={draw=red, fill=red!8, minimum width=1.9cm, minimum height=0.85cm, align=center, font=\scriptsize}
  }
  % input gradient
  \node[box] (g) at (0,0) {\texttt{gradient}\\$g_t$};
  % top branch: first moment
  \node[fbox] (m) at (3.2,1.35) {\texttt{first moment}\\$m_t$};
  \node[fbox] (mh) at (6.6,1.35) {\texttt{bias correct}\\$\hat m_t$};
  % bottom branch: second moment
  \node[vbox] (v) at (3.2,-1.35) {\texttt{second moment}\\$v_t$};
  \node[vbox] (vh) at (6.6,-1.35) {\texttt{bias correct}\\$\hat v_t$};
  % combine
  \node[box, draw=green, fill=green!8] (step) at (9.9,0) {\texttt{normalized}\\\texttt{step}};
  \node[box] (upd) at (9.9,-2.2) {\texttt{update}\\\texttt{theta}};
  % arrows
  \draw[->, thick] (g) -- (m);
  \draw[->, thick] (g) -- (v);
  \draw[->, acc, thick] (m) -- (mh);
  \draw[->, red, thick] (v) -- (vh);
  \draw[->, acc, thick] (mh) -- (step);
  \draw[->, red, thick] (vh) -- (step);
  \draw[->, green, thick] (step) -- (upd);
  % annotations
  \node[acc, anchor=south, font=\footnotesize] at (4.9,2.0) {\texttt{direction}};
  \node[red, anchor=north, font=\footnotesize] at (4.9,-2.0) {\texttt{scale}};
\end{tikzpicture}
$$

```algorithm
caption: $\textsc{Adam}(L, \theta, \eta, \beta_1, \beta_2, \epsilon)$ — adaptive moment estimation
initialize $\theta$ randomly,\ $m \gets 0,\ v \gets 0,\ t \gets 0$
repeat
  $t \gets t + 1$
  sample a minibatch; $g \gets \nabla_\theta L(\theta)$
  $m \gets \beta_1 \cdot m + (1 - \beta_1)\cdot g$ // first moment EMA
  $v \gets \beta_2 \cdot v + (1 - \beta_2)\cdot (g \odot g)$ // second moment EMA
  $\hat m \gets m / (1 - \beta_1^{\,t})$ // bias-correct
  $\hat v \gets v / (1 - \beta_2^{\,t})$
  $\theta \gets \theta - \eta \cdot \hat m / (\sqrt{\hat v} + \epsilon)$ // per-parameter step
until converged
return $\theta$
```

### AdamW: decoupled weight decay

Adding $L_2$ regularization to the loss is _not_ the same as weight decay once an
adaptive denominator is in play. The $L_2$ gradient $\lambda\theta$ enters $g$,
gets divided by $\sqrt{\hat v}$, and so decays large-gradient weights _less_ than
small-gradient ones, the opposite of the intent. **AdamW** fixes this by
applying the decay directly to the parameters, outside the adaptive update:[^chollet-optimizers]

$$
\theta_{t+1} = \theta_t - \eta\parens{\frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}
+ \lambda\,\theta_t}.
$$

> **Definition (AdamW).** Adam with **decoupled weight decay**: the term
> $\lambda\theta$ is subtracted from the parameters _after_ the adaptive step, not
> folded into the gradient. This restores the regularization strength to a uniform
> $\eta\lambda$ per weight, independent of $\hat v$.

### Reading the defaults

Each default addresses a specific concern:

- $\beta_1 = 0.9$ gives the first moment a memory of about $1/(1-\beta_1) = 10$
  steps — long enough to smooth minibatch noise in the direction, short enough to
  turn when the loss surface turns.
- $\beta_2 = 0.999$ gives the second moment a memory of about $1000$ steps. The
  per-parameter scale should reflect the _typical_ gradient magnitude over a long
  window, so it is deliberately slower than the direction. A $\beta_2$ set too
  low makes $\sqrt{\hat v}$ jumpy and the effective rate spikes on any single large
  gradient.
- $\epsilon = 10^{-8}$ only guards the division $\hat m / \sqrt{\hat v}$ from a
  zero denominator. It is not a learning-rate knob, though raising it to
  $10^{-4}$ or $10^{-3}$ caps the effective rate for coordinates whose gradients
  are tiny, which stabilizes some models.
- $\eta = 10^{-3}$ is the common starting point. Because $\hat m / \sqrt{\hat v}$
  is already normalized to roughly unit magnitude per coordinate, Adam's $\eta$
  transfers across problems far better than SGD's, which must absorb the raw
  gradient scale.

**Failure modes.** Adam is not uniformly safe. Three recur:

1. **Second-moment stalls.** A coordinate that saw one enormous gradient inflates
   $\hat v$; its effective rate collapses and can take the full $\beta_2$ horizon
   to recover. This is why gradient clipping and $\beta_2$ tuning matter on RNNs
   and transformers.
2. **Worse generalization than SGD.** On some vision benchmarks Adam converges
   faster but lands in sharper minima that test slightly worse; SGD with momentum
   remains a competitive final-accuracy baseline.[^gf-adam]
3. **The $L_2$/decay confusion.** Passing a `weight_decay` to a plain-Adam
   implementation applies coupled $L_2$, not true decay, and the effective
   regularization is uneven across parameters; AdamW is the fix and the modern
   transformer default.

## The optimizer zoo

Every method here is one skeleton (accumulate _something_ from the gradient
history, step by a function of it) with different choices of what to keep. The
master table collapses the family onto that axis.

| Optimizer | Update rule | State kept | Key hyperparameters | When to use |
| --- | --- | --- | --- | --- |
| SGD | $\theta \gets \theta - \eta g$ | none | $\eta$ | convex / well-tuned baselines |
| Momentum | $v \gets \mu v - \eta g;\ \theta \gets \theta + v$ | velocity $v$ | $\eta,\ \mu$ | ravines; large-batch vision |
| Nesterov | $v \gets \mu v - \eta\nabla L(\theta + \mu v);\ \theta \gets \theta + v$ | velocity $v$ | $\eta,\ \mu$ | smoother convex problems |
| AdaGrad | $r \gets r + g^2;\ \theta \gets \theta - \tfrac{\eta}{\sqrt r}g$ | sum of $g^2$ | $\eta$ | sparse features (NLP) |
| RMSProp | $r \gets \rho r + (1-\rho)g^2;\ \theta \gets \theta - \tfrac{\eta}{\sqrt r}g$ | EMA of $g^2$ | $\eta,\ \rho$ | RNNs; non-stationary loss |
| Adam | $\hat m / (\sqrt{\hat v} + \epsilon)$ step | EMAs $m,\ v$ | $\eta,\ \beta_1,\ \beta_2$ | default for almost everything |
| AdamW | Adam $+\ \lambda\theta$ decoupled | EMAs $m,\ v$ | $\eta,\ \beta_1,\ \beta_2,\ \lambda$ | transformers; default with decay |

On the ravine of the opening figure the three trajectories diverge in character:
SGD zig-zags across the steep axis, momentum cancels the oscillation and coasts
along the valley, and Adam heads almost straight at the minimum because its
per-parameter denominator has already rescaled the steep axis down.

$$
% caption: Trajectories on the same ravine. SGD (red) oscillates across the steep
% axis; momentum (green) damps the zig-zag; Adam (blue) rescales per axis and cuts a near-straight path.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  % elliptical contours (ravine: wide x, narrow y)
  \foreach \r in {0.6,1.2,1.8,2.4}
    \draw[black] (0,0) ellipse ({\r*2.3} and \r);
  \fill[black] (0,0) circle (2.4pt);
  \node[anchor=north west, font=\footnotesize] at (0.3,-0.08) {\texttt{min}};
  % SGD zig-zag (red)
  \draw[red, very thick, ->] (-5.2,1.5) -- (-4.1,-1.15);
  \draw[red, very thick, ->] (-4.1,-1.15) -- (-3.1,0.92);
  \draw[red, very thick, ->] (-3.1,0.92) -- (-2.25,-0.7);
  \draw[red, very thick, ->] (-2.25,-0.7) -- (-1.55,0.52);
  \node[red, anchor=south, font=\footnotesize] at (-5.2,1.56) {\texttt{SGD}};
  % momentum (green): damped coast
  \draw[green, very thick, ->] (-5.2,-1.5) .. controls (-3.8,-0.9) and (-2.4,-0.42) .. (-1.1,-0.16);
  \draw[green, very thick, ->] (-1.1,-0.16) -- (-0.1,-0.03);
  \node[green, anchor=north, font=\footnotesize] at (-5.2,-1.56) {\texttt{momentum}};
  % Adam (blue): near-straight from top-right
  \draw[acc, very thick, ->] (5.0,1.9) .. controls (3.0,1.0) and (1.2,0.35) .. (0.12,0.04);
  \node[acc, anchor=south, font=\footnotesize] at (5.0,1.96) {\texttt{Adam}};
\end{tikzpicture}
$$

The qualitative training curves separate the three regimes: SGD descends slowest
and ripples, momentum smooths and steepens the descent, Adam drops fastest in the
early phase by adapting per-parameter rates.

$$
% caption: Qualitative loss-vs-iteration. SGD (black) descends slowly, momentum
% (green) accelerates it, and Adam (blue) drops fastest early; all reach a similar floor.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % axes
  \draw[->, thick] (0,0) -- (7.6,0) node[right, font=\footnotesize] {\texttt{iteration}};
  \draw[->, thick] (0,0) -- (0,3.8) node[above, font=\footnotesize] {\texttt{loss}};
  % SGD (gray): slow decay with small ripple
  \draw[black, very thick] plot[domain=0:7.2, samples=90]
    (\x, {3.3*exp(-0.32*\x) + 0.35 + 0.07*sin(5*\x r)*exp(-0.2*\x)});
  \node[black, anchor=south west, font=\footnotesize] at (4.7,1.25) {\texttt{SGD}};
  % Momentum (green): faster, smoother
  \draw[green, very thick] plot[domain=0:7.2, samples=90]
    (\x, {3.3*exp(-0.52*\x) + 0.3});
  \node[green, anchor=east, font=\footnotesize] at (5.15,0.52) {\texttt{momentum}};
  % Adam (blue): fastest early drop
  \draw[acc, very thick] plot[domain=0:7.2, samples=90]
    (\x, {3.3*exp(-0.95*\x) + 0.28});
  \node[acc, anchor=west, font=\footnotesize] at (1.5,0.55) {\texttt{Adam}};
\end{tikzpicture}
$$

## Optimizer refinements

Goodfellow's chapter predates most of the optimizer refinements now standard in
practice; each is a public, named result that patches a specific failure listed
above.

- **AdamW.** The $L_2$-vs-decay distinction of the
  previous section matters in practice: Loshchilov and Hutter showed that _decoupled_
  weight decay measurably improves generalization and makes the decay strength
  independent of the adaptive denominator. AdamW is now the default for
  transformer training, and the coupled form the original Adam paper used is
  treated as a bug.[^loshchilov]
- **A convergence hole in the original proof.** Reddi, Kale, and Kumar found a
  counterexample on which Adam _fails to converge_ even on a convex problem,
  traced to the second moment being able to shrink between steps. Their **AMSGrad**
  fix enforces a non-decreasing $\hat v$, restoring the guarantee.[^reddi]
- **Decoupling direction from scale, taken further.** **LAMB** (You et al.)
  layer-wise normalizes the Adam step so that very large batches (used to train
  BERT in $76$ minutes) stay stable — the natural extension of the linear-scaling
  and warmup ideas to adaptive optimizers.[^lamb]
- **Lion: a learned optimizer.** Chen et al. used program search to _discover_ an
  optimizer, **Lion**, which keeps only a momentum buffer and steps by the sign of
  the interpolated update — cheaper state than Adam and competitive or better on
  large vision and language models.[^lion]

Every one of these keeps the momentum-plus-per-parameter-
scale skeleton and changes exactly one choice: how decay is applied, how $\hat v$
is bounded, how the step is normalized across layers, or what nonlinearity maps
the moments to the update.

## Takeaways

- Plain SGD fails on **ill-conditioned** surfaces ($\kappa = \lambda_{\max}/
  \lambda_{\min} \gg 1$): one global $\eta$ must satisfy $\eta < 2/\lambda_{\max}$
  for stability, which leaves convergence along $\lambda_{\min}$ very slow.
- **Momentum** accumulates a velocity $v_{t+1} = \mu v_t - \eta\nabla L$ and steps
  $\theta_{t+1} = \theta_t + v_{t+1}$; it amplifies persistent directions to an
  effective step $\eta/(1-\mu)$ and cancels oscillating ones, damping zig-zag.
- **Nesterov** evaluates the gradient at the lookahead $\theta + \mu v$, braking
  before overshoot and improving the convex rate to $O(1/t^2)$.
- **AdaGrad** gives per-parameter rates $\eta/\sqrt{\sum g^2}$ but the
  monotonically growing accumulator drives the rate to zero; **RMSProp** fixes it
  with an EMA of $g^2$.
- **Adam** fuses momentum (first moment) and RMSProp (second moment) with exact
  **bias correction** $1/(1-\beta^t)$, and is the default; **AdamW** decouples
  weight decay so regularization strength is uniform across parameters.

[^gf-basic]: **Goodfellow**, _Deep Learning_, §8.3 — Basic Algorithms: SGD, momentum, and Nesterov as successive cures for the geometry of the loss surface.
[^gf-momentum]: **Goodfellow**, _Deep Learning_, §8.3.2 — Momentum: the velocity recurrence $v \gets \mu v - \eta\nabla L$ read as a heavy ball with viscous friction.
[^gf-nesterov]: **Goodfellow**, _Deep Learning_, §8.3.3 — Nesterov Momentum: evaluating the gradient at the lookahead point and the $O(1/t^2)$ convex rate.
[^gf-adagrad]: **Goodfellow**, _Deep Learning_, §8.5.1 — AdaGrad: per-parameter rates scaled by the accumulated squared gradient, and the vanishing-rate flaw.
[^gf-adam]: **Goodfellow**, _Deep Learning_, §8.5.3 — Adam: first- and second-moment EMAs with bias correction, the default adaptive optimizer.
[^chollet-optimizers]: **Chollet**, _Deep Learning with Python_, §2.4, §3.6 — Optimizers in practice: choosing RMSProp/Adam and the role of weight decay in Keras training loops.
[^loshchilov]: **Loshchilov & Hutter**, _Decoupled Weight Decay Regularization_, ICLR 2019 — AdamW: applying weight decay to the parameters, not the gradient, so its strength is independent of $\hat v$.
[^reddi]: **Reddi, Kale & Kumar**, _On the Convergence of Adam and Beyond_, ICLR 2018 — a convex counterexample where Adam diverges, and the AMSGrad fix that keeps $\hat v$ non-decreasing.
[^lamb]: **You et al.**, _Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes_, ICLR 2020 — LAMB, a layer-wise normalized Adam variant for very large batches.
[^lion]: **Chen et al.**, _Symbolic Discovery of Optimization Algorithms_, NeurIPS 2023 — Lion, a program-searched optimizer using only a momentum buffer and a sign update.