Momentum & Adaptive Methods

Plain stochastic gradient descent takes the same step rule everywhere: $\theta \leftarrow \theta -\eta \nabla L$, one scalar $\eta$, applied identically to every coordinate. Two structural facts of the loss surface break this. First, curvature differs by direction: a ravine is steep across and shallow along, so a step large enough to make progress down the valley overshoots across it and the path zig-zags. Second, curvature differs by coordinate, since some weights see huge gradients, others tiny ones, yet a single $\eta$ serves all of them. This lesson addresses each in turn: momentum for the first, adaptive learning rates for the second, and Adam combining them into the modern default.¹

The pathology: ill-conditioning

Near a minimum the loss is locally quadratic, $L(\theta )\approx \frac{1}{2}(\theta -{\theta }^{\star }{)}^{\top }H(\theta -{\theta }^{\star })$, with Hessian $H$. Rotate to the eigenbasis of $H$ and the coordinates decouple: along eigenvector $i$ with eigenvalue ${\lambda }_i$, gradient descent with rate $\eta$ contracts the error by a fixed factor each step,

Stability demands $|1-\eta {\lambda }_i|<1$ for every axis, so the largest eigenvalue ${\lambda }_{\max }$ caps the step at $\eta <2/{\lambda }_{\max }$. But convergence along the flattest axis proceeds at rate $1-\eta {\lambda }_{\min }$, which ${\lambda }_{\min }\ll {\lambda }_{\max }$ drives very close to $1$.

The next figure shows the geometry: on elliptical contours plain SGD oscillates across the steep direction while making little progress along the shallow one.

Momentum

Momentum addresses the zig-zag with memory. Rather than stepping by the current gradient, accumulate an exponentially-weighted running sum of gradients (a velocity) and step by that. Oscillating components alternate sign and cancel in the sum; the consistent down-valley component survives and compounds.

The name is literal. Read $\theta$ as the position of a ball of unit mass, $-\nabla L$ as the force from the potential $L$, and $\mu$ as a friction (drag) coefficient: the velocity update is a discrete-time Newtonian dynamics with viscous damping. The ball does not teleport down the gradient; it coasts, carrying its momentum through small bumps and across narrow ravines.²

Effective step size

Unrolling the velocity recurrence with a constant gradient $g$ exposes why momentum accelerates. With $v_0=0$,

The terminal velocity is the geometric series ${\sum }_{k\ge 0}{\mu }^k=1/(1-\mu )$ times a single SGD step. The effective learning rate along any direction the gradient persistently points is therefore amplified:

That asymmetry is the entire mechanism. Down the shallow valley the gradient is consistent, so momentum compounds it and the path accelerates; across the steep ravine the gradient reverses every step, so the contributions annihilate and the oscillation is damped. One coefficient produces both behaviors.

For example, fix $\eta =0.1$, $\mu =0.9$, and a constant unit gradient $g=1$ along a persistent direction. The velocity magnitude grows step by step,

each term being $0.9$ of the previous velocity plus $0.1$. The steady state $0.1/(1-0.9)=1.0$ is ten times a single SGD step of $0.1$. Now flip the sign of $g$ every step, as it does across a ravine: the velocity oscillates within a small band around zero instead of growing, because each contribution partly cancels the last. The same recurrence, fed a consistent signal, integrates it; fed an alternating signal, filters it out.

The dependence on $\mu$ is steep near $1$: the effective step $1/(1-\mu )$ is a hyperbola, so the last tenth of $\mu$ carries most of the amplification. That is also why $\mu$ close to $1$ is delicate: it lengthens the memory (a horizon of $1/(1-\mu )$ steps) and can overshoot on curvature that turns before the velocity can respond.

Nesterov accelerated gradient

Momentum measures the force where the ball is; Nesterov's variant measures it where the ball is about to be. Take the momentum step first to the lookahead point $\tilde{\theta }=\theta +\mu v$, and evaluate the gradient there, so the correction anticipates the surface ahead instead of reacting to the surface behind.

The lookahead gives a stiffer, self-correcting response: if the velocity is about to overshoot, the gradient at the projected point already points back, so NAG brakes a step earlier than classical momentum. On smooth convex problems this sharpens the worst-case rate from $O(1/t)$ to $O(1/t^2)$.³

Adaptive learning rates

Momentum still uses one global $\eta$. The second pathology is that different parameters need different rates: a weight that consistently sees large gradients should take small steps, a weight that rarely receives gradient should take large ones. Adaptive methods give each coordinate $i$ its own effective rate, scaled by the history of $g_i$. The arithmetic is per-coordinate; the figure below shows the geometric effect: rescaling a stretched ellipse back toward a circle so a single step heads straight at the minimum.

AdaGrad

AdaGrad accumulates the sum of squared gradients per coordinate and divides the step by its square root. Let $r$ be the running accumulator (a vector, one entry per parameter), $\odot$ elementwise product, division elementwise:

A coordinate with a long history of large gradients gets a large $r_i$ and so a small effective rate; a rarely-updated coordinate keeps a large rate. The $ϵ\approx 10^{-8}$ only guards the division.⁴

The flaw is structural: $r$ only ever grows, so the effective rate $\eta /\sqrt{r_i}$ decays monotonically to zero. On a long training run AdaGrad's steps shrink toward zero before reaching the minimum.

RMSProp

RMSProp removes the flaw with one change: replace the cumulative sum by an exponential moving average, so old gradients decay out of the accumulator and $r$ tracks only the recent gradient magnitude.

The decay $\rho$ (typically $0.9$ or $0.99$) gives $r$ a finite effective memory of about $1/(1-\rho )$ steps. The accumulator no longer grows without bound, so the effective rate stabilizes instead of vanishing.

Adam

Adam (Adaptive Moment Estimation) is the synthesis: momentum's first moment and RMSProp's second moment, each an exponential moving average, each bias-corrected. It is the default optimizer of modern deep learning.⁵

The numerator ${\hat{m}}_t$ is the momentum direction; the denominator $\sqrt{{\hat{v}}_t}$ is the RMSProp per-parameter scale. Their ratio is a momentum step normalized by recent gradient magnitude: direction from the first moment, step-size from the second.

Why bias correction

The moving averages start at $m_0=v_0=0$, which biases the early estimates toward zero. The correction undoes this exactly. Unroll the first-moment recurrence from zero, assuming a stationary gradient with $\mathbb{E}[g_s]=g$:

Take expectations and pull out the constant mean:

The geometric sum ${\sum }_{s=1}^t{\beta }_1^{t-s}=(1-{\beta }_1^t)/(1-{\beta }_1)$ leaves the factor $(1-{\beta }_1^t)$, so $m_t$ underestimates $g$ by exactly that factor. Dividing it out gives an unbiased estimate, $\mathbb{E}[{\hat{m}}_t]=\mathbb{E}[m_t]/(1-{\beta }_1^t)=g$. The identical argument with ${\beta }_2$ corrects $v_t$.

The correction factor $1/(1-{\beta }_1^t)$ is enormous at $t=1$ and decays to $1$ within a few hundred steps. It is a warm-up baked into the math: early steps, when the averages are unreliable, are scaled up to compensate.

At the very first step the correction is essential. With $m_0=v_0=0$ and a single gradient $g_1$,

Without correction the update would be $\eta (1-{\beta }_1)g_1/\sqrt{(1-{\beta }_2)g_1^2}$, which for the defaults ${\beta }_1=0.9$, ${\beta }_2=0.999$ carries a raw scale of $0.1/\sqrt{0.001}\approx 3.16$ instead of the intended $|g_1|/|g_1|=1$ — the first step would be more than $3\times$ too large. Bias correction restores it:

so ${\hat{m}}_1/\sqrt{{\hat{v}}_1}=g_1/|g_1|=\pm 1$ and the first step has magnitude exactly $\eta$, as intended. The correction and the initialization bias cancel term for term.

The whole update is a five-stage pipeline: the gradient feeds two exponential moving averages, each is bias-corrected, and their ratio becomes a normalized step. The figure traces one gradient through it.

AdamW: decoupled weight decay

Adding $L_2$ regularization to the loss is not the same as weight decay once an adaptive denominator is in play. The $L_2$ gradient $\lambda \theta$ enters $g$, gets divided by $\sqrt{\hat{v}}$, and so decays large-gradient weights less than small-gradient ones, the opposite of the intent. AdamW fixes this by applying the decay directly to the parameters, outside the adaptive update:⁶

Reading the defaults

Each default addresses a specific concern:

• ${\beta }_1=0.9$ gives the first moment a memory of about $1/(1-{\beta }_1)=10$ steps — long enough to smooth minibatch noise in the direction, short enough to turn when the loss surface turns.
• ${\beta }_2=0.999$ gives the second moment a memory of about $1000$ steps. The per-parameter scale should reflect the typical gradient magnitude over a long window, so it is deliberately slower than the direction. A ${\beta }_2$ set too low makes $\sqrt{\hat{v}}$ jumpy and the effective rate spikes on any single large gradient.
• $ϵ=10^{-8}$ only guards the division $\hat{m}/\sqrt{\hat{v}}$ from a zero denominator. It is not a learning-rate knob, though raising it to $10^{-4}$ or $10^{-3}$ caps the effective rate for coordinates whose gradients are tiny, which stabilizes some models.
• $\eta =10^{-3}$ is the common starting point. Because $\hat{m}/\sqrt{\hat{v}}$ is already normalized to roughly unit magnitude per coordinate, Adam's $\eta$ transfers across problems far better than SGD's, which must absorb the raw gradient scale.

Failure modes. Adam is not uniformly safe. Three recur:

1. Second-moment stalls. A coordinate that saw one enormous gradient inflates $\hat{v}$; its effective rate collapses and can take the full ${\beta }_2$ horizon to recover. This is why gradient clipping and ${\beta }_2$ tuning matter on RNNs and transformers.
2. Worse generalization than SGD. On some vision benchmarks Adam converges faster but lands in sharper minima that test slightly worse; SGD with momentum remains a competitive final-accuracy baseline.⁵
3. The $L_2$/decay confusion. Passing a weight_decay to a plain-Adam implementation applies coupled $L_2$, not true decay, and the effective regularization is uneven across parameters; AdamW is the fix and the modern transformer default.

The optimizer zoo

Every method here is one skeleton (accumulate something from the gradient history, step by a function of it) with different choices of what to keep. The master table collapses the family onto that axis.

On the ravine of the opening figure the three trajectories diverge in character: SGD zig-zags across the steep axis, momentum cancels the oscillation and coasts along the valley, and Adam heads almost straight at the minimum because its per-parameter denominator has already rescaled the steep axis down.

The qualitative training curves separate the three regimes: SGD descends slowest and ripples, momentum smooths and steepens the descent, Adam drops fastest in the early phase by adapting per-parameter rates.

Optimizer refinements

Goodfellow's chapter predates most of the optimizer refinements now standard in practice; each is a public, named result that patches a specific failure listed above.

• AdamW. The $L_2$-vs-decay distinction of the previous section matters in practice: Loshchilov and Hutter showed that decoupled weight decay measurably improves generalization and makes the decay strength independent of the adaptive denominator. AdamW is now the default for transformer training, and the coupled form the original Adam paper used is treated as a bug.⁷
• A convergence hole in the original proof. Reddi, Kale, and Kumar found a counterexample on which Adam fails to converge even on a convex problem, traced to the second moment being able to shrink between steps. Their AMSGrad fix enforces a non-decreasing $\hat{v}$, restoring the guarantee.⁸
• Decoupling direction from scale, taken further. LAMB (You et al.) layer-wise normalizes the Adam step so that very large batches (used to train BERT in $76$ minutes) stay stable — the natural extension of the linear-scaling and warmup ideas to adaptive optimizers.⁹
• Lion: a learned optimizer. Chen et al. used program search to discover an optimizer, Lion, which keeps only a momentum buffer and steps by the sign of the interpolated update — cheaper state than Adam and competitive or better on large vision and language models.¹⁰

Every one of these keeps the momentum-plus-per-parameter- scale skeleton and changes exactly one choice: how decay is applied, how $\hat{v}$ is bounded, how the step is normalized across layers, or what nonlinearity maps the moments to the update.

Takeaways

• Plain SGD fails on ill-conditioned surfaces ($\kappa ={\lambda }_{\max }/{\lambda }_{\min }\gg 1$): one global $\eta$ must satisfy $\eta <2/{\lambda }_{\max }$ for stability, which leaves convergence along ${\lambda }_{\min }$ very slow.
• Momentum accumulates a velocity $v_{t+1}=\mu v_t-\eta \nabla L$ and steps ${\theta }_{t+1}={\theta }_t+v_{t+1}$; it amplifies persistent directions to an effective step $\eta /(1-\mu )$ and cancels oscillating ones, damping zig-zag.
• Nesterov evaluates the gradient at the lookahead $\theta +\mu v$, braking before overshoot and improving the convex rate to $O(1/t^2)$.
• AdaGrad gives per-parameter rates $\eta /\sqrt{\sum g^2}$ but the monotonically growing accumulator drives the rate to zero; RMSProp fixes it with an EMA of $g^2$.
• Adam fuses momentum (first moment) and RMSProp (second moment) with exact bias correction $1/(1-{\beta }^t)$, and is the default; AdamW decouples weight decay so regularization strength is uniform across parameters.