Regularization Overview

A model with enough capacity will fit its training set perfectly, noise and all, and then generalize badly. Regularization is the set of techniques that make a flexible model generalize, and the most effective approach is not making the model smaller but leaving it large and penalizing what it does with that size.

This lesson treats the oldest and most transparent family, the parameter norm penalties: add a term that penalizes large weights. Every such method adds a penalty $\Omega (w)$, scaled by a strength $\lambda \ge 0$, to the data-fit loss $L$, and minimizes the sum

The hyperparameter $\lambda$ trades the two off: $\lambda =0$ recovers ordinary fitting, large $\lambda$ forces the weights toward small values of $\Omega$. The bias term is conventionally left unpenalized (it shifts the whole function, and regularizing it tends to underfit), so throughout, $w$ denotes the weights only.¹

Why penalize weights at all: the bias–variance decomposition

Before the penalties, the reason they help. Fix an input $x$ and suppose the label is generated as $y=f(x)+\epsilon$ with zero-mean noise $\mathbb{E}[\epsilon ]=0$ and variance $\mathrm{Var}(\epsilon )={\sigma }^2$. We fit an estimator $\hat{f}$ on a random training set $\mathcal{D}$, so $\hat{f}(x)$ is itself a random quantity — draw a different training set and you get a different fit. The quantity we care about is the expected squared test error at $x$, averaged over both the label noise and the draw of $\mathcal{D}$:

Write $\bar{f}(x)={\mathbb{E}}_{\mathcal{D}}[\hat{f}(x)]$ for the average fit over training sets. Insert and subtract $\bar{f}(x)$ inside the square, then expand. The noise $\epsilon$ is independent of $\hat{f}$, so every cross term with $\mathbb{E}[\epsilon ]=0$ drops:

The middle cross term $2(f-\bar{f})\mathbb{E}[\bar{f}-\hat{f}]$ vanishes because ${\mathbb{E}}_{\mathcal{D}}[\hat{f}]=\bar{f}$. Three named pieces remain.

A large, unconstrained model can drive bias to nearly zero but at the cost of large variance: it fits the noise in each particular $\mathcal{D}$, so $\hat{f}$ varies greatly from one training set to the next. Regularization deliberately introduces a little bias — it prevents the fit from matching the data exactly — in exchange for a large reduction in variance. Because the two enter the error as a sum, the minimum test error sits at the balance point, not at zero bias. That balance point is what the penalty tunes.²

$L^2$ regularization (weight decay)

The canonical choice penalizes the squared Euclidean norm.

The gradient of the penalty is just $w$, so the regularized gradient adds a term pointing back toward the origin:

Deriving the multiplicative-shrinkage update

Substitute that gradient into a single gradient-descent step with learning rate $\eta$ and collect the $w$ terms:

The name is now visible: before taking the usual data-driven step, every weight is multiplied by the factor $(1-\eta \lambda )<1$. Each step decays the weights toward zero by a constant fraction; this is the shrinkage. With $\eta =0.1$ and $\lambda =0.5$, the factor is $0.95$, so a weight left untouched by the data gradient halves roughly every fourteen steps ($0.95^{14}\approx 0.49$).

Closed-form effect on a quadratic objective

To see which weights decay most, approximate $L$ near its unregularized minimizer $w^{\star }$ by a second-order Taylor expansion. With $H$ the (symmetric, positive semidefinite) Hessian at $w^{\star }$ and the gradient vanishing there,

The regularized minimizer $\hat{w}$ sets the full gradient to zero, $H(\hat{w}-w^{\star })+\lambda \hat{w}=0$, which solves to

Diagonalize $H=Q\Lambda Q^{\top }$ in its orthonormal eigenbasis, with eigenvalues ${\lambda }_i\ge 0$ measuring curvature along each eigenvector. In that basis the solution decouples coordinate by coordinate:

Each component of the optimum is rescaled by ${\lambda }_i/({\lambda }_i+\lambda )\in (0,1]$. Read the two extremes:

The interpretation matches the bias–variance decomposition: directions the data constrains weakly (small ${\lambda }_i$) are where overfitting occurs, and weight decay zeroes them out while leaving the well-determined directions alone. A high-curvature direction has a steep, narrow valley in the loss — moving off $w_i^{\star }$ there raises the loss sharply, so the penalty moves it very little. A low-curvature direction is a flat trough — the loss barely changes with the weight, so the penalty pulls it freely to zero.

At ${\lambda }_i=\lambda$ the factor is exactly $\frac{1}{2}$: the penalty and the curvature pull equally, and half the weight is kept. That crossover point is the scale $\lambda$ sets — everything much stiffer than it is kept, everything much flatter is discarded.

Why $L^2$ prefers small, spread-out weights. The squared norm charges the square of each weight, so it is much cheaper to represent a needed quantity as many small weights than as one large weight. To split a target of $10$ across two weights, a single $(10,0)$ costs $100$ while an even $(5,5)$ costs $50$; the even split is preferred whenever the loss is indifferent to how the target is divided. The marginal penalty ${\partial }_j\frac{1}{2}\Vert w{\Vert }_2^2=w_j$ grows with the weight, so pushing a large weight down saves more than pushing an already-small weight — the objective equalizes magnitudes and never quite reaches zero.

Weight decay vs $L^2$, and where they part: AdamW

Under plain gradient descent, adding $\frac{\lambda }{2}\Vert w{\Vert }_2^2$ to the loss and applying the multiplicative decay $w\leftarrow (1-\eta \lambda )w$ are the same operation, which is why the two names are used interchangeably. They stop agreeing the moment the optimizer rescales the gradient. Adam divides each gradient component by a running estimate of its own magnitude, ${\hat{v}}_j$. If the $L^2$ term is folded into the loss, its contribution $\lambda w_j$ is scaled by the same $1/\sqrt{{\hat{v}}_j}$, so weights on noisy, large-gradient directions get less decay than intended and the effective penalty depends on the gradient history. AdamW fixes this by decoupling: it applies the shrink $w\leftarrow (1-\eta \lambda )w$ directly to the weights, outside the adaptive rescaling, restoring a uniform $(1-\eta \lambda )$ pull on every coordinate. For adaptive optimizers, decoupled weight decay (AdamW) and loss-added $L^2$ are genuinely different, and the decoupled form is the one that behaves like the analysis above.³

$L^1$ regularization

Swap the squared norm for the absolute-value norm and the qualitative behavior changes completely.

Subgradient and the soft-threshold

The absolute value is not differentiable at zero, so the gradient becomes a subgradient: the slope is $+1$ for positive weights, $-1$ for negative, and any value in $[-1,1]$ at the kink.

The penalty's gradient no longer scales with $w_j$; it is a constant push of size $\lambda$ toward zero, independent of the weight's magnitude. For a quadratic fit term the stationarity condition ${\partial }_{w_j}L+\lambda \mathrm{sign}(w_j)=0$ can be satisfied at exactly $w_j=0$ whenever the data gradient is smaller than $\lambda$: the subgradient interval $[-\lambda ,\lambda ]$ contains it. That is the mechanism of sparsity, expressed by the soft-thresholding solution (decoupled, unit-curvature case):

Any coordinate whose unregularized value sits within $\lambda$ of zero is set exactly to zero, not merely small. Plotted as a map from the unregularized value $w_j^{\star }$ to the fitted value ${\hat{w}}_j$, this is a flat dead-zone of width $2\lambda$ centered at the origin, flanked by two lines of slope $1$ shifted inward by $\lambda$. Contrast the $L^2$ map ${\hat{w}}_j=w_j^{\star }/(1+\lambda )$: a single straight line through the origin, never flat, never crossing zero except at zero itself.

Why the corner induces sparsity

The geometry makes this inevitable. The $L^1$ ball $\{w:\Vert w{\Vert }_1\le t\}$ is a diamond whose vertices lie on the axes; the $L^2$ ball is a round circle. The constrained optimum is where the expanding loss contours first touch the ball, and a smooth elliptical contour overwhelmingly first meets the diamond at a corner, a point where one coordinate is zero.

The contrast is the chapter's first idea: $L^2$ makes weights small, $L^1$ makes them absent. Sparsity is why $L^1$ doubles as a feature selector: a zeroed weight is an input the model has switched off entirely.⁴

Elastic net

Combining the two penalties keeps $L^1$'s sparsity while inheriting $L^2$'s stability when features are correlated (where pure $L^1$ picks one arbitrarily).

The master table

The two penalties, side by side, with the prior each one corresponds to (derived in the MAP view below):

Two views of the same penalty

View 1 — constrained optimization (KKT)

Penalizing a norm is the Lagrangian of constraining it.⁵ Minimizing $L(w)+\lambda \Omega (w)$ for a fixed $\lambda >0$ produces the same solution as the hard-constrained problem

for some radius $t=t(\lambda )$. The link is the KKT conditions: the Lagrangian is $\mathcal{L}(w,\mu )=L(w)+\mu (\Omega (w)-t)$, and stationarity $\nabla L+\mu \nabla \Omega =0$ reproduces the regularized gradient, with the multiplier $\mu$ playing the role of $\lambda$. Larger $\lambda$ pulls the optimum inward, which is a smaller ball radius $t$; the two knobs are inverses.

View 2 — the prior (MAP estimation)

The same penalty is a log-prior. Treat the weights as random with prior $p(w)$ and maximize the posterior; maximum a posteriori estimation gives, after taking $-\log$,

The penalty is the negative log-prior. Take a zero-mean isotropic Gaussian prior $p(w)={\prod }_j\frac{1}{\sqrt{2\pi }\tau }\exp (-w_j^2/2{\tau }^2)$. Its negative log is

which is $L^2$ with $\lambda =1/{\tau }^2$: a tighter prior (small $\tau$) is a stronger penalty. A Laplace prior $p(w)={\prod }_j\frac{1}{2b}\exp (-\left|w_j\right|/b)$ gives $-\log p(w)=\frac{1}{b}\Vert w{\Vert }_1+\text{const}$, i.e. $L^1$ with $\lambda =1/b$. The two priors differ exactly where the penalties do. The Gaussian is smooth and rounded at the origin, so it puts no special mass at zero. The Laplace has a sharp peak at zero (its derivative jumps from $+1/b$ to $-1/b$ there), placing far more prior mass on tiny weights; that spike is the probabilistic counterpart of the diamond's corner, and it is why the $L^1$ posterior mode lands on exact zeros. This recap extends the MAP framing introduced under the machine-learning refresher.⁶

What each penalty does to the weight distribution

The two priors produce distinct histograms of fitted weights: unregularized weights spread with heavy tails; $L^2$ concentrates them in a Gaussian-like hump near zero; $L^1$ produces a tall spike at zero plus a few surviving nonzeros.

Regularization strength and the bias–variance tradeoff

Sweeping $\lambda$ from $0$ upward walks the model down the capacity axis: small $\lambda$ leaves the full flexible model (low bias, high variance, prone to overfitting); large $\lambda$ pins the weights near zero (high bias, low variance, underfitting). Test error traces the familiar U-curve, indexed by regularization strength rather than raw model size. This is the bias–variance decomposition made operational: the falling red curve is the variance term shrinking as the penalty tightens, the rising blue curve is the bias${}^2$ term growing, and their sum is the U whose floor is the best $\lambda$.

The minimum is found by holding out data: $\lambda$ is selected on a validation set, never on the training loss it is designed to ignore.⁷

Practical defaults, with reasons

• Use $L^2$ (weight decay) first. It is the default because its effect — a smooth, uniform shrink toward zero — improves almost every over-parameterized network and never zeroes a coordinate outright, so it does not risk deleting a feature the model still needs. A weight-decay coefficient around $10^{-4}$ to $10^{-2}$ is the usual starting band; sweep it on a log scale.
• Use decoupled weight decay (AdamW) with adaptive optimizers. With Adam, folding $L^2$ into the loss makes the true penalty depend on the gradient history; the decoupled shrink restores the uniform pull that the eigenbasis analysis assumes.
• Use $L^1$ only when you want sparsity. Feature selection, compressing a model by pruning zeroed weights, or making the fit interpretable are the cases where exact zeros are useful. When features are correlated, prefer the elastic net so the group is kept together instead of one member being chosen arbitrarily.
• Leave biases unpenalized. A bias shifts the whole function rather than steepening it, so shrinking it toward zero buys no variance reduction and only adds bias.
• Tune $\lambda$ on validation, not training. The penalty exists to raise training loss; judging it by that loss would always prefer $\lambda =0$.

Decoupled weight decay

Goodfellow Ch. 7 treats weight decay and $L^2$ as interchangeable — right for plain SGD, wrong for the adaptive optimizers that now dominate. The correction is decoupled weight decay (Loshchilov & Hutter, 2019, the AdamW paper): folding $\frac{\lambda }{2}\Vert w{\Vert }_2^2$ into the loss makes Adam rescale the penalty by each coordinate's gradient history, so the effective decay is no longer uniform; applying the shrink $w\leftarrow (1-\eta \lambda )w$ directly to the weights restores the clean ${\lambda }_i/({\lambda }_i+\lambda )$ behavior the eigenbasis analysis above assumes. AdamW is now the default optimizer for training transformers, and the distinction between loss-added $L^2$ and decoupled weight decay is one every practitioner has to get right.

Two further public results sharpen the picture. First, in networks with normalization layers, weight decay often works through an unexpected channel: scaling a weight before a normalized layer leaves the output unchanged, so the penalty does not shrink the function but instead controls the effective learning rate by keeping weight norms from drifting (van Laarhoven, 2017; Zhang et al., 2019). Second, the sparsity story of $L^1$ generalizes into structured pruning and the lottery-ticket hypothesis (Frankle & Carbin, 2019), which finds sparse subnetworks that train to full accuracy from the original initialization — the modern descendant of a zeroed weight is a feature switched off. The elastic net's grouping intuition survives here too: correlated features are kept or pruned together rather than one being chosen arbitrarily.

The rest of the chapter

Parameter norm penalties are only the first of many regularizers; the rest of the chapter is a tour of methods that constrain the model without an explicit weight penalty. Each gets its own lesson.

Several of these have an equivalence to weight decay. Early stopping limits how far the weights can travel from their initialization, which under the same quadratic model is again a shrinkage toward the origin — the constraint-ball picture from this lesson in another form. Every effective regularizer can be read as either a constraint or a prior, and that reading recurs throughout the chapter.