---
title: Regularization Overview
module: Regularization
moduleNumber: 4
lessonNumber: 1
order: 401
summary: >
  Regularization is any modification to a learning algorithm meant to lower test
  error at the possible expense of training error. We derive the bias–variance
  decomposition that explains why it helps, set up the two parameter-norm penalties,
  $L^2$ weight decay and $L^1$, derive their update rules and eigenbasis shrinkage,
  show geometrically why $L^1$ alone produces sparse weights (soft-thresholding),
  distinguish weight decay from loss-added $L^2$ under AdamW, and read both penalties
  through the two lenses that recur across the chapter: a norm-ball constraint via
  KKT, and a prior via MAP estimation.
topics: [Regularization]
sources:
  - book: Goodfellow
    ref: "Ch. 7 — Regularization for Deep Learning; §7.1 Parameter Norm Penalties"
  - book: Goodfellow
    ref: "§7.1.1 $L^2$ Regularization; §7.1.2 $L^1$ Regularization"
  - book: Chollet
    ref: "§4.4 — Adding Weight Regularization"
---

A model with enough capacity will fit its training set perfectly, noise and all,
and then generalize badly. **Regularization** is the set of techniques that make
a flexible model generalize, and the most effective approach is not making the
model smaller but leaving it large and _penalizing_ what it does
with that size.

> **Definition (Regularization).** Any modification made to a learning algorithm
> that is intended to reduce its generalization (test) error but not necessarily
> its training error. The training loss is allowed to rise; what must fall is the
> gap between training and test performance.

This lesson treats the oldest and most transparent family, the **parameter norm
penalties**: add a term that penalizes large weights. Every such method adds a
penalty $\Omega(w)$, scaled by a strength $\lambda \ge 0$, to the data-fit loss
$L$, and minimizes the sum

$$
\tilde{L}(w) \;=\; L(w) \;+\; \lambda\,\Omega(w).
$$

The hyperparameter $\lambda$ trades the two off: $\lambda = 0$ recovers ordinary
fitting, large $\lambda$ forces the weights toward small values of $\Omega$. The bias term is conventionally left unpenalized (it shifts the whole
function, and regularizing it tends to underfit), so throughout, $w$ denotes the
weights only.[^gf-norm]

## Why penalize weights at all: the bias–variance decomposition

Before the penalties, the reason they help. Fix an input $x$ and suppose the label
is generated as $y = f(x) + \varepsilon$ with zero-mean noise
$\mathbb{E}[\varepsilon] = 0$ and variance $\Var(\varepsilon) =
\sigma^2$. We fit an estimator $\hat{f}$ on a random training set $\mathcal{D}$, so
$\hat{f}(x)$ is itself a random quantity — draw a different training set and you
get a different fit. The quantity we care about is the expected squared test error
at $x$, averaged over both the label noise and the draw of $\mathcal{D}$:

$$
\Err(x) \;=\; \mathbb{E}\!\left[\bigl(y - \hat{f}(x)\bigr)^2\right].
$$

Write $\bar{f}(x) = \mathbb{E}_{\mathcal{D}}[\hat{f}(x)]$ for the average fit over
training sets. Insert and subtract $\bar f(x)$ inside the square, then expand. The
noise $\varepsilon$ is independent of $\hat f$, so every cross term with
$\mathbb{E}[\varepsilon] = 0$ drops:

$$
\begin{aligned}
\mathbb{E}\!\left[(y - \hat f)^2\right]
&= \mathbb{E}\!\left[(f + \varepsilon - \hat f)^2\right] \\
&= \mathbb{E}\!\left[(f - \hat f)^2\right] + \underbrace{2\,\mathbb{E}[\varepsilon]\,\mathbb{E}[f-\hat f]}_{=\,0} + \mathbb{E}[\varepsilon^2] \\
&= \mathbb{E}\!\left[(f - \bar f + \bar f - \hat f)^2\right] + \sigma^2 \\
&= \underbrace{(f - \bar f)^2}_{\text{bias}^2}
 + \underbrace{\mathbb{E}\!\left[(\hat f - \bar f)^2\right]}_{\text{variance}}
 + \underbrace{\sigma^2}_{\text{noise}}.
\end{aligned}
$$

The middle cross term $2(f - \bar f)\,\mathbb{E}[\bar f - \hat f]$ vanishes because
$\mathbb{E}_{\mathcal{D}}[\hat f] = \bar f$. Three named pieces remain.

> **Theorem (Bias–variance decomposition).** The expected squared test error of any
> estimator splits into three additive, non-negative parts,
> $$
> \Err(x) = \underbrace{\bigl(f(x) - \bar f(x)\bigr)^2}_{\text{bias}^2}
> + \underbrace{\mathbb{E}\!\left[\bigl(\hat f(x) - \bar f(x)\bigr)^2\right]}_{\text{variance}}
> + \sigma^2 .
> $$
> **Bias** measures how far the average fit sits from the truth; **variance**
> measures how much the fit jitters across training sets; **noise** $\sigma^2$ is
> irreducible.

A large, unconstrained model can drive bias to nearly zero but at the cost of large
variance: it fits the noise in each particular $\mathcal{D}$, so $\hat f$ varies
greatly from one training set to the next. Regularization deliberately introduces a
_little_ bias — it prevents the fit from matching the data exactly — in exchange
for a _large_ reduction in variance. Because the two enter the error as a sum, the
minimum test error sits at the balance point, not at zero bias. That balance point
is what the penalty tunes.[^gf-bv]

## $L^2$ regularization (weight decay)

The canonical choice penalizes the squared Euclidean norm.

> **Definition ($L^2$ regularization).** The penalty
> $\Omega(w) = \tfrac{1}{2}\norm{w}_2^2 = \tfrac{1}{2}\sum_j w_j^2$, giving the
> regularized objective $\tilde{L}(w) = L(w) + \tfrac{\lambda}{2}\norm{w}_2^2$.
> In deep learning it is universally called **weight decay**.[^gf-l2]

The gradient of the penalty is just $w$, so the regularized gradient adds a term
pointing back toward the origin:

$$
\nabla_w \tilde{L} \;=\; \nabla_w L \;+\; \lambda\,w.
$$

### Deriving the multiplicative-shrinkage update

Substitute that gradient into a single gradient-descent step with learning rate
$\eta$ and collect the $w$ terms:

$$
w \;\gets\; w - \eta\,\nabla_w \tilde{L}
\;=\; w - \eta\parens{\nabla_w L + \lambda w}
\;=\; \underbrace{(1 - \eta\lambda)}_{\text{shrink}}\,w \;-\; \eta\,\nabla_w L.
$$

The name is now visible: _before_ taking the usual data-driven step, every weight
is multiplied by the factor $(1 - \eta\lambda) < 1$. Each step decays the weights
toward zero by a constant fraction; this is the shrinkage. With $\eta = 0.1$ and
$\lambda = 0.5$, the factor is $0.95$, so a weight left untouched by the data
gradient halves roughly every fourteen steps ($0.95^{14} \approx 0.49$).

$$
% caption: Each step scales $w$ by $(1-\eta\lambda)$ toward the origin, then
% subtracts the data gradient $\eta\nabla_w L$.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, black] (-0.4,0) -- (5.6,0) node[right, font=\footnotesize, black] {$w_1$};
  \draw[->, black] (0,-0.4) -- (0,3.6) node[above, font=\footnotesize, black] {$w_2$};
  \fill[black] (0,0) circle (1.6pt);
  \node[font=\footnotesize, anchor=north east] at (-0.05,-0.05) {\texttt{origin}};
  % current weight vector
  \draw[->, acc, very thick] (0,0) -- (4.6,2.9);
  \node[anchor=south west, font=\footnotesize, text=acc] at (4.5,2.9) {\texttt{w}};
  % shrunk weight (0.7 factor)
  \draw[->, green, very thick] (0,0) -- (3.22,2.03);
  \node[green, font=\footnotesize, anchor=north west] at (3.0,1.75) {\texttt{shrunk}};
  % the decay arrow pulling w toward shrunk
  \draw[->, red, thick, dashed] (4.6,2.9) -- (3.32,2.10);
  \node[red, font=\footnotesize, anchor=south] at (4.4,3.12) {\texttt{decay}};
\end{tikzpicture}
$$

### Closed-form effect on a quadratic objective

To see _which_ weights decay most, approximate $L$ near its unregularized
minimizer $w^\star$ by a second-order Taylor expansion. With $H$ the (symmetric,
positive semidefinite) Hessian at $w^\star$ and the gradient vanishing there,

$$
L(w) \;\approx\; L(w^\star) + \tfrac{1}{2}(w - w^\star)^{\!\top} H\,(w - w^\star),
\qquad \nabla_w L \approx H\,(w - w^\star).
$$

The regularized minimizer $\hat{w}$ sets the full gradient to zero,
$H(\hat{w} - w^\star) + \lambda \hat{w} = 0$, which solves to

$$
\hat{w} \;=\; (H + \lambda I)^{-1} H\,w^\star.
$$

Diagonalize $H = Q\,\Lambda\,Q^{\top}$ in its orthonormal eigenbasis, with
eigenvalues $\lambda_i \ge 0$ measuring curvature along each eigenvector. In that
basis the solution decouples coordinate by coordinate:

$$
Q^{\top}\hat{w}
\;=\; (\Lambda + \lambda I)^{-1}\Lambda\;Q^{\top}w^\star,
\qquad\text{so}\qquad
(Q^{\top}\hat{w})_i \;=\; \frac{\lambda_i}{\lambda_i + \lambda}\,(Q^{\top}w^\star)_i.
$$

Each component of the optimum is rescaled by $\lambda_i/(\lambda_i + \lambda) \in
(0,1]$. Read the two extremes:

| Direction | Curvature $\lambda_i$ | Shrink factor $\tfrac{\lambda_i}{\lambda_i+\lambda}$ | Effect |
| --- | --- | --- | --- |
| high-curvature | $\lambda_i \gg \lambda$ | $\approx 1$ | barely moved — strongly constrained by the loss |
| low-curvature | $\lambda_i \ll \lambda$ | $\approx 0$ | collapsed toward zero |

> **Theorem ($L^2$ shrinks along low-curvature directions).** Under the quadratic
> approximation, $L^2$ regularization rescales the $i$-th eigencomponent of $w^\star$
> by $\lambda_i/(\lambda_i + \lambda)$. Components aligned with large-eigenvalue
> (high-curvature) directions of $H$ are nearly preserved; components aligned with
> small-eigenvalue (flat) directions are driven toward zero.

> **Proof.** With $H = Q\Lambda Q^\top$ and $Q$ orthogonal, $(H+\lambda I)^{-1}H =
> Q(\Lambda+\lambda I)^{-1}\Lambda Q^\top$, since both factors are diagonal in the
> shared eigenbasis. Projecting $\hat w = (H+\lambda I)^{-1}H\,w^\star$ onto
> eigenvector $q_i$ gives $(Q^\top\hat w)_i = \tfrac{\lambda_i}{\lambda_i+\lambda}
> (Q^\top w^\star)_i$. The map $\lambda_i \mapsto \lambda_i/(\lambda_i+\lambda)$ is
> increasing in $\lambda_i$, $\to 1$ as $\lambda_i\to\infty$ and $\to 0$ as
> $\lambda_i\to 0$, which is the claim. $\qed$

The interpretation matches the bias–variance decomposition: directions the data
constrains weakly (small $\lambda_i$) are where overfitting occurs, and weight decay
zeroes them out while leaving the well-determined directions
alone. A high-curvature direction has a steep, narrow valley in the loss — moving
off $w^\star_i$ there raises the loss sharply, so the penalty moves it very little.
A low-curvature direction is a flat trough — the loss barely changes with the
weight, so the penalty pulls it freely to zero.

$$
% caption: The shrink factor $\lambda_i/(\lambda_i+\lambda)$ rises from $0$ toward
% $1$ as curvature $\lambda_i$ grows past the penalty $\lambda$.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (0,0) -- (6.4,0) node[right, font=\footnotesize, black] {curvature};
  \draw[->, black] (0,0) -- (0,3.4) node[above, font=\footnotesize, black] {shrink};
  % shrink factor curve: x/(x+lam), lam=1.2, scaled so factor 1 -> y=2.8
  \draw[acc, very thick] plot[domain=0:6, samples=80] (\x, {2.8*\x/(\x+1.2)});
  % asymptote at factor = 1
  \draw[black, dashed] (0,2.8) -- (6.2,2.8);
  \node[font=\footnotesize, anchor=east, black] at (-0.06,2.8) {1};
  \node[font=\footnotesize, anchor=east, black] at (-0.06,0) {0};
  % penalty marker at x = lam
  \draw[red, dashed] (1.2,0) -- (1.2,1.4);
  \fill[red] (1.2,1.4) circle (2pt);
  \node[red, font=\footnotesize, anchor=north] at (1.2,-0.08) {\texttt{lambda}};
  \node[red, font=\footnotesize, anchor=west] at (1.35,1.25) {\texttt{half kept}};
  % annotations
  \node[green, font=\footnotesize, anchor=north] at (4.4,1.75) {\texttt{stiff: kept}};
  \node[acc, font=\footnotesize, anchor=west] at (0.35,0.55) {\texttt{flat: collapsed}};
\end{tikzpicture}
$$

At $\lambda_i = \lambda$ the factor is exactly $\tfrac{1}{2}$: the penalty and the
curvature pull equally, and half the weight is kept. That crossover point is the
scale $\lambda$ sets — everything much stiffer than it is kept, everything much
flatter is discarded.

**Why $L^2$ prefers small, spread-out weights.** The squared norm charges the
_square_ of each weight, so it is much cheaper to represent a needed quantity as
many small weights than as one large weight. To split a target of $10$ across two
weights, a single $(10, 0)$ costs $100$ while an even $(5, 5)$ costs $50$; the
even split is preferred whenever the loss is indifferent to how the target is
divided. The marginal penalty $\partial_j \tfrac12\norm{w}_2^2 = w_j$ grows with
the weight, so pushing a large weight down saves more than pushing an already-small
weight — the objective equalizes magnitudes and never quite reaches zero.

### Weight decay vs $L^2$, and where they part: AdamW

Under plain gradient descent, adding $\tfrac{\lambda}{2}\norm{w}_2^2$ to the loss
and applying the multiplicative decay $w \gets (1-\eta\lambda)w$ are the same
operation, which is why the two names are used interchangeably. They stop agreeing
the moment the optimizer rescales the gradient. Adam divides each gradient
component by a running estimate of its own magnitude, $\hat v_j$. If the $L^2$
term is folded into the loss, its contribution $\lambda w_j$ is scaled by the same
$1/\sqrt{\hat v_j}$, so weights on noisy, large-gradient directions get _less_
decay than intended and the effective penalty depends on the gradient history.
**AdamW** fixes this by decoupling: it applies the shrink $w \gets (1-\eta\lambda)w$
directly to the weights, outside the adaptive rescaling, restoring a uniform
$(1-\eta\lambda)$ pull on every coordinate. For adaptive optimizers, decoupled
weight decay (AdamW) and loss-added $L^2$ are genuinely different, and the decoupled
form is the one that behaves like the analysis above.[^gf-l2]

## $L^1$ regularization

Swap the squared norm for the absolute-value norm and the qualitative behavior
changes completely.

> **Definition ($L^1$ regularization).** The penalty
> $\Omega(w) = \norm{w}_1 = \sum_j \abs{w_j}$, giving
> $\tilde{L}(w) = L(w) + \lambda\norm{w}_1$.

### Subgradient and the soft-threshold

The absolute value is not differentiable at zero, so the gradient becomes a
**subgradient**: the slope is $+1$ for positive weights, $-1$ for negative, and any
value in $[-1, 1]$ at the kink.

$$
\partial_{w_j}\norm{w}_1 \;=\;
\begin{cases}
\sign(w_j) & w_j \ne 0,\\[2pt]
[-1, 1] & w_j = 0.
\end{cases}
$$

The penalty's gradient no longer scales with $w_j$; it is a _constant_ push of
size $\lambda$ toward zero, independent of the weight's magnitude. For a quadratic
fit term the stationarity condition $\partial_{w_j}L + \lambda\,\sign(w_j) = 0$
can be _satisfied at exactly_ $w_j = 0$ whenever the data gradient is smaller than
$\lambda$: the subgradient interval $[-\lambda, \lambda]$ contains it. That is the
mechanism of sparsity, expressed by the **soft-thresholding** solution (decoupled,
unit-curvature case):

$$
\hat{w}_j \;=\; \sign(w^\star_j)\,\max\!\parens{\abs{w^\star_j} - \lambda,\; 0}.
$$

Any coordinate whose unregularized value sits within $\lambda$ of zero is set
_exactly_ to zero, not merely small. Plotted as a map from the unregularized value
$w^\star_j$ to the fitted value $\hat w_j$, this is a flat dead-zone of width
$2\lambda$ centered at the origin, flanked by two lines of slope $1$ shifted inward
by $\lambda$. Contrast the $L^2$ map $\hat w_j = w^\star_j/(1+\lambda)$: a single
straight line through the origin, never flat, never crossing zero except at zero
itself.

$$
% caption: Soft-threshold (L1) zeroes a dead-band $[-\lambda,\lambda]$ then shifts
% inward by $\lambda$; L2 only rescales the slope. The dashed line is the identity.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (-3.2,0) -- (3.4,0);
  \node[right, font=\footnotesize, black] at (3.4,0.02) {\texttt{unreg. w}};
  \draw[->, black] (0,-2.6) -- (0,2.8);
  \node[above, font=\footnotesize, black] at (0,2.8) {\texttt{fitted w}};
  % identity reference
  \draw[black, dashed] (-2.4,-2.4) -- (2.4,2.4);
  \node[black, font=\footnotesize, anchor=south east] at (2.35,2.35) {\texttt{identity}};
  % lambda ticks
  \draw[red, dashed] (1.0,-0.12) -- (1.0,0.12);
  \draw[red, dashed] (-1.0,-0.12) -- (-1.0,0.12);
  \node[red, font=\footnotesize, anchor=north] at (1.0,-0.14) {\texttt{lambda}};
  % L1 soft-threshold: 0 on [-1,1], then slope 1 shifted in
  \draw[green, very thick] (-2.9,-1.9) -- (-1.0,0) -- (1.0,0) -- (2.9,1.9);
  \node[green, font=\footnotesize, anchor=south east] at (2.7,1.45) {\texttt{L1}};
  % L2 rescale: line of slope 1/(1+lam), lam=0.7 -> slope ~0.59
  \draw[acc, very thick] (-2.9,-1.7) -- (2.9,1.7);
  \node[acc, font=\footnotesize, anchor=north west] at (2.55,1.15) {\texttt{L2}};
\end{tikzpicture}
$$

### Why the corner induces sparsity

The geometry makes this inevitable. The $L^1$ ball $\{w : \norm{w}_1 \le t\}$
is a diamond whose vertices lie _on the axes_; the $L^2$ ball is a round circle.
The constrained optimum is where the expanding loss contours first touch the ball,
and a smooth elliptical contour overwhelmingly first meets the diamond at a
**corner**, a point where one coordinate is zero.

$$
% caption: Loss contours meet the $L^1$ diamond at an axis corner ($w_1=0$, sparse)
% but the $L^2$ circle at a generic point (both coordinates nonzero).
\begin{tikzpicture}[>=stealth, font=\small, scale=0.95]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % ===== LEFT: L1 diamond =====
  \begin{scope}
    \draw[->, black] (-2.2,0) -- (2.4,0) node[right, font=\footnotesize, black] {$w_1$};
    \draw[->, black] (0,-2.2) -- (0,2.4) node[above, font=\footnotesize, black] {$w_2$};
    % L1 ball (diamond)
    \draw[acc, very thick] (1.3,0) -- (0,1.3) -- (-1.3,0) -- (0,-1.3) -- cycle;
    \node[acc, font=\footnotesize, anchor=east] at (-1.5,-1.05) {\texttt{L1 ball}};
    % loss contours (ellipses) centered up-right, touching the top vertex
    \foreach \r in {0.55,1.0,1.45} \draw[black] (0.95,1.95) ellipse ({\r*1.35} and \r);
    \fill[black] (0.95,1.95) circle (1.4pt);
    \node[font=\footnotesize, anchor=south] at (0.95,3.5) {\texttt{loss min}};
    % optimum at the top corner -> sparse (w1 = 0)
    \fill[green] (0,1.3) circle (2.6pt);
    \node[green, font=\footnotesize, anchor=east] at (-0.15,1.35) {\texttt{corner}};
    \node[green, font=\footnotesize, anchor=west] at (0.55,1.55) {$w_1 = 0$};
  \end{scope}
  % ===== RIGHT: L2 circle =====
  \begin{scope}[xshift=7.2cm]
    \draw[->, black] (-2.2,0) -- (2.4,0) node[right, font=\footnotesize, black] {$w_1$};
    \draw[->, black] (0,-2.2) -- (0,2.4) node[above, font=\footnotesize, black] {$w_2$};
    \draw[acc, very thick] (0,0) circle (1.3);
    \node[acc, font=\footnotesize, anchor=east] at (-1.5,-1.05) {\texttt{L2 ball}};
    \foreach \r in {0.55,1.0,1.45} \draw[black] (1.5,1.5) ellipse ({\r*1.35} and \r);
    \fill[black] (1.5,1.5) circle (1.4pt);
    \node[font=\footnotesize, anchor=south] at (1.5,3.05) {\texttt{loss min}};
    % optimum on the round arc -> both nonzero
    \fill[red] (0.92,0.92) circle (2.6pt);
    \node[red, font=\footnotesize, anchor=west] at (1.05,0.72) {\texttt{both nonzero}};
  \end{scope}
\end{tikzpicture}
$$

> **Theorem ($L^1$ induces sparsity).** Minimizing $L(w) + \lambda\norm{w}_1$
> drives a subset of the weights to exactly zero. A coordinate $w_j$ vanishes
> whenever $\abs{\partial_{w_j}L}_{w_j=0} \le \lambda$, because the
> subgradient of $\lambda\norm{w}_1$ at zero spans $[-\lambda, \lambda]$ and
> can cancel any data gradient inside that band. $L^2$ has no such band — its
> gradient $\lambda w_j$ shrinks to nothing as $w_j \to 0$, so it never forces an
> exact zero.

The contrast is the chapter's first idea: $L^2$ makes weights
_small_, $L^1$ makes them _absent_. Sparsity is why $L^1$ doubles as a feature
selector: a zeroed weight is an input the model has switched off entirely.[^gf-l1]

## Elastic net

Combining the two penalties keeps $L^1$'s sparsity while inheriting $L^2$'s
stability when features are correlated (where pure $L^1$ picks one arbitrarily).

> **Definition (Elastic net).** The mixed penalty
> $\Omega(w) = \alpha\norm{w}_1 + \tfrac{1-\alpha}{2}\norm{w}_2^2$ with
> mixing parameter $\alpha \in [0, 1]$. It interpolates between pure $L^2$
> ($\alpha = 0$) and pure $L^1$ ($\alpha = 1$).

## The master table

The two penalties, side by side, with the prior each one corresponds to (derived
in the MAP view below):

| Penalty | $\Omega(w)$ | Gradient / subgradient | Effect on weights | Equivalent prior |
| --- | --- | --- | --- | --- |
| $L^2$ (weight decay) | $\tfrac{1}{2}\norm{w}_2^2$ | $w$ | multiplicative shrinkage, all small | Gaussian $\mathcal{N}(0, \tau^2 I)$ |
| $L^1$ | $\norm{w}_1$ | $\sign(w)$, $[-1,1]$ at $0$ | soft-threshold, many exactly $0$ (sparse) | Laplace $\propto e^{-\abs{w}/b}$ |
| Elastic net | $\alpha\norm{w}_1 + \tfrac{1-\alpha}{2}\norm{w}_2^2$ | $\alpha\sign(w) + (1-\alpha)w$ | sparse _and_ grouped | Gaussian–Laplace mixture |

## Two views of the same penalty

### View 1 — constrained optimization (KKT)

Penalizing a norm is the Lagrangian of _constraining_ it.[^gf-kkt] Minimizing
$L(w) + \lambda\Omega(w)$ for a fixed $\lambda > 0$ produces the same solution as
the hard-constrained problem

$$
\min_w\; L(w) \quad\text{subject to}\quad \Omega(w) \le t,
$$

for some radius $t = t(\lambda)$. The link is the **KKT** conditions: the
Lagrangian is $\mathcal{L}(w, \mu) = L(w) + \mu\,(\Omega(w) - t)$, and stationarity
$\nabla L + \mu\nabla\Omega = 0$ reproduces the regularized gradient, with the
multiplier $\mu$ playing the role of $\lambda$. Larger $\lambda$ pulls the optimum
inward, which is a smaller ball radius $t$; the two knobs are inverses.

$$
% caption: Constraint $\leftrightarrow$ penalty equivalence: the optimum sits where a
% loss contour is tangent to the ball, so $\nabla L$ and $\nabla\Omega$ are antiparallel.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \draw[->, black] (-2.4,0) -- (2.6,0) node[right, font=\footnotesize, black] {$w_1$};
  \draw[->, black] (0,-2.4) -- (0,2.6) node[above left, font=\footnotesize, black] {$w_2$};
  % norm ball (circle, radius t)
  \draw[acc, very thick] (0,0) circle (1.5);
  \node[acc, font=\footnotesize, anchor=north] at (-1.65,-2.55) {\texttt{ball radius t}};
  % loss contours centered up-right
  \foreach \r in {0.5,0.95,1.4,1.85} \draw[black] (2.0,1.7) ellipse ({\r*1.3} and \r);
  \fill[black] (2.0,1.7) circle (1.5pt);
  \node[font=\footnotesize, anchor=south] at (2.0,3.65) {\texttt{loss min}};
  % tangent point on ball (toward loss min): direction (2,1.7) normalized * 1.5
  \coordinate (tp) at (1.144,0.972);
  \fill[green] (tp) circle (2.6pt);
  % outward normal of the ball at tp = gradient of penalty
  \draw[->, acc, thick] (tp) -- ++(0.72,0.61);
  \node[acc, font=\footnotesize, anchor=west] at (2.05,1.42) {\texttt{grad penalty}};
  % loss gradient at tp points the opposite way (toward decreasing loss is inward)
  \draw[->, green, thick] (tp) -- ++(-0.6,-0.51);
  \node[green, font=\footnotesize, anchor=north] at (1.7,-2.55) {\texttt{neg grad loss}};
\end{tikzpicture}
$$

> **Theorem (Penalty–constraint equivalence).** For convex $L$ and $\Omega$ and any
> $\lambda > 0$, the minimizer of $L(w) + \lambda\Omega(w)$ also minimizes $L(w)$
> subject to $\Omega(w) \le t$ for $t = \Omega(\hat{w})$. At the optimum the loss
> contour is tangent to the constraint surface $\{\Omega(w) = t\}$.

> **Proof.** Let $\hat w$ minimize $L + \lambda\Omega$; then $\nabla L(\hat w) +
> \lambda\nabla\Omega(\hat w) = 0$. Set $t = \Omega(\hat w)$. The KKT conditions for
> the constrained problem — stationarity $\nabla L + \mu\nabla\Omega = 0$, primal
> feasibility $\Omega \le t$, dual feasibility $\mu \ge 0$, complementary slackness
> $\mu(\Omega - t) = 0$ — are all met at $\hat w$ with multiplier $\mu = \lambda$:
> stationarity is the penalized gradient, $\Omega(\hat w) = t$ holds with equality,
> and $\lambda > 0$. By convexity these conditions are sufficient, so $\hat w$ solves
> the constrained problem. Antiparallel gradients state that the contour is tangent
> to the constraint surface. $\qed$

### View 2 — the prior (MAP estimation)

The same penalty is a log-prior. Treat the weights as random with prior $p(w)$ and
maximize the posterior; **maximum a posteriori** estimation gives, after taking
$-\log$,

$$
\hat{w}_{\text{MAP}}
= \arg\max_w\; p(w \mid \mathcal{D})
= \arg\min_w\; \underbrace{-\log p(\mathcal{D}\mid w)}_{\text{loss } L(w)}
\;\underbrace{-\;\log p(w)}_{\text{penalty } \lambda\Omega(w)}.
$$

The penalty _is_ the negative log-prior. Take a zero-mean isotropic Gaussian prior
$p(w) = \prod_j \tfrac{1}{\sqrt{2\pi}\,\tau}\exp\!\bigl(-w_j^2/2\tau^2\bigr)$. Its
negative log is

$$
-\log p(w) = \frac{1}{2\tau^2}\sum_j w_j^2 + \text{const}
= \frac{1}{2\tau^2}\norm{w}_2^2 + \text{const},
$$

which is $L^2$ with $\lambda = 1/\tau^2$: a tighter prior (small $\tau$) is a
stronger penalty. A Laplace prior $p(w) = \prod_j \tfrac{1}{2b}\exp\!\bigl(-\abs{w_j}/b\bigr)$
gives $-\log p(w) = \tfrac{1}{b}\norm{w}_1 + \text{const}$, i.e. $L^1$ with
$\lambda = 1/b$. The two priors differ exactly where the penalties do. The Gaussian
is smooth and rounded at the origin, so it puts no special mass at zero. The
Laplace has a sharp peak _at_ zero (its derivative jumps from
$+1/b$ to $-1/b$ there), placing far more prior mass on tiny weights; that spike is
the probabilistic counterpart of the diamond's corner, and it is why the $L^1$
posterior mode lands on exact zeros. This recap extends the MAP framing introduced under
[the machine-learning refresher](/deep-learning/foundations/machine-learning-refresher).[^gf-map]

$$
% caption: Gaussian prior (L2) is rounded at $0$; Laplace prior (L1) has a sharp
% spike at $0$, placing more mass on near-zero weights.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \draw[->, black] (-3.2,0) -- (3.4,0) node[right, font=\footnotesize, black] {$w$};
  \draw[->, black] (0,-0.2) -- (0,2.9) node[above, font=\footnotesize, black] {$p(w)$};
  % Gaussian: rounded top
  \draw[acc, very thick] plot[domain=-3:3, samples=90] (\x, {2.3*exp(-0.7*\x*\x)});
  \node[acc, font=\footnotesize, anchor=west] at (1.05,1.15) {\texttt{Gaussian (L2)}};
  % Laplace: sharp spike (higher, decays as exp(-|x|))
  \draw[green, very thick] plot[domain=-3:0, samples=60] (\x, {2.6*exp(1.15*\x)});
  \draw[green, very thick] plot[domain=0:3, samples=60] (\x, {2.6*exp(-1.15*\x)});
  \node[green, font=\footnotesize, anchor=west] at (0.55,2.35) {\texttt{Laplace (L1)}};
\end{tikzpicture}
$$

| Prior $p(w)$ | $-\log p(w)$ | Penalty | $\lambda$ |
| --- | --- | --- | --- |
| Gaussian $\mathcal{N}(0,\tau^2 I)$ | $\tfrac{1}{2\tau^2}\norm{w}_2^2$ | $L^2$ | $1/\tau^2$ |
| Laplace, scale $b$ | $\tfrac{1}{b}\norm{w}_1$ | $L^1$ | $1/b$ |

## What each penalty does to the weight distribution

The two priors produce distinct histograms of fitted weights:
unregularized weights spread with heavy tails; $L^2$ concentrates them in a
Gaussian-like hump near zero; $L^1$ produces a tall spike _at_ zero plus a few
surviving nonzeros.

$$
% caption: Histograms of fitted weights: unregularized is wide, $L^2$ a narrow hump
% near $0$, $L^1$ a sparse spike at $0$.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % ---- panel 1: unregularized ----
  \begin{scope}
    \draw[->, black] (-1.7,0) -- (1.9,0) node[right, black] {$w$};
    \draw[->, black] (0,0) -- (0,2.4);
    \foreach \x/\h in {-1.5/0.5,-1.1/0.7,-0.7/1.0,-0.3/1.25,0.1/1.3,0.5/1.05,0.9/0.75,1.3/0.5}
      \draw[red, thick, fill=red!15] (\x,0) rectangle ++(0.34,\h);
    \node[anchor=north] at (0,-0.25) {\texttt{unregularized}};
    \node[red, anchor=south west] at (0.2,2.05) {\texttt{heavy tails}};
  \end{scope}
  % ---- panel 2: L2 ----
  \begin{scope}[xshift=5.0cm]
    \draw[->, black] (-1.7,0) -- (1.9,0) node[right, black] {$w$};
    \draw[->, black] (0,0) -- (0,2.4);
    \foreach \x/\h in {-1.5/0.1,-1.1/0.25,-0.7/0.6,-0.3/1.5,0.1/1.65,0.5/0.95,0.9/0.35,1.3/0.12}
      \draw[acc, thick, fill=acc!15] (\x,0) rectangle ++(0.34,\h);
    \node[anchor=north] at (0,-0.25) {\texttt{L2 weight decay}};
    \node[acc, anchor=south west] at (0.2,2.05) {\texttt{near zero}};
  \end{scope}
  % ---- panel 3: L1 ----
  \begin{scope}[xshift=10.0cm]
    \draw[->, black] (-1.7,0) -- (1.9,0) node[right, black] {$w$};
    \draw[->, black] (0,0) -- (0,2.4);
    % spike at zero plus a few survivors
    \draw[green, very thick, fill=green!22] (-0.17,0) rectangle ++(0.34,2.2);
    \foreach \x/\h in {-1.5/0.18,-0.7/0.45,0.5/0.5,1.3/0.22}
      \draw[green, thick, fill=green!15] (\x,0) rectangle ++(0.34,\h);
    \node[anchor=north] at (0,-0.25) {\texttt{L1}};
    \node[green, anchor=south west] at (0.2,1.9) {\texttt{spike sparse}};
  \end{scope}
\end{tikzpicture}
$$

## Regularization strength and the bias–variance tradeoff

Sweeping $\lambda$ from $0$ upward walks the model down the capacity axis: small
$\lambda$ leaves the full flexible model (low bias, high variance, prone to
overfitting); large $\lambda$ pins the weights near zero (high bias, low
variance, underfitting). Test error traces the familiar U-curve, indexed
by regularization strength rather than raw model size. This is the bias–variance
decomposition made operational: the falling red curve is the variance term shrinking
as the penalty tightens, the rising blue curve is the bias$^2$ term growing, and
their sum is the U whose floor is the best $\lambda$.

$$
% caption: Test error versus regularization strength $\lambda$: small over-fits,
% large under-fits, and the U-curve's minimum balances them.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, thick] (0,0) -- (7.4,0);
  \node[font=\footnotesize, anchor=south east] at (7.4,0.62) {\texttt{reg. strength}};
  \draw[->, thick] (0,0) -- (0,4.0) node[above, font=\footnotesize] {\texttt{error}};
  % variance term: high at small lambda, falls
  \draw[red, thick] plot[domain=0.2:6, samples=60] (\x, {3.2*exp(-0.55*\x)+0.25});
  \node[red, font=\footnotesize, anchor=south west] at (0.2,1.25) {\texttt{variance}};
  % bias term: low at small lambda, rises
  \draw[acc, thick] plot[domain=0.2:6, samples=60] (\x, {0.18*exp(0.42*\x)+0.2});
  \node[acc, font=\footnotesize, anchor=south] at (5.4,2.7) {$\text{bias}^2$};
  % test error = sum (U-shape)
  \draw[green, very thick] plot[domain=0.2:6, samples=80] (\x, {3.2*exp(-0.55*\x)+0.18*exp(0.42*\x)+0.45});
  \node[green, font=\footnotesize, anchor=south west] at (3.4,2.5) {\texttt{test error}};
  % mark the minimum (around lambda where derivative ~ 0, ~3.0)
  \fill[green] (3.0,1.27) circle (2.4pt);
  \draw[black, dashed] (3.0,0) -- (3.0,1.27);
  \node[font=\footnotesize, anchor=north] at (3.0,-0.08) {\texttt{best}};
  \node[font=\footnotesize, text=red, anchor=north west] at (0.1,-0.1) {\texttt{over-fit}};
  \node[font=\footnotesize, text=acc, anchor=north] at (5.7,-0.1) {\texttt{under-fit}};
\end{tikzpicture}
$$

The minimum is found by holding out data: $\lambda$ is selected on a validation
set, never on the training loss it is designed to ignore.[^chollet-wd]

## Practical defaults, with reasons

- **Use $L^2$ (weight decay) first.** It is the default because its effect —
  a smooth, uniform shrink toward zero — improves almost every over-parameterized
  network and never zeroes a coordinate outright, so it does not risk deleting a
  feature the model still needs. A weight-decay coefficient around $10^{-4}$ to
  $10^{-2}$ is the usual starting band; sweep it on a log scale.
- **Use decoupled weight decay (AdamW) with adaptive optimizers.** With Adam, folding
  $L^2$ into the loss makes the true penalty depend on the gradient history; the
  decoupled shrink restores the uniform pull that the eigenbasis analysis assumes.
- **Use $L^1$ only when you want sparsity.** Feature selection, compressing a
  model by pruning zeroed weights, or making the fit interpretable are the cases
  where exact zeros are useful. When features are correlated, prefer the elastic net so
  the group is kept together instead of one member being chosen arbitrarily.
- **Leave biases unpenalized.** A bias shifts the whole function rather than steepening
  it, so shrinking it toward zero buys no variance reduction and only adds bias.
- **Tune $\lambda$ on validation, not training.** The penalty exists to raise training
  loss; judging it by that loss would always prefer $\lambda = 0$.

## Decoupled weight decay

Goodfellow Ch. 7 treats weight decay and $L^2$ as interchangeable — right for plain
SGD, wrong for the adaptive optimizers that now dominate.
The correction is **decoupled weight decay** (Loshchilov & Hutter, 2019, the AdamW
paper): folding $\tfrac\lambda2\norm{w}_2^2$ into the loss makes Adam rescale the
penalty by each coordinate's gradient history, so the effective decay is no longer
uniform; applying the shrink $w \gets (1-\eta\lambda)w$ directly to the weights
restores the clean $\lambda_i/(\lambda_i+\lambda)$ behavior the eigenbasis analysis
above assumes. AdamW is now the default optimizer for training transformers, and the
distinction between "loss-added $L^2$" and "decoupled weight decay" is one every
practitioner has to get right.

Two further public results sharpen the picture. First, in networks with
[normalization](/deep-learning/regularization/normalization) layers, weight decay
often works through an unexpected channel: scaling a weight before a normalized layer
leaves the output unchanged, so the penalty does not shrink the _function_ but
instead controls the **effective learning rate** by keeping weight norms from
drifting (van Laarhoven, 2017; Zhang et al., 2019). Second, the sparsity story of
$L^1$ generalizes into **structured pruning** and the **lottery-ticket hypothesis**
(Frankle & Carbin, 2019), which finds sparse subnetworks that train to full accuracy
from the original initialization — the modern descendant of "a zeroed weight is a
feature switched off." The elastic net's grouping intuition survives here too:
correlated features are kept or pruned together rather than one being chosen
arbitrarily.

## The rest of the chapter

Parameter norm penalties are only the first of many regularizers; the rest of the
chapter is a tour of methods that constrain the model without an explicit weight
penalty. Each gets its own lesson.

| Method | Mechanism | What it constrains | Lesson |
| --- | --- | --- | --- |
| Dropout | randomly zero units each step | co-adaptation; approximates an ensemble | [dropout & data augmentation](/deep-learning/regularization/dropout-and-data-augmentation) |
| Data augmentation | synthesize label-preserving inputs | invariance to nuisance transforms | [dropout & data augmentation](/deep-learning/regularization/dropout-and-data-augmentation) |
| Early stopping | halt before training loss bottoms out | effective training time $\approx$ an $L^2$ budget | [early stopping & parameter sharing](/deep-learning/regularization/early-stopping-and-parameter-sharing) |
| Parameter sharing | tie weights across positions | parameter count, baked-in symmetry | [early stopping & parameter sharing](/deep-learning/regularization/early-stopping-and-parameter-sharing) |
| Normalization | rescale activations per batch/layer | internal covariate shift; smoother loss | [normalization](/deep-learning/regularization/normalization) |

Several of these have an equivalence to
weight decay. Early stopping limits how far the weights can travel from their
initialization, which under the same quadratic model is _again_ a shrinkage toward
the origin — the constraint-ball picture from this lesson in another form.
Every effective regularizer can be read as either a constraint or a prior, and
that reading recurs throughout the chapter.

[^gf-norm]: **Goodfellow**, _Deep Learning_, §7.1 — Parameter Norm Penalties: the regularized objective $\tilde L = L + \lambda\Omega$ and the convention of leaving biases unpenalized to avoid underfitting.
[^gf-bv]: **Goodfellow**, _Deep Learning_, §5.4.4 — the Bias–Variance Tradeoff: the decomposition of expected generalization error into squared bias, variance, and irreducible noise, and how capacity control moves along the resulting U-curve.
[^gf-l2]: **Goodfellow**, _Deep Learning_, §7.1.1 — $L^2$ Regularization (weight decay): the per-step multiplicative shrinkage $(1-\eta\lambda)w$ and the eigenbasis analysis $\lambda_i/(\lambda_i+\lambda)$ that shrinks low-curvature directions.
[^gf-l1]: **Goodfellow**, _Deep Learning_, §7.1.2 — $L^1$ Regularization: the constant-magnitude subgradient, soft-thresholding solution, and why $L^1$ yields exact zeros (sparsity) where $L^2$ yields only small weights.
[^gf-kkt]: **Goodfellow**, _Deep Learning_, §7.2 — Norm Penalties as Constrained Optimization: the KKT view in which $\lambda\Omega$ is the Lagrangian of the hard constraint $\Omega(w)\le t$.
[^gf-map]: **Goodfellow**, _Deep Learning_, §5.6.1 — MAP Estimation: a weight penalty is a negative log-prior, with the Gaussian prior giving $L^2$ and the Laplace prior giving $L^1$.
[^chollet-wd]: **Chollet**, _Deep Learning with Python_, §4.4 — Adding Weight Regularization: the practitioner's view of weight decay and tuning its strength on a held-out validation set.