---
title: Early Stopping & Parameter Sharing
module: Regularization
moduleNumber: 4
lessonNumber: 3
order: 403
summary: >
  Two cheap regularizers that cost no extra term in the loss. Early stopping
  treats training time itself as a hyperparameter (watch the validation curve,
  halt at its minimum, keep the best checkpoint), and for a quadratic objective
  it is provably equivalent to $L^2$ weight decay. Parameter sharing goes the
  other way: it constrains many weights to be _equal_, the prior behind every
  convolution and every recurrent step, and the reason a CNN has orders of
  magnitude fewer parameters than the dense net it replaces.
topics: [Regularization]
sources:
  - book: Goodfellow
    ref: "§7.8 — Early Stopping"
  - book: Goodfellow
    ref: "§7.9 — Parameter Tying and Parameter Sharing; §7.11 Bagging and Other Ensemble Methods"
  - book: Chollet
    ref: "§4.4 — Overfitting and Underfitting; §7.3.3 — Early Stopping Callbacks"
---

The regularizers of the previous lessons add a term to the loss: an
[$L^2$ penalty](/deep-learning/regularization/regularization-overview), a
[dropout mask](/deep-learning/regularization/dropout-and-data-augmentation).
This lesson covers two that add _no_ term and still constrain the hypothesis
space. **Early stopping** controls capacity through the optimization _trajectory_:
it limits how far the weights travel from their initialization. **Parameter
sharing** controls it through the parameterization _itself_: it forces distinct
weights to take a single shared value. Both shrink the effective number of free
parameters without ever appearing in $\mathcal{L}(\theta)$.

## Early stopping

Train a high-capacity network long enough and the training loss falls
monotonically while the validation loss traces a **U**: down as the model learns
real structure, then back up as it begins to fit noise. The gap between the two
curves is the [generalization gap](/deep-learning/theory/generalization-theory),
and it widens precisely in the regime where training continues to pay off but
validation does not.

$$
% caption: Early stopping. Training loss falls monotonically while validation loss
% bottoms out at $t^\star$ then climbs; the dashed line marks $t^\star$, and the
% shaded region right of it is where training keeps paying off but validation does not.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  % axes
  \draw[->, thick] (0,0) -- (8.6,0) node[right, font=\footnotesize] {\texttt{training step} $t$};
  \draw[->, thick] (0,0) -- (0,4.4) node[above, font=\footnotesize] {\texttt{loss}};
  % overfitting region shaded (to the right of t*)
  \fill[red!10] (4.0,0) rectangle (8.2,4.2);
  \node[red, font=\footnotesize, align=center] at (6.1,3.8) {\texttt{over-}\\\texttt{f\/itting}\\\texttt{region}};
  % training loss: monotone decreasing
  \draw[acc, very thick] plot[domain=0.25:8.2, samples=80] (\x, {0.6 + 3.3*exp(-0.55*\x)});
  \node[acc, font=\footnotesize, anchor=south] at (5.6,0.12) {\texttt{training loss}};
  % validation loss: U-shape, min near x=4
  \draw[red, very thick] plot[domain=0.25:8.2, samples=90] (\x, {1.15 + 2.6*exp(-0.6*\x) + 0.085*(\x-4)*(\x-4)});
  \node[red, font=\footnotesize, anchor=south] at (2.0,3.55) {\texttt{validation loss}};
  % stop point at validation min
  \draw[green, dashed, thick] (4.0,0) -- (4.0,4.0);
  \fill[green] (4.0,{1.15 + 2.6*exp(-0.6*4) + 0}) circle (2.6pt);
  \node[green, font=\footnotesize, anchor=north] at (4.0,-0.1) {\texttt{stop}};
  \node[green, font=\footnotesize, anchor=west, align=center] at (4.2,2.3) {\texttt{keep this}\\\texttt{checkpoint}};
\end{tikzpicture}
$$

The recipe is to treat the number of training steps as a hyperparameter and tune
it on the validation set: evaluate validation loss every epoch, remember the
parameters that achieved the lowest value seen, and return _those_ rather than the
final ones.[^gf-earlystop]

> **Definition (Early stopping).** A regularization strategy that selects the
> training duration $t^{\star}$ minimizing validation loss. The returned model is
> the checkpoint $\theta^{(t^{\star})}$ with the smallest validation error
> observed, not the last iterate. Training time becomes a hyperparameter chosen by
> the validation curve rather than fixed in advance.

Because the validation curve is noisy, we do not halt at the _first_ uptick. The
**patience** parameter $p$ specifies how many consecutive non-improving
evaluations to tolerate before stopping, a guard against quitting on a transient
bump.

> **Definition (Patience).** The number of consecutive validation evaluations
> permitted without improvement before training halts. Patience $p$ trades
> compute against the risk of stopping on noise: small $p$ stops eagerly and may
> quit a temporary plateau; large $p$ wastes steps past the true minimum but
> tolerates fluctuations.

The full procedure keeps a running best checkpoint and a counter of stale
evaluations.

```algorithm
caption: $\textsc{EarlyStopping}(f_\theta, \mathcal{D}_{\text{train}}, \mathcal{D}_{\text{val}}, p)$ — train with a patience-$p$ stopping rule
initialize $\theta$ randomly
$\theta^{\star} \gets \theta$ // best checkpoint so far
$v^{\star} \gets +\infty$ // best validation loss so far
$j \gets 0$ // evaluations since last improvement
repeat
  train one epoch on $\mathcal{D}_{\text{train}}$ // standard gradient steps
  $v \gets \text{loss}(f_\theta, \mathcal{D}_{\text{val}})$ // validate
  if $v < v^{\star}$ then // improved
    $\theta^{\star} \gets \theta$
    $v^{\star} \gets v$
    $j \gets 0$
  else // stale
    $j \gets j + 1$
until $j \ge p$
return $\theta^{\star}$ // the best checkpoint, not the last
```

Two practical points fall out of the algorithm. First, early stopping is nearly
free: the only overhead is one validation pass per epoch and a copy of the best
weights. Second, it composes with every other regularizer (it is a stopping rule,
not a loss term) and is the one regularizer essentially every training run uses.[^chollet-earlystop]

The reason the validation curve turns upward is that **effective capacity grows
with training time**. At initialization the weights are near the origin and the
function the network computes is nearly constant; each gradient step lets the
weights travel farther and the function it can express becomes richer. Training
time therefore behaves like a continuous capacity knob. Early in training the
extra capacity captures real structure and both losses fall; past $t^{\star}$ the extra
capacity goes toward memorizing the training set, and validation loss climbs while
training loss keeps falling. Stopping at $t^{\star}$ caps the effective capacity at
the level that best matches the data.

$$
% caption: Effective capacity rises with training time. Early stopping caps it at
% $t^\star$, where validation loss is minimized; training past that point adds
% capacity that fits noise.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \draw[->, thick] (0,0) -- (8.4,0) node[right, font=\footnotesize] {\texttt{training step} $t$};
  \draw[->, thick] (0,0) -- (0,4.2) node[above, font=\footnotesize] {\texttt{eff. capacity}};
  % capacity: saturating increase with t
  \draw[acc, very thick] plot[domain=0:8.0, samples=90] (\x, {3.7*(1 - exp(-0.42*\x))});
  \node[acc, font=\footnotesize, anchor=south east] at (7.9,{3.7*(1-exp(-0.42*8.0))-0.05}) {\texttt{eff. capacity}};
  % stop marker
  \draw[green, dashed, thick] (4.0,0) -- (4.0,{3.7*(1-exp(-0.42*4.0))});
  \fill[green] (4.0,{3.7*(1-exp(-0.42*4.0))}) circle (2.6pt);
  \node[green, font=\footnotesize, anchor=north] at (4.0,-0.1) {\texttt{stop}};
  \node[green, font=\footnotesize, anchor=west, align=center] at (4.2,1.5) {\texttt{capacity}\\\texttt{capped here}};
\end{tikzpicture}
$$

| Property | Early stopping |
| --- | --- |
| Cost | one validation pass per epoch; one stored checkpoint |
| Extra loss term | none |
| Hyperparameters | patience $p$, evaluation frequency |
| Returns | $\theta^{(t^{\star})}$ at validation minimum |
| Data cost | a held-out validation split (retrain on full data after, optionally) |
| Composes with | $L^2$, dropout, augmentation — all of them |

A subtlety: early stopping spends a validation split that could otherwise be
training data. One remedy is to record $t^{\star}$, then retrain from scratch on
the union $\mathcal{D}_{\text{train}} \cup \mathcal{D}_{\text{val}}$ for $t^{\star}$
steps; another is to keep training the existing model on the validation data until
its loss falls below the level reached at $t^{\star}$. Both recover the held-out
examples at the cost of a second training run.

## Early stopping is $L^2$ regularization

Early stopping looks like a heuristic about wall-clock time, but for a quadratic
objective it is _exactly_ a norm penalty. The mechanism: gradient descent started
at $\theta = 0$ can only move so far in a finite number of steps, so halting early
keeps $\theta$ near the origin: the same constraint
[$L^2$ weight decay](/deep-learning/regularization/regularization-overview)
imposes by penalizing $\norm{\theta}^2$.[^gf-es-l2]

> **Theorem (Early stopping $\approx$ $L^2$).** Let the objective be the quadratic
> approximation $\hat{\mathcal{L}}(\theta) = \mathcal{L}(\theta^{\star}) +
> \tfrac12 (\theta - \theta^{\star})^{\top} H (\theta - \theta^{\star})$ about the
> minimizer $\theta^{\star}$, with $H \succeq 0$. Run gradient descent from
> $\theta^{(0)} = 0$ with learning rate $\eta$ small enough that $\eta \lambda_i <
> 1$ for every eigenvalue $\lambda_i$ of $H$. After $\tau$ steps the iterate
> equals the $L^2$-regularized solution with penalty strength $\alpha \approx
> 1/(\eta\tau)$. Early stopping after $\tau$ steps is $L^2$ regularization with
> $\alpha \approx 1/(\eta\tau)$.

> **Proof.** Diagonalize $H = Q\Lambda Q^{\top}$ with $Q$ orthonormal and
> $\Lambda = \diag(\lambda_i)$, and work in the rotated coordinates
> $u = Q^{\top}\theta$, $u^{\star} = Q^{\top}\theta^{\star}$, where the dynamics
> decouple per axis. Gradient descent on $\hat{\mathcal{L}}$ uses
> $\nabla\hat{\mathcal{L}}(\theta) = H(\theta - \theta^{\star})$, so the update
> $\theta \gets \theta - \eta H(\theta - \theta^{\star})$ becomes, coordinate-wise,
> $u_i^{(t)} - u_i^{\star} = (1 - \eta\lambda_i)\,(u_i^{(t-1)} - u_i^{\star})$.
> Iterating from $u^{(0)} = 0$ collapses the geometric recursion:
>
> $$
> u_i^{(\tau)} = \parens{1 - (1 - \eta\lambda_i)^{\tau}}\,u_i^{\star}.
> $$
>
> Now the $L^2$ side. The penalized objective $\hat{\mathcal{L}}(\theta) +
> \tfrac{\alpha}{2}\norm{\theta}^2$ has minimizer $\tilde\theta = (H +
> \alpha I)^{-1} H\,\theta^{\star}$, which in the same coordinates is
>
> $$
> \tilde u_i = \frac{\lambda_i}{\lambda_i + \alpha}\,u_i^{\star}
> = \parens{1 - \frac{\alpha}{\lambda_i + \alpha}}\,u_i^{\star}.
> $$
>
> The two shrinkage factors match when $(1 - \eta\lambda_i)^{\tau} =
> \alpha/(\lambda_i + \alpha)$. Take logs and use $\log(1 - \eta\lambda_i) \approx
> -\eta\lambda_i$ for small $\eta\lambda_i$; if additionally $\lambda_i \ll
> \alpha^{-1}\!$-scale terms are dropped, the relation reduces to
> $\eta\lambda_i\tau \approx \lambda_i/\alpha$, i.e. $\alpha \approx 1/(\eta\tau)$,
> uniformly in $i$. The number of steps $\tau$ and the inverse penalty $\eta\alpha$
> play interchangeable roles. $\qed$

The correspondence $\alpha \approx 1/(\eta\tau)$ is exact in its consequences for
each eigendirection. Along a **stiff** direction (large $\lambda_i$) the factor
$1 - \eta\lambda_i$ is small, so $u_i$ reaches $u_i^{\star}$ in a few steps: these
weights are learned early and unaffected by the penalty. Along a **flat**
direction (small $\lambda_i$) progress is slow; halting at step $\tau$ leaves
$u_i$ short of $u_i^{\star}$, exactly the directions $L^2$ shrinks most.

The per-axis factor $1 - (1 - \eta\lambda_i)^{\tau}$, read as a function of
$\lambda_i$, shows the selectivity. A stiff axis has $\eta\lambda_i$ close
to $1$, so $(1 - \eta\lambda_i)^{\tau} \to 0$ within a handful of steps and the
factor saturates at $1$: the coordinate is fully learned. A flat axis has
$\eta\lambda_i \approx 0$, so $(1 - \eta\lambda_i)^{\tau} \approx 1$ still after
$\tau$ steps and the factor stays near $0$: the coordinate barely moves off the
origin. The $L^2$ factor $\lambda_i/(\lambda_i + \alpha)$ traces the same S-curve
against $\lambda_i$ — near $1$ for $\lambda_i \gg \alpha$, near $0$ for
$\lambda_i \ll \alpha$ — with the crossover set by $\alpha$ instead of by $\tau$.

$$
% caption: Per-eigendirection shrinkage. Both early stopping (factor
% $1-(1-\eta\lambda)^\tau$) and $L^2$ (factor $\lambda/(\lambda+\alpha)$) fully keep
% stiff directions (large $\lambda$) and suppress flat ones; the crossover is set by
% $\tau$ for stopping and by $\alpha$ for weight decay.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \draw[->, thick] (0,0) -- (8.4,0) node[right, font=\footnotesize] {\texttt{Hessian eigenvalue}};
  \draw[->, thick] (0,0) -- (0,4.2) node[above, font=\footnotesize, align=left] {\texttt{fraction of target}\\\texttt{retained}};
  % gridline at 1.0
  \draw[black, dashed] (0,3.6) -- (8.2,3.6);
  \node[black, font=\scriptsize, anchor=east] at (-0.08,3.6) {$1$};
  % early-stopping factor: 1-(1-eta*lambda)^tau, eta*tau chosen so knee near lambda=2
  \draw[acc, very thick] plot[domain=0:8.0, samples=120] (\x, {3.6*(1 - exp(-0.9*\x))});
  \node[acc, font=\footnotesize, anchor=south east] at (2.6,3.15) {\texttt{early stopping}};
  % L2 factor lambda/(lambda+alpha), alpha chosen for a similar knee
  \draw[green, very thick, dashed] plot[domain=0:8.0, samples=120] (\x, {3.6*\x/(\x + 1.1)});
  \node[green, font=\footnotesize, anchor=north west] at (5.0,1.9) {$L^2$ \texttt{decay}};
  \node[font=\footnotesize, anchor=north] at (1.2,-0.12) {\texttt{flat}};
  \node[font=\footnotesize, anchor=north] at (7.0,-0.12) {\texttt{stiff}};
\end{tikzpicture}
$$

| | Early stopping | $L^2$ weight decay |
| --- | --- | --- |
| Control knob | number of steps $\tau$ | penalty $\alpha$ |
| Effective strength | $\alpha \approx 1/(\eta\tau)$ | $\alpha$ |
| Stiff directions ($\lambda_i$ large) | learned fully | barely shrunk |
| Flat directions ($\lambda_i$ small) | left near origin | shrunk hard |
| Knob set by | validation curve | validation grid search |
| Cost | single training run | one run per $\alpha$ |

Early stopping wins on cost: it discovers the effective penalty during a _single_
training run by reading it off the validation curve, whereas tuning $\alpha$
requires a separate run for each candidate value.

$$
% caption: Geometry of the equivalence: descent halted at $\theta^{(\tau)}$ lands
% short of $\theta^\star$, on the locus an $L^2$ ball (dashed) would enforce.
\begin{tikzpicture}[>=stealth, font=\small, scale=1.08]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % unregularized minimum, off-origin
  \coordinate (O) at (0,0);
  \coordinate (wstar) at (3.2,1.7);
  % elliptical contours of the quadratic objective, centered at wstar (tilted bowl)
  \foreach \r in {0.7,1.4,2.1,2.8}
    \draw[black, rotate around={28:(wstar)}] (wstar) ellipse ({\r*1.5} and \r);
  \fill[red] (wstar) circle (2.6pt);
  \node[red, font=\footnotesize, anchor=south west] at (3.3,1.8) {unreg. min};
  % L2 ball around origin
  \draw[green, dashed, thick] (O) circle (1.85);
  \node[green, font=\footnotesize, anchor=north east] at (-0.05,-1.25) {$L^2$ ball};
  % GD trajectory from origin, curving toward wstar but halted
  \coordinate (p1) at (0.55,0.18);
  \coordinate (p2) at (1.05,0.5);
  \coordinate (p3) at (1.45,0.92);
  \coordinate (p4) at (1.7,1.28);
  \draw[acc, very thick, ->] (O) -- (p1);
  \draw[acc, very thick, ->] (p1) -- (p2);
  \draw[acc, very thick, ->] (p2) -- (p3);
  \draw[acc, very thick, ->] (p3) -- (p4);
  \fill[acc] (O) circle (2.4pt);
  \node[acc, font=\footnotesize, anchor=north east] at (-0.05,0.05) {start $0$};
  \fill[green] (p4) circle (2.8pt);
  \node[green, font=\footnotesize, anchor=south] at (1.7,1.42) {\texttt{early stop}};
\end{tikzpicture}
$$

## Parameter sharing

Early stopping shrinks weights toward a point. **Parameter sharing** instead ties
weights to _each other_: it declares a set of weights to be one and the same
free parameter, so the optimizer adjusts them in lockstep. Where an $L^2$ penalty
expresses a soft preference that two weights be _close_ (**parameter tying**,
$\Omega = \norm{w^{(A)} - w^{(B)}}^2$), sharing is the hard constraint that
they be _identical_.

> **Definition (Parameter sharing).** A regularization strategy that forces
> disjoint subsets of weights to be equal, so they are represented by a single
> stored value. The model has fewer free parameters than connections; the
> constraint encodes a prior that the same computation is useful at many positions.

> **Definition (Parameter tying).** A softer relative: a penalty
> $\Omega(w^{(A)}, w^{(B)}) = \norm{w^{(A)} - w^{(B)}}_2^2$ pulling two
> parameter sets toward each other without forcing equality. Sharing is its hard
> limit, the constraint $w^{(A)} = w^{(B)}$.

The prior behind sharing is **equivariance**: if a feature detector is useful at
one location in the input, it is useful at every location. Two architectures are
built entirely on this idea.[^gf-sharing]

**The gradient of a shared weight is a sum.** Suppose a single free parameter $w$
is used at $m$ locations, appearing in the forward pass as $m$ copies
$w_1 = w_2 = \dots = w_m = w$. Treat the copies as independent for a moment and let
$\partial \mathcal{L} / \partial w_j$ be the local gradient at location $j$. By the
chain rule, the derivative of the loss with respect to the one shared value is the
sum of the per-location contributions,

$$
\frac{\partial \mathcal{L}}{\partial w}
= \sum_{j=1}^{m} \frac{\partial \mathcal{L}}{\partial w_j}
  \frac{\partial w_j}{\partial w}
= \sum_{j=1}^{m} \frac{\partial \mathcal{L}}{\partial w_j},
$$

since $\partial w_j/\partial w = 1$ for every copy. Backpropagation through a shared
weight computes the ordinary local gradient at each location where the weight is
used and **accumulates** them into one number. Every location contributes to the
single shared value's update; a convolution kernel is updated by the summed signal from
every patch it slid over, an RNN weight by the summed signal from every time step.
This accumulation is why a shared parameter, seeing $m$ locations worth of gradient
per example, is more **sample-efficient** than $m$ independent parameters that each
see one.

$$
% caption: Gradient of a shared weight. The forward pass reuses $w$ at $m$
% locations; the backward pass sums the local gradients into one update
% $\partial\mathcal{L}/\partial w = \sum_j \partial\mathcal{L}/\partial w_j$.
\begin{tikzpicture}[>=stealth, font=\small,
  loc/.style={draw, black, minimum width=13mm, minimum height=8mm, font=\scriptsize, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[draw=acc, thick, text=acc, align=center, font=\footnotesize, minimum width=20mm, minimum height=11mm] (w) at (0,0) {\texttt{shared}\\ $w$};
  \node[loc] (l1) at (4.4,1.7) {\texttt{location} 1};
  \node[loc] (l2) at (4.4,0)   {\texttt{location} 2};
  \node[loc] (l3) at (4.4,-1.7) {\texttt{location} $m$};
  % forward: w to each location (upper edge of each link)
  \draw[->, acc, thick] (w.north east) to[bend left=10] (l1.west);
  \draw[->, acc, thick] ([yshift=1.6mm]w.east) -- ([yshift=1.6mm]l2.west);
  \draw[->, acc, thick] (w.south east) to[bend right=10] (l3.west);
  \node[acc, font=\footnotesize, anchor=south] at (2.2,1.35) {\texttt{reuse (forward)}};
  % backward: gradients sum back (lower edge of each link)
  \draw[->, black, thick, dashed] (l1.south west) to[bend right=22] (w.north);
  \draw[->, black, thick, dashed] ([yshift=-1.6mm]l2.west) -- ([yshift=-1.6mm]w.east);
  \draw[->, black, thick, dashed] (l3.north west) to[bend left=14] (w.south);
  \node[black, font=\footnotesize, anchor=north] at (2.0,-1.9) {\texttt{sum grads (backward)}};
\end{tikzpicture}
$$

### Convolutions share across space

A [convolutional layer](/deep-learning/architectures/convolutional-networks)
applies one small filter at every spatial position. The same weights are reused
across the whole image, so a filter that finds a vertical edge finds it
everywhere: translation equivariance, baked into the parameterization.

$$
% caption: A CNN slides one shared filter (blue) over every spatial location, so
% the parameter count is the filter size, independent of image size.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  % input grid 5x5
  \foreach \x in {0,...,4}
    \foreach \y in {0,...,4}
      \draw[black, fill=black!8] (\x*0.55,\y*0.55) rectangle ++(0.55,0.55);
  \node[font=\footnotesize, anchor=north] at (1.375,-0.1) {\texttt{input feature map}};
  % shared filter at position A (top-left)
  \draw[acc, very thick] (0,2.2) rectangle ++(1.1,1.1);
  \node[acc, font=\footnotesize] at (0.55,3.55) {\texttt{f\/ilter}};
  % shared filter at position B (bottom-right) -- SAME weights
  \draw[acc, very thick] (1.65,0.55) rectangle ++(1.1,1.1);
  % arrows from both filter placements to one shared weight store
  \node[draw, align=center, font=\footnotesize, minimum width=18mm, minimum height=11mm] (store) at (5.4,1.9) {\texttt{one shared}\\\texttt{weight set}};
  \draw[->, thick] (1.1,2.75) to[bend left=12] (store.west);
  \draw[->, thick] (2.75,1.1) to[bend right=18] (store.south west);
  \node[font=\footnotesize, anchor=south] at (4.0,2.7) {\texttt{same weights}};
\end{tikzpicture}
$$

### Recurrence shares across time

A [recurrent network](/deep-learning/architectures/recurrent-networks) applies one
cell at every time step. Unrolled, the network looks deep, but every step reuses
the _same_ weight matrices: sharing across time, the sequence analogue of the
CNN's sharing across space.

$$
% caption: An RNN unrolled in time: one cell with weights $W$, $U$ is reused at
% every step, the recurrent analogue of the convolutional filter.
\begin{tikzpicture}[>=stealth, font=\small,
  cell/.style={draw, acc, very thick, minimum width=12mm, minimum height=12mm, align=center},
  io/.style={draw, black, minimum width=9mm, minimum height=8mm, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  % three time steps, all the SAME cell
  \node[cell] (c1) at (0,0)   {\texttt{cell}};
  \node[cell] (c2) at (3,0)   {\texttt{cell}};
  \node[cell] (c3) at (6,0)   {\texttt{cell}};
  % inputs
  \node[io] (x1) at (0,-1.9) {$x_1$};
  \node[io] (x2) at (3,-1.9) {$x_2$};
  \node[io] (x3) at (6,-1.9) {$x_3$};
  % outputs
  \node[io] (h1) at (0,1.9) {$h_1$};
  \node[io] (h2) at (3,1.9) {$h_2$};
  \node[io] (h3) at (6,1.9) {$h_3$};
  \draw[->, thick] (x1) -- (c1);  \draw[->, thick] (c1) -- (h1);
  \draw[->, thick] (x2) -- (c2);  \draw[->, thick] (c2) -- (h2);
  \draw[->, thick] (x3) -- (c3);  \draw[->, thick] (c3) -- (h3);
  % recurrent edges, labelled with the shared weights
  \draw[->, acc, thick] (c1) -- (c2) node[midway, above, font=\footnotesize, text=acc] {\texttt{same}\hspace{1mm}$W$};
  \draw[->, acc, thick] (c2) -- (c3) node[midway, above, font=\footnotesize, text=acc] {\texttt{same}\hspace{1mm}$W$};
  \node[font=\footnotesize, anchor=north] at (3,-2.5) {\texttt{one cell, reused across time steps}};
\end{tikzpicture}
$$

### What sharing buys

The payoff is a collapse in the parameter count. A dense layer connecting every
input to every output scales as the _product_ of their sizes; a shared layer
scales only as the size of the shared unit. For a single channel mapping an
$n \times n$ image with a $k \times k$ filter, the contrast is stark.

| Layer | Free parameters | $n = 32,\ k = 3$ |
| --- | --- | --- |
| Dense ($n^2 \to n^2$) | $n^2 \cdot n^2 = n^4$ | $1{,}048{,}576$ |
| Convolution ($k \times k$ filter) | $k^2$ | $9$ |
| RNN step (hidden $d$, input $m$) | $d^2 + d m$ | — |
| Unrolled RNN over $T$ steps | $d^2 + d m$ (shared, not $T \cdot$) | — |

Five orders of magnitude separate the dense layer from the convolution computing a
comparable feature. The savings is not merely memory: fewer free parameters is
_directly_ lower capacity, hence the regularizing effect. The model cannot fit a
position-specific quirk because it has no position-specific weights to fit it with.[^gf-cnn-params]

## Weight tying, multi-task learning, and ensembles

Three further regularizers reuse parameters or models. Each constrains the
hypothesis space by sharing.

### Weight tying

A network may reuse one weight matrix in two roles. The **tied autoencoder** sets
the decoder weights to the transpose of the encoder's, $W_{\text{dec}} =
W_{\text{enc}}^{T}$, halving the parameters and enforcing a clean encode/decode
symmetry. In language models, **input–output embedding tying** shares the
token-embedding matrix with the output projection that scores the vocabulary: the
two are the same $V \times d$ matrix, a large saving when the vocabulary $V$ is
tens of thousands.

> **Definition (Weight tying).** Reusing a single weight tensor in two places in
> the network (e.g. encoder and decoder of an autoencoder, or the input embedding
> and output softmax of a language model), so that one stored matrix serves both
> roles and the gradient from both flows into it.

### Multi-task learning

When several related tasks share a **trunk** of layers and branch only at
task-specific **heads**, the shared parameters must serve every task at once. That
shared pressure is a regularizer: a representation good for many tasks is less able
to overfit the idiosyncrasies of any one of them.

$$
% caption: Multi-task learning: a shared trunk feeds task-specific heads, its
% weights regularized by having to satisfy every task at once.
\begin{tikzpicture}[>=stealth, font=\small,
  box/.style={draw, minimum width=20mm, minimum height=10mm, align=center, font=\footnotesize},
  head/.style={draw, minimum width=18mm, minimum height=9mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box] (in)  at (0,0) {\texttt{input}};
  \node[box, draw=acc, thick, text=acc] (tr) at (2.9,0) {\texttt{shared}\\\texttt{trunk}};
  \node[head] (h1) at (6.4,1.7) {\texttt{task 1 head}};
  \node[head] (h2) at (6.4,0)   {\texttt{task 2 head}};
  \node[head] (h3) at (6.4,-1.7) {\texttt{task 3 head}};
  \draw[->, thick] (in) -- (tr);
  \draw[->, thick] (tr) -- (h1);
  \draw[->, thick] (tr) -- (h2);
  \draw[->, thick] (tr) -- (h3);
  \node[font=\footnotesize, anchor=west] at (8.0,1.7) {\texttt{output 1}};
  \node[font=\footnotesize, anchor=west] at (8.0,0)   {\texttt{output 2}};
  \node[font=\footnotesize, anchor=west] at (8.0,-1.7) {\texttt{output 3}};
  \draw[->, thick] (h1) -- (8.0,1.7);
  \draw[->, thick] (h2) -- (8.0,0);
  \draw[->, thick] (h3) -- (8.0,-1.7);
\end{tikzpicture}
$$

### Bagging and the implicit ensemble of dropout

**Bagging** (bootstrap aggregating) trains $k$ models on $k$ resampled datasets
and averages their predictions. Independent errors partly cancel under averaging,
so the ensemble's variance falls.

> **Theorem (Variance reduction by averaging).** Let $k$ models have errors
> $\varepsilon_i$ with $\mathbb{E}[\varepsilon_i^2] = v$ and pairwise covariance
> $\mathbb{E}[\varepsilon_i \varepsilon_j] = c$ for $i \ne j$. The expected squared
> error of the average prediction is
> $$
> \mathbb{E}\brackets{\parens{\tfrac1k\textstyle\sum_i \varepsilon_i}^2}
> = \tfrac1k v + \tfrac{k-1}{k} c.
> $$

> **Proof.** Expand the square: $\parens{\tfrac1k\sum_i\varepsilon_i}^2 =
> \tfrac{1}{k^2}\parens{\sum_i \varepsilon_i^2 + \sum_{i\ne j}
> \varepsilon_i\varepsilon_j}$. Taking expectations, the $k$ diagonal terms
> each contribute $v$ and the $k(k-1)$ off-diagonal terms each contribute $c$:
> $\tfrac{1}{k^2}(kv + k(k-1)c) = \tfrac1k v + \tfrac{k-1}{k}c$. When the errors
> are uncorrelated ($c = 0$) this is $v/k$ — averaging cuts the error variance by a
> factor of $k$. When perfectly correlated ($c = v$) it stays $v$ — averaging
> identical models gives no reduction. $\qed$

[Dropout](/deep-learning/regularization/dropout-and-data-augmentation) is bagging
made implicit and cheap. Each forward pass samples a random sub-network by masking
units; over training, exponentially many sub-networks are trained, all _sharing_
the underlying weights. Test-time weight scaling approximates averaging their
predictions: an ensemble of $2^{n}$ models with the parameter budget of one.

| Method | Models | Parameters | Members trained | Combination |
| --- | --- | --- | --- | --- |
| Bagging | $k$ explicit, independent | $k \times$ one model | each on a bootstrap sample | average predictions |
| Dropout | $2^{n}$ implicit, weight-shared | $1\times$ one model | each on one minibatch | weight-scaling at test |

Bagging pays $k$ times the parameters and compute for genuinely independent
members; dropout reuses one parameter set across an exponential family of
sub-networks, trading independence for near-zero overhead — parameter sharing once
more, now across the members of an ensemble.[^gf-bagging]

## Early stopping in the overparameterized regime

The early-stopping-equals-$L^2$ result is classical (it traces to Bishop, 1995, and
the quadratic analysis Goodfellow §7.8 reproduces). What has changed is the setting
it operates in.
In the [overparameterized regime](/deep-learning/theory/generalization-theory), where
training error reaches zero and validation error can descend a _second_ time past the
interpolation threshold, the validation curve is no longer a clean U, so the "halt at
the first minimum" rule can stop too early; modern practice relies on patience and,
increasingly, on training to a fixed compute budget rather than to a validation
minimum at all.

Parameter sharing, meanwhile, has only grown more central. Beyond the CNN and RNN
cases above, **cross-layer weight sharing** in transformers (ALBERT, Lan et al.,
2020) ties the parameters of every transformer block to one shared set, cutting a
large model's parameter count by an order of magnitude with modest accuracy cost —
the DEQ from [deep equilibrium models](/deep-learning/theory/deep-equilibrium-models)
is the extreme limit of this idea, a single shared block iterated to convergence.
**Input-output embedding tying** (Press & Wolf, 2017; Inan et al., 2017) is now
standard in language models for the vocabulary-scale saving described above, and the
sample-efficiency argument for shared parameters — one weight, many gradient signals
per example — explains why convolution and attention, both built on sharing, scale
where dense layers cannot.

## Takeaways

- **Early stopping** halts at the validation-loss minimum and returns the best
  checkpoint, guarded by a **patience** parameter; it adds no loss term and one
  validation pass per epoch.
- For a quadratic objective, $\tau$ steps of gradient descent from the origin
  equals $L^2$ regularization with $\alpha \approx 1/(\eta\tau)$: stopping early
  shrinks the flat (small-$\lambda$) directions exactly as a norm penalty does,
  but discovers the strength in a single run.
- **Parameter sharing** forces weights to be _equal_ (the hard limit of soft
  **parameter tying**); it is the prior of translation/time equivariance behind
  **CNNs** (shared filters across space) and **RNNs** (shared cell across time),
  collapsing parameter counts by orders of magnitude.
- **Weight tying** (tied autoencoders, input–output embeddings), **multi-task
  learning** (shared trunk), and **dropout** (an implicit weight-shared ensemble,
  the cheap cousin of explicit **bagging**) all regularize by sharing parameters or
  models rather than by penalizing them.

[^gf-earlystop]: **Goodfellow**, _Deep Learning_, §7.8 — Early Stopping: training time as a hyperparameter, the patience-guarded best-checkpoint rule, and the strategies for recovering the held-out validation data afterward.
[^gf-es-l2]: **Goodfellow**, _Deep Learning_, §7.8 — Early Stopping as $L^2$: for a quadratic objective, $\tau$ gradient steps from the origin equal weight decay with $\alpha\approx 1/(\eta\tau)$, shrinking the flat eigendirections.
[^gf-sharing]: **Goodfellow**, _Deep Learning_, §7.9 — Parameter Tying and Parameter Sharing: the equivariance prior, the hard equality constraint of sharing versus the soft tying penalty $\norm{w^{(A)}-w^{(B)}}^2$.
[^gf-cnn-params]: **Goodfellow**, _Deep Learning_, §9.2 — Convolution and Parameter Sharing: one filter reused across every spatial position collapses the parameter count to the filter size and bakes in translation equivariance.
[^gf-bagging]: **Goodfellow**, _Deep Learning_, §7.11 — Bagging and Other Ensemble Methods: averaging $k$ models cuts error variance by up to $k$, and dropout is the implicit weight-shared limit of this idea.
[^chollet-earlystop]: **Chollet**, _Deep Learning with Python_, §7.3.3 — Using Callbacks: the `EarlyStopping` and `ModelCheckpoint` callbacks that monitor validation loss and restore the best weights.