Early Stopping & Parameter Sharing

The regularizers of the previous lessons add a term to the loss: an $L^2$ penalty, a dropout mask. This lesson covers two that add no term and still constrain the hypothesis space. Early stopping controls capacity through the optimization trajectory: it limits how far the weights travel from their initialization. Parameter sharing controls it through the parameterization itself: it forces distinct weights to take a single shared value. Both shrink the effective number of free parameters without ever appearing in $\mathcal{L}(\theta )$.

Early stopping

Train a high-capacity network long enough and the training loss falls monotonically while the validation loss traces a U: down as the model learns real structure, then back up as it begins to fit noise. The gap between the two curves is the generalization gap, and it widens precisely in the regime where training continues to pay off but validation does not.

The recipe is to treat the number of training steps as a hyperparameter and tune it on the validation set: evaluate validation loss every epoch, remember the parameters that achieved the lowest value seen, and return those rather than the final ones.¹

Because the validation curve is noisy, we do not halt at the first uptick. The patience parameter $p$ specifies how many consecutive non-improving evaluations to tolerate before stopping, a guard against quitting on a transient bump.

The full procedure keeps a running best checkpoint and a counter of stale evaluations.

Two practical points fall out of the algorithm. First, early stopping is nearly free: the only overhead is one validation pass per epoch and a copy of the best weights. Second, it composes with every other regularizer (it is a stopping rule, not a loss term) and is the one regularizer essentially every training run uses.²

The reason the validation curve turns upward is that effective capacity grows with training time. At initialization the weights are near the origin and the function the network computes is nearly constant; each gradient step lets the weights travel farther and the function it can express becomes richer. Training time therefore behaves like a continuous capacity knob. Early in training the extra capacity captures real structure and both losses fall; past $t^{\star }$ the extra capacity goes toward memorizing the training set, and validation loss climbs while training loss keeps falling. Stopping at $t^{\star }$ caps the effective capacity at the level that best matches the data.

A subtlety: early stopping spends a validation split that could otherwise be training data. One remedy is to record $t^{\star }$, then retrain from scratch on the union ${\mathcal{D}}_{\text{train}}\cup {\mathcal{D}}_{\text{val}}$ for $t^{\star }$ steps; another is to keep training the existing model on the validation data until its loss falls below the level reached at $t^{\star }$. Both recover the held-out examples at the cost of a second training run.

Early stopping is $L^2$ regularization

Early stopping looks like a heuristic about wall-clock time, but for a quadratic objective it is exactly a norm penalty. The mechanism: gradient descent started at $\theta =0$ can only move so far in a finite number of steps, so halting early keeps $\theta$ near the origin: the same constraint $L^2$ weight decay imposes by penalizing $\Vert \theta {\Vert }^2$.³

The correspondence $\alpha \approx 1/(\eta \tau )$ is exact in its consequences for each eigendirection. Along a stiff direction (large ${\lambda }_i$) the factor $1-\eta {\lambda }_i$ is small, so $u_i$ reaches $u_i^{\star }$ in a few steps: these weights are learned early and unaffected by the penalty. Along a flat direction (small ${\lambda }_i$) progress is slow; halting at step $\tau$ leaves $u_i$ short of $u_i^{\star }$, exactly the directions $L^2$ shrinks most.

The per-axis factor $1-(1-\eta {\lambda }_i{)}^{\tau }$, read as a function of ${\lambda }_i$, shows the selectivity. A stiff axis has $\eta {\lambda }_i$ close to $1$, so $(1-\eta {\lambda }_i{)}^{\tau }\to 0$ within a handful of steps and the factor saturates at $1$: the coordinate is fully learned. A flat axis has $\eta {\lambda }_i\approx 0$, so $(1-\eta {\lambda }_i{)}^{\tau }\approx 1$ still after $\tau$ steps and the factor stays near $0$: the coordinate barely moves off the origin. The $L^2$ factor ${\lambda }_i/({\lambda }_i+\alpha )$ traces the same S-curve against ${\lambda }_i$ — near $1$ for ${\lambda }_i\gg \alpha$, near $0$ for ${\lambda }_i\ll \alpha$ — with the crossover set by $\alpha$ instead of by $\tau$.

Early stopping wins on cost: it discovers the effective penalty during a single training run by reading it off the validation curve, whereas tuning $\alpha$ requires a separate run for each candidate value.

Parameter sharing

Early stopping shrinks weights toward a point. Parameter sharing instead ties weights to each other: it declares a set of weights to be one and the same free parameter, so the optimizer adjusts them in lockstep. Where an $L^2$ penalty expresses a soft preference that two weights be close (parameter tying, $\Omega =\Vert w^{(A)}-w^{(B)}{\Vert }^2$), sharing is the hard constraint that they be identical.

The prior behind sharing is equivariance: if a feature detector is useful at one location in the input, it is useful at every location. Two architectures are built entirely on this idea.⁴

The gradient of a shared weight is a sum. Suppose a single free parameter $w$ is used at $m$ locations, appearing in the forward pass as $m$ copies $w_1=w_2=\cdots =w_m=w$. Treat the copies as independent for a moment and let $\partial \mathcal{L}/\partial w_j$ be the local gradient at location $j$. By the chain rule, the derivative of the loss with respect to the one shared value is the sum of the per-location contributions,

since $\partial w_j/\partial w=1$ for every copy. Backpropagation through a shared weight computes the ordinary local gradient at each location where the weight is used and accumulates them into one number. Every location contributes to the single shared value's update; a convolution kernel is updated by the summed signal from every patch it slid over, an RNN weight by the summed signal from every time step. This accumulation is why a shared parameter, seeing $m$ locations worth of gradient per example, is more sample-efficient than $m$ independent parameters that each see one.

Convolutions share across space

A convolutional layer applies one small filter at every spatial position. The same weights are reused across the whole image, so a filter that finds a vertical edge finds it everywhere: translation equivariance, baked into the parameterization.

Recurrence shares across time

A recurrent network applies one cell at every time step. Unrolled, the network looks deep, but every step reuses the same weight matrices: sharing across time, the sequence analogue of the CNN's sharing across space.

What sharing buys

The payoff is a collapse in the parameter count. A dense layer connecting every input to every output scales as the product of their sizes; a shared layer scales only as the size of the shared unit. For a single channel mapping an $n\times n$ image with a $k\times k$ filter, the contrast is stark.

Five orders of magnitude separate the dense layer from the convolution computing a comparable feature. The savings is not merely memory: fewer free parameters is directly lower capacity, hence the regularizing effect. The model cannot fit a position-specific quirk because it has no position-specific weights to fit it with.⁵

Weight tying, multi-task learning, and ensembles

Three further regularizers reuse parameters or models. Each constrains the hypothesis space by sharing.

Weight tying

A network may reuse one weight matrix in two roles. The tied autoencoder sets the decoder weights to the transpose of the encoder's, $W_{\text{dec}}=W_{\text{enc}}^T$, halving the parameters and enforcing a clean encode/decode symmetry. In language models, input–output embedding tying shares the token-embedding matrix with the output projection that scores the vocabulary: the two are the same $V\times d$ matrix, a large saving when the vocabulary $V$ is tens of thousands.

Multi-task learning

When several related tasks share a trunk of layers and branch only at task-specific heads, the shared parameters must serve every task at once. That shared pressure is a regularizer: a representation good for many tasks is less able to overfit the idiosyncrasies of any one of them.

Bagging and the implicit ensemble of dropout

Bagging (bootstrap aggregating) trains $k$ models on $k$ resampled datasets and averages their predictions. Independent errors partly cancel under averaging, so the ensemble's variance falls.

Dropout is bagging made implicit and cheap. Each forward pass samples a random sub-network by masking units; over training, exponentially many sub-networks are trained, all sharing the underlying weights. Test-time weight scaling approximates averaging their predictions: an ensemble of $2^n$ models with the parameter budget of one.

Bagging pays $k$ times the parameters and compute for genuinely independent members; dropout reuses one parameter set across an exponential family of sub-networks, trading independence for near-zero overhead — parameter sharing once more, now across the members of an ensemble.⁶

Early stopping in the overparameterized regime

The early-stopping-equals-$L^2$ result is classical (it traces to Bishop, 1995, and the quadratic analysis Goodfellow §7.8 reproduces). What has changed is the setting it operates in. In the overparameterized regime, where training error reaches zero and validation error can descend a second time past the interpolation threshold, the validation curve is no longer a clean U, so the halt at the first minimum rule can stop too early; modern practice relies on patience and, increasingly, on training to a fixed compute budget rather than to a validation minimum at all.

Parameter sharing, meanwhile, has only grown more central. Beyond the CNN and RNN cases above, cross-layer weight sharing in transformers (ALBERT, Lan et al., 2020) ties the parameters of every transformer block to one shared set, cutting a large model's parameter count by an order of magnitude with modest accuracy cost — the DEQ from deep equilibrium models is the extreme limit of this idea, a single shared block iterated to convergence. Input-output embedding tying (Press & Wolf, 2017; Inan et al., 2017) is now standard in language models for the vocabulary-scale saving described above, and the sample-efficiency argument for shared parameters — one weight, many gradient signals per example — explains why convolution and attention, both built on sharing, scale where dense layers cannot.

Takeaways

• Early stopping halts at the validation-loss minimum and returns the best checkpoint, guarded by a patience parameter; it adds no loss term and one validation pass per epoch.
• For a quadratic objective, $\tau$ steps of gradient descent from the origin equals $L^2$ regularization with $\alpha \approx 1/(\eta \tau )$: stopping early shrinks the flat (small-$\lambda$) directions exactly as a norm penalty does, but discovers the strength in a single run.
• Parameter sharing forces weights to be equal (the hard limit of soft parameter tying); it is the prior of translation/time equivariance behind CNNs (shared filters across space) and RNNs (shared cell across time), collapsing parameter counts by orders of magnitude.
• Weight tying (tied autoencoders, input–output embeddings), multi-task learning (shared trunk), and dropout (an implicit weight-shared ensemble, the cheap cousin of explicit bagging) all regularize by sharing parameters or models rather than by penalizing them.