Dropout & Data Augmentation

The previous lesson built regularization as a penalty added to the loss: an $L^2$ term that pulls weights toward zero, an $L^1$ term that sparsifies. Dropout and data augmentation belong to a different family: they leave the loss alone and perturb the computation instead. Dropout injects multiplicative noise into the hidden units; augmentation injects label-preserving noise into the inputs. Both reduce the gap between training and test error: rather than shrinking parameters, they prevent the network from memorizing any single fragile configuration.

Dropout: multiplicative Bernoulli noise

At each training step, dropout independently sets each hidden unit to zero with probability $p$. Concretely, for a layer's pre-activation output $h\in {\mathbb{R}}^d$, draw a binary mask and apply it element-wise.

A unit survives with probability $q=1-p$ (the keep probability) and is zeroed otherwise. Because masking is multiplicative, the expected value of each surviving coordinate before any correction is

so the mask attenuates the layer's expected output by a factor $q$. A network trained with dropped units but evaluated with all units present would therefore see inputs that are, on average, a factor $1/q$ too large at every layer, a mismatch that compounds with depth. The next two sections give the two equivalent ways to reconcile train-time and test-time scale.

Inverted dropout: scale at training time

The standard implementation, inverted dropout, rescales the surviving units by $1/q$ during training so that the expected output is preserved unchanged.

Because the train-time activations already carry the correct expected scale, the test-time forward pass needs no change at all: masking is disabled and every unit is kept. This is why every modern framework implements inverted dropout: inference is the ordinary dense network, with zero scaling code on the hot path.²

The rescaling only fixes the mean; it leaves a residual variance that is the actual regularizing signal. Treat the scaled mask factor $r_j=m_j/q$ as a random multiplier with $\mathbb{E}[r_j]=1$. Its variance follows from $m_j\sim \text{Bernoulli}(q)$, whose variance is $q(1-q)$:

Each activation is therefore multiplied by unit-mean noise whose spread grows with the drop probability: $p=0.5$ gives multiplier variance $1$, while a light $p=0.1$ gives only $0.11$. Dropout injects multiplicative noise with a tunable amplitude, and that injected variance is what forces the downstream units to stay robust to any single input vanishing. The next figure traces this on a single layer.

The weight-scaling inference rule

The historical alternative leaves the masks unscaled at training time and instead corrects the scale at inference by multiplying each layer's outgoing weights by the keep probability $q$. For a unit feeding weight $w$ into the next layer, the expected contribution under training-time dropout is $\mathbb{E}[m]wh=qwh$; the weight-scaling inference rule reproduces this by using the deterministic weight $qw$ on the full network.

In expectation the two recipes coincide (inverted dropout moves the factor to training and divides, the weight-scaling rule moves it to inference and multiplies), and both make a single dense forward pass at test time approximate the full dropout ensemble. The next section explains what that ensemble is.

Dropout as an exponential ensemble

Each sampled mask defines a different thinned subnetwork, the full architecture with some units (and all their edges) deleted. Over a layer of $n$ droppable units there are $2^n$ possible masks, hence $2^n$ subnetworks, and a single training step is one stochastic-gradient step on one of them, sampled uniformly. Dropout trains all $2^n$ of them at once, with massive parameter sharing: every subnetwork inherits its weights from the same shared pool, so each weight is updated by whichever subnetworks happen to contain it.

Why a single pass approximates the ensemble

A true ensemble prediction would average over masks. For a classifier producing a distribution $p(y\mid x,m)$ under mask $m$, the principled ensemble is the geometric mean over all masks, renormalized,

a product of $2^n$ terms that is intractable to evaluate directly.

The proof for the linear-softmax case is short: the geometric mean of softmaxes over masks reduces, after taking logs, to a softmax of the mean pre-activation, and the mean pre-activation under Bernoulli$(q)$ masking equals the weight scaled by $q$, precisely the weight-scaling rule. Dropout thus trains an ensemble of exponentially many models at the cost of training and evaluating one.³

Dropout versus bagging

Dropout is a form of bagging (bootstrap aggregating, which trains many models on resampled datasets and averages them) but with the constraints that make an exponential ensemble affordable.

Bagging's models are independent and fully trained; dropout's are coupled through shared weights and each sees only a few gradient steps. The shared parameters are what make $2^n$ models tractable, and what make dropout a regularizer rather than just an ensemble: a unit cannot rely on any particular other unit being present, so the network is pushed to learn redundant, distributed features instead of brittle co-adapted ones.

Preventing co-adaptation

The ensemble view explains why the test-time average generalizes; the co-adaptation view explains what changes inside the network during training. Without dropout, a hidden unit is free to learn a feature that is only useful in combination with a specific other unit — one detector fires a nonsense value that a partner corrects, and the pair works only because both are always present. Such a feature is fragile: it encodes a fact about the training set (that unit $k$ is reliably available) rather than about the data.

Dropout removes that guarantee. Because any given partner is absent with probability $p$ on each step, a unit that depends on a specific collaborator is penalized every time the collaborator is dropped. The gradient pressure pushes each unit to be useful on its own, against a randomly changing set of other units. The learned features become individually meaningful and mutually redundant, which is the same property that makes the ensemble prediction stable.³

DropConnect and structured dropout

Standard dropout zeroes units — whole activations. Two variants change the granularity of the mask.

DropConnect drops individual weights rather than units: it samples a Bernoulli mask over the weight matrix $W$ and trains on $\tilde{W}=M\odot W$. Because a unit now loses a random subset of its incoming connections instead of its whole output, the subnetwork family is larger — $2^{|W|}$ rather than $2^n$ — and the masking is finer grained. In practice it regularizes slightly more strongly than unit dropout at higher compute cost, and unit dropout remains the default.

Spatial dropout targets convolutional feature maps. A convolutional activation is a tensor with shape $(C,H,W)$: $C$ channels of an $H\times W$ spatial grid. Ordinary dropout on such a tensor zeroes individual pixels, but neighboring pixels in a feature map are strongly correlated, so a dropped pixel is nearly recoverable from its neighbors and the noise barely regularizes. Spatial dropout instead zeroes entire channels at once — an all-or-nothing mask over the channel axis — which removes a whole feature detector and injects noise the network cannot trivially interpolate around. This is the correct dropout for convolutional layers when dropout is used there at all.

The dropout forward pass

The full forward pass simply branches on whether the network is in training or evaluation mode. In training it samples a mask and applies inverted scaling; in evaluation it passes activations through untouched.

The branch on training is the single most common source of train/test bugs: forgetting to switch a model to evaluation mode leaves dropout active at inference, injecting noise into every prediction. Frameworks expose this as an explicit mode flag: model.eval() in PyTorch, the training argument in Keras, so the mask can be disabled deterministically.⁴

Practical defaults

The drop probability trades bias against variance of the injected noise. Too small and the regularization is negligible; too large and each step trains on so few units that optimization slows and the network underfits. The conventional defaults follow the structure of the layer.

The value $p=0.5$ for hidden fully connected layers is not arbitrary: it is the drop probability that makes the number of distinct subnetworks $\left(\genfrac{}{}{0ex}{}{n}{n/2}\right)$-weighted around its maximum, so training samples the widest possible ensemble.¹

Data augmentation

Dropout perturbs the hidden units; data augmentation perturbs the inputs. The principle is to enlarge the training set with label-preserving transforms: operations $T$ for which the label of $T(x)$ is the same as the label of $x$.

The transforms must be chosen to match the invariances the task actually has. Horizontally flipping a photo of a cat yields another valid cat, so flip is label-preserving for natural-image classification, but flipping a photo of the digit $6$ produces something closer to a $9$, so horizontal flip is not label-preserving for digit recognition. Augmentation is the mechanism by which a practitioner injects domain knowledge about which variations are nuisance and which are signal.⁵

Augmentation is most effective exactly where it is cheap and the invariances are well understood, vision above all, where it is now standard practice and often worth several percentage points of accuracy on its own.⁶

Why augmentation regularizes

The regularizing mechanism is a widening of the effective training distribution. A raw dataset is a finite sample of $N$ points from the true data distribution $p_{\text{data}}(x,y)$; the network can, given enough capacity, memorize those $N$ points exactly. Augmentation replaces each point $x$ with a whole orbit $\{T(x):T\sim \mathcal{T}\}$ of transformed versions that share its label, so the network never sees the same input twice and cannot memorize a lookup table. Formally, training on $(T(x),y)$ with $T$ resampled every epoch minimizes the loss under an augmented distribution

which is $p_{\text{data}}$ smeared out along the directions the transforms move. Every augmented example is a valid draw from a distribution that is broader than, but consistent with, the real one.

Augmentation also encodes a prior on the function class. By declaring that $x$ and $T(x)$ carry the same label, it constrains the network to be invariant to $T$: to produce the same output across the entire orbit. That constraint shrinks the space of functions the network can represent to those that already respect the task's symmetries, exactly the role a regularizer plays. Choosing the transform set is choosing which invariances to build in, and choosing them wrong (horizontal flip on digits) builds in a false symmetry that hurts.

Noise injection as regularization

Both dropout and augmentation are special cases of a broader principle: injecting noise during training regularizes. Noise can enter at three points, each with a distinct effect.

The equivalence is a short calculation. Take a linear model $f(x)=w^{\top }x$ under squared loss, and corrupt the input as $x+\epsilon$ with $\epsilon \sim \mathcal{N}(0,{\sigma }^{2}I)$, so $\mathbb{E}[\epsilon ]=0$ and $\mathbb{E}[\epsilon {\epsilon }^{\top }]={\sigma }^{2}I$. The expected loss on a single example expands as

The cross term vanishes because the noise is zero-mean, and the last term is ${\sigma }^2\Vert w{\Vert }_2^2$. So the noisy objective equals the clean squared loss plus a weight-decay penalty of strength ${\sigma }^2$ — the same $L^2$ term the previous lesson added by hand. Input noise and weight decay are the same regularizer viewed from two sides. Noise injection therefore belongs alongside explicit penalties: in the simplest case it is an explicit penalty, and in the general case it does the same job of penalizing sensitivity to small input changes.⁷

Label smoothing

Label smoothing injects noise into the targets rather than the inputs. A one-hot target requires the correct logit to be driven to $+\infty$ and the rest to $-\infty$: unachievable, and an incentive toward overconfidence. Label smoothing softens the target by mixing in a uniform distribution over the $K$ classes.

where $\alpha \in (0,1)$ is the smoothing strength and $y$ is the one-hot label. The target for the correct class drops from $1$ to $1-\alpha +\alpha /K$, and each wrong class rises from $0$ to $\alpha /K$. The network is no longer rewarded for unbounded confidence, which improves calibration and generalization.

Mixup and CutMix

Mixup is the most aggressive form of input-and-label noise: it trains on convex combinations of pairs of examples, blending both the inputs and their one-hot labels by the same coefficient.

Because both the input and the soft label are blended by the same $\lambda$, mixup trains the model to behave linearly between training examples, a strong prior that smooths the decision boundary and sharply curbs memorization of individual points. The $\text{Beta}(\alpha ,\alpha )$ distribution controls how aggressive the blending is: small $\alpha$ (say $0.2$) puts most mass near $\lambda =0$ or $\lambda =1$, so most mixed examples stay close to one endpoint and only occasionally sit near the midpoint; $\alpha =1$ gives a uniform $\lambda$ and the strongest mixing. The default $\alpha \in [0.1,0.4]$ keeps the augmentation mild.

CutMix is the spatial variant: instead of blending pixel intensities, it pastes a rectangular patch from $x_j$ into $x_i$ and sets $\lambda$ to the patch's area fraction, mixing the labels by that same area. Because the composite image contains intact patches of both sources rather than a blurred average, it preserves local texture that mixup blurs, and it usually transfers better to detection-style tasks.

Learned augmentation policies

Goodfellow §7.4 and §7.12 predate the augmentation and dropout developments that now matter most in practice; the public papers fill the gap. On the augmentation side, the hand-chosen transform list has been replaced by learned or randomized policies: AutoAugment (Cubuk et al., 2019) searches for the best augmentation policy per dataset, and its cheaper successor RandAugment (Cubuk et al., 2020) drops the search entirely, applying a fixed number of random transforms at a single tunable magnitude and matching the searched policies at a fraction of the cost. The mixing augmentations sketched above are also post-Goodfellow: Mixup (Zhang et al., 2018) and CutMix (Yun et al., 2019) are the citations of record, and label smoothing was analyzed as a calibration tool by Müller et al. (2019).

On the dropout side, dropout has largely receded from convolutional vision architectures. Once batch normalization became standard, stacking it with dropout was found to hurt (the two interact badly through the variance shift between train and test, per Li et al., 2019), and heavy augmentation plus weight decay took over the regularization budget. Dropout remains standard in transformers and other fully connected stacks, where its ensemble effect still pays, but the field's default regularizer for image models shifted from dropout everywhere to normalize, augment aggressively, decay the weights. The noise-injection view still applies: each of these prevents the network from memorizing a fragile configuration.

Takeaways

• Dropout zeroes each hidden unit with probability $p$ via a Bernoulli mask $m$, training on $\tilde{h}=m\odot h$; inverted dropout rescales survivors by $1/(1-p)$ so that inference is an ordinary unscaled dense pass.
• The weight-scaling inference rule is the equivalent: train with unscaled masks, then multiply outgoing weights by $(1-p)$ at test time. Both make one forward pass approximate the dropout ensemble.
• Dropout trains $2^n$ thinned subnetworks with shared weights — a form of bagging — and a weight-scaled pass approximates the geometric-mean ensemble, exactly for a linear softmax.
• Data augmentation enlarges the data with label-preserving transforms, injecting the task's invariances; the transform set must match the invariance (flip is fine for cats, wrong for digits).
• Noise injection generalizes both: input noise equals weight decay for linear squared loss, label smoothing curbs overconfidence, and Mixup/CutMix train on convex combinations $\lambda x_i+(1-\lambda )x_j$ to enforce linear behavior between examples.