---
title: Dropout & Data Augmentation
module: Regularization
moduleNumber: 4
lessonNumber: 2
order: 402
summary: >
  Two of the most effective regularizers add no penalty term at all; they
  perturb the computation instead. Dropout multiplies hidden units by a random
  Bernoulli mask, training an exponential ensemble of thinned subnetworks that
  share weights; inverted scaling collapses that ensemble into one cheap forward
  pass at test time. Data augmentation enlarges the training set with
  label-preserving transforms, injecting the invariances the task demands, and
  noise injection (input, weight, label smoothing, Mixup) generalizes the same
  idea into a continuous family.
topics: [Regularization]
sources:
  - book: Goodfellow
    ref: "§7.12 Dropout"
  - book: Goodfellow
    ref: "§7.4 Dataset Augmentation; §7.5 Noise Robustness"
  - book: Chollet
    ref: "§4.4.3 Adding Dropout; §5.2.5 Data Augmentation"
---

The previous lesson built regularization as a penalty added to the loss: an
$L^2$ term that pulls weights toward zero, an $L^1$ term that sparsifies. Dropout
and data augmentation belong to a different family: they leave the loss alone and
**perturb the computation** instead. Dropout injects multiplicative noise into the
hidden units; augmentation injects label-preserving noise into the inputs. Both
reduce the gap between training and test error: rather than shrinking parameters,
they prevent the network from memorizing any single fragile configuration.

## Dropout: multiplicative Bernoulli noise

At each training step, dropout independently sets each hidden unit to zero with
probability $p$. Concretely, for a layer's pre-activation output $h \in
\mathbb{R}^{d}$, draw a binary mask and apply it element-wise.

> **Definition (Dropout).** Sample a mask $m \in \{0,1\}^{d}$ with components
> $m_j \sim \text{Bernoulli}(1-p)$ independently, where $p$ is the **drop
> probability**. Replace the layer output $h$ by the masked output
> $\tilde h = m \odot h$, where $\odot$ is the element-wise (Hadamard) product. A
> resampled mask is drawn for every minibatch and every layer.[^gf-dropout]

A unit survives with probability $q = 1-p$ (the **keep probability**) and is zeroed
otherwise. Because masking is multiplicative, the expected value of each surviving
coordinate before any correction is

$$
\mathbb{E}[\tilde h_j] = \mathbb{E}[m_j]\,h_j = (1-p)\,h_j = q\,h_j,
$$

so the mask attenuates the layer's expected output by a factor $q$. A network
trained with dropped units but evaluated with all units present would therefore see
inputs that are, on average, a factor $1/q$ too large at every layer, a mismatch
that compounds with depth. The next two sections give the two equivalent ways to
reconcile train-time and test-time scale.

$$
% caption: Dropout on one hidden layer. Left: the full network. Right: the same
% network for one training step, with two units zeroed (drawn crossed out); their
% incoming and outgoing edges carry no signal.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  u/.style={circle, draw, minimum size=6mm, inner sep=0pt},
  d/.style={circle, draw=black, minimum size=6mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  % ---------- LEFT: full network ----------
  \begin{scope}
    \node[u] (i1) at (0,1.2) {};
    \node[u] (i2) at (0,-1.2) {};
    \foreach \i/\y in {1/2.4, 2/1.2, 3/0, 4/-1.2, 5/-2.4} \node[u] (h\i) at (1.7,\y) {};
    \node[u] (o1) at (3.4,0.6) {};
    \node[u] (o2) at (3.4,-0.6) {};
    \foreach \a in {i1,i2} \foreach \b in {h1,h2,h3,h4,h5} \draw[black] (\a) -- (\b);
    \foreach \a in {h1,h2,h3,h4,h5} \foreach \b in {o1,o2} \draw[black] (\a) -- (\b);
    \node at (1.7,-3.4) {\texttt{full network}};
  \end{scope}
  % ---------- RIGHT: thinned network ----------
  \begin{scope}[xshift=7cm]
    \node[u] (j1) at (0,1.2) {};
    \node[u] (j2) at (0,-1.2) {};
    % keep g1, g3, g4 ; drop g2, g5
    \node[u] (g1) at (1.7,2.4) {};
    \node[d] (g2) at (1.7,1.2) {};
    \node[u] (g3) at (1.7,0) {};
    \node[u] (g4) at (1.7,-1.2) {};
    \node[d] (g5) at (1.7,-2.4) {};
    \node[u] (p1) at (3.4,0.6) {};
    \node[u] (p2) at (3.4,-0.6) {};
    % live edges to kept units only
    \foreach \a in {j1,j2} \foreach \b in {g1,g3,g4} \draw[acc, thick] (\a) -- (\b);
    \foreach \a in {g1,g3,g4} \foreach \b in {p1,p2} \draw[acc, thick] (\a) -- (\b);
    % dropped edges faded + dashed
    \foreach \a in {j1,j2} \foreach \b in {g2,g5} \draw[black, dashed] (\a) -- (\b);
    \foreach \a in {g2,g5} \foreach \b in {p1,p2} \draw[black, dashed] (\a) -- (\b);
    % X strokes over the two dropped units
    \foreach \g in {g2,g5} {
      \draw[black!70, thick] ($(\g)+(-0.2,-0.2)$) -- ($(\g)+(0.2,0.2)$);
      \draw[black!70, thick] ($(\g)+(-0.2,0.2)$) -- ($(\g)+(0.2,-0.2)$);
    }
    \node at (1.7,-3.4) {\texttt{thinned} (one step)};
  \end{scope}
\end{tikzpicture}
$$

### Inverted dropout: scale at training time

The standard implementation, **inverted dropout**, rescales the surviving units by
$1/q$ during training so that the expected output is preserved unchanged.

$$
\tilde h = \frac{1}{1-p}\,(m \odot h)
\qquad\Longrightarrow\qquad
\mathbb{E}[\tilde h_j] = \frac{1}{1-p}\,(1-p)\,h_j = h_j.
$$

Because the train-time activations already carry the correct expected scale, the
test-time forward pass needs **no change at all**: masking is disabled and every unit
is kept. This is why every modern framework implements inverted dropout: inference is
the ordinary dense network, with zero scaling code on the hot path.[^chollet-dropout]

The rescaling only fixes the _mean_; it leaves a residual variance that is the actual
regularizing signal. Treat the scaled mask factor $r_j = m_j/q$ as a random multiplier
with $\mathbb{E}[r_j] = 1$. Its variance follows from $m_j \sim \text{Bernoulli}(q)$,
whose variance is $q(1-q)$:

$$
\Var[r_j] \;=\; \frac{\Var[m_j]}{q^2}
\;=\; \frac{q(1-q)}{q^2} \;=\; \frac{1-q}{q} \;=\; \frac{p}{1-p}.
$$

Each activation is therefore multiplied by unit-mean noise whose spread grows with the
drop probability: $p = 0.5$ gives multiplier variance $1$, while a light $p = 0.1$
gives only $0.11$. Dropout injects _multiplicative_ noise with a tunable amplitude, and
that injected variance is what forces the downstream units to stay robust to any single
input vanishing. The next figure traces this on a single layer.

$$
% caption: Inverted-dropout dataflow for one layer at training time. The pre-activation
% $h$ is multiplied elementwise by a Bernoulli mask, then divided by the keep
% probability $q=1-p$ so the surviving units carry the full expected scale.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  op/.style={draw, minimum width=17mm, minimum height=9mm, align=center},
  val/.style={draw, circle, minimum size=8mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  % input activation
  \node[val] (h) at (0,0) {$h$};
  \node[anchor=north, font=\footnotesize] at (0,-0.65) {\texttt{layer output}};
  % mask draw
  \node[op, draw=acc, text=acc] (mask) at (2.5,0) {\texttt{sample}\\\texttt{mask}};
  \node[anchor=south, font=\footnotesize] at (2.5,0.75) {\texttt{Bernoulli}};
  % hadamard
  \node[op] (had) at (5.2,0) {\texttt{mask}\\\texttt{times} $h$};
  % rescale
  \node[op, draw=acc, text=acc] (sc) at (7.9,0) {\texttt{rescale}\\\texttt{up}};
  % output
  \node[val] (out) at (10.2,0) {$\tilde h$};
  \node[anchor=north, font=\footnotesize] at (10.2,-0.65) {\texttt{to next layer}};
  % arrows
  \draw[->, thick] (h) -- (had);
  \draw[->, thick, acc] (mask) -- (had);
  \draw[->, thick] (had) -- (sc);
  \draw[->, thick] (sc) -- (out);
\end{tikzpicture}
$$

$$
% caption: Reconciling train and test scale. Top: inverted dropout scales surviving
% units up by $1/(1-p)$ during training, so the test pass keeps every unit unchanged.
% Bottom: the classical scheme leaves training alone and scales outgoing weights by
% $1-p$ at test time. Both leave the expected activation matched.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=36mm, minimum height=14mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % train side
  \node[box, draw=acc, text=acc, thick] (tr) at (0,0)
    {\texttt{TRAIN}\\\texttt{mask,} then scale\\\texttt{survivors} up};
  % test side
  \node[box, draw=green, text=green, thick] (te) at (6.4,0)
    {\texttt{TEST}\\keep all \texttt{units}\\no \texttt{scaling}};
  \draw[->, thick] (tr) -- (te)
    node[midway, above, font=\footnotesize] {\texttt{expected}};
  \node[font=\footnotesize] at (3.2,-0.6) {\texttt{activation matches}};
  % alternative at bottom
  \node[box] (alt1) at (0,-2.6) {\texttt{TRAIN}\\\texttt{mask only}\\(no \texttt{scaling})};
  \node[box] (alt2) at (6.4,-2.6) {\texttt{TEST}\\scale \texttt{outgoing}\\\texttt{weights} down};
  \draw[->, thick] (alt1) -- (alt2)
    node[midway, above, font=\footnotesize] {\texttt{weight-scaling rule}};
  \node[font=\footnotesize, text=black] at (3.2,-3.65) {\texttt{classical alternative}};
\end{tikzpicture}
$$

### The weight-scaling inference rule

The historical alternative leaves the masks unscaled at training time and instead
corrects the scale at inference by multiplying each layer's outgoing weights by the
keep probability $q$. For a unit feeding weight $w$ into the next layer, the
expected contribution under training-time dropout is $\mathbb{E}[m]\,w\,h =
q\,w\,h$; the **weight-scaling inference rule** reproduces this by using the
deterministic weight $q\,w$ on the full network.

> **Definition (Weight-scaling inference rule).** Train with unscaled masks
> $\tilde h = m \odot h$. At test time, run the full network with every outgoing
> weight multiplied by the keep probability: $w \mapsto (1-p)\,w$. This makes each
> unit's expected test-time input equal to its expected train-time input.

In expectation the two recipes coincide (inverted dropout moves the
factor to training and divides, the weight-scaling rule moves it to inference and
multiplies), and both make a single dense forward pass at test time approximate the
full dropout ensemble. The next section explains what that ensemble is.

## Dropout as an exponential ensemble

Each sampled mask defines a different **thinned subnetwork**, the
full architecture with some units (and all their edges) deleted. Over a layer of
$n$ droppable units there are $2^{n}$ possible masks, hence $2^{n}$ subnetworks,
and a single training step is one stochastic-gradient step on **one** of them,
sampled uniformly. Dropout trains all $2^{n}$ of them at once, with massive
parameter sharing: every subnetwork inherits its weights from the same shared pool,
so each weight is updated by whichever subnetworks happen to contain it.

$$
% caption: Dropout as an ensemble: training samples one thinned subnetwork (left);
% inference approximates the ensemble average with one weight-scaled pass (right).
\begin{tikzpicture}[>=stealth, font=\scriptsize,
  u/.style={circle, draw, minimum size=3.4mm, inner sep=0pt},
  d/.style={circle, draw=black, minimum size=3.4mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  % three small thinned subnets
  \foreach \k/\dx/\drop in {1/0/2, 2/2.7/0, 3/5.4/1} {
    \begin{scope}[xshift=\dx cm]
      \node[u] (a\k) at (0,0.55) {};
      \node[u] (b\k) at (0,-0.55) {};
      \foreach \i/\y in {1/1.0, 2/0, 3/-1.0} {
        \ifnum\i=\drop
          \node[d] (m\k\i) at (0.9,\y) {};
        \else
          \node[u] (m\k\i) at (0.9,\y) {};
        \fi
      }
      \node[u] (c\k) at (1.8,0) {};
      \foreach \src in {a\k,b\k} \foreach \i in {1,2,3} {
        \ifnum\i=\drop \draw[black, dashed] (\src) -- (m\k\i);
        \else \draw[acc, thick] (\src) -- (m\k\i); \fi }
      \foreach \i in {1,2,3} {
        \ifnum\i=\drop \draw[black, dashed] (m\k\i) -- (c\k);
        \else \draw[acc, thick] (m\k\i) -- (c\k); \fi }
      % X stroke on the dropped unit
      \ifnum\drop>0
        \draw[black!70, thick] ($(m\k\drop)+(-0.14,-0.14)$) -- ($(m\k\drop)+(0.14,0.14)$);
        \draw[black!70, thick] ($(m\k\drop)+(-0.14,0.14)$) -- ($(m\k\drop)+(0.14,-0.14)$);
      \fi
    \end{scope}
  }
  \node at (3.6,-1.75) {\texttt{sampled subnetworks (one per step)}};
  % averaging arrow
  \draw[->, very thick, black] (7.05,0) -- (8.05,0);
  % averaged dense net
  \begin{scope}[xshift=8.6cm]
    \node[u] (qa) at (0,0.55) {};
    \node[u] (qb) at (0,-0.55) {};
    \foreach \y in {1.0, 0, -1.0} \node[u] at (0.9,\y) {};
    \node[u] (qc) at (1.8,0) {};
    \foreach \src in {qa,qb} \foreach \y in {1.0,0,-1.0} \draw[acc, thick] (\src) -- (0.9,\y);
    \foreach \y in {1.0,0,-1.0} \draw[acc, thick] (0.9,\y) -- (qc);
    \node at (0.9,-1.75) {\texttt{weight-scaled average}};
  \end{scope}
\end{tikzpicture}
$$

### Why a single pass approximates the ensemble

A true ensemble prediction would average over masks. For a classifier producing a
distribution $p(y \mid x, m)$ under mask $m$, the principled ensemble is the
**geometric mean** over all masks, renormalized,

$$
p_{\text{ens}}(y \mid x) \;=\;
\frac{\tilde p_{\text{ens}}(y \mid x)}{\sum_{y'} \tilde p_{\text{ens}}(y' \mid x)},
\qquad
\tilde p_{\text{ens}}(y \mid x) \;=\;
\brackets{\,\prod_{m} p(y \mid x, m)\,}^{1/2^{n}},
$$

a product of $2^{n}$ terms that is intractable to evaluate directly.

> **Theorem (Weight-scaling approximates the ensemble).** For a single softmax
> layer with linear pre-activations, the weight-scaling inference rule computes the
> geometric-mean ensemble $p_{\text{ens}}$ **exactly**. For deeper nonlinear
> networks it is an approximation, but an empirically excellent one: a single
> forward pass with weights scaled by the keep probability matches the average over
> sampled masks closely enough to recover almost all of the ensemble's benefit.

The proof for the linear-softmax case is short: the geometric mean of softmaxes
over masks reduces, after taking logs, to a softmax of the **mean** pre-activation,
and the mean pre-activation under Bernoulli$(q)$ masking equals the weight scaled by
$q$, precisely the weight-scaling rule. Dropout thus trains an ensemble of
exponentially many models at the cost of training and evaluating one.[^gf-ensemble]

### Dropout versus bagging

Dropout is a form of **bagging** (bootstrap aggregating, which trains many models
on resampled datasets and averages them) but with the constraints that make an
exponential ensemble affordable.

| | Bagging | Dropout |
| --- | --- | --- |
| Number of models | $k$ (a handful, explicit) | $2^{n}$ (implicit, one per mask) |
| Model architectures | independent, can differ | one architecture, units removed |
| Parameters | each model has its own | all share one weight pool |
| Training per model | full convergence on a resample | one minibatch step per sampled mask |
| Data per model | a bootstrap resample of the data | the full minibatch |
| Inference | average / vote over $k$ models | one weight-scaled forward pass |
| Memory cost | $k\times$ the parameters | $1\times$ the parameters |

Bagging's models are independent and fully trained; dropout's are coupled through
shared weights and each sees only a few gradient steps. The shared parameters are
what make $2^{n}$ models tractable, and what make dropout a regularizer rather than
just an ensemble: a unit cannot rely on any particular other unit being present, so
the network is pushed to learn **redundant, distributed** features instead of
brittle co-adapted ones.

### Preventing co-adaptation

The ensemble view explains why the test-time average generalizes; the
**co-adaptation** view explains what changes inside the network during training.
Without dropout, a hidden unit is free to learn a feature that is only useful in
combination with a specific other unit — one detector fires a nonsense value that a
partner corrects, and the pair works only because both are always present. Such a
feature is fragile: it encodes a fact about the training set (that unit $k$ is
reliably available) rather than about the data.

Dropout removes that guarantee. Because any given partner is absent with probability
$p$ on each step, a unit that depends on a specific collaborator is penalized every
time the collaborator is dropped. The gradient pressure pushes each unit to be useful
_on its own_, against a randomly changing set of other units. The learned
features become individually meaningful and mutually redundant, which is the same
property that makes the ensemble prediction stable.[^gf-ensemble]

### DropConnect and structured dropout

Standard dropout zeroes _units_ — whole activations. Two variants change the
granularity of the mask.

**DropConnect** drops individual _weights_ rather than units: it samples a Bernoulli
mask over the weight matrix $W$ and trains on $\tilde W = M \odot W$. Because a unit
now loses a random subset of its incoming connections instead of its whole output, the
subnetwork family is larger — $2^{|W|}$ rather than $2^{n}$ — and the masking is finer
grained. In practice it regularizes slightly more strongly than unit dropout at higher
compute cost, and unit dropout remains the default.

**Spatial dropout** targets convolutional feature maps. A convolutional activation is
a tensor with shape $(C, H, W)$: $C$ channels of an $H \times W$ spatial grid. Ordinary
dropout on such a tensor zeroes individual pixels, but neighboring pixels in a feature
map are strongly correlated, so a dropped pixel is nearly recoverable from its
neighbors and the noise barely regularizes. Spatial dropout instead zeroes _entire
channels_ at once — an all-or-nothing mask over the channel axis — which removes a
whole feature detector and injects noise the network cannot trivially interpolate
around. This is the correct dropout for convolutional layers when dropout is used
there at all.

## The dropout forward pass

The full forward pass simply branches on whether the network is in training or
evaluation mode. In training it samples a mask and applies inverted scaling; in
evaluation it passes activations through untouched.

```algorithm
caption: $\textsc{DropoutForward}(h, p, \text{training})$ — inverted dropout for one layer
if not training then // evaluation: dense pass, no scaling
  return $h$
$q \gets 1 - p$ // keep probability
for each unit $j$ do
  sample $m_j \sim \text{Bernoulli}(q)$ // 1 with prob. $q$, else 0
  $\tilde h_j \gets (m_j \cdot h_j) / q$ // mask, then rescale by $1/q$
return $\tilde h$
```

The branch on `training` is the single most common source of train/test bugs:
forgetting to switch a model to evaluation mode leaves dropout active at inference,
injecting noise into every prediction. Frameworks expose this as an explicit mode
flag: `model.eval()` in PyTorch, the `training` argument in Keras, so the mask can be
disabled deterministically.[^stevens-eval]

### Practical defaults

The drop probability trades bias against variance of the injected noise. Too small and
the regularization is negligible; too large and each step trains on so few units that
optimization slows and the network underfits. The conventional defaults follow the
structure of the layer.

| Layer type | Typical $p$ | Reason |
| --- | --- | --- |
| Fully connected (hidden) | $0.5$ | Dense layers have the most parameters per unit and overfit most; $p=0.5$ maximizes the subnetwork count $2^{n}$ and is the value the original analysis studied. |
| Input layer | $0.1$–$0.2$ | Dropping raw inputs discards signal directly, so only light noise is safe. |
| Convolutional | $0.0$–$0.2$ | Conv layers already have few parameters and are regularized by weight sharing; heavy dropout wastes their spatial structure. Prefer spatial dropout if any. |
| After batch norm | often $0.0$ | Batch normalization contributes its own noise, and stacking the two can hurt; many modern architectures drop dropout from normalized conv stacks entirely. |

The value $p = 0.5$ for hidden fully connected layers is not arbitrary: it is the
drop probability that makes the number of distinct
subnetworks $\binom{n}{n/2}$-weighted around its maximum, so training samples the
widest possible ensemble.[^gf-dropout]

## Data augmentation

Dropout perturbs the hidden units; **data augmentation** perturbs the inputs.
The principle is to enlarge the training set with **label-preserving transforms**:
operations $T$ for which the label of $T(x)$ is the same as the label of $x$.

> **Definition (Data augmentation).** Given a training pair $(x, y)$ and a
> distribution over label-preserving transforms $T$, train on the augmented pairs
> $(T(x), y)$ with a fresh $T$ sampled each epoch. Because the label is invariant
> under $T$, every augmented example is a valid new training point, drawn from a
> richer effective data distribution than the raw sample provides.

The transforms must be chosen to match the **invariances the task actually has**.
Horizontally flipping a photo of a cat yields another valid cat, so flip is
label-preserving for natural-image classification, but flipping a photo of the
digit "$6$" produces something closer to a "$9$", so horizontal flip is **not**
label-preserving for digit recognition. Augmentation is the mechanism by which a
practitioner injects domain knowledge about which variations are nuisance and which
are signal.[^gf-augment]

| Modality | Transform | Invariance injected |
| --- | --- | --- |
| Images | horizontal flip | left–right symmetry |
| Images | random crop / pad | translation, framing |
| Images | small rotation / shear | pose, camera tilt |
| Images | color jitter (brightness, contrast, hue) | lighting, white balance |
| Images | cutout (mask a random patch) | occlusion robustness |
| Audio | time shift / time stretch | onset timing, tempo |
| Audio | pitch shift; add background noise | speaker pitch, channel noise |
| Text | synonym swap; back-translation | lexical, paraphrastic variation |

$$
% caption: Image augmentation: one labeled source transformed several
% label-preserving ways — flip, crop, and small rotation.
\begin{tikzpicture}[>=stealth, font=\scriptsize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % ---- source ----
  \draw[thick] (0,0) rectangle (1.6,1.6);
  \draw[acc, very thick] (0.55,0.3) -- (0.55,1.3) -- (1.05,1.3);
  \node[anchor=north] at (0.8,-0.1) {\texttt{source}};
  % arrow fan
  \draw[->, black, thick] (1.8,0.8) -- (2.6,0.8);
  % ---- flip ----
  \begin{scope}[xshift=2.8cm]
    \draw[thick] (0,0) rectangle (1.6,1.6);
    \draw[acc, very thick] (1.05,0.3) -- (1.05,1.3) -- (0.55,1.3);
    \node[anchor=north] at (0.8,-0.1) {\texttt{flip}};
    \node[green, anchor=south, font=\footnotesize] at (0.8,1.65) {\texttt{same label}};
  \end{scope}
  % ---- crop ----
  \begin{scope}[xshift=5.6cm]
    \draw[black, thin] (0,0) rectangle (1.6,1.6);
    \draw[thick] (0.35,0.2) rectangle (1.45,1.3);
    \draw[acc, very thick] (0.75,0.4) -- (0.75,1.15) -- (1.15,1.15);
    \node[anchor=north] at (0.8,-0.1) {\texttt{crop}};
    \node[green, anchor=south, font=\footnotesize] at (0.8,1.65) {\texttt{same label}};
  \end{scope}
  % ---- rotate ----
  \begin{scope}[xshift=8.4cm]
    \begin{scope}[rotate around={14:(0.8,0.8)}]
      \draw[thick] (0,0) rectangle (1.6,1.6);
      \draw[acc, very thick] (0.55,0.3) -- (0.55,1.3) -- (1.05,1.3);
    \end{scope}
    \node[anchor=north] at (0.8,-0.15) {\texttt{rotate}};
    \node[green, anchor=south, font=\footnotesize] at (0.8,1.75) {\texttt{same label}};
  \end{scope}
\end{tikzpicture}
$$

Augmentation is most effective exactly where it is cheap and the invariances are
well understood, vision above all, where it is now standard practice and often
worth several percentage points of accuracy on its own.[^chollet-augment]

### Why augmentation regularizes

The regularizing mechanism is a widening of the _effective_ training distribution. A
raw dataset is a finite sample of $N$ points from the true data distribution
$p_{\text{data}}(x, y)$; the network can, given enough capacity, memorize those $N$
points exactly. Augmentation replaces each point $x$ with a whole orbit
$\{T(x) : T \sim \mathcal{T}\}$ of transformed versions that share its label, so the
network never sees the same input twice and cannot memorize a lookup table. Formally,
training on $(T(x), y)$ with $T$ resampled every epoch minimizes the loss under an
_augmented_ distribution

$$
p_{\text{aug}}(x', y) \;=\; \mathbb{E}_{T \sim \mathcal{T}}\,
p_{\text{data}}\!\big(T^{-1}(x'),\, y\big),
$$

which is $p_{\text{data}}$ smeared out along the directions the transforms move. Every
augmented example is a valid draw from a distribution that is broader than, but
consistent with, the real one.

Augmentation also encodes a **prior on the function class**. By
declaring that $x$ and $T(x)$ carry the same label, it constrains the network to be
_invariant_ to $T$: to produce the same output across the entire orbit. That constraint
shrinks the space of functions the network can represent to those that already respect
the task's symmetries, exactly the role a regularizer plays. Choosing the transform set
is choosing which invariances to build in, and choosing them wrong (horizontal flip on
digits) builds in a false symmetry that hurts.

$$
% caption: Augmentation widens the effective training distribution. Each raw sample
% (filled) is replaced by an orbit of label-preserving variants (open), so the network
% fits a smeared distribution that is broader than the finite sample.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  % ---- left: raw samples ----
  \begin{scope}
    \draw[black] (0,0) rectangle (3,2.4);
    \foreach \x/\y in {0.6/0.7, 1.2/1.8, 2.1/0.5, 2.4/1.7, 1.5/1.1} \fill[acc] (\x,\y) circle (2.4pt);
    \node[anchor=north, font=\footnotesize] at (1.5,-0.15) {\texttt{raw samples}};
  \end{scope}
  % arrow
  \draw[->, thick] (3.4,1.2) -- (4.4,1.2);
  \node[anchor=south, font=\footnotesize] at (3.9,1.25) {\texttt{augment}};
  % ---- right: orbits ----
  \begin{scope}[xshift=4.8cm]
    \draw[black] (0,0) rectangle (3,2.4);
    \foreach \x/\y in {0.6/0.7, 1.2/1.8, 2.1/0.5, 2.4/1.7, 1.5/1.1} {
      \fill[acc] (\x,\y) circle (2.4pt);
      \draw[acc!55] (\x,\y) circle (5pt);
      \foreach \a in {40, 140, 240, 320} {
        \fill[acc!45] ($(\x,\y)+(\a:5pt)$) circle (1.4pt);
      }
    }
    \node[anchor=north, font=\footnotesize] at (1.5,-0.15) {\texttt{augmented orbits}};
  \end{scope}
\end{tikzpicture}
$$

## Noise injection as regularization

Both dropout and augmentation are special cases of a broader principle: **injecting
noise during training regularizes**. Noise can enter at three points, each with a
distinct effect.

| Injection site | Operation | Regularizing effect |
| --- | --- | --- |
| Input | $x \mapsto x + \varepsilon$, $\varepsilon \sim \mathcal{N}(0,\sigma^2 I)$ | for squared loss, equivalent to an $L^2$ weight penalty |
| Hidden units | dropout mask $m \odot h$ | ensemble of thinned subnetworks |
| Weights | $w \mapsto w + \varepsilon$ each step | pushes toward flat minima, robust to perturbation |
| Labels | label smoothing | discourages overconfident logits |

> **Theorem (Input noise equals weight decay).** For a linear model under squared
> loss, training on inputs corrupted by isotropic Gaussian noise $\varepsilon \sim
> \mathcal{N}(0, \sigma^2 I)$ is equivalent in expectation to training on clean
> inputs with an added $L^2$ penalty $\sigma^2 \norm{w}_2^2$ on the weights.

The equivalence is a short calculation. Take a linear model $f(x) = w^\top x$ under
squared loss, and corrupt the input as $x + \varepsilon$ with
$\varepsilon \sim \mathcal{N}(0, \sigma^2 I)$, so $\mathbb{E}[\varepsilon] = 0$ and
$\mathbb{E}[\varepsilon \varepsilon^\top] = \sigma^2 I$. The expected loss on a single
example expands as

$$
\mathbb{E}_\varepsilon\big[(w^\top(x+\varepsilon) - y)^2\big]
= (w^\top x - y)^2
+ 2(w^\top x - y)\,w^\top \underbrace{\mathbb{E}[\varepsilon]}_{0}
+ w^\top \underbrace{\mathbb{E}[\varepsilon\varepsilon^\top]}_{\sigma^2 I}\, w.
$$

The cross term vanishes because the noise is zero-mean, and the last term is
$\sigma^2 \norm{w}_2^2$. So the noisy objective equals the clean squared loss plus a
weight-decay penalty of strength $\sigma^2$ — the same $L^2$ term the previous lesson
added by hand. Input noise and weight decay are the same regularizer viewed from two
sides. Noise injection therefore belongs alongside explicit penalties: in the
simplest case it _is_ an explicit penalty, and in the general case it does the same job
of penalizing sensitivity to small input changes.[^gf-noise]

### Label smoothing

**Label smoothing** injects noise into the targets rather than the inputs. A
one-hot target requires the correct logit to be driven to $+\infty$ and the
rest to $-\infty$: unachievable, and an incentive toward overconfidence. Label smoothing
softens the target by mixing in a uniform distribution over the $K$ classes.

$$
y^{\text{LS}}_k = (1 - \alpha)\,y_k + \frac{\alpha}{K},
$$

where $\alpha \in (0,1)$ is the smoothing strength and $y$ is the one-hot label. The
target for the correct class drops from $1$ to $1 - \alpha + \alpha/K$, and each
wrong class rises from $0$ to $\alpha/K$. The network is no longer rewarded for
unbounded confidence, which improves calibration and generalization.

### Mixup and CutMix

**Mixup** is the most aggressive form of input-and-label noise: it trains on
**convex combinations** of pairs of examples, blending both the inputs and their
one-hot labels by the same coefficient.

> **Definition (Mixup).** Sample two training pairs $(x_i, y_i)$, $(x_j, y_j)$ and a
> mixing coefficient $\lambda \sim \text{Beta}(\alpha, \alpha)$, then train on the
> blended pair
> $$
> \tilde x = \lambda\,x_i + (1-\lambda)\,x_j,
> \qquad
> \tilde y = \lambda\,y_i + (1-\lambda)\,y_j.
> $$

Because both the input and the soft label are blended by the same $\lambda$, mixup
trains the model to behave **linearly between training examples**, a strong prior
that smooths the decision boundary and sharply curbs memorization of individual
points. The $\text{Beta}(\alpha, \alpha)$ distribution controls how aggressive the
blending is: small $\alpha$ (say $0.2$) puts most mass near $\lambda = 0$ or
$\lambda = 1$, so most mixed examples stay close to one endpoint and only occasionally
sit near the midpoint; $\alpha = 1$ gives a uniform $\lambda$ and the strongest mixing.
The default $\alpha \in [0.1, 0.4]$ keeps the augmentation mild.

**CutMix** is the spatial variant: instead of blending pixel intensities, it pastes a
rectangular patch from $x_j$ into $x_i$ and sets $\lambda$ to the patch's area
fraction, mixing the labels by that same area. Because the composite image contains
intact patches of both sources rather than a blurred average, it preserves local
texture that mixup blurs, and it usually transfers better to detection-style tasks.

$$
% caption: Mixup. The blended point $\tilde x = \lambda x_i + (1-\lambda) x_j$ lies on
% the segment between two examples of different classes: weight $\lambda$ pulls toward
% class $A$, weight $1-\lambda$ toward class $B$. Its soft label is mixed by the same $\lambda$.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  % two class endpoints
  \coordinate (A) at (0,0);
  \coordinate (B) at (6,1.4);
  % connecting segment
  \draw[black, thick] (A) -- (B);
  % blended point at lambda from A toward B (about 0.62 of the way)
  \coordinate (M) at (3.7,0.86);
  % endpoints
  \fill[acc] (A) circle (3.2pt);
  \node[acc, anchor=north east, align=center] at (-0.05,0.15) {\texttt{class A}\\point $x_i$};
  \fill[red] (B) circle (3.2pt);
  \node[red, anchor=south west, align=center] at (6.05,1.3) {\texttt{class B}\\point $x_j$};
  % blended
  \fill[green] (M) circle (3.4pt);
  \node[green, anchor=south east] at (3.55,0.95) {\texttt{blend}};
  % labels for the two convex weights, kept clear of the segment
  \node[acc, anchor=south, font=\footnotesize] at (1.6,0.82) {\texttt{toward A}};
  \node[red, anchor=north, font=\footnotesize] at (5.15,0.5) {\texttt{toward B}};
\end{tikzpicture}
$$

## Learned augmentation policies

Goodfellow §7.4 and §7.12 predate the augmentation and dropout developments that now
matter most in practice; the public papers fill the gap. On the augmentation side,
the hand-chosen transform list has been replaced by **learned or randomized
policies**: **AutoAugment** (Cubuk et al., 2019) searches for the best augmentation
policy per dataset, and its cheaper successor **RandAugment** (Cubuk et al., 2020)
drops the search entirely, applying a fixed number of random transforms at a single
tunable magnitude and matching the searched policies at a fraction of the cost. The
mixing augmentations sketched above are also post-Goodfellow: **Mixup** (Zhang et
al., 2018) and **CutMix** (Yun et al., 2019) are the citations of record, and
**label smoothing** was analyzed as a calibration tool by Müller et al. (2019).

On the dropout side, **dropout has largely
receded** from convolutional vision architectures. Once
[batch normalization](/deep-learning/regularization/normalization) became standard,
stacking it with dropout was found to hurt (the two interact badly through the
variance shift between train and test, per Li et al., 2019), and heavy augmentation
plus weight decay took over the regularization budget. Dropout remains standard in
transformers and other fully connected stacks, where its ensemble effect still pays,
but the field's default regularizer for image models shifted from "dropout
everywhere" to "normalize, augment aggressively, decay the weights." The
noise-injection view still applies: each of these prevents the
network from memorizing a fragile configuration.

## Takeaways

- **Dropout** zeroes each hidden unit with probability $p$ via a Bernoulli mask
  $m$, training on $\tilde h = m \odot h$; **inverted dropout** rescales survivors
  by $1/(1-p)$ so that inference is an ordinary unscaled dense pass.
- The **weight-scaling inference rule** is the equivalent: train with unscaled
  masks, then multiply outgoing weights by $(1-p)$ at test time. Both make one
  forward pass approximate the dropout ensemble.
- Dropout trains $2^{n}$ **thinned subnetworks** with shared weights — a form of
  bagging — and a weight-scaled pass approximates the geometric-mean ensemble,
  exactly for a linear softmax.
- **Data augmentation** enlarges the data with **label-preserving** transforms,
  injecting the task's invariances; the transform set must match the invariance
  (flip is fine for cats, wrong for digits).
- **Noise injection** generalizes both: input noise equals weight decay for linear
  squared loss, **label smoothing** curbs overconfidence, and **Mixup**/**CutMix**
  train on convex combinations $\lambda x_i + (1-\lambda) x_j$ to enforce linear
  behavior between examples.

[^gf-dropout]: **Goodfellow**, _Deep Learning_, §7.12 — Dropout: the Bernoulli mask $m\odot h$, the keep probability $q=1-p$, and the expected-scale attenuation the inference rule must correct.
[^gf-ensemble]: **Goodfellow**, _Deep Learning_, §7.12 — Dropout as bagging: the $2^{n}$ thinned subnetworks with shared weights and the geometric-mean ensemble that weight scaling computes exactly for a linear softmax.
[^gf-augment]: **Goodfellow**, _Deep Learning_, §7.4 — Dataset Augmentation: label-preserving transforms inject the task's invariances, with the cat-flip-versus-digit-flip caution on choosing the transform set.
[^gf-noise]: **Goodfellow**, _Deep Learning_, §7.5 — Noise Robustness: injecting noise at the inputs, weights, or labels regularizes; for a linear model under squared loss, input noise equals an $L^2$ weight penalty.
[^chollet-dropout]: **Chollet**, _Deep Learning with Python_, §4.4.3 — Adding Dropout: inverted dropout as implemented in Keras, scaling survivors at train time so inference is an unscaled dense pass.
[^chollet-augment]: **Chollet**, _Deep Learning with Python_, §5.2.5 — Using Data Augmentation: the `ImageDataGenerator` pipeline and why augmentation is worth several points of accuracy on small image datasets.
[^stevens-eval]: **Stevens et al.**, _Deep Learning with PyTorch_, ch. 8 — the `model.train()` / `model.eval()` mode switch that disables dropout (and freezes batch-norm statistics) for inference, and the bugs that follow from forgetting it.