---
title: CNN Architectures
module: Architectures
moduleNumber: 5
lessonNumber: 2
order: 502
summary: >
  Six landmark networks, each contributing exactly one idea: LeNet's conv-pool
  stack, AlexNet's ReLU-and-dropout scale, VGG's $3\times3$ uniformity, Inception's
  multi-scale module, ResNet's residual skip, and DenseNet's dense connectivity.
  The common thread is the degradation problem (why plain deeper nets train worse,
  not just overfit) and the residual block that solved it by keeping a $+1$ path
  open for the gradient.
topics: [Architectures]
sources:
  - book: Goodfellow
    ref: "Ch. 9 — Convolutional Networks; §9.11 — The Neuroscientific Basis / Historical"
  - book: Chollet
    ref: "Ch. 5 — Deep Learning for Computer Vision"
---

The [convolutional layer](/deep-learning/architectures/convolutional-networks)
fixes the primitive (local, weight-shared, translation-equivariant filtering),
but it does not fix the wiring. _How many_ filters, _how deep_, _connected how?_
The answer came from a decade of architectures, each one
isolating a single structural idea and pushing it until it broke. This lesson is
that sequence: six landmark networks, the one idea each contributed, and the
degradation problem that the most important of them solved.

## The landmark architectures

Every modern vision backbone is a recombination of six prior moves. Read the table
as a list of _contributions_, not models; each row is one idea that survived.

| Architecture | Year | Depth | Key idea | Approx. params |
| --- | --- | --- | --- | --- |
| LeNet-5 | 1998 | 7 | conv-pool stacks for digits | $60\text{K}$ |
| AlexNet | 2012 | 8 | ReLU + dropout + GPU scale | $60\text{M}$ |
| VGG-16 | 2014 | 16 | uniform $3\times3$ stacks, depth | $138\text{M}$ |
| GoogLeNet (Inception) | 2014 | 22 | multi-scale module, $1\times1$ bottleneck | $5\text{M}$ |
| ResNet-50 | 2015 | 50 | residual / skip connections | $25\text{M}$ |
| DenseNet-121 | 2017 | 121 | dense connectivity (concat all prior) | $8\text{M}$ |

Two numbers in that table summarize the trend. Depth climbs from $7$ to $121$;
parameter count _peaks_ at VGG's $138$M and then _falls_ — Inception and ResNet are
an order of magnitude deeper than VGG yet carry far fewer weights. Depth and
parameter count decoupled, and every architecture after VGG is a way of adding
depth without adding parameters.

### LeNet-5: the template

LeNet-5 set the pattern every later network inherits: alternate **convolution**
(detect local features) with **subsampling/pooling** (shrink spatial extent, grow
receptive field), then finish with a few dense layers. Spatial resolution falls
while channel depth rises: features get coarser and more abstract toward the
output.[^gf-history]

$$
% caption: The LeNet-5 template: conv-pool stacks halve resolution as channels grow, then dense layers read off the class. Every later CNN varies this skeleton.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  conv/.style={draw, fill=acclo!30, minimum width=8mm, minimum height=16mm, align=center, font=\scriptsize},
  pool/.style={draw, fill=black!8, minimum width=8mm, minimum height=12mm, align=center, font=\scriptsize},
  fc/.style={draw, fill=black!6, minimum width=7mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{acclo}{HTML}{A7B5FB}
  \node[draw, minimum width=10mm, minimum height=18mm, align=center, font=\scriptsize] (in) at (0,0) {input\\image};
  \node[conv] (c1) at (1.5,0)  {conv};
  \node[pool] (p1) at (2.7,0)  {pool};
  \node[conv] (c2) at (3.9,0)  {conv};
  \node[pool] (p2) at (5.1,0)  {pool};
  \node[fc]   (f1) at (6.4,0)  {fc};
  \node[fc]   (f2) at (7.5,0)  {fc};
  \node[draw=acc, text=acc, thick, minimum width=9mm, minimum height=8mm, align=center, font=\scriptsize] (out) at (8.9,0) {class};
  \draw[->, thick] (in) -- (c1); \draw[->, thick] (c1) -- (p1);
  \draw[->, thick] (p1) -- (c2); \draw[->, thick] (c2) -- (p2);
  \draw[->, thick] (p2) -- (f1); \draw[->, thick] (f1) -- (f2);
  \draw[->, acc, thick] (f2) -- (out);
  \draw[->, black, thick] (0,-1.6) -- (8.9,-1.6);
  \node[black, anchor=south, font=\footnotesize] at (4.45,-1.55) {\texttt{resolution down, channels up}};
\end{tikzpicture}
$$

> **Definition (Conv-pool stack).** A repeated unit of one or more convolutions
> followed by a spatial downsampling (pooling or strided convolution). Stacking
> the unit deepens the network while geometrically shrinking the feature map, so
> each successive filter sees a larger fraction of the original image: its
> _receptive field_ grows.

### AlexNet: scale, ReLU, dropout

AlexNet is LeNet, larger, trained on two GPUs over ImageNet's $1.2$M images. Its
contribution is the three choices that made depth
_trainable at scale_: the [ReLU](/deep-learning/neural-networks/activation-functions)
nonlinearity, which does not saturate and so keeps gradients alive; aggressive
[dropout](/deep-learning/regularization/dropout-and-data-augmentation) in the dense
layers to fight overfitting; and GPU implementation to make the compute feasible.[^chollet-vision]

| Choice | Replaces | Why it mattered |
| --- | --- | --- |
| ReLU $\max(0, z)$ | $\tanh$ / sigmoid | no saturation $\Rightarrow$ no vanishing gradient; $\sim 6\times$ faster convergence |
| dropout $p = 0.5$ | none | decorrelates dense units, cuts overfitting on $60$M params |
| GPU training | CPU | makes $60$M params over $1.2$M images tractable in days, not months |

Where the $60$M parameters actually sit is worth tracing, because it explains
every design move that follows. AlexNet takes a $224\times224\times3$ image through
five convolutional stages and three dense layers. The convolutions are cheap in
_weights_ but expensive in _activations_; the dense layers are the reverse.

| Layer | Output shape $H\times W\times C$ | Kernel / stride | Weights |
| --- | --- | --- | --- |
| input | $224\times224\times3$ | — | $0$ |
| conv1 + pool | $27\times27\times96$ | $11\times11$, s$4$ | $35\text{K}$ |
| conv2 + pool | $13\times13\times256$ | $5\times5$ | $614\text{K}$ |
| conv3 | $13\times13\times384$ | $3\times3$ | $885\text{K}$ |
| conv4 | $13\times13\times384$ | $3\times3$ | $1.3\text{M}$ |
| conv5 + pool | $6\times6\times256$ | $3\times3$ | $442\text{K}$ |
| fc6 | $4096$ | dense | $37.7\text{M}$ |
| fc7 | $4096$ | dense | $16.8\text{M}$ |
| fc8 (softmax) | $1000$ | dense | $4.1\text{M}$ |

The first dense layer alone, `fc6`, holds $6\cdot6\cdot256 \times 4096 = 37.7$M
weights — $62\%$ of the whole network — because it flattens the entire final feature
map and connects it fully to $4096$ units. That single number is why later
architectures replace the flatten-then-dense head with **global average pooling**:
collapsing the $6\times6\times256$ map to a $256$-vector by averaging removes the
$37.7$M-weight layer outright. Inception and ResNet both do exactly this, which is a
large part of why they carry an order of magnitude fewer parameters than VGG.

### VGG: depth through uniformity

VGG replaces every large filter with a stack of $3\times3$ convolutions, and proves
the substitution is _strictly better_: two stacked $3\times3$ layers see the same
$5\times5$ receptive field as one $5\times5$ layer, but with fewer parameters and an
extra nonlinearity in between.

$$
% caption: Two stacked $3\times3$ convolutions cover the same $5\times5$ receptive field as one $5\times5$ filter, at fewer parameters and with an extra nonlinearity.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % one 5x5
  \draw[black] (0,0) grid (2.5,2.5);
  \draw[acc, very thick] (0,0) rectangle (2.5,2.5);
  \node[anchor=north, font=\scriptsize] at (1.25,-0.15) {one 5x5};
  \node[anchor=south, font=\scriptsize, text=acc] at (1.25,2.6) {$25\,C^2$ weights};
  % arrow
  \draw[->, thick] (3.1,1.25) -- (4.1,1.25) node[midway, above, font=\scriptsize] {$=$};
  % two 3x3
  \draw[black] (4.7,0) grid (7.2,2.5);
  \draw[red, very thick] (4.7,0) rectangle (6.2,1.5);
  \draw[acc, very thick] (5.2,0.5) rectangle (6.7,2.0);
  \node[anchor=north, font=\scriptsize] at (5.95,-0.15) {two 3x3};
  \node[anchor=south, font=\scriptsize, text=acc] at (5.95,2.6) {$18\,C^2$ weights};
\end{tikzpicture}
$$

> **Lemma (Stacked small filters).** A stack of $k$ convolutions with $3\times3$
> kernels has receptive field $(2k+1)\times(2k+1)$ and, on $C$ channels in and out,
> costs $k\cdot 9C^2$ weights, versus $(2k+1)^2 C^2$ for the single equivalent
> filter. For $k=2$ this is $18C^2$ against $25C^2$: fewer parameters _and_ one
> more nonlinearity, so deeper-and-narrower dominates wider-and-shallower.

The uniformity yields more than a parameter saving. Because every stage keeps the same
$3\times3$ kernel and halves the spatial resolution at each pooling step, VGG-16 is a
single rule applied five times: two-or-three $3\times3$ convolutions, then a
$2\times2$ pool that halves $H$ and $W$ and doubles the channel count. The tensor
shape follows a clean geometric progression.

| Block | Convs | Output shape $H\times W\times C$ | Receptive field |
| --- | --- | --- | --- |
| input | — | $224\times224\times3$ | $1\times1$ |
| block 1 (+ pool) | $2\times$ conv $64$ | $112\times112\times64$ | $5\times5$ |
| block 2 (+ pool) | $2\times$ conv $128$ | $56\times56\times128$ | $14\times14$ |
| block 3 (+ pool) | $3\times$ conv $256$ | $28\times28\times256$ | $40\times40$ |
| block 4 (+ pool) | $3\times$ conv $512$ | $14\times14\times512$ | $92\times92$ |
| block 5 (+ pool) | $3\times$ conv $512$ | $7\times7\times512$ | $196\times196$ |
| fc6 / fc7 / fc8 | dense | $4096 / 4096 / 1000$ | full image |

Resolution falls $224 \to 7$ (a $32\times$ reduction, five halvings) while channels
climb $3 \to 512$. By block 5 a single unit's receptive field ($196\times196$)
covers nearly the whole $224\times224$ input, so the final feature map sees
the entire image at once. Each pooling step both grows that field and, by doubling
$C$, keeps the per-layer compute roughly constant as the map shrinks.

$$
% caption: Receptive field growth by depth. Stacking $3\times3$ convolutions expands the field linearly ($2\ell+1$); each $2\times2$ pool then multiplies the stride, so the field a deep unit sees grows to cover the whole image.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % axes
  \draw[->, thick] (0,0) -- (7.2,0) node[right, font=\footnotesize] {\texttt{depth (layers)}};
  \draw[->, thick] (0,0) -- (0,4.3) node[above, font=\footnotesize] {\texttt{receptive field}};
  % gridlines
  \foreach \y/\lab in {1/50, 2/100, 3/150} {
    \draw[black] (0,\y) -- (6.9,\y);
    \node[black, anchor=east, font=\scriptsize] at (-0.1,\y) {\lab};
  }
  % linear (no pooling): 3x3 stack, field = 2L+1, shallow slope
  \draw[acc, very thick] (0,0.1) -- (6.6,1.5);
  \node[acc, anchor=south, font=\footnotesize] at (5.2,1.55) {\texttt{3x3 stack only}};
  \draw[acc, thin] (5.2,1.5) -- (5.6,1.28);
  % with pooling: superlinear growth
  \draw[green, very thick] plot[smooth, domain=0:6.6, samples=40] (\x, {0.1 + 0.09*\x*\x});
  \node[green, anchor=south east, font=\footnotesize] at (6.4,3.9) {\texttt{3x3 + pooling}};
\end{tikzpicture}
$$

VGG pushed this to $16$–$19$ layers and won by depth alone — but at $138$M
parameters, most of them in the dense layers, it is the heaviest model in the
table. It also exposed a limit: stacking past $\sim20$ plain layers stopped helping.

## The degradation problem

The natural hypothesis — _deeper is at least as good, because the extra layers can
always learn the identity_ — is false in practice. Plain stacked networks past a
depth threshold train to _higher_ error than their shallower counterparts. This is
not overfitting: the **training** error itself is worse, so the failure lies in
optimization, not in the generalization gap.[^gf-depth]

$$
% caption: The degradation problem: a plain 56-layer net reaches higher training error than a 20-layer net, so deeper is worse even on the training set.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, thick] (0,0) -- (6.4,0) node[right, font=\scriptsize] {iterations};
  \draw[->, thick] (0,0) -- (0,4.2) node[above, font=\scriptsize] {training error};
  % 20-layer (lower, blue)
  \draw[acc, very thick] plot[smooth, domain=0.2:6, samples=60] (\x, {0.9 + 2.4*exp(-0.9*\x)});
  \node[acc, anchor=west, font=\footnotesize] at (4.3,1.15) {\texttt{20-layer (plain)}};
  % 56-layer (higher, red) -- degraded
  \draw[red, very thick] plot[smooth, domain=0.2:6, samples=60] (\x, {1.8 + 2.2*exp(-0.8*\x)});
  \node[red, anchor=west, font=\footnotesize] at (4.3,2.05) {\texttt{56-layer (plain)}};
  % gap marker
  \draw[black, dashed] (4.0,1.05) -- (4.0,2.0);
  \node[black, anchor=west, font=\footnotesize] at (1.2,3.6) {\texttt{deeper means higher train error}};
\end{tikzpicture}
$$

> **Definition (Degradation).** The empirical phenomenon that increasing the depth
> of a plain feed-forward network beyond some point raises its **training** error,
> even though a deeper net strictly contains the shallower one as a sub-network
> (set the extra layers to identity). The added layers should be able to do no
> harm, yet the optimizer fails to find that solution.

The diagnosis is that the identity map is _hard to represent_ with a stack of
nonlinear layers. Producing $H(x) = x$ from a conv-ReLU-conv block requires the
weights to combine into an exact identity, a thin target in weight space that
gradient descent does not reliably reach.

## ResNet: learn the residual

ResNet's fix is a reparameterization. Instead of asking a block to compute the
target map $H(x)$ directly, ask it to compute the **residual** $F(x) = H(x) - x$,
and add the input back via an identity skip:

$$
H(x) = F(x) + x.
$$

Now the identity is _free_: if the optimal block is the identity, the network only
needs $F(x) \to 0$, driving a stack of weights to zero, which weight decay does
anyway, rather than constructing an exact identity from nonlinearities.[^gf-residual]

$$
% caption: The residual block. Input $x$ splits into a learned path $F(x)$ and an identity skip, summed before the final nonlinearity.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  layer/.style={draw, fill=acclo!25, minimum width=26mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{acclo}{HTML}{A7B5FB}
  \definecolor{green}{HTML}{1F9D4D}
  \node (x) at (0,0) {$x$};
  \node[layer] (c1) at (0,1.3) {conv 3x3, ReLU};
  \node[layer] (c2) at (0,2.8) {conv 3x3};
  \node[circle, draw, thick, minimum size=7mm, inner sep=0pt] (sum) at (0,4.2) {$+$};
  \node[draw=acc, text=acc, thick, minimum width=20mm, minimum height=8mm] (out) at (0,5.6) {ReLU};
  % main path
  \draw[->, thick] (x) -- (c1);
  \draw[->, thick] (c1) -- (c2);
  \draw[->, thick] (c2) -- (sum);
  \draw[->, acc, thick] (sum) -- (out);
  % identity skip (right side)
  \draw[->, green, very thick] (0.6,0.15) -- (2.6,0.15) -- (2.6,4.2) -- (sum.east);
  \node[green, anchor=west, font=\footnotesize] at (2.7,2.2) {\texttt{identity skip} $x$};
  % F(x) label — the learned path bracket
  \draw[acc, thick] (-1.55,0.8) -- (-1.75,0.8) -- (-1.75,3.3) -- (-1.55,3.3);
  \node[acc, anchor=east, font=\scriptsize] at (-1.8,2.05) {$F(x)$};
\end{tikzpicture}
$$

### Why the skip keeps gradients alive

The skip's deeper payoff is on the backward pass. Consider a block $y = x + F(x, W)$.
The Jacobian of the output with respect to the input is

$$
\frac{\partial y}{\partial x}
= I + \frac{\partial F(x, W)}{\partial x}.
$$

Chaining $L$ such blocks from layer $\ell$ to the loss $\mathcal{L}$ at the top, the
gradient flowing back to $x_\ell$ is, by the chain rule,

$$
\frac{\partial \mathcal{L}}{\partial x_\ell}
= \frac{\partial \mathcal{L}}{\partial x_L}
\prod_{i=\ell}^{L-1}\parens{ I + \frac{\partial F_i}{\partial x_i} }
= \frac{\partial \mathcal{L}}{\partial x_L}
\parens{ I + \sum_{i} \frac{\partial F_i}{\partial x_i} + \cdots }.
$$

Expanding the product, the leading term is the bare $\dfrac{\partial
\mathcal{L}}{\partial x_L}$ — the gradient reaches the early layer **undiminished**,
along a path of all-identity factors. The deep multiplicative product of
$\partial F_i/\partial x_i$ Jacobians that vanishes in a plain net is now an
_additive correction_ on top of a guaranteed $+1$ path.

> **Theorem (Gradient highway).** In a residual stack the gradient
> $\partial\mathcal{L}/\partial x_\ell$ contains an additive term
> $\partial\mathcal{L}/\partial x_L$ that is independent of the intermediate
> weights. The signal can never vanish entirely across depth unless
> $\partial\mathcal{L}/\partial x_L$ itself vanishes — the identity skip is a
> short-circuit for the gradient.

> **Proof.** Write $x_{i+1} = x_i + F_i(x_i)$, so $\partial x_{i+1}/\partial x_i =
> I + \partial F_i/\partial x_i$. By the chain rule $\partial\mathcal{L}/\partial
> x_\ell = (\partial\mathcal{L}/\partial x_L)\prod_{i=\ell}^{L-1}(I + \partial
> F_i/\partial x_i)$. Distribute the product over the $L-\ell$ factors: every term
> picks either $I$ or $\partial F_i/\partial x_i$ from each factor. The unique term
> that picks $I$ from _all_ factors is the identity, contributing
> $\partial\mathcal{L}/\partial x_L \cdot I$ verbatim. Hence
> $\partial\mathcal{L}/\partial x_\ell = \partial\mathcal{L}/\partial x_L + (\text{terms in }\partial F)$,
> and the first summand survives regardless of how small the $\partial F_i$ grow. $\qed$

For a concrete number, suppose each block's Jacobian
$\partial F_i/\partial x_i$ has spectral norm $\approx 0.8$, a modest per-layer
contraction. In a **plain** net the signal is the product of these factors: across
$50$ blocks the gradient shrinks by $0.8^{50} \approx 1.4\times10^{-5}$, five orders
of magnitude, so the early layers get almost no signal and stop learning. In a
**residual** net the same $50$ blocks contribute $\prod(I + \partial F_i/\partial
x_i)$, whose expansion keeps the bare identity term at magnitude $1$ no matter how
many factors multiply. The gradient at the first block is $1 + (\text{small
corrections})$, not $10^{-5}$. That is the entire difference between a net that
trains at depth $50$ and one that does not.

This is the same mechanism as the LSTM's cell state in
[recurrent networks](/deep-learning/architectures/lstm-and-gru): a near-identity
path along which the gradient propagates without repeated multiplication. With it,
ResNet trained $152$ layers where plain nets degraded past $20$.

ResNet-50 stacks the bottleneck block (below) into four stages, halving resolution
and doubling width at each stage boundary. The tensor shape follows the same ladder
VGG used, but the flatten-then-dense head is gone: a global average pool collapses
the final $7\times7\times2048$ map to a $2048$-vector, so the classifier is a single
$2048\times1000$ layer instead of VGG's $100$M-weight dense stack.

| Stage | Blocks | Output shape $H\times W\times C$ | Block channels ($1{\times}1, 3{\times}3, 1{\times}1$) |
| --- | --- | --- | --- |
| conv1 + pool | — | $56\times56\times64$ | $7\times7$ conv, s$2$ |
| stage 2 | $3$ | $56\times56\times256$ | $64, 64, 256$ |
| stage 3 | $4$ | $28\times28\times512$ | $128, 128, 512$ |
| stage 4 | $6$ | $14\times14\times1024$ | $256, 256, 1024$ |
| stage 5 | $3$ | $7\times7\times2048$ | $512, 512, 2048$ |
| global avg pool + fc | — | $2048 \to 1000$ | dense |

Fifty weight layers, $25$M parameters: a fifth of VGG's count at more than three
times the depth. The bottleneck (next section) is what keeps each of those $16$
stacked blocks cheap.

```algorithm
caption: $\textsc{ResidualBlock}(x, W_1, W_2)$ — forward pass of a basic block
$a \gets \text{ReLU}(\text{conv}(x, W_1))$ // first conv path
$F \gets \text{conv}(a, W_2)$ // residual, no activation yet
if $\dim(F) \ne \dim(x)$ then // spatial / channel mismatch
  $x \gets \text{conv}_{1\times1}(x)$ // project skip to match
$y \gets \text{ReLU}(F + x)$ // add identity, then activate
return $y$
```

## 1×1 convolutions and the bottleneck

A $1\times1$ convolution touches one spatial location at a time; it has no spatial
extent at all. What it does is mix **channels**: it is a per-pixel linear map from
$C_{\text{in}}$ channels to $C_{\text{out}}$, applied identically everywhere. That
makes it a cheap dimensionality knob on the channel axis.[^gf-tensor]

> **Definition ($1\times1$ convolution).** A convolution with a $1\times1$ kernel:
> for each spatial position it computes $C_{\text{out}}$ linear combinations of the
> $C_{\text{in}}$ input channels, sharing the same $C_{\text{in}}\times
> C_{\text{out}}$ weight matrix across all positions. It changes channel depth
> without touching spatial resolution: a learned, per-pixel channel projection.

The payoff is the **bottleneck**: squeeze channels down with a $1\times1$, do the
expensive $3\times3$ convolution in the reduced space, then expand back. Count the
multiply-adds on an $H\times W$ map to see why it cuts compute.

$$
\underbrace{H W\,(256 \cdot 3 \cdot 3 \cdot 256)}_{\text{plain }3\times3:\ 589\text{K}\,HW}
\;\gg\;
\underbrace{H W\,(256 \cdot 64 + 64 \cdot 3 \cdot 3 \cdot 64 + 64 \cdot 256)}_{\text{bottleneck } 1\times1,\,3\times3,\,1\times1:\ 70\text{K}\,HW}.
$$

The $3\times3$ — the costly part, quadratic in channels — runs on $64$ channels
instead of $256$, an $8\times$ saving on the dominant term. The two $1\times1$ caps
restore the width for free by comparison.

$$
% caption: The bottleneck block: a $1\times1$ squeezes $256$ channels to $64$, the $3\times3$ runs in the reduced space, and a $1\times1$ expands back to $256$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  blk/.style={draw, minimum height=10mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{acclo}{HTML}{A7B5FB}
  \node[blk, fill=acclo!20, minimum width=22mm] (in) at (0,0) {in: 256 ch};
  \node[blk, fill=black!8, minimum width=22mm] (sq) at (2.9,0) {1x1\\squeeze to 64};
  \node[blk, fill=acc!18, minimum width=22mm] (cv) at (5.8,0) {3x3\\on 64 ch};
  \node[blk, fill=black!8, minimum width=22mm] (ex) at (8.7,0) {1x1\\expand to 256};
  \node[blk, draw=acc, text=acc, thick, minimum width=20mm] (out) at (11.5,0) {out: 256};
  \draw[->, thick] (in) -- (sq);
  \draw[->, thick] (sq) -- (cv);
  \draw[->, thick] (cv) -- (ex);
  \draw[->, acc, thick] (ex) -- (out);
  % width annotation
  \draw[black, thick] (4.7,-1.1) -- (6.9,-1.1);
  \node[black, anchor=north, font=\footnotesize] at (5.8,-1.1) {\texttt{narrow waist}};
\end{tikzpicture}
$$

### Inception: multi-scale in one module

GoogLeNet's Inception module takes a different approach: rather than commit to one
filter size per layer, run several in **parallel** — $1\times1$, $3\times3$,
$5\times5$, and a pooling branch — and concatenate their outputs. Each branch is
prefaced by a $1\times1$ bottleneck so the parallel paths stay cheap. The network
learns, per layer, how much of each scale to use.

$$
% caption: A VGG stack (one filter size, in series) versus an Inception module (several scales in parallel, then concatenated). VGG picks a scale; Inception offers all at once.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  b/.style={draw, minimum width=15mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{acclo}{HTML}{A7B5FB}
  % --- VGG (left): serial 3x3 ---
  \node[b, fill=acclo!25] (v1) at (0,0)   {3x3};
  \node[b, fill=acclo!25] (v2) at (0,1.3) {3x3};
  \node[b, fill=acclo!25] (v3) at (0,2.6) {3x3};
  \draw[->, thick] (v1) -- (v2); \draw[->, thick] (v2) -- (v3);
  \node[anchor=north, font=\footnotesize] at (0,-0.65) {\texttt{VGG: serial}};
  % --- Inception (right): parallel ---
  \begin{scope}[xshift=5.8cm]
    \node[b, fill=black!8] (in) at (0,-0.2) {input};
    \node[b, fill=acc!16]   (p1) at (-2.4,1.4) {1x1};
    \node[b, fill=acc!16]   (p2) at (-0.8,1.4) {3x3};
    \node[b, fill=acc!16]   (p3) at (0.8,1.4)  {5x5};
    \node[b, fill=acc!16]   (p4) at (2.4,1.4)  {pool};
    \node[b, draw=acc, text=acc, thick] (cc) at (0,2.9) {concat};
    \draw[->, thick] (in) -- (p1); \draw[->, thick] (in) -- (p2);
    \draw[->, thick] (in) -- (p3); \draw[->, thick] (in) -- (p4);
    \draw[->, acc, thick] (p1) -- (cc); \draw[->, acc, thick] (p2) -- (cc);
    \draw[->, acc, thick] (p3) -- (cc); \draw[->, acc, thick] (p4) -- (cc);
    \node[anchor=north, font=\footnotesize] at (0,-0.85) {\texttt{Inception: parallel scales}};
  \end{scope}
\end{tikzpicture}
$$

Inception's $5$M parameters (against VGG's $138$M at comparable accuracy) come
almost entirely from the $1\times1$ bottlenecks pruning each branch before the
expensive convolution.

## DenseNet: dense connectivity

ResNet adds the skip; DenseNet generalizes it. Each layer receives, by
**concatenation**, the feature maps of _all_ preceding layers in its block:

$$
x_\ell = H_\ell\parens{ [\,x_0, x_1, \dots, x_{\ell-1}\,] },
$$

where $[\cdot]$ is channel-wise concatenation. Where ResNet _sums_ a single skip,
DenseNet _stacks_ every prior output, giving each layer direct access to the raw
features below it and to the loss gradient above. Because features are reused
rather than relearned, each layer adds only a few channels (the **growth rate**
$k$), so a $121$-layer DenseNet carries just $8$M parameters.

| Connectivity | Combine rule | Skip to layer $\ell$ | Params |
| --- | --- | --- | --- |
| Plain | $x_\ell = H_\ell(x_{\ell-1})$ | none | high |
| Residual | $x_\ell = H_\ell(x_{\ell-1}) + x_{\ell-1}$ | sum, one back | medium |
| Dense | $x_\ell = H_\ell([x_0,\dots,x_{\ell-1}])$ | concat, all back | low |

## The accuracy/compute frontier

Depthwise-separable convolution, MobileNet's core, pushes the bottleneck idea to
its limit. It factorizes a standard convolution into a **depthwise** step (one
$3\times3$ filter _per channel_, no cross-channel mixing) followed by a $1\times1$ **pointwise**
step (mix channels, no spatial extent). The standard conv does both
at once and pays $D_K^2 \cdot C_{\text{in}} \cdot C_{\text{out}}$ per pixel; the
factorized version pays only

$$
\frac{D_K^2 \cdot C_{\text{in}} + C_{\text{in}} \cdot C_{\text{out}}}{D_K^2 \cdot C_{\text{in}} \cdot C_{\text{out}}}
= \frac{1}{C_{\text{out}}} + \frac{1}{D_K^2}
$$

of the cost, roughly $\tfrac19$ for a $3\times3$ kernel, at a small accuracy
penalty. This is the **accuracy/compute frontier**: the same recognition quality at
a fraction of the multiply-adds, which is what makes CNNs run on phones.

$$
% caption: ImageNet top-1 accuracy climbs across the landmark models as depth grows, from AlexNet through VGG and Inception to ResNet.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % axes
  \draw[->, thick] (0,0) -- (8.6,0) node[right, font=\scriptsize] {year};
  \draw[->, thick] (0,0) -- (0,4.3) node[above, font=\scriptsize] {top-1 accuracy};
  % y gridlines
  \foreach \y/\lab in {1/60, 2/70, 3/80} {
    \draw[black] (0,\y) -- (8.4,\y);
    \node[black, anchor=east, font=\scriptsize] at (-0.1,\y) {\lab};
  }
  % bars: AlexNet 2012 (63), VGG 2014 (71), Inception 2014 (72), ResNet 2015 (76)
  \draw[acc, thick, fill=acc!15] (0.8,0) rectangle (1.8,1.3);
  \node[anchor=north, font=\footnotesize] at (1.3,-0.1) {\texttt{AlexNet}};
  \draw[acc, thick, fill=acc!15] (2.6,0) rectangle (3.6,2.1);
  \node[anchor=north, font=\footnotesize] at (3.1,-0.1) {\texttt{VGG}};
  \draw[acc, thick, fill=acc!15] (4.4,0) rectangle (5.4,2.2);
  \node[anchor=north, font=\footnotesize] at (4.9,-0.1) {\texttt{Inception}};
  \draw[acc, thick, fill=acc!15] (6.2,0) rectangle (7.2,2.6);
  \node[anchor=north, font=\footnotesize] at (6.7,-0.1) {\texttt{ResNet}};
  % trend
  \draw[green, very thick, ->] (1.3,1.45) -- (6.7,2.75);
  \definecolor{acclo}{HTML}{A7B5FB}
\end{tikzpicture}
$$

## Pushing the CNN frontier

The six landmarks end with ResNet and DenseNet (2015–2017); the standard references stop
there. Two later results changed how the frontier is pushed and are worth naming.

- **Compound scaling (EfficientNet).** VGG-to-ResNet progress came from scaling one
  axis at a time — deeper (VGG), then a better block (ResNet). Tan & Le
  ("EfficientNet," ICML 2019) showed the axes should move _together_: depth $d$,
  width $w$, and input resolution $r$ scaled by a shared coefficient $\phi$ under the
  constraint $d^\alpha w^\beta r^\gamma \approx 2$, with $\alpha, \beta, \gamma$
  found by a small grid search. For example, doubling the compute budget
  ($\phi \to \phi+1$) is best spent as roughly $1.2\times$ depth, $1.1\times$ width,
  and $1.15\times$ resolution together, not as a $2\times$ on any single axis.
  EfficientNet-B7 matched the then-best accuracy with $8.4\times$ fewer parameters
  than the previous record, by scaling in balance rather than blindly deepening.
- **CNNs after the Transformer (ConvNeXt).** When the
  [Vision Transformer](/deep-learning/architectures/transformers-in-practice)
  overtook CNNs, it was unclear whether attention or merely a decade of better
  training recipes was responsible. Liu et al. ("A ConvNet for the 2020s," CVPR
  2022) modernized a plain ResNet one change at a time — larger kernels, fewer
  activations, LayerNorm, the AdamW schedule — until a pure convolutional network
  (ConvNeXt) matched a comparable ViT. The result is a useful control: much of the
  ViT's early advantage was training protocol, not the attention block itself.

The residual skip these landmarks introduced is the one idea that survived intact
into every architecture since — it is the same $+1$ path that stabilizes the deep
[Transformer stack](/deep-learning/architectures/the-transformer-architecture).

## Takeaways

- The six landmarks each isolate **one idea**: LeNet's conv-pool stack, AlexNet's
  ReLU-dropout-GPU scale, VGG's uniform $3\times3$ depth, Inception's parallel
  multi-scale module, ResNet's residual skip, DenseNet's dense concatenation.
- After VGG, **depth and parameter count decoupled** — Inception and ResNet are far
  deeper than VGG with an order of magnitude fewer weights.
- **Degradation** is an optimization failure, not overfitting: plain nets past
  $\sim20$ layers reach higher _training_ error because a stack of nonlinearities
  cannot easily represent the identity.
- **ResNet** reparameterizes the block to learn the residual $F(x) = H(x) - x$, so
  $H(x) = F(x) + x$; the identity skip injects a $+1$ into the backward Jacobian,
  giving the gradient an additive path that never vanishes across depth.
- **$1\times1$ convolutions** mix channels at zero spatial cost; squeeze-then-expand
  **bottlenecks** run the costly $3\times3$ in a reduced channel space, cutting
  compute by $\sim8\times$.
- **Depthwise-separable** convolution factorizes spatial and channel mixing for a
  $\sim9\times$ saving, defining the **accuracy/compute frontier** that puts CNNs on
  mobile hardware.

[^gf-history]: **Goodfellow**, _Deep Learning_, §9.11 — The Neuroscientific Basis / historical notes: LeNet and the lineage of conv-pool stacks that every modern vision backbone inherits.
[^chollet-vision]: **Chollet**, _Deep Learning with Python_, Ch. 5 — Deep Learning for Computer Vision: ReLU, dropout, and data augmentation as the practical tricks that made deep convnets trainable at ImageNet scale.
[^gf-depth]: **Goodfellow**, _Deep Learning_, §8.2 — Challenges in Neural Network Optimization: why adding depth can raise _training_ error, framing degradation as an optimization failure rather than overfitting.
[^gf-residual]: **Goodfellow**, _Deep Learning_, §8.7.5 — skip/residual connections and §9.x design: re-parameterizing a block to learn $F(x)=H(x)-x$ so the identity is free.
[^gf-tensor]: **Goodfellow**, _Deep Learning_, §9.5 — Convolution and Pooling as an Infinitely Strong Prior / variants: $1\times1$ convolutions as a per-pixel channel projection and the bottleneck they enable.