CNN Architectures

The convolutional layer fixes the primitive (local, weight-shared, translation-equivariant filtering), but it does not fix the wiring. How many filters, how deep, connected how? The answer came from a decade of architectures, each one isolating a single structural idea and pushing it until it broke. This lesson is that sequence: six landmark networks, the one idea each contributed, and the degradation problem that the most important of them solved.

The landmark architectures

Every modern vision backbone is a recombination of six prior moves. Read the table as a list of contributions, not models; each row is one idea that survived.

Two numbers in that table summarize the trend. Depth climbs from $7$ to $121$; parameter count peaks at VGG's $138$M and then falls — Inception and ResNet are an order of magnitude deeper than VGG yet carry far fewer weights. Depth and parameter count decoupled, and every architecture after VGG is a way of adding depth without adding parameters.

LeNet-5: the template

LeNet-5 set the pattern every later network inherits: alternate convolution (detect local features) with subsampling/pooling (shrink spatial extent, grow receptive field), then finish with a few dense layers. Spatial resolution falls while channel depth rises: features get coarser and more abstract toward the output.¹

AlexNet: scale, ReLU, dropout

AlexNet is LeNet, larger, trained on two GPUs over ImageNet's $1.2$M images. Its contribution is the three choices that made depth trainable at scale: the ReLU nonlinearity, which does not saturate and so keeps gradients alive; aggressive dropout in the dense layers to fight overfitting; and GPU implementation to make the compute feasible.²

Where the $60$M parameters actually sit is worth tracing, because it explains every design move that follows. AlexNet takes a $224\times 224\times 3$ image through five convolutional stages and three dense layers. The convolutions are cheap in weights but expensive in activations; the dense layers are the reverse.

The first dense layer alone, fc6, holds $6\cdot 6\cdot 256\times 4096=37.7$M weights — $62\%$ of the whole network — because it flattens the entire final feature map and connects it fully to $4096$ units. That single number is why later architectures replace the flatten-then-dense head with global average pooling: collapsing the $6\times 6\times 256$ map to a $256$-vector by averaging removes the $37.7$M-weight layer outright. Inception and ResNet both do exactly this, which is a large part of why they carry an order of magnitude fewer parameters than VGG.

VGG: depth through uniformity

VGG replaces every large filter with a stack of $3\times 3$ convolutions, and proves the substitution is strictly better: two stacked $3\times 3$ layers see the same $5\times 5$ receptive field as one $5\times 5$ layer, but with fewer parameters and an extra nonlinearity in between.

The uniformity yields more than a parameter saving. Because every stage keeps the same $3\times 3$ kernel and halves the spatial resolution at each pooling step, VGG-16 is a single rule applied five times: two-or-three $3\times 3$ convolutions, then a $2\times 2$ pool that halves $H$ and $W$ and doubles the channel count. The tensor shape follows a clean geometric progression.

Resolution falls $224\to 7$ (a $32\times$ reduction, five halvings) while channels climb $3\to 512$. By block 5 a single unit's receptive field ($196\times 196$) covers nearly the whole $224\times 224$ input, so the final feature map sees the entire image at once. Each pooling step both grows that field and, by doubling $C$, keeps the per-layer compute roughly constant as the map shrinks.

VGG pushed this to $16$–$19$ layers and won by depth alone — but at $138$M parameters, most of them in the dense layers, it is the heaviest model in the table. It also exposed a limit: stacking past $\sim 20$ plain layers stopped helping.

The degradation problem

The natural hypothesis — deeper is at least as good, because the extra layers can always learn the identity — is false in practice. Plain stacked networks past a depth threshold train to higher error than their shallower counterparts. This is not overfitting: the training error itself is worse, so the failure lies in optimization, not in the generalization gap.³

The diagnosis is that the identity map is hard to represent with a stack of nonlinear layers. Producing $H(x)=x$ from a conv-ReLU-conv block requires the weights to combine into an exact identity, a thin target in weight space that gradient descent does not reliably reach.

ResNet: learn the residual

ResNet's fix is a reparameterization. Instead of asking a block to compute the target map $H(x)$ directly, ask it to compute the residual $F(x)=H(x)-x$, and add the input back via an identity skip:

Now the identity is free: if the optimal block is the identity, the network only needs $F(x)\to 0$, driving a stack of weights to zero, which weight decay does anyway, rather than constructing an exact identity from nonlinearities.⁴

Why the skip keeps gradients alive

The skip's deeper payoff is on the backward pass. Consider a block $y=x+F(x,W)$. The Jacobian of the output with respect to the input is

Chaining $L$ such blocks from layer $\ell$ to the loss $\mathcal{L}$ at the top, the gradient flowing back to $x_{\ell }$ is, by the chain rule,

Expanding the product, the leading term is the bare $\frac{\partial \mathcal{L}}{\partial x_L}$ — the gradient reaches the early layer undiminished, along a path of all-identity factors. The deep multiplicative product of $\partial F_i/\partial x_i$ Jacobians that vanishes in a plain net is now an additive correction on top of a guaranteed $+1$ path.

For a concrete number, suppose each block's Jacobian $\partial F_i/\partial x_i$ has spectral norm $\approx 0.8$, a modest per-layer contraction. In a plain net the signal is the product of these factors: across $50$ blocks the gradient shrinks by $0.8^{50}\approx 1.4\times 10^{-5}$, five orders of magnitude, so the early layers get almost no signal and stop learning. In a residual net the same $50$ blocks contribute $\prod (I+\partial F_i/\partial x_i)$, whose expansion keeps the bare identity term at magnitude $1$ no matter how many factors multiply. The gradient at the first block is $1+(\text{small corrections})$, not $10^{-5}$. That is the entire difference between a net that trains at depth $50$ and one that does not.

This is the same mechanism as the LSTM's cell state in recurrent networks: a near-identity path along which the gradient propagates without repeated multiplication. With it, ResNet trained $152$ layers where plain nets degraded past $20$.

ResNet-50 stacks the bottleneck block (below) into four stages, halving resolution and doubling width at each stage boundary. The tensor shape follows the same ladder VGG used, but the flatten-then-dense head is gone: a global average pool collapses the final $7\times 7\times 2048$ map to a $2048$-vector, so the classifier is a single $2048\times 1000$ layer instead of VGG's $100$M-weight dense stack.

Fifty weight layers, $25$M parameters: a fifth of VGG's count at more than three times the depth. The bottleneck (next section) is what keeps each of those $16$ stacked blocks cheap.

1×1 convolutions and the bottleneck

A $1\times 1$ convolution touches one spatial location at a time; it has no spatial extent at all. What it does is mix channels: it is a per-pixel linear map from $C_{\text{in}}$ channels to $C_{\text{out}}$, applied identically everywhere. That makes it a cheap dimensionality knob on the channel axis.⁵

The payoff is the bottleneck: squeeze channels down with a $1\times 1$, do the expensive $3\times 3$ convolution in the reduced space, then expand back. Count the multiply-adds on an $H\times W$ map to see why it cuts compute.

The $3\times 3$ — the costly part, quadratic in channels — runs on $64$ channels instead of $256$, an $8\times$ saving on the dominant term. The two $1\times 1$ caps restore the width for free by comparison.

Inception: multi-scale in one module

GoogLeNet's Inception module takes a different approach: rather than commit to one filter size per layer, run several in parallel — $1\times 1$, $3\times 3$, $5\times 5$, and a pooling branch — and concatenate their outputs. Each branch is prefaced by a $1\times 1$ bottleneck so the parallel paths stay cheap. The network learns, per layer, how much of each scale to use.

Inception's $5$M parameters (against VGG's $138$M at comparable accuracy) come almost entirely from the $1\times 1$ bottlenecks pruning each branch before the expensive convolution.

DenseNet: dense connectivity

ResNet adds the skip; DenseNet generalizes it. Each layer receives, by concatenation, the feature maps of all preceding layers in its block:

where $[\cdot ]$ is channel-wise concatenation. Where ResNet sums a single skip, DenseNet stacks every prior output, giving each layer direct access to the raw features below it and to the loss gradient above. Because features are reused rather than relearned, each layer adds only a few channels (the growth rate$k$), so a $121$-layer DenseNet carries just $8$M parameters.

The accuracy/compute frontier

Depthwise-separable convolution, MobileNet's core, pushes the bottleneck idea to its limit. It factorizes a standard convolution into a depthwise step (one $3\times 3$ filter per channel, no cross-channel mixing) followed by a $1\times 1$ pointwise step (mix channels, no spatial extent). The standard conv does both at once and pays $D_K^2\cdot C_{\text{in}}\cdot C_{\text{out}}$ per pixel; the factorized version pays only

of the cost, roughly $\frac{1}{9}$ for a $3\times 3$ kernel, at a small accuracy penalty. This is the accuracy/compute frontier: the same recognition quality at a fraction of the multiply-adds, which is what makes CNNs run on phones.

Pushing the CNN frontier

The six landmarks end with ResNet and DenseNet (2015–2017); the standard references stop there. Two later results changed how the frontier is pushed and are worth naming.

• Compound scaling (EfficientNet). VGG-to-ResNet progress came from scaling one axis at a time — deeper (VGG), then a better block (ResNet). Tan & Le (EfficientNet, ICML 2019) showed the axes should move together: depth $d$, width $w$, and input resolution $r$ scaled by a shared coefficient $\varphi$ under the constraint $d^{\alpha }{w}^{\beta }{r}^{\gamma }\approx 2$, with $\alpha ,\beta ,\gamma$ found by a small grid search. For example, doubling the compute budget ($\varphi \to \varphi +1$) is best spent as roughly $1.2\times$ depth, $1.1\times$ width, and $1.15\times$ resolution together, not as a $2\times$ on any single axis. EfficientNet-B7 matched the then-best accuracy with $8.4\times$ fewer parameters than the previous record, by scaling in balance rather than blindly deepening.
• CNNs after the Transformer (ConvNeXt). When the Vision Transformer overtook CNNs, it was unclear whether attention or merely a decade of better training recipes was responsible. Liu et al. (A ConvNet for the 2020s, CVPR 2022) modernized a plain ResNet one change at a time — larger kernels, fewer activations, LayerNorm, the AdamW schedule — until a pure convolutional network (ConvNeXt) matched a comparable ViT. The result is a useful control: much of the ViT's early advantage was training protocol, not the attention block itself.

The residual skip these landmarks introduced is the one idea that survived intact into every architecture since — it is the same $+1$ path that stabilizes the deep Transformer stack.

Takeaways

• The six landmarks each isolate one idea: LeNet's conv-pool stack, AlexNet's ReLU-dropout-GPU scale, VGG's uniform $3\times 3$ depth, Inception's parallel multi-scale module, ResNet's residual skip, DenseNet's dense concatenation.
• After VGG, depth and parameter count decoupled — Inception and ResNet are far deeper than VGG with an order of magnitude fewer weights.
• Degradation is an optimization failure, not overfitting: plain nets past $\sim 20$ layers reach higher training error because a stack of nonlinearities cannot easily represent the identity.
• ResNet reparameterizes the block to learn the residual $F(x)=H(x)-x$, so $H(x)=F(x)+x$; the identity skip injects a $+1$ into the backward Jacobian, giving the gradient an additive path that never vanishes across depth.
• $1\times 1$ convolutions mix channels at zero spatial cost; squeeze-then-expand bottlenecks run the costly $3\times 3$ in a reduced channel space, cutting compute by $\sim 8\times$.
• Depthwise-separable convolution factorizes spatial and channel mixing for a $\sim 9\times$ saving, defining the accuracy/compute frontier that puts CNNs on mobile hardware.