Convolutional Networks

A dense layer treats an image as a flat vector and connects every input pixel to every output unit. For a $224\times 224\times 3$ image that is $150,528$ inputs; a single dense layer of $1000$ units carries $1.5\times 10^8$ weights, and learns nothing about the fact that a cat shifted three pixels right is still a cat. A convolutional network discards that wasteful matrix. It connects each output to a small local patch of the input and reuses one tiny set of weights, a kernel, at every position. The cat-detector learned at the top-left corner is, for free, a cat-detector everywhere.

The convolution operation

The discrete 2D convolution slides a kernel $K$ of size $k_h\times k_w$ across an input $I$ and, at each position, takes the sum of element-wise products of the kernel with the underlying patch. Writing $(i,j)$ for the output position:

Each output value is one inner product between the kernel and a window of the input. The kernel is the layer's only learnable object; sliding it produces a feature map $S$, one scalar per spatial position.¹

The canonical picture: place the kernel over a patch, multiply, sum, write one cell of the feature map, then step over by the stride and repeat.

Channels

Real inputs are not flat grids but stacks of them: an RGB image is $H\times W\times 3$. A kernel spans all input channels, so a kernel for a $C_{\text{in}}$- channel input has shape $k_h\times k_w\times C_{\text{in}}$ and produces a single feature map by summing over channels too:

A convolutional layer learns $C_{\text{out}}$ such kernels, stacking their feature maps into an output volume of depth $C_{\text{out}}$. The layer's full weight tensor is $k_h\times k_w\times C_{\text{in}}\times C_{\text{out}}$ — note it is independent of the input's spatial size $H,W$.

For a worked shape trace, take an RGB input of $32\times 32\times 3$ and a conv layer with $C_{\text{out}}=16$ kernels of size $3\times 3$, same padding ($p=1$), stride $1$. Each kernel has shape $3\times 3\times 3=27$ weights plus one bias, so the layer's weight tensor is $3\times 3\times 3\times 16=432$ weights and $16$ biases, $448$ parameters total. The output geometry from $n_{\text{out}}=\lfloor (32+2-3)/1\rfloor +1=32$ per spatial axis gives an output volume of $32\times 32\times 16$. Every one of those $32\cdot 32\cdot 16=16,384$ output scalars is an inner product over $3\cdot 3\cdot 3=27$ input values, so the layer costs $16,384\cdot 27\approx 4.4\times 10^5$ multiply-adds. The single fact to carry: the parameter count $k_h{k}_w{C}_{\text{in}}{C}_{\text{out}}$ does not mention $H$ or $W$, so the same $448$ weights process a $32\times 32$ image or a $512\times 512$ one; only the compute scales with resolution.

Three ideas that beat the dense layer

Convolution wins for images because of three structural commitments, each cutting a different cost the dense layer pays.²

Sparse connectivity

A dense layer connects every output to every input: $m$ inputs and $n$ outputs cost $m\cdot n$ weights and $O(m\cdot n)$ multiply-adds. A conv layer connects each output only to a $k_h\times k_w$ patch, so each output unit has just $k_h{k}_w$ incoming weights regardless of input size.

Parameter sharing

The deeper saving is parameter reuse: the conv layer uses the same weights at every position. The dense layer learns an independent weight for every (input, output) pair; the conv layer learns one kernel and applies it everywhere. The weights are tied across space.

One kernel slides across the input and writes each feature-map cell. Every output position is computed by the same three weights; the network stores one filter, not one weight per location.

Take a $32\times 32\times 1$ input mapped to a $32\times 32\times 1$ output, comparing a dense layer to a $3\times 3$ conv:

The conv layer carries $9$ weights against the dense layer's million, a $10^5\times$ reduction, and the gap widens with input size, since the dense count grows as $m\cdot n$ while the kernel stays fixed at $k_h{k}_w$.

Translation equivariance

Parameter sharing gives a structural property for free: shifting the input shifts the output the same way. Let $T_{\delta }$ be a translation by $\delta$ pixels. Then

so detecting a feature and shifting commute. The kernel responds identically to a pattern no matter where it sits — exactly the right prior for images, where an object's identity is independent of its position.³

Equivariance is not invariance: the output moves with the input. Pooling, below, converts a small amount of this equivariance into outright invariance.

Convolution arithmetic

The kernel cannot be centered on the boundary pixels, so a valid convolution shrinks the output. Two knobs control the output geometry: the stride $s$ (how far the kernel hops between positions) and the padding $p$ (how many zero rows and columns ring the input).

Deriving the output size

Consider one spatial dimension of size $n$, padded to $n+2p$. A kernel of width $k$ has its leftmost valid position at index $0$ and its rightmost when its right edge hits the padded boundary, i.e. at start index $n+2p-k$. Stepping by $s$, the valid start positions are $0,s,2s,\dots$ up to $n+2p-k$, so their count is

The floor handles a final partial step that does not fit; the $+1$ counts the starting position itself. Several special cases recover familiar layers:

A worked instance: $n=7$, $k=3$, $p=1$, $s=2$ gives $\lfloor (7+2-3)/2\rfloor +1=\lfloor 6/2\rfloor +1=4$.

Pooling

A pooling layer summarizes each small region of a feature map by a single statistic (its maximum for max pooling, or mean for average pooling) over a window, typically with stride equal to the window so the regions tile without overlap. Pooling has no learnable weights; it exists to downsample and to add local invariance.⁴

The invariance is the point. If the strongest activation shifts by one position but stays inside the same pooling window, the max is unchanged — the output is locally invariant to small translations, not merely equivariant. Max pooling keeps the strongest evidence; average pooling keeps the typical level.

The receptive field

The receptive field of a unit is the region of the original input that can influence it. A single $3\times 3$ conv sees a $3\times 3$ patch; stack another on top and each of its units sees a $3\times 3$ patch of the layer below, which already each saw $3\times 3$ of the input, so the second layer's units see $5\times 5$ of the input. The field grows linearly with depth for stride-$1$ convs.

Stride and pooling accelerate the growth: a stride-$s$ layer multiplies the spacing, so receptive fields expand geometrically rather than linearly once downsampling is interleaved. This is why deep stacks of small $3\times 3$ convs can, with a few pooling steps, end up seeing the entire image — the late layers have receptive fields spanning the input, which is what lets them represent whole objects.

The convolution algorithm

The forward pass of a single convolutional layer is a direct transcription of the definition: for each output channel, slide that channel's kernel over the padded input, summing products across the window and the input channels.

The five nested loops make the cost explicit: $C_{\text{out}}\cdot H^{\prime }{W}^{\prime }\cdot C_{\text{in}}{k}_h{k}_w$ multiply-adds. Optimized implementations recast the inner loops as a single matrix multiply (im2col), but the arithmetic is unchanged.⁵

A typical convolutional stack

A convolutional network alternates convolution, a nonlinear activation, and pooling, each block shrinking spatial size while growing channel depth, then flattens and hands off to a dense classifier head.⁶

The geometry follows a consistent pattern: each conv block enlarges receptive fields and adds channels (more pattern detectors), while pooling discards spatial resolution that is no longer needed. By the flatten, each unit summarizes a large region through a deep feature, and a small dense head suffices to read off the class.

A worked shape and parameter trace

Instantiate the stack with real numbers on a $32\times 32\times 3$ input, two conv-pool blocks with same-padded $3\times 3$ convs ($C_1=32$, $C_2=64$) and $2\times 2$ stride-$2$ pooling, then a dense head to $10$ classes. Following one tensor through, with the parameter count $k_h{k}_w{C}_{\text{in}}{C}_{\text{out}}+C_{\text{out}}$ at each conv:

Two observations fall out of the numbers. First, spatial size halves at every pool ($32\to 16\to 8$) while depth grows ($3\to 32\to 64$): the network trades resolution for richer per-position features. Second, the convolutional feature extractor holds $896+18,496=19,392$ parameters against the dense head's $40,970$ — most of the weights live in that final flatten-to-dense step, which is exactly why the flatten happens only after pooling has shrunk the map to $8\times 8$; flattening the $32\times 32\times 32$ map straight after the first conv would feed $32,768$ features into the dense layer and inflate that head by a factor of eight.

Convolution variants in production

Goodfellow, Chollet, and Stevens stop at the standard dense convolution; two variants that reshape the cost and the receptive field are now everywhere in production vision models.

• Depthwise-separable convolution. A standard $k\times k$ layer with $C_{\text{in}}$ input and $C_{\text{out}}$ output channels costs $k^2{C}_{\text{in}}{C}_{\text{out}}$ multiply-adds per output position. Split it into a depthwise step (one $k\times k$ filter per input channel, no channel mixing) followed by a pointwise$1\times 1$ step (mixing channels, no spatial extent), and the cost drops to $k^2{C}_{\text{in}}+C_{\text{in}}{C}_{\text{out}}$. For a $3\times 3$ layer with $C_{\text{in}}=C_{\text{out}}=256$, that is $9\cdot 256\cdot 256\approx 590\text{k}$ versus $9\cdot 256+256\cdot 256\approx 68\text{k}$, an $8.6\times$ reduction, with almost no accuracy loss. This factorization underlies MobileNet (Howard et al., 2017) and Xception (Chollet, 2017), the standard for on-device vision.
• Dilated (atrous) convolution. Insert gaps of size $r-1$ between kernel taps, so a $3\times 3$ kernel covers a $(2r+1)\times (2r+1)$ span while touching only $9$ weights. The receptive field then grows exponentially with depth if $r$ doubles each layer, instead of the linear $1+{\sum }_{\ell }(k_{\ell }-1)$ derived above, which is why dilation dominates dense-prediction tasks like segmentation (Yu & Koltun, Multi-Scale Context Aggregation by Dilated Convolutions, ICLR 2016) and audio generation (WaveNet, van den Oord et al., 2016).

The Vision Transformer drops this same locality prior that the three ideas encode, trading the built-in bias for data; the convolution and the attention block are two ends of one design axis.

Takeaways

• A convolution $(I\ast K)[i,j]={\sum }_{m,n}I[i+m,j+n]K[m,n]$ slides a small learnable kernel over the input; each kernel yields one feature map, and a layer stacks $C_{\text{out}}$ of them across $C_{\text{in}}$ input channels.
• Three commitments beat the dense layer: sparse connectivity (each output sees only a $k_h{k}_w$ patch), parameter sharing (one kernel reused everywhere), and translation equivariance ($(T_{\delta }I)\ast K=T_{\delta }(I\ast K)$) — together collapsing a million weights to nine for a $3\times 3$ kernel.
• Output geometry obeys $n_{\text{out}}=\lfloor (n+2p-k)/s\rfloor +1$; same padding $p=(k-1)/2$ with $s=1$ preserves size, stride $s>1$ downsamples.
• Pooling (max or average) is parameter-free downsampling that turns local equivariance into local invariance; the receptive field grows as $1+{\sum }_{\ell }(k_{\ell }-1)$ with depth, so deep stacks eventually see the whole image.
• The canonical stack — conv → activation → pool → … → flatten → dense — deepens channels while shrinking space, trained end-to-end, and is the backbone the next lesson's architectures refine.