---
title: Convolutional Networks
module: Architectures
moduleNumber: 5
lessonNumber: 1
order: 501
summary: >
  A convolutional network replaces the dense layer's all-to-all weight matrix
  with a small kernel slid across the input. Three structural commitments
  (sparse connectivity, parameter sharing, and translation equivariance) collapse
  the parameter count by orders of magnitude and bake the right prior for images
  directly into the architecture. We derive the convolution arithmetic, the output
  geometry, pooling, and the receptive field, then assemble the canonical stack.
topics: [Architectures]
sources:
  - book: Goodfellow
    ref: "Ch. 9 — Convolutional Networks"
  - book: Chollet
    ref: "Ch. 5 — Deep Learning for Computer Vision"
  - book: Stevens
    ref: "Ch. 8 — Using Convolutions to Generalize"
---

A [dense layer](/deep-learning/neural-networks/the-multilayer-perceptron) treats
an image as a flat vector and connects every input pixel to every output unit. For
a $224 \times 224 \times 3$ image that is $150{,}528$ inputs; a single dense layer
of $1000$ units carries $1.5 \times 10^{8}$ weights, and learns nothing about the
fact that a cat shifted three pixels right is still a cat. A **convolutional
network** discards that wasteful matrix. It connects each output to a small _local_
patch of the input and reuses one tiny set of weights, a **kernel**, at every
position. The cat-detector learned at the top-left corner is, for free, a
cat-detector everywhere.

## The convolution operation

The discrete 2D convolution slides a kernel $K$ of size $k_h \times k_w$ across an
input $I$ and, at each position, takes the sum of element-wise products of the
kernel with the underlying patch. Writing $(i,j)$ for the output position:

$$
S[i, j] \;=\; (I \ast K)[i, j] \;=\; \sum_{m}\sum_{n} I[\,i + m,\; j + n\,]\,K[m, n].
$$

Each output value is one inner product between the kernel and a window of the
input. The kernel is the layer's only learnable object; sliding it produces a
**feature map** $S$, one scalar per spatial position.[^gf-conv]

> **Definition (Kernel / filter).** A small learnable array $K \in \mathbb{R}^{k_h
> \times k_w}$ (extended over input channels) whose weights are shared across all
> spatial positions. One kernel detects one local pattern (an edge, a corner, a
> texture) wherever it occurs in the input.

> **Definition (Feature map).** The output array $S = I \ast K$ produced by convolving
> one kernel over the whole input. Cell $S[i,j]$ measures how strongly the kernel's
> pattern is present in the input patch anchored at $(i,j)$.

The canonical picture: place the kernel over a patch, multiply, sum, write one cell
of the feature map, then step over by the stride and repeat.

$$
% caption: One step of a 2D convolution: the kernel $K$ overlays a $3\times 3$ patch of input $I$, and the products sum into a single output cell.
\begin{tikzpicture}[font=\small, >=stealth]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{acclo}{HTML}{A7B5FB}
  \definecolor{green}{HTML}{1F9D4D}
  % --- input grid 5x5 ---
  \def\cw{0.62}
  \foreach \r in {0,...,4}
    \foreach \c in {0,...,4}
      \node[draw, black, minimum size=\cw cm, inner sep=0pt] (I\r\c) at (\c*\cw, -\r*\cw) {};
  % highlight 3x3 receptive patch (rows 0-2, cols 0-2)
  \foreach \r in {0,1,2}
    \foreach \c in {0,1,2}
      \node[draw, acc, very thick, fill=acclo!35, minimum size=\cw cm, inner sep=0pt] at (\c*\cw, -\r*\cw) {};
  % patch values
  \foreach \r/\c/\v in {0/0/3,0/1/1,0/2/2,1/0/0,1/1/4,1/2/1,2/0/2,2/1/2,2/2/0}
    \node[font=\scriptsize] at (\c*\cw, -\r*\cw) {\v};
  \node[anchor=south, font=\footnotesize] at (1.24,0.55) {input $I$};
  % --- kernel 3x3 ---
  \def\kx{4.6}
  \foreach \r/\c/\v in {0/0/1,0/1/0,0/2/1,1/0/0,1/1/1,1/2/0,2/0/1,2/1/0,2/2/1}
    \node[draw, green, very thick, minimum size=\cw cm, inner sep=0pt, font=\scriptsize, text=green] at (\kx + \c*\cw, -\r*\cw) {\v};
  \node[anchor=south, font=\footnotesize, text=green] at (\kx + \cw, 0.55) {kernel $K$};
  % --- output cell ---
  \def\ox{7.7}
  \node[draw, acc, very thick, fill=acclo!35, minimum size=\cw cm, inner sep=0pt, font=\footnotesize, text=acc] (out) at (\ox, -1.24) {11};
  \node[anchor=south, font=\footnotesize, text=acc] at (\ox, -0.55) {output $S$[$i$,$j$]};
  % arrows
  \draw[->, acc, thick] (3.0,-1.24) -- (\kx - 0.55, -1.24);
  \draw[->, acc, thick] (\kx + 2*\cw + 0.45, -1.24) -- (\ox - 0.55, -1.24);
  \node[font=\footnotesize, anchor=south] at (3.85,-1.18) {\texttt{sum of}};
  \node[font=\footnotesize, anchor=north] at (3.85,-1.30) {\texttt{products}};
\end{tikzpicture}
$$

### Channels

Real inputs are not flat grids but stacks of them: an RGB image is $H \times W
\times 3$. A kernel spans _all_ input channels, so a kernel for a $C_{\text{in}}$-
channel input has shape $k_h \times k_w \times C_{\text{in}}$ and produces a single
feature map by summing over channels too:

$$
S[i, j] \;=\; \sum_{c=1}^{C_{\text{in}}} \sum_{m}\sum_{n}
  I[\,i + m,\; j + n,\; c\,]\,K[m, n, c].
$$

A convolutional _layer_ learns $C_{\text{out}}$ such kernels, stacking their
feature maps into an output volume of depth $C_{\text{out}}$. The layer's full
weight tensor is $k_h \times k_w \times C_{\text{in}} \times C_{\text{out}}$ — note
it is independent of the input's spatial size $H, W$.

> **Definition (Channel).** One slice of depth in an input or feature volume. Input
> channels are the raw components (red, green, blue); output channels are the
> feature maps, one per learned kernel, each a different learned pattern detector.

For a worked shape trace, take an RGB input of
$32 \times 32 \times 3$ and a conv layer with $C_{\text{out}} = 16$ kernels of size
$3 \times 3$, "same" padding ($p = 1$), stride $1$. Each kernel has shape
$3 \times 3 \times 3 = 27$ weights plus one bias, so the layer's weight tensor is
$3 \times 3 \times 3 \times 16 = 432$ weights and $16$ biases, $448$ parameters
total. The output geometry from $n_{\text{out}} = \lfloor (32 + 2 - 3)/1 \rfloor + 1
= 32$ per spatial axis gives an output volume of $32 \times 32 \times 16$. Every one
of those $32 \cdot 32 \cdot 16 = 16{,}384$ output scalars is an inner product over
$3 \cdot 3 \cdot 3 = 27$ input values, so the layer costs $16{,}384 \cdot 27
\approx 4.4 \times 10^{5}$ multiply-adds. The single fact to carry: the parameter
count $k_h k_w C_{\text{in}} C_{\text{out}}$ does not mention $H$ or $W$, so the same
$448$ weights process a $32 \times 32$ image or a $512 \times 512$ one; only the
compute scales with resolution.

## Three ideas that beat the dense layer

Convolution wins for images because of three structural commitments, each cutting a
different cost the dense layer pays.[^gf-motivation]

### Sparse connectivity

A dense layer connects every output to every input: $m$ inputs and $n$ outputs cost
$m \cdot n$ weights and $O(m \cdot n)$ multiply-adds. A conv layer connects each
output only to a $k_h \times k_w$ patch, so each output unit has just $k_h k_w$
incoming weights regardless of input size.

$$
% caption: Sparse (left) versus dense (right) connectivity. Each conv output draws from a local $3$-wide input patch; each dense output draws from all inputs.
\begin{tikzpicture}[font=\small, >=stealth,
  u/.style={circle, draw, minimum size=5mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{acclo}{HTML}{A7B5FB}
  % --- LEFT: sparse ---
  \foreach \i in {0,...,4} \node[u] (sin\i) at (\i*0.9, 0) {};
  \foreach \i in {0,...,4} \node[u, draw=acc, thick] (sout\i) at (\i*0.9, 2.0) {};
  \node[anchor=east, font=\footnotesize] at (-0.45,0) {input};
  \node[anchor=east, font=\footnotesize, text=acc] at (-0.45,2.0) {conv};
  % local edges: each output to a 3-window (clamped)
  \foreach \o in {0,...,4} {
    \foreach \d in {-1,0,1} {
      \pgfmathtruncatemacro{\s}{\o+\d}
      \ifnum\s>-1 \ifnum\s<5
        \draw[acc, thick] (sin\s) -- (sout\o);
      \fi\fi
    }
  }
  \node[font=\footnotesize, anchor=north] at (1.8,-0.55) {\texttt{sparse:} $k$ \texttt{edges/output}};
  % --- RIGHT: dense ---
  \begin{scope}[xshift=6.5cm]
    \foreach \i in {0,...,4} \node[u] (din\i) at (\i*0.9, 0) {};
    \foreach \i in {0,...,4} \node[u, draw=black, thick] (dout\i) at (\i*0.9, 2.0) {};
    \node[anchor=east, font=\footnotesize] at (-0.45,0) {input};
    \node[anchor=east, font=\footnotesize] at (-0.45,2.0) {dense};
    \foreach \o in {0,...,4}
      \foreach \s in {0,...,4}
        \draw[black] (din\s) -- (dout\o);
    \node[font=\footnotesize, anchor=north] at (1.8,-0.55) {\texttt{dense:} $m$ \texttt{edges/output}};
  \end{scope}
\end{tikzpicture}
$$

> **Definition (Sparse connectivity).** Each output unit depends only on a small
> contiguous patch of the input (its **receptive field**) rather than the entire
> input. This shrinks both the weight count and the compute per output from $O(m)$
> to $O(k_h k_w)$.

### Parameter sharing

The deeper saving is parameter reuse: the conv layer uses the
_same_ weights at every position. The dense layer learns an independent weight for
every (input, output) pair; the conv layer learns one kernel and applies it
everywhere. The weights are **tied** across space.

> **Definition (Parameter sharing).** Using one kernel's weights at every spatial
> position rather than a distinct weight per position. A pattern learned once
> applies everywhere, so the parameter count depends on the kernel size, not the
> input size.

One kernel slides across the input and writes each feature-map cell. Every output
position is computed by the _same_ three weights; the network stores one filter,
not one weight per location.

$$
% caption: Parameter sharing. One kernel $K$ is reused at every position: each
% output cell $S_1,S_2,S_3$ comes from the same weights on a shifted patch.
\begin{tikzpicture}[font=\small, >=stealth,
  cellin/.style={draw, black, thick, minimum size=8mm, inner sep=0pt, font=\footnotesize},
  cellout/.style={draw=acc, thick, minimum size=8mm, inner sep=0pt, font=\footnotesize, text=acc}]
  \definecolor{acc}{HTML}{2348F2}
  % input row
  \foreach \i/\v in {0/3,1/1,2/2,3/0,4/4} \node[cellin] (in\i) at (\i*0.8,0) {\v};
  \node[anchor=east, font=\footnotesize] at (-0.35,0) {input};
  % kernel
  \node[draw=acc, thick, fill=acc!12, minimum width=24mm, minimum height=8mm,
        font=\footnotesize, text=acc] (K) at (1.6,2.0) {kernel $K$ (one f\/ilter)};
  % output row
  \foreach \i/\v in {0/11,1/7,2/9} \node[cellout] (out\i) at (\i*0.8+0.4,3.8) {\v};
  \node[anchor=west, font=\footnotesize, text=acc] at (2.7,3.8) {feature map};
  % solid curved edges: same kernel to each output
  \draw[->, acc, thick] (K.north) to[out=120,in=270] (out0.south);
  \draw[->, acc, thick] (K.north) to[out=90,in=270]  (out1.south);
  \draw[->, acc, thick] (K.north) to[out=60,in=270]  (out2.south);
  % input patches feeding the kernel (solid black)
  \draw[->, black, thick] (in1.north) to[out=90,in=250] (K.south);
  \node[font=\footnotesize, text=acc, anchor=west] at (3.0,2.0) {same weights};
\end{tikzpicture}
$$

Take a $32 \times 32 \times 1$ input mapped to a
$32 \times 32 \times 1$ output, comparing a dense layer to a $3 \times 3$ conv:

| Quantity | Dense layer | Conv layer ($3\times 3$) |
| --- | --- | --- |
| Inputs $m$ | $1024$ | $1024$ |
| Outputs $n$ | $1024$ | $1024$ |
| Weights | $m \cdot n = 1{,}048{,}576$ | $k_h k_w = 9$ |
| Weights scale with input size? | yes ($\propto m n$) | no (fixed at $k_h k_w$) |
| Multiply-adds | $m \cdot n = 1{,}048{,}576$ | $n \cdot k_h k_w = 9216$ |

The conv layer carries $9$ weights against the dense layer's million, a
$10^{5}\times$ reduction, and the gap _widens_ with input size, since the dense
count grows as $m \cdot n$ while the kernel stays fixed at $k_h k_w$.

### Translation equivariance

Parameter sharing gives a structural property for free: shifting the input shifts
the output the same way. Let $T_\delta$ be a translation by $\delta$ pixels. Then

$$
(T_\delta I) \ast K \;=\; T_\delta\,(I \ast K),
$$

so detecting a feature and shifting commute. The kernel responds identically to a
pattern no matter where it sits — exactly the right prior for images, where an
object's identity is independent of its position.[^gf-equivariance]

> **Theorem (Translation equivariance of convolution).** For any integer shift
> $\delta$, convolution commutes with translation: $(T_\delta I) * K = T_\delta\,(I *
> K)$.

> **Proof.** Write $J = T_\delta I$, so $J[a, b] = I[a - \delta_1,\, b - \delta_2]$.
> Then
> $$
> (J \ast K)[i, j] = \sum_{m, n} J[i + m,\, j + n]\,K[m, n]
> = \sum_{m, n} I[i - \delta_1 + m,\; j - \delta_2 + n]\,K[m, n].
> $$
> The right-hand side is exactly $(I \ast K)[i - \delta_1,\, j - \delta_2] = \parens{T_\delta\,(I \ast K)}[i, j]$. Equality holds at every $(i, j)$. $\qed$

Equivariance is not invariance: the output _moves_ with the input. Pooling, below,
converts a small amount of this equivariance into outright invariance.

## Convolution arithmetic

The kernel cannot be centered on the boundary pixels, so a valid convolution
shrinks the output. Two knobs control the output geometry: the **stride** $s$ (how
far the kernel hops between positions) and the **padding** $p$ (how many zero rows
and columns ring the input).

### Deriving the output size

Consider one spatial dimension of size $n$, padded to $n + 2p$. A kernel of width
$k$ has its leftmost valid position at index $0$ and its rightmost when its right
edge hits the padded boundary, i.e. at start index $n + 2p - k$. Stepping by $s$,
the valid start positions are $0, s, 2s, \dots$ up to $n + 2p - k$, so their count
is

$$
n_{\text{out}} \;=\; \left\lfloor \frac{(n + 2p - k)}{s} \right\rfloor + 1.
$$

The floor handles a final partial step that does not fit; the $+1$ counts the
starting position itself. Several special cases recover familiar layers:

| Configuration | $p$ | $s$ | Output size | Effect |
| --- | --- | --- | --- | --- |
| Valid (no pad) | $0$ | $1$ | $n - k + 1$ | shrinks by $k - 1$ |
| Same (preserve size) | $\tfrac{k-1}{2}$ | $1$ | $n$ | output matches input |
| Strided downsample | $\tfrac{k-1}{2}$ | $2$ | $\approx n/2$ | halves resolution |
| No pad, stride $2$ | $0$ | $2$ | $\lfloor (n-k)/2\rfloor + 1$ | shrink + downsample |

> **Definition (Stride).** The step size $s$ by which the kernel moves between
> successive positions. $s = 1$ visits every position; $s > 1$ skips positions and
> downsamples the output by roughly a factor of $s$.

> **Definition (Padding).** A border of $p$ zero rows and columns added around the
> input before convolving. With $p = (k-1)/2$ and $s = 1$ ("same" padding) the
> output keeps the input's spatial size, so depth can be stacked without the maps
> vanishing.

A worked instance: $n = 7$, $k = 3$, $p = 1$, $s = 2$ gives $\lfloor (7 + 2 - 3)/2
\rfloor + 1 = \lfloor 6/2 \rfloor + 1 = 4$.

## Pooling

A **pooling** layer summarizes each small region of a feature map by a single
statistic (its maximum for max pooling, or mean for average pooling) over a window,
typically with stride equal to the window so the regions tile without overlap.
Pooling has no learnable weights; it exists to downsample and to add local
invariance.[^gf-pooling]

$$
% caption: Max versus average pooling, $2\times 2$ window, stride $2$. Max keeps each region's strongest activation; average keeps the mean; both halve the map.
\begin{tikzpicture}[font=\small, >=stealth]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{acclo}{HTML}{A7B5FB}
  \definecolor{green}{HTML}{1F9D4D}
  \def\cw{0.62}
  % --- input 4x4 ---
  \foreach \r/\c/\v in {
    0/0/1,0/1/3,0/2/2,0/3/1,
    1/0/4,1/1/2,1/2/0,1/3/1,
    2/0/0,2/1/1,2/2/3,2/3/2,
    3/0/2,3/1/0,3/2/1,3/3/4}
    \node[draw, black, minimum size=\cw cm, inner sep=0pt, font=\scriptsize] at (\c*\cw, -\r*\cw) {\v};
  % shade the four 2x2 quadrants faintly with colored borders
  \draw[acc, thick] (-0.5*\cw,0.5*\cw) rectangle (1.5*\cw,-1.5*\cw);
  \draw[acc, thick] (1.5*\cw,0.5*\cw) rectangle (3.5*\cw,-1.5*\cw);
  \draw[acc, thick] (-0.5*\cw,-1.5*\cw) rectangle (1.5*\cw,-3.5*\cw);
  \draw[acc, thick] (1.5*\cw,-1.5*\cw) rectangle (3.5*\cw,-3.5*\cw);
  \node[anchor=south, font=\footnotesize] at (1.05,0.62) {feature map};
  % --- max pool output ---
  \def\mx{3.6}
  \foreach \r/\c/\v in {0/0/4,0/1/2,1/0/2,1/1/4}
    \node[draw, green, very thick, minimum size=\cw cm, inner sep=0pt, font=\scriptsize, text=green] at (\mx + \c*\cw, -\r*\cw) {\v};
  \node[anchor=south, font=\footnotesize, text=green] at (\mx + 0.5*\cw, 0.62) {max pool};
  % --- avg pool output ---
  \def\ax{5.9}
  \foreach \r/\c/\v in {0/0/2.5,0/1/1.0,1/0/0.75,1/1/2.5}
    \node[draw, acc, very thick, minimum size=\cw cm, inner sep=0pt, font=\scriptsize, text=acc] at (\ax + \c*\cw, -\r*\cw) {\v};
  \node[anchor=south, font=\footnotesize, text=acc] at (\ax + 0.5*\cw, 0.62) {avg pool};
  % arrows
  \draw[->, green, thick] (2.5,-0.93) -- (\mx - 0.45, -0.31);
  \draw[->, acc, thick] (2.5,-2.0) -- (\ax - 0.45, -0.31);
\end{tikzpicture}
$$

> **Definition (Pooling).** A parameter-free downsampling layer that replaces each
> window of a feature map by a summary statistic: the maximum (max pooling) or
> the average (average pooling). It reduces spatial resolution and grants
> invariance to small translations within each window.

The invariance is the point. If the strongest activation shifts by one position but
stays inside the same pooling window, the max is unchanged — the output is _locally
invariant_ to small translations, not merely equivariant. Max pooling keeps the
strongest evidence; average pooling keeps the typical level.

| Property | Max pooling | Average pooling |
| --- | --- | --- |
| Statistic | $\max$ over window | mean over window |
| Keeps | strongest activation | overall level |
| Gradient | flows to the argmax only | spread evenly |
| Best for | sharp feature presence | smooth global context |

## The receptive field

The **receptive field** of a unit is the region of the _original input_ that can
influence it. A single $3 \times 3$ conv sees a $3 \times 3$ patch; stack another
on top and each of its units sees a $3 \times 3$ patch of the layer below, which
already each saw $3 \times 3$ of the input, so the second layer's units see $5
\times 5$ of the input. The field grows linearly with depth for stride-$1$ convs.

$$
% caption: The receptive field grows with depth: stacking two $3$-wide layers lets one layer-2 unit see a $5$-wide span of the input.
\begin{tikzpicture}[font=\small, >=stealth,
  u/.style={circle, draw, minimum size=4.5mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{acclo}{HTML}{A7B5FB}
  \definecolor{green}{HTML}{1F9D4D}
  % input row (7 units)
  \foreach \i in {0,...,6} \node[u] (in\i) at (\i*0.95, 0) {};
  % layer 1 row (5 units), centered
  \foreach \i in {0,...,4} \node[u, draw=acc, thick] (l1\i) at (\i*0.95 + 0.95, 1.7) {};
  % layer 2 single highlighted unit
  \node[u, draw=green, very thick, fill=green!18] (l2) at (3*0.95 + 0.95, 3.4) {};
  \node[anchor=east, font=\footnotesize] at (-0.55,0) {input};
  \node[anchor=east, font=\footnotesize, text=acc] at (0.4,1.7) {layer 1};
  \node[anchor=west, font=\footnotesize, text=green] at (4.6,3.4) {layer 2};
  % layer2 -> 3 units of layer1 (l1 2,3,4)
  \foreach \i in {2,3,4} \draw[green, thick] (l2) -- (l1\i);
  % each of those layer1 units -> its 3-window in input
  % l1_2 sits over input 1,2,3 ; l1_3 over 2,3,4 ; l1_4 over 3,4,5
  \foreach \i in {1,2,3} \draw[acc] (l12) -- (in\i);
  \foreach \i in {2,3,4} \draw[acc] (l13) -- (in\i);
  \foreach \i in {3,4,5} \draw[acc] (l14) -- (in\i);
  % bracket marking the 5-wide receptive field on input (in 1..5)
  \draw[green, thick] (0.95 - 0.28, -0.6) -- (0.95 - 0.28, -0.42) -- (5*0.95 + 0.28, -0.42) -- (5*0.95 + 0.28, -0.6);
  \node[anchor=north, font=\footnotesize, text=green] at (3*0.95, -0.7) {\texttt{receptive field: 5 wide}};
\end{tikzpicture}
$$

> **Definition (Receptive field).** The set of input positions that affect a given
> unit's value. For a stack of stride-$1$ convolutions with kernel sizes $k_1, k_2,
> \dots, k_L$ the field width after $L$ layers is $r_L = 1 + \sum_{\ell=1}^{L}
> (k_\ell - 1)$.

Stride and pooling accelerate the growth: a stride-$s$ layer multiplies the
spacing, so receptive fields expand geometrically rather than linearly once
downsampling is interleaved. This is why deep stacks of small $3 \times 3$ convs
can, with a few pooling steps, end up "seeing" the entire image — the late layers
have receptive fields spanning the input, which is what lets them represent whole
objects.

## The convolution algorithm

The forward pass of a single convolutional layer is a direct transcription of the
definition: for each output channel, slide that channel's kernel over the padded
input, summing products across the window and the input channels.

```algorithm
caption: $\textsc{Conv2D}(I, K, s, p)$ — forward pass of one convolutional layer
pad $I$ with $p$ zeros on each side
$H' \gets \lfloor (H + 2p - k_h)/s \rfloor + 1$
$W' \gets \lfloor (W + 2p - k_w)/s \rfloor + 1$
for each output channel $d \gets 1$ to $C_{\text{out}}$ do
  for $i \gets 0$ to $H' - 1$ do
    for $j \gets 0$ to $W' - 1$ do
      $a \gets 0$ // accumulator for cell $(i,j,d)$
      for each input channel $c$, offset $(m, n)$ do
        $a \gets a + I[s i + m,\ s j + n,\ c] \cdot K[m, n, c, d]$
      $S[i, j, d] \gets a + b_d$ // add per-channel bias
return $S$
```

The five nested loops make the cost explicit: $C_{\text{out}} \cdot H' W' \cdot
C_{\text{in}} k_h k_w$ multiply-adds. Optimized implementations recast the inner
loops as a single matrix multiply ("im2col"), but the arithmetic is unchanged.[^stevens-conv]

## A typical convolutional stack

A convolutional network alternates convolution, a nonlinear
[activation](/deep-learning/neural-networks/activation-functions), and pooling,
each block shrinking spatial size while growing channel depth, then flattens and
hands off to a [dense](/deep-learning/neural-networks/the-multilayer-perceptron)
classifier head.[^chollet-cnn]

$$
% caption: A typical CNN pipeline. Conv-ReLU-pool blocks downsample the image and deepen the channel stack; a flatten and dense head produce class scores.
\begin{tikzpicture}[font=\small, >=stealth,
  blk/.style={draw, minimum width=15mm, minimum height=11mm, align=center, font=\scriptsize},
  vol/.style={draw, fill=acclo!25, minimum width=10mm, minimum height=13mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{acclo}{HTML}{A7B5FB}
  \definecolor{green}{HTML}{1F9D4D}
  \node[vol, minimum height=16mm] (img) at (0,0) {image\\(H by W, 3 ch)};
  \node[blk, draw=acc, text=acc] (c1) at (2.1,0) {conv\\+ ReLU};
  \node[blk] (p1) at (3.9,0) {pool};
  \node[blk, draw=acc, text=acc] (c2) at (5.7,0) {conv\\+ ReLU};
  \node[blk] (p2) at (7.5,0) {pool};
  \node[blk] (fl) at (9.3,0) {f\/latten};
  \node[blk, draw=green, text=green] (fc) at (11.1,0) {dense};
  \node[font=\scriptsize, anchor=west, text=green] (cls) at (12.1,0) {class};
  \draw[->, acc, thick] (img) -- (c1);
  \draw[->, thick] (c1) -- (p1);
  \draw[->, acc, thick] (p1) -- (c2);
  \draw[->, thick] (c2) -- (p2);
  \draw[->, thick] (p2) -- (fl);
  \draw[->, green, thick] (fl) -- (fc);
  \draw[->, green, thick] (fc) -- (cls);
  \node[font=\scriptsize, text=black, anchor=north] at (3.0,-1.0) {spatial size down, depth up};
  \draw[black, ->] (1.3,-0.85) -- (8.0,-0.85);
\end{tikzpicture}
$$

The geometry follows a consistent pattern: each conv block enlarges receptive fields
and adds channels (more pattern detectors), while pooling discards spatial
resolution that is no longer needed. By the flatten, each unit summarizes a large
region through a deep feature, and a small dense head suffices to read off the
class.

| Stage | Spatial size | Channels | Role |
| --- | --- | --- | --- |
| input | $H \times W$ | $3$ | raw pixels |
| conv + ReLU | $H \times W$ | $C_1$ | local feature detectors |
| pool | $H/2 \times W/2$ | $C_1$ | downsample, local invariance |
| conv + ReLU | $H/2 \times W/2$ | $C_2 > C_1$ | compose into richer features |
| pool | $H/4 \times W/4$ | $C_2$ | downsample again |
| flatten + dense | $1$ | classes | global readout |

### A worked shape and parameter trace

Instantiate the stack with real numbers on a $32 \times 32 \times 3$ input, two
conv-pool blocks with "same"-padded $3 \times 3$ convs ($C_1 = 32$, $C_2 = 64$) and
$2 \times 2$ stride-$2$ pooling, then a dense head to $10$ classes. Following one
tensor through, with the parameter count $k_h k_w C_{\text{in}} C_{\text{out}} +
C_{\text{out}}$ at each conv:

| Layer | Output shape | Parameters |
| --- | --- | --- |
| input | $32 \times 32 \times 3$ | $0$ |
| conv $3\times3$, $32$ | $32 \times 32 \times 32$ | $3\cdot3\cdot3\cdot32 + 32 = 896$ |
| max pool $2\times2$ | $16 \times 16 \times 32$ | $0$ |
| conv $3\times3$, $64$ | $16 \times 16 \times 64$ | $3\cdot3\cdot32\cdot64 + 64 = 18{,}496$ |
| max pool $2\times2$ | $8 \times 8 \times 64$ | $0$ |
| flatten | $4096$ | $0$ |
| dense $\to 10$ | $10$ | $4096\cdot10 + 10 = 40{,}970$ |

Two observations fall out of the numbers. First, spatial size halves at every pool
($32 \to 16 \to 8$) while depth grows ($3 \to 32 \to 64$): the network trades
resolution for richer per-position features. Second, the convolutional feature
extractor holds $896 + 18{,}496 = 19{,}392$ parameters against the dense head's
$40{,}970$ — most of the weights live in that final flatten-to-dense step, which is
exactly why the flatten happens only after pooling has shrunk the map to $8 \times 8$;
flattening the $32 \times 32 \times 32$ map straight after the first conv would feed
$32{,}768$ features into the dense layer and inflate that head by a factor of eight.

$$
% caption: Tensor-shape flow through one conv-pool block. Each conv (same-padded) preserves $H \times W$ and grows the channel depth; each stride-$2$ pool halves $H \times W$ and preserves depth.
\begin{tikzpicture}[font=\footnotesize, >=stealth,
  vol/.style={draw=acc, fill=acclo!22, minimum width=13mm, align=center, font=\scriptsize},
  op/.style={font=\scriptsize, text=black, anchor=south}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{acclo}{HTML}{A7B5FB}
  \node[vol, minimum height=18mm] (v0) at (0,0) {32 by 32\\3 ch};
  \node[vol, minimum height=18mm] (v1) at (3.2,0) {32 by 32\\32 ch};
  \node[vol, minimum height=13mm] (v2) at (6.4,0) {16 by 16\\32 ch};
  \node[vol, minimum height=13mm] (v3) at (9.6,0) {16 by 16\\64 ch};
  \draw[->, acc, thick] (v0) -- (v1);
  \draw[->, black, thick] (v1) -- (v2);
  \draw[->, acc, thick] (v2) -- (v3);
  \node[op, text=acc] at (1.6,0.6) {\texttt{conv 3x3}};
  \node[op] at (4.8,0.6) {\texttt{pool 2x2}};
  \node[op, text=acc] at (8.0,0.6) {\texttt{conv 3x3}};
  \node[font=\footnotesize, anchor=north, text=acc] at (1.6,-1.15) {\texttt{depth up}};
  \node[font=\footnotesize, anchor=north] at (4.8,-1.15) {\texttt{space halved}};
  \node[font=\footnotesize, anchor=north, text=acc] at (8.0,-1.15) {\texttt{depth up}};
\end{tikzpicture}
$$

## Convolution variants in production

Goodfellow, Chollet, and Stevens stop at the standard dense convolution; two variants that
reshape the cost and the receptive field are now everywhere in production vision
models.

- **Depthwise-separable convolution.** A standard $k\times k$ layer with $C_{\text{in}}$
  input and $C_{\text{out}}$ output channels costs $k^2 C_{\text{in}} C_{\text{out}}$
  multiply-adds per output position. Split it into a **depthwise** step (one
  $k\times k$ filter per input channel, no channel mixing) followed by a **pointwise**
  $1\times1$ step (mixing channels, no spatial extent), and the cost drops to
  $k^2 C_{\text{in}} + C_{\text{in}} C_{\text{out}}$. For a $3\times3$ layer with
  $C_{\text{in}} = C_{\text{out}} = 256$, that is $9 \cdot 256 \cdot 256 \approx
  590\text{k}$ versus $9 \cdot 256 + 256 \cdot 256 \approx 68\text{k}$, an
  $8.6\times$ reduction, with almost no accuracy loss. This factorization underlies
  MobileNet (Howard et al., 2017) and Xception (Chollet, 2017), the
  standard for on-device vision.
- **Dilated (atrous) convolution.** Insert gaps of size $r-1$ between kernel taps,
  so a $3\times3$ kernel covers a $ (2r+1)\times(2r+1)$ span while touching only $9$
  weights. The receptive field then grows _exponentially_ with depth if $r$ doubles
  each layer, instead of the linear $1 + \sum_\ell (k_\ell - 1)$ derived above, which
  is why dilation dominates dense-prediction tasks like segmentation (Yu & Koltun,
  "Multi-Scale Context Aggregation by Dilated Convolutions," ICLR 2016) and audio
  generation (WaveNet, van den Oord et al., 2016).

The [Vision Transformer](/deep-learning/architectures/transformers-in-practice)
drops this same locality prior that the three ideas encode,
trading the built-in bias for data; the convolution and the attention
block are two ends of one design axis.

## Takeaways

* A **convolution** $(I \ast K)[i,j] = \sum_{m,n} I[i+m, j+n]\,K[m,n]$ slides a small
  learnable kernel over the input; each kernel yields one **feature map**, and a
  layer stacks $C_{\text{out}}$ of them across $C_{\text{in}}$ input channels.
* Three commitments beat the dense layer: **sparse connectivity** (each output sees
  only a $k_h k_w$ patch), **parameter sharing** (one kernel reused everywhere), and
  **translation equivariance** ($(T_\delta I) \ast K = T_\delta(I\ast{}K)$) — together
  collapsing a million weights to nine for a $3 \times 3$ kernel.
* Output geometry obeys $n_{\text{out}} = \lfloor (n + 2p - k)/s \rfloor + 1$;
  "same" padding $p = (k-1)/2$ with $s = 1$ preserves size, stride $s > 1$
  downsamples.
* **Pooling** (max or average) is parameter-free downsampling that turns local
  equivariance into local _invariance_; the **receptive field** grows as $1 +
  \sum_\ell (k_\ell - 1)$ with depth, so deep stacks eventually see the whole image.
* The canonical stack — conv → activation → pool → … → flatten → dense — deepens
  channels while shrinking space, trained end-to-end, and is the backbone the
  [next lesson's architectures](/deep-learning/architectures/cnn-architectures)
  refine.

[^gf-conv]: **Goodfellow**, _Deep Learning_, §9.1 — The Convolution Operation: the discrete cross-correlation a "convolution" layer actually computes, and the feature map it produces.
[^gf-motivation]: **Goodfellow**, _Deep Learning_, §9.2 — Motivation: sparse interactions, parameter sharing, and equivariant representations as the three levers convolution pulls against the dense layer.
[^gf-equivariance]: **Goodfellow**, _Deep Learning_, §9.2 — Equivariance to translation: why shared kernels commute with shifts, and why this is equivariance rather than invariance.
[^gf-pooling]: **Goodfellow**, _Deep Learning_, §9.3 — Pooling: max/average pooling as parameter-free downsampling that buys approximate invariance to small translations.
[^stevens-conv]: **Stevens**, _Deep Learning with PyTorch_, Ch. 8 — Using Convolutions to Generalize: the `Conv2d` forward pass, padding/stride arithmetic, and channel bookkeeping in practice.
[^chollet-cnn]: **Chollet**, _Deep Learning with Python_, Ch. 5 — Deep Learning for Computer Vision: the canonical conv → ReLU → pool → flatten → dense convnet and why it dominates image tasks.