--- title: The Multilayer Perceptron module: Neural Networks moduleNumber: 2 lessonNumber: 1 order: 201 summary: > Stacking linear layers with a nonlinearity between them removes the limitation that stopped the perceptron. We build the multilayer perceptron in explicit matrix form (the forward pass, its dimensions, a worked XOR network with concrete weights) and prove why the nonlinearity is essential: without it the deepest stack collapses to a single hyperplane. topics: [Neural Networks] sources: - book: Goodfellow ref: "Ch. 6 — Deep Feedforward Networks; §6.1 Example: Learning XOR" - book: Chollet ref: "Ch. 2–3 — The Mathematical Building Blocks; Getting Started with Neural Networks" - book: Stevens ref: "Ch. 5–6 — The Mechanics of Learning; Using a Neural Network" --- The [linear model](/deep-learning/foundations/linear-models-and-the-perceptron) failed XOR because a single hyperplane cannot carve up an interleaved space. The **multilayer perceptron** (MLP) fixes this by inserting a layer of neurons _between_ the input and the output and passing their combined signal through a nonlinearity before the next layer. The result is a stack of affine maps interleaved with elementwise nonlinearities, the canonical _deep feedforward network_.[^gf-ffn] ## The architecture An MLP is a stack of **fully connected layers**. Layer $l$ takes the vector $h^{(l-1)}$ below it, applies an affine map (a weight matrix plus a bias), and passes the result componentwise through a nonlinear **activation** $g$: $$ h^{(l)} = g\!\parens{W^{(l)} h^{(l-1)} + b^{(l)}}, \qquad l = 1, \dots, L, $$ with the convention $h^{(0)} = x$ and a final layer usually left _linear_ so the output can take any value a regression target or logit needs. Writing the four layer equations out shows the chain explicitly, $$ h^{(1)} = g\parens{W^{(1)} x + b^{(1)}}, \quad h^{(2)} = g\parens{W^{(2)} h^{(1)} + b^{(2)}}, \quad \dots, \quad \hat{y} = W^{(L)} h^{(L-1)} + b^{(L)}. $$ $$ % caption: A fully connected multilayer perceptron; the hidden layer learns a % representation in which the output layer's job becomes linear. \begin{tikzpicture}[>=stealth, font=\footnotesize, unit/.style={circle, draw, minimum size=8mm, inner sep=0pt}, hid/.style={circle, draw, thick, minimum size=8mm, inner sep=0pt}] \definecolor{acc}{HTML}{2348F2} % layers: input (3), hidden (4), output (2) \foreach \i/\y in {1/2.4, 2/0.8, 3/-0.8} \node[unit] (i\i) at (0,\y) {}; \foreach \i/\y in {1/3.2, 2/1.6, 3/0, 4/-1.6} \node[hid] (h\i) at (3,\y) {}; \foreach \i/\y in {1/1.6, 2/0} \node[unit] (o\i) at (6,\y) {}; % fully connected edges \foreach \a in {1,2,3} \foreach \b in {1,2,3,4} \draw[black] (i\a) -- (h\b); \foreach \a in {1,2,3,4} \foreach \b in {1,2} \draw[black] (h\a) -- (o\b); % layer labels \node[align=center] at (0,-2.3) {input\\$x$}; \node[align=center] at (3,-2.9) {hidden\\$h = g(Wx+b)$}; \node[align=center] at (6,-2.3) {output\\$y$ (hat)}; \end{tikzpicture} $$ Stack as many hidden layers as you like; the count $L$ is the network's **depth**, the number of units in a layer its **width**. > **Definition (Multilayer perceptron).** A function $f : \mathbb{R}^{n_0} \to > \mathbb{R}^{n_L}$ built by composing $L$ affine maps with an elementwise > nonlinearity $g$ between consecutive maps: > $f(x) = W^{(L)} g(\dots g(W^{(1)}x + b^{(1)}) \dots) + b^{(L)}$. The parameters > are the weight matrices $W^{(l)}$ and bias vectors $b^{(l)}$; the hidden vectors > $h^{(l)}$ are _learned representations_, not chosen by hand. ## The forward pass in matrix form The whole computation is a sequence of matrix–vector products. The layer **widths** $n_0, n_1, \dots, n_L$ determine every shape: the weight matrix maps the layer below into the layer above, so it has one row per output unit and one column per input unit. $$ W^{(l)} \in \mathbb{R}^{n_l \times n_{l-1}}, \qquad b^{(l)} \in \mathbb{R}^{n_l}, \qquad h^{(l)} \in \mathbb{R}^{n_l}, \qquad h^{(0)} = x \in \mathbb{R}^{n_0}. $$ It is convenient to name the **pre-activation** $z^{(l)} = W^{(l)} h^{(l-1)} + b^{(l)}$ separately from the **post-activation** $h^{(l)} = g(z^{(l)})$; the split is what [backpropagation](/deep-learning/neural-networks/backpropagation) later differentiates through. The dimension chain must telescope, each layer's output width becoming the next layer's input width, or the products are undefined: $$ \underbrace{x}_{n_0} \;\xrightarrow{\,W^{(1)}\,}\; \underbrace{z^{(1)}, h^{(1)}}_{n_1} \;\xrightarrow{\,W^{(2)}\,}\; \underbrace{z^{(2)}, h^{(2)}}_{n_2} \;\xrightarrow{\,\dots\,}\; \underbrace{z^{(L)} = \hat{y}}_{n_L}. $$ $$ % caption: Layer-by-layer dataflow. Each affine map reshapes the width while % the elementwise $g$ preserves it; widths must telescope for the maps to compose. \begin{tikzpicture}[>=stealth, font=\footnotesize, vec/.style={draw, minimum width=11mm, minimum height=15mm, align=center}, op/.style={draw, minimum width=13mm, minimum height=8mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} % input vector \node[vec] (x) at (0,0) {$x$\\\scriptsize$n_0$}; % layer 1 \node[op, draw=acc, text=acc] (w1) at (1.9,0) {$W^{(1)}$\\$+\,b^{(1)}$}; \node[op] (g1) at (3.6,0) {$g$}; \node[vec] (h1) at (5.1,0) {$h^{(1)}$\\\scriptsize$n_1$}; % layer 2 \node[op, draw=acc, text=acc] (w2) at (7.0,0) {$W^{(2)}$\\$+\,b^{(2)}$}; \node[op] (g2) at (8.7,0) {$g$}; \node[vec] (h2) at (10.2,0) {$h^{(2)}$\\\scriptsize$n_2$}; % output map \node[op, draw=acc, text=acc] (w3) at (12.1,0) {$W^{(3)}$\\$+\,b^{(3)}$}; \node[vec, draw=acc, text=acc] (yh) at (13.9,0) {out\\\scriptsize$n_3$}; \draw[->, thick] (x) -- (w1); \draw[->, thick] (w1) -- (g1); \draw[->, thick] (g1) -- (h1); \draw[->, thick] (h1) -- (w2); \draw[->, thick] (w2) -- (g2); \draw[->, thick] (g2) -- (h2); \draw[->, thick] (h2) -- (w3); \draw[->, thick] (w3) -- (yh); % shape annotations under the maps (rows by cols) \node[font=\scriptsize, text=acc] at (1.9,-1.4) {$n_1$ rows, $n_0$ cols}; \node[font=\scriptsize, text=acc] at (7.0,-1.4) {$n_2$ rows, $n_1$ cols}; \node[font=\scriptsize, text=acc] at (12.1,-1.4) {$n_3$ rows, $n_2$ cols}; \end{tikzpicture} $$ Reading off the figure: each blue box is a matrix multiply whose shape is printed beneath it, and the bare $g$ box leaves the width untouched. The forward pass is the literal traversal of this chain. ```algorithm caption: $\textsc{Forward}(x; \{W^{(l)}, b^{(l)}\}_{l=1}^{L})$ — evaluate an MLP $h^{(0)} \gets x$ for $l \gets 1$ to $L$ do $z^{(l)} \gets W^{(l)} h^{(l-1)} + b^{(l)}$ // affine map, shape $n_l \times n_{l-1}$ if $l < L$ then $h^{(l)} \gets g(z^{(l)})$ // elementwise nonlinearity else $h^{(l)} \gets z^{(l)}$ // linear output layer return $\hat{y} \gets h^{(L)}$ ``` In practice the input is not one vector but a **batch** of $m$ examples stacked into a matrix $X \in \mathbb{R}^{m \times n_0}$ (the row-major, [PyTorch](/deep-learning/foundations/what-is-deep-learning)-shaped convention), and the layer becomes $H^{(l)} = g\parens{H^{(l-1)} W^{(l)\,T} + b^{(l)}}$ with the bias **broadcast** across rows.[^chollet-tensor] The same matrix multiply that processes one example processes the whole batch at once, the reason a GPU is the natural host for the forward pass.[^stevens-batch] For example, track the shapes through the $784$–$128$–$64$–$10$ net with a batch of $m = 32$. The input is $X \in \mathbb{R}^{32 \times 784}$; layer $l$ right-multiplies by $W^{(l)\,T} \in \mathbb{R}^{n_{l-1} \times n_l}$, so the batch dimension $32$ rides untouched through every product: $$ \underbrace{X}_{32 \times 784} \;W^{(1)\,T}\;\underbrace{\phantom{X}}_{784 \times 128} =\; \underbrace{Z^{(1)}}_{32 \times 128} \;\to\; \underbrace{H^{(2)}}_{32 \times 64} \;\to\; \underbrace{\hat{Y}}_{32 \times 10}. $$ The bias $b^{(1)} \in \mathbb{R}^{128}$ is added to each of the $32$ rows of $XW^{(1)\,T}$ by broadcasting — the library copies the length-$128$ vector down the batch axis rather than materializing a $32 \times 128$ bias. Every intermediate is a matrix whose first axis is the batch and whose second axis is the layer width, and the widths telescope exactly as in the single-example chain. ## A forward pass, traced with numbers For example, take a $2$–$3$–$1$ net with **tanh** hidden units and a linear output, feed it $x = (1,\, -2)$, and read every intermediate off the algorithm above. The parameters are $$ W^{(1)} = \begin{pmatrix} 0.5 & -0.3 \\ 0.1 & 0.8 \\ -0.4 & 0.2 \end{pmatrix},\quad b^{(1)} = \begin{pmatrix} 0.1 \\ -0.2 \\ 0.0 \end{pmatrix},\quad W^{(2)} = \begin{pmatrix} 0.7 & -0.5 & 0.3 \end{pmatrix},\quad b^{(2)} = 0.2. $$ **Layer 1.** The pre-activation $z^{(1)} = W^{(1)} x + b^{(1)}$ is one dot product per hidden unit, and $h^{(1)} = \tanh(z^{(1)})$ squashes each into $(-1, 1)$: | unit | $z^{(1)}_i = W^{(1)}_{i,:}\cdot x + b^{(1)}_i$ | value | $h^{(1)}_i = \tanh(z^{(1)}_i)$ | | --- | --- | --- | --- | | $1$ | $0.5\cdot 1 + (-0.3)(-2) + 0.1$ | $1.20$ | $0.8337$ | | $2$ | $0.1\cdot 1 + 0.8\cdot(-2) - 0.2$ | $-1.70$ | $-0.9354$ | | $3$ | $-0.4\cdot 1 + 0.2\cdot(-2) + 0.0$ | $-0.80$ | $-0.6640$ | **Layer 2.** The output is a single linear readout $\hat y = W^{(2)} h^{(1)} + b^{(2)}$, a weighted sum of the three hidden activations: $$ \hat y = 0.7\,(0.8337) - 0.5\,(-0.9354) + 0.3\,(-0.6640) + 0.2 = 0.5836 + 0.4677 - 0.1992 + 0.2 = 1.0521. $$ Nothing here is more than a dot product and an elementwise squash, repeated once per layer; the whole net is that pattern scaled up to hundreds of units. Notice unit $2$ has saturated close to $-1$ (its pre-activation $-1.70$ sits well into tanh's flat tail), a detail that will matter when [backpropagation](/deep-learning/neural-networks/backpropagation) multiplies by the small derivative there. ## Counting parameters Every entry of every $W^{(l)}$ and $b^{(l)}$ is a free parameter. A layer mapping width $n_{l-1}$ into width $n_l$ contributes $n_l \, n_{l-1}$ weights plus $n_l$ biases, so the total is $$ P = \sum_{l=1}^{L} \parens{n_l \, n_{l-1} + n_l} = \sum_{l=1}^{L} n_l\,(n_{l-1} + 1). $$ For a concrete $784$–$128$–$64$–$10$ classifier (the shape of a small MNIST net), the count is dominated by the first layer, where the wide input meets the first hidden layer: | Layer | Shape $n_l \times n_{l-1}$ | Weights $n_l n_{l-1}$ | Biases $n_l$ | Params | | --- | --- | --- | --- | --- | | $W^{(1)}$ | $128 \times 784$ | $100{,}352$ | $128$ | $100{,}480$ | | $W^{(2)}$ | $64 \times 128$ | $8{,}192$ | $64$ | $8{,}256$ | | $W^{(3)}$ | $10 \times 64$ | $640$ | $10$ | $650$ | | **Total** | | $109{,}184$ | $202$ | $\mathbf{109{,}386}$ | > **Remark (Where the weights live).** Parameter count grows with the _product_ of > adjacent widths. The first layer of a net fed raw high-dimensional input (pixels, > tokens) usually holds most of the weights; narrowing the input or the first > hidden width is the cheapest way to shrink a model. ## Why the nonlinearity is non-negotiable Drop the activation $g$ and the whole tower collapses. A composition of affine maps is itself affine; multiply two layers and the product is one weight matrix times the input plus one bias: $$ W^{(2)}\parens{W^{(1)} x + b^{(1)}} + b^{(2)} = \underbrace{W^{(2)}W^{(1)}}_{W'}\,x + \underbrace{W^{(2)} b^{(1)} + b^{(2)}}_{b'}. $$ Induction extends this to any depth: $L$ stacked linear layers compute exactly what one linear layer computes. > **Theorem (Affine collapse).** Let $f(x) = W^{(L)} \cdots W^{(1)} x + b'$ be a > composition of $L$ affine maps with _no_ nonlinearity between them. Then $f$ is > affine: there exist a single matrix $W' = W^{(L)} \cdots W^{(1)}$ and vector $b'$ > with $f(x) = W' x + b'$ for all $x$. > **Proof.** Induct on $L$. For $L=1$, $f(x) = W^{(1)}x + b^{(1)}$ is affine by > definition. Assume $g(x) = A x + c$ collects the first $L-1$ maps. Then > $f(x) = W^{(L)} g(x) + b^{(L)} = W^{(L)}(Ax + c) + b^{(L)} = (W^{(L)}A)\,x + > (W^{(L)} c + b^{(L)})$, again of the form $W'x + b'$. $\qed$ A hundred stacked linear layers therefore still compute a single hyperplane and still fail XOR. **The nonlinearity is the only reason depth adds expressivity**: it lets each layer transform the space nonlinearly before the next one acts, so interleaved classes can be separated. For example, take two linear layers, $W^{(1)} = \left(\begin{smallmatrix} 2 & 0 \\ 1 & 3 \end{smallmatrix}\right)$, $b^{(1)} = \left(\begin{smallmatrix} 1 \\ -1 \end{smallmatrix}\right)$ and $W^{(2)} = \left(\begin{smallmatrix} 1 & -1 \end{smallmatrix}\right)$, $b^{(2)} = 4$, with no activation between them. Their composite has the single equivalent weight and bias $$ W' = W^{(2)}W^{(1)} = \begin{pmatrix} 1 & -1 \end{pmatrix}\! \begin{pmatrix} 2 & 0 \\ 1 & 3 \end{pmatrix} = \begin{pmatrix} 1 & -3 \end{pmatrix}, \qquad b' = W^{(2)}b^{(1)} + b^{(2)} = (1)(1) + (-1)(-1) + 4 = 6. $$ Check on $x = (2, 1)$: layer by layer, $h^{(1)} = W^{(1)}x + b^{(1)} = (5, 4)$ and then $\hat y = W^{(2)}h^{(1)} + b^{(2)} = 5 - 4 + 4 = 5$; the collapsed form gives $W'x + b' = (1)(2) + (-3)(1) + 6 = 5$, the same value. The two-layer stack is exactly the single map $x \mapsto x_1 - 3x_2 + 6$. With a nonlinearity between the layers, no such single matrix exists. ## A network that computes XOR Here is a fully explicit $2$–$2$–$1$ MLP with **ReLU** hidden units, $g(z) = \max(0, z)$, that computes XOR exactly.[^gf-xor] The hidden layer has two units; the output weights place a $-2$ on the second, the cancellation that carves out the interior: $$ W^{(1)} = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix},\quad b^{(1)} = \begin{bmatrix} 0 \\ -1 \end{bmatrix},\quad W^{(2)} = \begin{bmatrix} 1 & -2 \end{bmatrix},\quad b^{(2)} = 0. $$ $$ % caption: An explicit $2$–$2$–$1$ ReLU network computing XOR: $h_1$ counts active % inputs, $h_2$ fires only when both are on, and $h_1-2h_2$ isolates the odd case. \begin{tikzpicture}[>=stealth, font=\footnotesize, unit/.style={circle, draw, minimum size=9mm, inner sep=0pt}, hid/.style={circle, draw, thick, minimum size=11mm, inner sep=0pt}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} % nodes \node[unit] (x1) at (0,1.3) {$x_1$}; \node[unit] (x2) at (0,-1.3) {$x_2$}; \node[hid] (h1) at (3.4,1.3) {$h_1$}; \node[hid] (h2) at (3.4,-1.3) {$h_2$}; \node[unit, draw=green, text=green] (y) at (6.8,0) {out}; % input -> hidden, weights labeled \draw[->, acc, thick] (x1) -- (h1) node[pos=0.55, above, font=\scriptsize, text=acc] {$1$}; \draw[->, acc, thick] (x2) -- (h1) node[pos=0.30, below, font=\scriptsize, text=acc] {$1$}; \draw[->, acc, thick] (x1) -- (h2) node[pos=0.30, above, font=\scriptsize, text=acc] {$1$}; \draw[->, acc, thick] (x2) -- (h2) node[pos=0.55, below, font=\scriptsize, text=acc] {$1$}; % biases (negative sign kept out of node text; exact values in caption + table) \node[font=\footnotesize] at (3.4,2.55) {\texttt{h1 bias 0}}; \node[font=\footnotesize] at (3.4,-2.55) {\texttt{h2 bias -1}}; % hidden -> output \draw[->, green, thick] (h1) -- (y) node[pos=0.5, above, font=\scriptsize, text=green] {add $1$}; \draw[->, green, thick] (h2) -- (y) node[pos=0.5, below, font=\scriptsize, text=green] {sub $2$}; % ReLU labels \node[font=\scriptsize] at (3.4,0.0) {ReLU}; \end{tikzpicture} $$ Tracing all four inputs through the algorithm above verifies the claim. The pre-activation is $z^{(1)} = W^{(1)} x + b^{(1)}$, the hidden vector is $h = \max(0, z^{(1)})$, and the output is $\hat{y} = h_1 - 2 h_2$: | $(x_1, x_2)$ | $z^{(1)} = (x_1{+}x_2,\; x_1{+}x_2{-}1)$ | $h = \max(0, z^{(1)})$ | $\hat{y} = h_1 - 2h_2$ | XOR | | --- | --- | --- | --- | --- | | $(0,0)$ | $(0,\,-1)$ | $(0,\,0)$ | $0 - 0 = 0$ | $0$ | | $(0,1)$ | $(1,\,0)$ | $(1,\,0)$ | $1 - 0 = 1$ | $1$ | | $(1,0)$ | $(1,\,0)$ | $(1,\,0)$ | $1 - 0 = 1$ | $1$ | | $(1,1)$ | $(2,\,1)$ | $(2,\,1)$ | $2 - 2 = 0$ | $0$ | The last row is the key case: when both inputs are on, $h_1 = 2$ would wrongly fire, but $h_2 = 1$ switches on and its $-2$ weight cancels the surplus exactly. The ReLU is essential: it clips $z^{(1)}_2 = -1$ to $0$ on the $(0,0)$ input so the second unit stays silent there. ## The geometry: intersecting half-planes Each ReLU unit is a **half-plane detector**: it outputs $0$ on one side of the line $w^\top x + b = 0$ and grows linearly on the other. Unit $h_1$ switches on above $x_1 + x_2 = 0$; unit $h_2$ switches on above $x_1 + x_2 = 1$ and, through its $-2$ weight, _subtracts twice as fast_. Writing $s = x_1 + x_2$, the output is $$ \hat{y} = \max(0, s) - 2\max(0, s-1) = \begin{cases} 0 & s \le 0,\\ s & 0 < s \le 1,\\ 2 - s & s > 1, \end{cases} $$ which is positive precisely on the **strip** $0 < s < 2$ — between the line through $(0,0)$ and the line through $(1,1)$. That strip contains the two XOR-positive corners $(0,1), (1,0)$ (where $s=1$) and excludes the two negatives, which sit exactly on its boundary lines ($s=0$ and $s=2$). $$ % caption: Two ReLU units carve the input square into a strip $\hat y>0$ that % contains the XOR-positive corners $(0,1)$ and $(1,0)$ but not the negatives. \begin{tikzpicture}[>=stealth, font=\small] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \definecolor{green}{HTML}{1F9D4D} % coordinate scale: 1 input unit = 2.5 drawing units; corner (1,1) -> (2.5,2.5) % shaded strip between line A (X+Y=0, through origin) and line B (X+Y=5, through (2.5,2.5)), % clipped to the viewing box so the band has clean parallel edges \begin{scope} \clip (-0.8,-0.8) rectangle (3.9,3.9); \fill[green!16] (-0.8,0.8) -- (0.8,-0.8) -- (3.9,1.1) -- (1.1,3.9) -- cycle; \end{scope} % axes (drawn over the shading) \draw[->, thick] (-0.6,0) -- (3.7,0) node[right, font=\footnotesize] {$x_1$}; \draw[->, thick] (0,-0.6) -- (0,3.7) node[above, font=\footnotesize] {$x_2$}; % line A: X + Y = 0 (h1 = 0 boundary), through the origin \draw[acc, thick] (1.3,-1.3) -- (-0.5,0.5) node[pos=0.95, left, font=\scriptsize, text=acc] {$x_1{+}x_2 = 0$}; % line B: X + Y = 5 (the s = 2 boundary), through corner (2.5,2.5) \draw[acc, thick] (3.6,1.4) -- (1.4,3.6) node[pos=0.05, right, font=\scriptsize, text=acc] {$x_1{+}x_2 = 2$}; % data points (XOR corners), scaled by 2.5 \fill[acc] (0,0) circle (3.4pt); \fill[acc] (2.5,2.5) circle (3.4pt); \fill[red] (0,2.5) circle (3.4pt); \fill[red] (2.5,0) circle (3.4pt); \node[acc, font=\scriptsize, anchor=north east] at (-0.05,-0.05) {(0, 0)}; \node[acc, font=\scriptsize, anchor=south west] at (2.58,2.58) {(1, 1)}; \node[red, font=\scriptsize, anchor=south east] at (-0.1,2.62) {(0, 1)}; \node[red, font=\scriptsize, anchor=north west] at (2.62,-0.08) {(1, 0)}; % strip label \node[green!60!black, font=\scriptsize, align=center] at (1.25,0.85) {accept\\strip}; \end{tikzpicture} $$ The two **red** corners (the XOR-positive points $(0,1)$ and $(1,0)$) sit inside the shaded strip; the two **blue** corners sit exactly on its boundary lines, where $\hat y = 0$. Composition has turned two flat half-plane cuts into a bounded slab, a non-convex labelling no single hyperplane could produce. ## Hidden space makes XOR linear The strip picture lives in the input space. The dual picture lives in the **hidden space** $(h_1, h_2)$, where the ReLU layer has re-coordinated the four points so that a single output line separates them: $$ % caption: A hidden layer re-coordinates the inputs: in the learned space % $(h_1,h_2)$ the output line $h_1-2h_2=\tfrac12$ linearly separates the classes. \begin{tikzpicture}[>=stealth, font=\small] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \definecolor{green}{HTML}{1F9D4D} % axes for hidden space \draw[->, thick] (-0.5,0) -- (3.4,0) node[right, font=\footnotesize] {$h_1$}; \draw[->, thick] (0,-0.5) -- (0,3.4) node[above, font=\footnotesize] {$h_2$}; % after the ReLU transform, the four points land at their (h1,h2) images: % (0,0)->(0,0), (1,1)->(2,1) are XOR-negative; (0,1),(1,0)->(1,0) are XOR-positive. % output line h1 - 2 h2 = 1/2 separates them cleanly. % after the ReLU transform the four points land at their (h1,h2) images: % (0,0)->(0,0), (1,1)->(2,1) are XOR-negative; (0,1),(1,0)->(1,0) are XOR-positive. % drawing scale: 1 hidden unit = 1.1 drawing units. % XOR-negative (blue): (0,0) and (2,1) \fill[acc] (0.0,0.0) circle (3.6pt) node[anchor=north east, font=\scriptsize, text=acc] {(0, 0)}; \fill[acc] (2.2,1.1) circle (3.6pt) node[anchor=south, font=\scriptsize, text=acc] {(2, 1)}; % XOR-positive (red): both inputs map to (1,0) \fill[red] (1.1,0.0) circle (3.6pt) node[anchor=north, font=\scriptsize, text=red] {(1, 0)}; % separating line h1 - 2 h2 = 1/2. Drawing units x=1.1 h1, y=1.1 h2 give x=2y+0.55. % y=-0.2 -> x=0.15 ; y=1.6 -> x=3.75. (1,0) is right of it (h1-2h2=1>1/2); % (0,0) and (2,1) are left of it (=0<1/2), cleanly separated. \draw[green, thick] (0.15,-0.2) -- (3.75,1.6) node[pos=1.0, anchor=west, font=\footnotesize, text=green] {\texttt{h1 - 2h2 = 1/2}}; \end{tikzpicture} $$ This is the central idea of deep learning: **instead of fitting a complicated boundary in the input space, learn a representation in which a simple boundary works.** The hidden layers build the representation; the output layer is the same linear model from the last lesson, now applied in coordinates where it succeeds. ## Depth versus width The XOR net used _width_ (two hidden units) to gain expressivity; it could also have used _depth_. Both knobs increase the function class, but they trade off differently, and the difference is a theorem rather than a heuristic. | Knob | Increases | Cost | Effect on the function | | --- | --- | --- | --- | | Width $n_l$ | units per layer | $O(n_l n_{l-1})$ params | more linear regions per layer | | Depth $L$ | layers | $O(n^2)$ params per layer | regions _multiply_ across layers | A shallow-but-wide net can represent any of these functions in principle (that is the [universal approximation](/deep-learning/neural-networks/universal-approximation) theorem), but often only at the cost of exponentially many units. A deep net produces the same number of linear regions by _composition_: each layer folds the regions of the layer below, so region count grows multiplicatively with depth and only additively with the parameters that buy it.[^gf-depth] > **Remark (Why stack layers at all).** Universal approximation guarantees a single > hidden layer _can_ fit any continuous target, but with no bound on its width. > Depth is the resource that makes expressive functions _compact_: a function a > deep net represents with $O(L)$ layers may demand width exponential in the input > dimension from a shallow one. We make this precise in > [universal approximation](/deep-learning/neural-networks/universal-approximation). The next lessons fill in the remaining pieces: _which_ nonlinearity ([activations](/deep-learning/neural-networks/activation-functions)), _how_ the weights get learned ([backpropagation](/deep-learning/neural-networks/backpropagation)), and _how expressive_ the stack is ([universal approximation](/deep-learning/neural-networks/universal-approximation)). ## What a layer became Chollet presents the MLP as a stack of `Dense` layers, the standard picture around 2015. Two developments since changed what a "layer" is without changing the affine-plus-nonlinearity unit. **Residual connections let the stack go deep.** The affine-collapse argument shows depth is useless without a nonlinearity; a second obstacle is that _training_ a very deep plain stack degrades even when it should not, because the gradient weakens and the identity map becomes hard to represent. He, Zhang, Ren & Sun's _residual network_ (2015) adds a shortcut so each block computes $h^{(l)} = h^{(l-1)} + F\!\parens{h^{(l-1)}}$ instead of $h^{(l)} = F(h^{(l-1)})$, where $F$ is the usual affine-plus-nonlinearity. The block now only has to learn the _residual_ correction on top of the input it already has, and the shortcut gives the backward pass a gradient path that skips the weight layers entirely. That single change took trainable depth from tens of layers to hundreds and is now standard in every deep architecture. **The MLP is the compute inside the Transformer.** The dominant architecture of the last decade, the Transformer (Vaswani et al., 2017), interleaves attention with a **position-wise feed-forward network** applied identically to every token: a two-layer MLP, $\max(0,\, x W_1 + b_1)\,W_2 + b_2$, typically widening the hidden dimension by $4\times$ before projecting back. Roughly two-thirds of a Transformer's parameters live in these MLP blocks; attention mixes information _across_ tokens while the MLP does the per-token nonlinear computation this lesson built. Every idea here — the affine map, the elementwise nonlinearity, the shape telescoping — is what that block runs. **Even the mixing can be an MLP.** Tolstikhin et al.'s _MLP-Mixer_ (2021) dropped attention entirely and showed a network built _only_ from MLPs — one MLP mixing across spatial positions, another mixing across channels — reaches competitive image-classification accuracy. Given enough scale and the right connectivity, the stack of affine maps and nonlinearities is a competitive architecture in its own right.[^mixer] ## Takeaways - An **MLP** composes affine maps $W^{(l)} h^{(l-1)} + b^{(l)}$ with an elementwise nonlinearity $g$ between them; the **forward pass** is the literal traversal $x \to h^{(1)} \to \dots \to \hat{y}$ with weight shapes $W^{(l)} \in \mathbb{R}^{n_l \times n_{l-1}}$ that must telescope. - Without $g$, the stack **collapses** to a single affine map (affine-collapse theorem); depth buys nothing, and the nonlinearity is the entire reason it helps. - A concrete $2$–$2$–$1$ **ReLU** net with $W^{(1)} = [[1,1],[1,1]]$, $b^{(1)} = (0,-1)$, $W^{(2)} = [1,-2]$ computes XOR exactly; the second unit's $-2$ weight cancels the surplus on the $(1,1)$ input. - Geometrically, two ReLU units carve **intersecting half-planes** whose strip is the XOR-positive region; equivalently, the hidden layer maps the four points into a space where one line separates them. - **Parameter count** is $\sum_l n_l(n_{l-1}+1)$, dominated by the layer facing the widest input; **depth** vs **width** both add expressivity, but depth multiplies linear regions while costing only additively in parameters. [^gf-ffn]: **Goodfellow**, _Deep Learning_, Ch. 6 — Deep Feedforward Networks: the MLP as a composition of affine maps and elementwise nonlinearities, learning hidden representations rather than hand-designed features. [^gf-xor]: **Goodfellow**, _Deep Learning_, §6.1 — Example: Learning XOR: the explicit two-unit ReLU network that solves the canonical non-linearly-separable problem a single perceptron cannot. [^gf-depth]: **Goodfellow**, _Deep Learning_, §6.4 — Architecture Design: depth multiplies the number of linear regions a piecewise-linear network can express, an exponential advantage over width. [^chollet-tensor]: **Chollet**, _Deep Learning with Python_, Ch. 2 — The Mathematical Building Blocks: tensor operations, broadcasting, and the `Dense` layer as $g(W x + b)$. [^stevens-batch]: **Stevens**, _Deep Learning with PyTorch_, Ch. 5–6 — batched tensors `(N, features)` flow through `nn.Linear` as one matrix multiply, the form a GPU executes in parallel. [^mixer]: Primary sources: He, Zhang, Ren & Sun, "Deep Residual Learning for Image Recognition" (CVPR 2016) for the residual block $h + F(h)$; Vaswani et al., "Attention Is All You Need" (NeurIPS 2017) for the position-wise feed-forward MLP inside each Transformer layer; Tolstikhin et al., "MLP-Mixer: An all-MLP Architecture for Vision" (NeurIPS 2021) for an attention-free network built from MLPs alone.