The Multilayer Perceptron

The linear model failed XOR because a single hyperplane cannot carve up an interleaved space. The multilayer perceptron (MLP) fixes this by inserting a layer of neurons between the input and the output and passing their combined signal through a nonlinearity before the next layer. The result is a stack of affine maps interleaved with elementwise nonlinearities, the canonical deep feedforward network.¹

The architecture

An MLP is a stack of fully connected layers. Layer $l$ takes the vector $h^{(l-1)}$ below it, applies an affine map (a weight matrix plus a bias), and passes the result componentwise through a nonlinear activation $g$:

with the convention $h^{(0)}=x$ and a final layer usually left linear so the output can take any value a regression target or logit needs. Writing the four layer equations out shows the chain explicitly,

Stack as many hidden layers as you like; the count $L$ is the network's depth, the number of units in a layer its width.

The forward pass in matrix form

The whole computation is a sequence of matrix–vector products. The layer widths $n_0,n_1,\dots ,n_L$ determine every shape: the weight matrix maps the layer below into the layer above, so it has one row per output unit and one column per input unit.

It is convenient to name the pre-activation $z^{(l)}=W^{(l)}{h}^{(l-1)}+b^{(l)}$ separately from the post-activation $h^{(l)}=g(z^{(l)})$; the split is what backpropagation later differentiates through. The dimension chain must telescope, each layer's output width becoming the next layer's input width, or the products are undefined:

Reading off the figure: each blue box is a matrix multiply whose shape is printed beneath it, and the bare $g$ box leaves the width untouched. The forward pass is the literal traversal of this chain.

In practice the input is not one vector but a batch of $m$ examples stacked into a matrix $X\in {\mathbb{R}}^{m\times n_0}$ (the row-major, PyTorch-shaped convention), and the layer becomes $H^{(l)}=g\left(H^{(l-1)}{W}^{(l)T}+b^{(l)}\right)$ with the bias broadcast across rows.² The same matrix multiply that processes one example processes the whole batch at once, the reason a GPU is the natural host for the forward pass.³

For example, track the shapes through the $784$–$128$–$64$–$10$ net with a batch of $m=32$. The input is $X\in {\mathbb{R}}^{32\times 784}$; layer $l$ right-multiplies by $W^{(l)T}\in {\mathbb{R}}^{n_{l-1}\times n_l}$, so the batch dimension $32$ rides untouched through every product:

The bias $b^{(1)}\in {\mathbb{R}}^{128}$ is added to each of the $32$ rows of $X{W}^{(1)T}$ by broadcasting — the library copies the length-$128$ vector down the batch axis rather than materializing a $32\times 128$ bias. Every intermediate is a matrix whose first axis is the batch and whose second axis is the layer width, and the widths telescope exactly as in the single-example chain.

A forward pass, traced with numbers

For example, take a $2$–$3$–$1$ net with tanh hidden units and a linear output, feed it $x=(1,-2)$, and read every intermediate off the algorithm above. The parameters are

Layer 1. The pre-activation $z^{(1)}=W^{(1)}x+b^{(1)}$ is one dot product per hidden unit, and $h^{(1)}=\tanh (z^{(1)})$ squashes each into $(-1,1)$:

Layer 2. The output is a single linear readout $\hat{y}=W^{(2)}{h}^{(1)}+b^{(2)}$, a weighted sum of the three hidden activations:

Nothing here is more than a dot product and an elementwise squash, repeated once per layer; the whole net is that pattern scaled up to hundreds of units. Notice unit $2$ has saturated close to $-1$ (its pre-activation $-1.70$ sits well into tanh's flat tail), a detail that will matter when backpropagation multiplies by the small derivative there.

Counting parameters

Every entry of every $W^{(l)}$ and $b^{(l)}$ is a free parameter. A layer mapping width $n_{l-1}$ into width $n_l$ contributes $n_l{n}_{l-1}$ weights plus $n_l$ biases, so the total is

For a concrete $784$–$128$–$64$–$10$ classifier (the shape of a small MNIST net), the count is dominated by the first layer, where the wide input meets the first hidden layer:

Why the nonlinearity is non-negotiable

Drop the activation $g$ and the whole tower collapses. A composition of affine maps is itself affine; multiply two layers and the product is one weight matrix times the input plus one bias:

Induction extends this to any depth: $L$ stacked linear layers compute exactly what one linear layer computes.

A hundred stacked linear layers therefore still compute a single hyperplane and still fail XOR. The nonlinearity is the only reason depth adds expressivity: it lets each layer transform the space nonlinearly before the next one acts, so interleaved classes can be separated.

For example, take two linear layers, $W^{(1)}=\left(\begin{array}{cc}2& 0\\ 1& 3\end{array}\right)$, $b^{(1)}=\left(\begin{array}{c}1\\ -1\end{array}\right)$ and $W^{(2)}=\left(\begin{array}{cc}1& -1\end{array}\right)$, $b^{(2)}=4$, with no activation between them. Their composite has the single equivalent weight and bias

Check on $x=(2,1)$: layer by layer, $h^{(1)}=W^{(1)}x+b^{(1)}=(5,4)$ and then $\hat{y}=W^{(2)}{h}^{(1)}+b^{(2)}=5-4+4=5$; the collapsed form gives $W^{\prime }x+b^{\prime }=(1)(2)+(-3)(1)+6=5$, the same value. The two-layer stack is exactly the single map $x\mapsto x_1-3x_2+6$. With a nonlinearity between the layers, no such single matrix exists.

A network that computes XOR

Here is a fully explicit $2$–$2$–$1$ MLP with ReLU hidden units, $g(z)=\max (0,z)$, that computes XOR exactly.⁴ The hidden layer has two units; the output weights place a $-2$ on the second, the cancellation that carves out the interior:

Tracing all four inputs through the algorithm above verifies the claim. The pre-activation is $z^{(1)}=W^{(1)}x+b^{(1)}$, the hidden vector is $h=\max (0,z^{(1)})$, and the output is $\hat{y}=h_1-2h_2$:

The last row is the key case: when both inputs are on, $h_1=2$ would wrongly fire, but $h_2=1$ switches on and its $-2$ weight cancels the surplus exactly. The ReLU is essential: it clips $z_2^{(1)}=-1$ to $0$ on the $(0,0)$ input so the second unit stays silent there.

The geometry: intersecting half-planes

Each ReLU unit is a half-plane detector: it outputs $0$ on one side of the line $w^{\top }x+b=0$ and grows linearly on the other. Unit $h_1$ switches on above $x_1+x_2=0$; unit $h_2$ switches on above $x_1+x_2=1$ and, through its $-2$ weight, subtracts twice as fast. Writing $s=x_1+x_2$, the output is

which is positive precisely on the strip $0<s<2$ — between the line through $(0,0)$ and the line through $(1,1)$. That strip contains the two XOR-positive corners $(0,1),(1,0)$ (where $s=1$) and excludes the two negatives, which sit exactly on its boundary lines ($s=0$ and $s=2$).

The two red corners (the XOR-positive points $(0,1)$ and $(1,0)$) sit inside the shaded strip; the two blue corners sit exactly on its boundary lines, where $\hat{y}=0$. Composition has turned two flat half-plane cuts into a bounded slab, a non-convex labelling no single hyperplane could produce.

Hidden space makes XOR linear

The strip picture lives in the input space. The dual picture lives in the hidden space $(h_1,h_2)$, where the ReLU layer has re-coordinated the four points so that a single output line separates them:

This is the central idea of deep learning: instead of fitting a complicated boundary in the input space, learn a representation in which a simple boundary works. The hidden layers build the representation; the output layer is the same linear model from the last lesson, now applied in coordinates where it succeeds.

Depth versus width

The XOR net used width (two hidden units) to gain expressivity; it could also have used depth. Both knobs increase the function class, but they trade off differently, and the difference is a theorem rather than a heuristic.

A shallow-but-wide net can represent any of these functions in principle (that is the universal approximation theorem), but often only at the cost of exponentially many units. A deep net produces the same number of linear regions by composition: each layer folds the regions of the layer below, so region count grows multiplicatively with depth and only additively with the parameters that buy it.⁵

The next lessons fill in the remaining pieces: which nonlinearity (activations), how the weights get learned (backpropagation), and how expressive the stack is (universal approximation).

What a layer became

Chollet presents the MLP as a stack of Dense layers, the standard picture around 2015. Two developments since changed what a layer is without changing the affine-plus-nonlinearity unit.

Residual connections let the stack go deep. The affine-collapse argument shows depth is useless without a nonlinearity; a second obstacle is that training a very deep plain stack degrades even when it should not, because the gradient weakens and the identity map becomes hard to represent. He, Zhang, Ren & Sun's residual network (2015) adds a shortcut so each block computes $h^{(l)}=h^{(l-1)}+F\left(h^{(l-1)}\right)$ instead of $h^{(l)}=F(h^{(l-1)})$, where $F$ is the usual affine-plus-nonlinearity. The block now only has to learn the residual correction on top of the input it already has, and the shortcut gives the backward pass a gradient path that skips the weight layers entirely. That single change took trainable depth from tens of layers to hundreds and is now standard in every deep architecture.

The MLP is the compute inside the Transformer. The dominant architecture of the last decade, the Transformer (Vaswani et al., 2017), interleaves attention with a position-wise feed-forward network applied identically to every token: a two-layer MLP, $\max (0,x{W}_1+b_1)W_2+b_2$, typically widening the hidden dimension by $4\times$ before projecting back. Roughly two-thirds of a Transformer's parameters live in these MLP blocks; attention mixes information across tokens while the MLP does the per-token nonlinear computation this lesson built. Every idea here — the affine map, the elementwise nonlinearity, the shape telescoping — is what that block runs.

Even the mixing can be an MLP. Tolstikhin et al.'s MLP-Mixer (2021) dropped attention entirely and showed a network built only from MLPs — one MLP mixing across spatial positions, another mixing across channels — reaches competitive image-classification accuracy. Given enough scale and the right connectivity, the stack of affine maps and nonlinearities is a competitive architecture in its own right.⁶

Takeaways

• An MLP composes affine maps $W^{(l)}{h}^{(l-1)}+b^{(l)}$ with an elementwise nonlinearity $g$ between them; the forward pass is the literal traversal $x\to h^{(1)}\to \cdots \to \hat{y}$ with weight shapes $W^{(l)}\in {\mathbb{R}}^{n_l\times n_{l-1}}$ that must telescope.
• Without $g$, the stack collapses to a single affine map (affine-collapse theorem); depth buys nothing, and the nonlinearity is the entire reason it helps.
• A concrete $2$–$2$–$1$ ReLU net with $W^{(1)}=[[1,1],[1,1]]$, $b^{(1)}=(0,-1)$, $W^{(2)}=[1,-2]$ computes XOR exactly; the second unit's $-2$ weight cancels the surplus on the $(1,1)$ input.
• Geometrically, two ReLU units carve intersecting half-planes whose strip is the XOR-positive region; equivalently, the hidden layer maps the four points into a space where one line separates them.
• Parameter count is ${\sum }_l{n}_l(n_{l-1}+1)$, dominated by the layer facing the widest input; depth vs width both add expressivity, but depth multiplies linear regions while costing only additively in parameters.