--- title: Neural Networks and Neural Language Models module: Semantics moduleNumber: 3 lessonNumber: 3 order: 303 summary: > A neural network is a stack of units, each a weighted sum passed through a non-linearity — a single unit on its own is logistic regression. We build the network up from that unit: the activation functions that give it power, the XOR problem that forces a hidden layer, the feedforward forward pass in matrix form, and the Bengio-style feedforward neural language model that concatenates word embeddings and predicts the next word with a softmax. Training is cross-entropy minimized by gradient descent, with backpropagation supplying the gradient. Embeddings let the model share statistical strength across similar words, avoiding the sparsity that limits n-gram models. topics: [Semantics] sources: - book: Jurafsky ref: "Ch. 7 — Neural Networks and Neural Language Models; §7.1 Units; §7.2 The XOR Problem" - book: Jurafsky ref: "§7.3 Feedforward Networks; §7.5 Feedforward Neural Language Modeling; §7.6 Training Neural Nets; §7.7 Training the Neural Language Model" --- An [n-gram language model](/natural-language-processing/foundations/n-gram-language-models) predicts the next word by counting: it stores how often "the cat gets" was followed by each word and normalizes. That works until the context is one it never saw, and in a large vocabulary almost every long context is unseen — the counts are sparse and the estimate collapses. A neural language model replaces the count table with a function. It represents each context word as a dense vector, feeds those vectors through a small network, and reads a next-word distribution off a softmax. Because similar words get similar vectors, a context the model never saw can borrow strength from one it did. The rest of this lesson builds this model. The model is a **neural network**: a stack of simple computing units, each a [logistic regression](/natural-language-processing/classification/logistic-regression) classifier in its own right. We start with that single unit. ## The unit The building block is one **unit**. It takes real-valued inputs $x_1, \ldots, x_n$, forms a weighted sum with a set of weights $w_1, \ldots, w_n$ and a **bias** $b$, and passes the result through a non-linear function. Written with the dot product, the weighted sum is $$ z \;=\; \mathbf{w} \cdot \mathbf{x} + b \;=\; b + \sum_i w_i x_i, $$ a single real number. The unit then applies a non-linear **activation** function $f$ to get its output, the activation value $a = f(z)$. For a lone unit that activation is the whole network's output, which we write $y$: $$ y \;=\; a \;=\; f(\mathbf{w} \cdot \mathbf{x} + b). $$ > **Definition (Unit).** A single computational node that outputs > $a = f(\mathbf{w}\cdot\mathbf{x} + b)$: a weighted sum of its inputs plus a bias, > passed through a non-linear activation $f$. With the sigmoid for $f$ it is > precisely a logistic regression classifier. $$ % caption: A neural unit takes inputs $x_1, x_2, x_3$ (and a bias $b$ carried as a % weight on a $+1$ input), forms the weighted sum $z = \mathbf{w}\cdot\mathbf{x}+b$, % and passes it through the activation $f$ to produce the output $a=y$. \begin{tikzpicture}[>=stealth, font=\small, sumnode/.style={circle, draw, minimum size=8mm, inner sep=0pt}, actnode/.style={draw, minimum width=8mm, minimum height=8mm, inner sep=0pt}] \definecolor{acc}{HTML}{2348F2} \node (x1) at (0,1.4) {$x_1$}; \node (x2) at (0,0.4) {$x_2$}; \node (x3) at (0,-0.6) {$x_3$}; \node (b) at (0,-1.6) {$+1$}; \node[sumnode] (sum) at (3.0,0) {$+$}; \node[actnode, draw=acc, text=acc] (act) at (4.6,0) {$f$}; \node (out) at (6.3,0) {$a = y$}; \draw[->] (x1) -- (sum) node[midway, above, font=\scriptsize] {$w_1$}; \draw[->] (x2) -- (sum) node[midway, above, font=\scriptsize] {$w_2$}; \draw[->] (x3) -- (sum) node[midway, above, font=\scriptsize] {$w_3$}; \draw[->, black] (b) -- (sum) node[midway, below, font=\scriptsize] {$b$}; \draw[->] (sum) -- (act) node[midway, above, font=\scriptsize] {$z$}; \draw[->, acc] (act) -- (out); \end{tikzpicture} $$ The bias is a weight on a dummy input clamped at $+1$. That convention lets us fold $b$ into $\mathbf{w}$ and write every layer uniformly as a matrix multiply, which matters once we stack units. ### Activation functions The activation $f$ is what makes a network more than linear algebra. Three choices dominate.[^jm-units] The **sigmoid** maps any real value into $(0,1)$, squashing outliers toward $0$ or $1$ and staying nearly linear near the origin: $$ \sigma(z) \;=\; \frac{1}{1 + e^{-z}}. $$ It is differentiable everywhere — handy for learning — and it is the same function that turns a logistic regression score into a probability. Substituting the weighted sum gives the full output of a sigmoid unit, $y = \sigma(\mathbf{w}\cdot\mathbf{x} + b) = 1 / (1 + e^{-(\mathbf{w}\cdot\mathbf{x}+b)})$. The **tanh** is a rescaled sigmoid ranging over $(-1, 1)$: $$ \tanh(z) \;=\; \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}}. $$ Being centered at zero, it usually trains better than the sigmoid at internal layers. The **rectified linear unit** or **ReLU** is the simplest and most common, the identity for positive inputs and zero otherwise: $$ \mathrm{ReLU}(z) \;=\; \max(z, 0). $$ $$ % caption: The three standard activations. The sigmoid $\sigma$ saturates toward % $0$ and $1$; tanh saturates toward $-1$ and $+1$; ReLU is flat below $0$ and % linear above it, so its slope never shrinks for large positive $z$. \begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \definecolor{grn}{HTML}{1F9D4D} % axes \draw[black, ->] (-2.6,0) -- (2.9,0) node[right] {$z$}; \draw[black, ->] (0,-1.4) -- (0,1.7); \node[black, anchor=north east, font=\scriptsize] at (0,0) {0}; \node[black, anchor=east, font=\scriptsize] at (0,1.5) {1}; \node[black, anchor=east, font=\scriptsize] at (0,-1.2) {-1}; % sigmoid: 1/(1+e^-2.2z), scaled to [0,1.5] \draw[acc, thick, domain=-2.5:2.7, samples=60, variable=\z] plot ({\z}, {1.5/(1+exp(-2.2*\z))}); \node[acc, anchor=west] at (2.72,1.42) {sigmoid}; % tanh, scaled to [-1.2,1.2] \draw[red, thick, domain=-2.5:2.7, samples=60, variable=\z] plot ({\z}, {1.2*tanh(1.6*\z)}); \node[red, anchor=west] at (2.72,1.12) {tanh}; % relu \draw[grn, thick] (-2.5,0) -- (0,0) -- (1.5,1.5); \node[grn, anchor=south] at (1.15,1.58) {ReLU}; \end{tikzpicture} $$ The differences matter for learning. The sigmoid and tanh **saturate**: for large $|z|$ their outputs flatten and their derivatives fall near zero. Since training propagates an error signal backward by multiplying local derivatives, a chain of near-zero derivatives shrinks the signal until it vanishes — the **vanishing gradient** problem. ReLU has derivative $1$ for all positive $z$, so it avoids this failure, which is why it is the default at internal layers. ## The XOR problem A single unit is a linear classifier, and linear classifiers have a hard limit. The classic demonstration is that no single unit can compute the logical **XOR** of two binary inputs.[^jm-xor] AND and OR are easy — a **perceptron** (a unit with a hard threshold and no non-linearity, outputting $1$ when $\mathbf{w}\cdot\mathbf{x}+b > 0$ and $0$ otherwise) computes each with the right weights. XOR it cannot. The reason is geometric. A unit's decision boundary is the set where $\mathbf{w}\cdot\mathbf{x} + b = 0$, which for two inputs is a straight line. It splits the plane into a positive side and a negative side. For AND and OR you can draw such a line; for XOR the two positive points $(0,1)$ and $(1,0)$ sit on one diagonal and the two negative points $(0,0)$ and $(1,1)$ on the other, so no single straight line separates the classes. XOR is not **linearly separable**. $$ % caption: AND and OR are linearly separable — one line cuts the positive points % (filled) from the negative (open). XOR is not: its positive points $(0,1)$ and % $(1,0)$ and negative points $(0,0)$, $(1,1)$ interleave, so no single line works. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % ---- AND ---- \begin{scope} \draw[black, ->] (-0.3,0) -- (1.9,0) node[right, font=\scriptsize] {x1}; \draw[black, ->] (0,-0.3) -- (0,1.9) node[above, font=\scriptsize] {x2}; \draw[fill=white] (0,0) circle (2.2pt); \draw[fill=white] (1.4,0) circle (2.2pt); \draw[fill=white] (0,1.4) circle (2.2pt); \fill[acc] (1.4,1.4) circle (2.4pt); \draw[red, thick, dashed] (0.2,1.75) -- (1.75,0.2); \node[anchor=north, font=\scriptsize] at (0.8,-0.35) {AND}; \end{scope} % ---- OR ---- \begin{scope}[xshift=3.6cm] \draw[black, ->] (-0.3,0) -- (1.9,0) node[right, font=\scriptsize] {x1}; \draw[black, ->] (0,-0.3) -- (0,1.9) node[above, font=\scriptsize] {x2}; \draw[fill=white] (0,0) circle (2.2pt); \fill[acc] (1.4,0) circle (2.4pt); \fill[acc] (0,1.4) circle (2.4pt); \fill[acc] (1.4,1.4) circle (2.4pt); \draw[red, thick, dashed] (-0.05,0.75) -- (0.75,-0.05); \node[anchor=north, font=\scriptsize] at (0.8,-0.35) {OR}; \end{scope} % ---- XOR ---- \begin{scope}[xshift=7.2cm] \draw[black, ->] (-0.3,0) -- (1.9,0) node[right, font=\scriptsize] {x1}; \draw[black, ->] (0,-0.3) -- (0,1.9) node[above, font=\scriptsize] {x2}; \draw[fill=white] (0,0) circle (2.2pt); \fill[acc] (1.4,0) circle (2.4pt); \fill[acc] (0,1.4) circle (2.4pt); \draw[fill=white] (1.4,1.4) circle (2.2pt); \node[red, font=\large] at (1.0,1.0) {?}; \node[anchor=north, font=\scriptsize] at (0.8,-0.35) {XOR}; \end{scope} \end{tikzpicture} $$ The fix is a **hidden layer**. Stack a second layer of units on top of the first and XOR becomes solvable — Goodfellow's construction uses two ReLU units in a middle layer feeding one output unit.[^jm-xor] The hidden units transform the input into new coordinates, an $\mathbf{h}$ space, in which the two positive input points collapse together and the classes _become_ linearly separable. Forming a **representation** of the input in which the final linear readout can succeed is the hidden layer's whole job. That is the recurring theme of neural networks, and it only works because the units are non-linear — a stack of purely linear units collapses back to a single linear map, as we show below. ### Worked example: the XOR network, unit by unit Goodfellow's XOR network is small enough to run by hand, and doing so shows the hidden layer building the separable representation. Two ReLU hidden units feed one output unit, with these weights and biases:[^jm-xor] $$ \mathbf{W} = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}, \quad \mathbf{b} = \begin{bmatrix} 0 \\ -1 \end{bmatrix}, \qquad \mathbf{u} = \begin{bmatrix} 1 \\ -2 \end{bmatrix}, \quad b_y = 0. $$ The hidden layer computes $\mathbf{h} = \mathrm{ReLU}(\mathbf{W}\mathbf{x} + \mathbf{b})$, and the output $y = \mathbf{u} \cdot \mathbf{h} + b_y$. Take $\mathbf{x} = [0, 0]$. The pre-activation is $\mathbf{W}\mathbf{x} + \mathbf{b} = [0, 0] + [0, -1] = [0, -1]$, and ReLU clips the negative coordinate, so $\mathbf{h} = [0, 0]$. Then $y = 1(0) + (-2)(0) = 0$ — correct, since $0 \oplus 0 = 0$. Run all four inputs: | $\mathbf{x}$ | $\mathbf{W}\mathbf{x} + \mathbf{b}$ | $\mathbf{h} = \mathrm{ReLU}$ | $y = \mathbf{u}\cdot\mathbf{h}$ | XOR | | --- | --- | --- | --- | --- | | $[0,0]$ | $[0, -1]$ | $[0, 0]$ | $0$ | $0$ | | $[0,1]$ | $[1, 0]$ | $[1, 0]$ | $1$ | $1$ | | $[1,0]$ | $[1, 0]$ | $[1, 0]$ | $1$ | $1$ | | $[1,1]$ | $[2, 1]$ | $[2, 1]$ | $0$ | $0$ | The decisive row pair is the middle two. In the input space, $[0,1]$ and $[1,0]$ sit on opposite corners; the hidden layer maps _both_ to the same point $\mathbf{h} = [1, 0]$. The two positive cases have been collapsed onto one location, and now a single line in $\mathbf{h}$-space — the readout $y = h_1 - 2h_2$ — separates the positives (at $[1,0]$, where $y = 1$) from the negatives (at $[0,0]$ and $[2,1]$, where $y = 0$). The hidden layer did not classify; it mapped the input into a space where the final linear unit could. $$ % caption: The XOR hidden layer as a change of coordinates. In the input $x$-space % (left) the two positive points $(0,1)$ and $(1,0)$ cannot be split from the % negatives by one line. The ReLU hidden layer maps them to the same point $(1,0)$ % in $h$-space (right), where a single line separates positives from negatives. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % ---- input space ---- \begin{scope} \draw[black, ->] (-0.3,0) -- (2.4,0) node[right, font=\scriptsize] {x1}; \draw[black, ->] (0,-0.3) -- (0,2.4) node[above, font=\scriptsize] {x2}; \draw[fill=white] (0,0) circle (2.2pt); \fill[acc] (1.8,0) circle (2.4pt); \fill[acc] (0,1.8) circle (2.4pt); \draw[fill=white] (1.8,1.8) circle (2.2pt); \node[red, font=\large] at (0.9,0.9) {?}; \node[anchor=north, font=\scriptsize] at (0.9,-0.4) {input x-space}; \end{scope} % ---- h space ---- \begin{scope}[xshift=5.2cm] \draw[black, ->] (-0.3,0) -- (2.9,0) node[right, font=\scriptsize] {h1}; \draw[black, ->] (0,-0.3) -- (0,2.4) node[above, font=\scriptsize] {h2}; % negatives: (0,0) and (2,1); positive: (1,0) merged \draw[fill=white] (0,0) circle (2.2pt); \draw[fill=white] (2.0,1.0) circle (2.2pt); \fill[acc] (1.0,0) circle (2.4pt); \node[acc, anchor=south, font=\scriptsize] at (1.0,0.12) {(0,1) and (1,0)}; % separating line: y = h1 - 2 h2 = 0.5 (a line splitting them) \draw[red, thick, dashed] (0.5,-0.25) -- (2.25,0.62); \node[anchor=north, font=\scriptsize] at (1.1,-0.4) {separable h-space}; \end{scope} \end{tikzpicture} $$ ## Feedforward networks A **feedforward network** is a multilayer network with no cycles: each layer's outputs feed the next, never backward. It has three kinds of layers — an input layer, one or more **hidden layers**, and an output layer — and in the standard architecture each layer is **fully connected**, so every unit reads every output of the layer below.[^jm-ff] (Historically these are also called **multi-layer perceptrons** or MLPs, though modern units are not perceptrons.) $$ % caption: A 2-layer feedforward network. The input $\mathbf{x}$ (with a bias input % $+1$) connects through weight matrix $\mathbf{W}$ to the hidden layer % $\mathbf{h}$, which connects through $\mathbf{U}$ to the output layer % $\mathbf{y}$. Every unit reads every output of the layer below. \begin{tikzpicture}[>=stealth, font=\footnotesize, u/.style={circle, draw, minimum size=6mm, inner sep=0pt}, hu/.style={circle, draw, fill=black!6, minimum size=6mm, inner sep=0pt}] \definecolor{acc}{HTML}{2348F2} % input layer \node (x1) at (0,2.1) {$x_1$}; \node (x2) at (0,1.1) {$x_2$}; \node (x3) at (0,0.1) {$x_3$}; \node[black] (bx) at (0,-1.0) {$+1$}; % hidden layer \node[hu] (h1) at (2.6,2.3) {}; \node[hu] (h2) at (2.6,1.3) {}; \node[hu] (h3) at (2.6,0.3) {}; \node[hu] (h4) at (2.6,-0.7) {}; % output layer \node[u, draw=acc] (y1) at (5.2,1.8) {}; \node[u, draw=acc] (y2) at (5.2,0.8) {}; \node[u, draw=acc] (y3) at (5.2,-0.2) {}; % edges input -> hidden \foreach \i in {x1,x2,x3} \foreach \j in {h1,h2,h3,h4} \draw[black] (\i) -- (\j); \foreach \j in {h1,h2,h3,h4} \draw[black] (bx) -- (\j); % edges hidden -> output \foreach \i in {h1,h2,h3,h4} \foreach \j in {y1,y2,y3} \draw[acc!45] (\i) -- (\j); % layer labels \node[font=\scriptsize, anchor=north] at (0,-1.5) {input}; \node[font=\scriptsize, anchor=north] at (2.6,-1.5) {hidden $\mathbf{h}$}; \node[font=\scriptsize, anchor=north] at (5.2,-1.5) {output $\mathbf{y}$}; \node[acc, font=\scriptsize] at (1.3,2.6) {W}; \node[acc, font=\scriptsize] at (3.9,2.6) {U}; \end{tikzpicture} $$ Collecting each hidden unit's weight vector as a row of a matrix $\mathbf{W}$ and its bias into a vector $\mathbf{b}$, the entire hidden layer is one matrix operation followed by an element-wise activation. For an input $\mathbf{x} \in \mathbb{R}^{n_0}$ and a hidden layer of $n_1$ units, $$ \mathbf{h} \;=\; \sigma(\mathbf{W}\mathbf{x} + \mathbf{b}), \qquad \mathbf{W} \in \mathbb{R}^{n_1 \times n_0},\; \mathbf{b} \in \mathbb{R}^{n_1}, $$ where $\sigma$ (or ReLU, or tanh) is applied to each coordinate. Element $\mathbf{W}_{ji}$ is the weight from input $x_i$ to hidden unit $h_j$, so the $j$-th coordinate is just the familiar $h_j = \sigma(\sum_i \mathbf{W}_{ji} x_i + b_j)$. The output layer applies a second weight matrix $\mathbf{U} \in \mathbb{R}^{n_2 \times n_1}$ to the hidden vector, producing an intermediate score vector $\mathbf{z}$, and then a **softmax** normalizes those scores into a probability distribution over the $n_2$ output classes: $$ \mathbf{z} = \mathbf{U}\mathbf{h}, \qquad \mathrm{softmax}(\mathbf{z})_i = \frac{\exp(z_i)}{\sum_{j=1}^{n_2} \exp(z_j)}. $$ Put together, a two-layer network — one hidden layer, one output layer — computes $$ \mathbf{h} = \sigma(\mathbf{W}\mathbf{x} + \mathbf{b}), \qquad \mathbf{z} = \mathbf{U}\mathbf{h}, \qquad \mathbf{y} = \mathrm{softmax}(\mathbf{z}). $$ By the convention of counting only layers with weights (not the input), this is a **2-layer network**, and logistic regression — one weight layer, one softmax — is a **1-layer network**. A neural classifier is thus logistic regression run on features $\mathbf{h}$ that the earlier layers _learned_, rather than features a human designed by hand. ### The forward pass in matrix form Deeper networks reuse the same two operations at every layer. Writing bracketed superscripts for layer index and $\mathbf{a}^{[0]} = \mathbf{x}$ for the input, an $n$-layer network computes its output by a single loop, each layer a matrix multiply plus an activation.[^jm-ff] ```algorithm caption: $\textsc{Forward}(\mathbf{x})$ — forward pass through an $n$-layer feedforward network $\mathbf{a}^{[0]} \gets \mathbf{x}$ for $i = 1$ to $n$ do $\mathbf{z}^{[i]} \gets \mathbf{W}^{[i]}\mathbf{a}^{[i-1]} + \mathbf{b}^{[i]}$ // affine map $\mathbf{a}^{[i]} \gets g^{[i]}(\mathbf{z}^{[i]})$ // element-wise activation return $\hat{\mathbf{y}} \gets \mathbf{a}^{[n]}$ ``` The activation $g^{[i]}$ differs by layer: an internal layer uses ReLU or tanh, and the final layer uses softmax for multiclass output or the sigmoid for a binary decision. ### Why the non-linearity is not optional Without activations, the stack collapses to one linear map. Suppose two layers were purely linear, $\mathbf{z}^{[1]} = \mathbf{W}^{[1]}\mathbf{x} + \mathbf{b}^{[1]}$ and $\mathbf{z}^{[2]} = \mathbf{W}^{[2]}\mathbf{z}^{[1]} + \mathbf{b}^{[2]}$. Substituting, $$ \mathbf{z}^{[2]} = \mathbf{W}^{[2]}\!\left(\mathbf{W}^{[1]}\mathbf{x} + \mathbf{b}^{[1]}\right) + \mathbf{b}^{[2]} = \underbrace{\mathbf{W}^{[2]}\mathbf{W}^{[1]}}_{\mathbf{W}'}\,\mathbf{x} + \underbrace{\mathbf{W}^{[2]}\mathbf{b}^{[1]} + \mathbf{b}^{[2]}}_{\mathbf{b}'} = \mathbf{W}'\mathbf{x} + \mathbf{b}'. $$ The composition is a single affine map $\mathbf{W}'\mathbf{x} + \mathbf{b}'$. This generalizes to any depth: without non-linear activations a deep network is just a notational variant of one linear layer, and all the representational power of depth is lost. The non-linearity is what lets each hidden layer bend the space into a form the next layer can use. ## The feedforward neural language model Now the target application. A **neural language model** predicts the next word from the previous ones, exactly the task of an n-gram model, but it represents the context by **embeddings** rather than word identity.[^jm-flm] Like the n-gram model it makes a Markov approximation, using only the last $N-1$ words: $$ P(w_t \mid w_1, \ldots, w_{t-1}) \;\approx\; P(w_t \mid w_{t-N+1}, \ldots, w_{t-1}). $$ The architecture is the feedforward network of Bengio and colleagues.[^jm-flm] Take a window of the $N-1$ previous words. Each word is first a **one-hot vector** of length $|V|$ — all zeros except a single $1$ at the word's index in the vocabulary. Multiplying a one-hot vector by an **embedding matrix** $\mathbf{E} \in \mathbb{R}^{d \times |V|}$ selects one column, the $d$-dimensional embedding $\mathbf{e}_w$ of that word. The $N-1$ embeddings are concatenated into the embedding layer $\mathbf{e}$, passed through a hidden layer, and finished with a softmax over the whole vocabulary. $$ % caption: The feedforward neural language model with a window of $N-1=3$ words. % One-hot inputs select columns of $\mathbf{E}$ to give embeddings, concatenated % into $\mathbf{e}$, mapped through $\mathbf{W}$ to the hidden layer $\mathbf{h}$ % and through $\mathbf{U}$ to a softmax over the vocabulary; here the winner is the % index for "fish". \begin{tikzpicture}[>=stealth, font=\footnotesize, wbox/.style={draw, minimum width=11mm, minimum height=6mm, inner sep=1pt, font=\scriptsize}, ebox/.style={draw, fill=black!6, minimum width=5mm, minimum height=11mm, inner sep=0pt}, hu/.style={circle, draw, fill=black!6, minimum size=5mm, inner sep=0pt}, ou/.style={circle, draw, minimum size=5mm, inner sep=0pt}] \definecolor{acc}{HTML}{2348F2} % context words (one-hot) \node[wbox] (w1) at (0,2.0) {for}; \node[wbox] (w2) at (0,0.9) {all}; \node[wbox] (w3) at (0,-0.2) {the}; \node[font=\scriptsize, anchor=north, align=center] at (0,-0.75) {input\\one-hot}; % embeddings \node[ebox] (e1) at (2.0,2.0) {}; \node[ebox] (e2) at (2.0,0.9) {}; \node[ebox] (e3) at (2.0,-0.2) {}; \node[font=\scriptsize, anchor=north, align=center] at (2.0,-0.85) {embedding e}; \draw[->] (w1) -- (e1) node[midway, above, font=\scriptsize] {E}; \draw[->] (w2) -- (e2) node[midway, above, font=\scriptsize] {E}; \draw[->] (w3) -- (e3) node[midway, above, font=\scriptsize] {E}; % hidden layer \node[hu] (h1) at (4.2,1.9) {}; \node[hu] (h2) at (4.2,1.0) {}; \node[hu] (h3) at (4.2,0.1) {}; \node[font=\scriptsize, anchor=north, align=center] at (4.2,-0.75) {hidden h}; \foreach \i in {e1,e2,e3} \foreach \j in {h1,h2,h3} \draw[black] (\i) -- (\j); \node[acc, font=\scriptsize] at (3.1,2.35) {W}; % output softmax \node[ou, draw=acc] (y1) at (6.6,2.4) {}; \node[ou, draw=acc] (y2) at (6.6,1.5) {}; \node[ou, draw=acc, fill=acc!12] (y3) at (6.6,0.6) {}; \node[ou, draw=acc] (y4) at (6.6,-0.3) {}; \foreach \i in {h1,h2,h3} \foreach \j in {y1,y2,y3,y4} \draw[acc!40] (\i) -- (\j); \node[acc, font=\scriptsize] at (5.4,2.55) {U}; % output labels \node[anchor=west, font=\scriptsize] at (7.0,2.4) {p(aardvark ...)}; \node[anchor=west, font=\scriptsize] at (7.0,1.5) {p(do)}; \node[anchor=west, acc, font=\scriptsize] at (7.0,0.6) {p(f\/ish)}; \node[anchor=west, font=\scriptsize] at (7.0,-0.3) {p(zebra)}; \node[font=\scriptsize, anchor=north, align=center] at (7.7,-0.75) {output softmax}; \end{tikzpicture} $$ Writing the semicolon for concatenation, the model with a window of $3$ computes $$ \mathbf{e} = [\mathbf{E}\mathbf{x}_{t-3};\, \mathbf{E}\mathbf{x}_{t-2};\, \mathbf{E}\mathbf{x}_{t-1}], \qquad \mathbf{h} = \sigma(\mathbf{W}\mathbf{e} + \mathbf{b}), \qquad \mathbf{z} = \mathbf{U}\mathbf{h}, \qquad \hat{\mathbf{y}} = \mathrm{softmax}(\mathbf{z}), $$ where each $\mathbf{x}_{t-k}$ is the one-hot vector of a context word, so $\mathbf{E}\mathbf{x}_{t-k} = \mathbf{e}_{w_{t-k}}$ is its embedding. Output node $\hat{y}_i$ gives $P(w_t = V_i \mid w_{t-3}, w_{t-2}, w_{t-1})$, the probability that the next word is the $i$-th vocabulary item. #### Worked example: shapes and the forward pass Fix the sizes to trace the forward pass end to end. Say the vocabulary is $|V| = 20000$, the embedding dimension $d = 50$, the window $N-1 = 3$, and the hidden layer $d_h = 200$. The one-hot of each context word is a $20000$-vector with a single $1$. Multiplying by $\mathbf{E} \in \mathbb{R}^{50 \times 20000}$ selects one column, a $50$-vector — no arithmetic actually happens, it is a table lookup. Concatenating three gives $\mathbf{e} \in \mathbb{R}^{150}$. Then $\mathbf{W} \in \mathbb{R}^{200 \times 150}$ maps that to the hidden layer $\mathbf{h} \in \mathbb{R}^{200}$, and $\mathbf{U} \in \mathbb{R}^{20000 \times 200}$ maps $\mathbf{h}$ to the score vector $\mathbf{z} \in \mathbb{R}^{20000}$, one score per vocabulary word, which softmax normalizes. The parameter count is dominated by the two matrices touching the vocabulary. The embedding matrix has $50 \times 20000 = 1{,}000{,}000$ weights; the output matrix has $200 \times 20000 = 4{,}000{,}000$; the hidden matrix only $200 \times 150 = 30{,}000$. The softmax over $20000$ classes is the expensive step at both training and inference, which is why later models economize on it — and why the byproduct $\mathbf{E}$, cheap to store, is the piece most often reused elsewhere. > **Definition (Feedforward neural language model).** A feedforward network that > maps the embeddings of the previous $N-1$ words, concatenated, through a hidden > layer to a softmax over the vocabulary, yielding > $P(w_t \mid w_{t-N+1}, \ldots, w_{t-1})$. The embedding matrix $\mathbf{E}$ is > **shared** across the context positions — one dictionary of word vectors, reused > wherever a word appears. One matrix $\mathbf{E}$ serves all $N-1$ positions rather than a separate matrix per slot: over a long text every word turns up in every position, and we want a single vector per word regardless of where it lands. The embeddings can be **pretrained** (initialized from [word2vec or GloVe](/natural-language-processing/semantics/static-word-embeddings) and either frozen or fine-tuned) or learned from scratch during language-model training. Either way the parameter set is $\theta = \{\mathbf{E}, \mathbf{W}, \mathbf{U}, \mathbf{b}\}$. ## Training Training is the same procedure as [logistic regression](/natural-language-processing/classification/logistic-regression), scaled up: a **cross-entropy** loss minimized by **stochastic gradient descent**, with the gradient supplied by **backpropagation**.[^jm-train] For language modeling the classes are the vocabulary words, and only one is correct at each step — the actual next word $w_t$. With a one-hot gold vector the multiclass cross-entropy collapses to the negative log probability the model assigns to that one correct word: $$ L_{\mathrm{CE}} \;=\; -\log \hat{y}_i \;=\; -\log p(w_t \mid w_{t-1}, \ldots, w_{t-N+1}), \qquad i = \text{index of } w_t. $$ This is the **negative log likelihood** loss. Minimizing it pushes probability mass onto the word that actually occurred. Substituting the softmax makes the score $z_i$ of the correct word explicit, $L_{\mathrm{CE}} = -\log\frac{\exp(z_i)}{\sum_{j} \exp(z_j)}$, so the loss rises whenever a wrong word's score competes with the right one's. #### Worked example: softmax, loss, and its gradient Take a toy vocabulary of three words $\{\textit{fish}, \textit{do}, \textit{zebra}\}$ and suppose the output scores at one step are $\mathbf{z} = [2.0, 1.0, 0.1]$, with the true next word _fish_ (index $1$). Exponentiate: $e^{2.0} = 7.39$, $e^{1.0} = 2.72$, $e^{0.1} = 1.11$, summing to $11.22$. The softmax is $$ \hat{\mathbf{y}} = \left[\frac{7.39}{11.22},\, \frac{2.72}{11.22},\, \frac{1.11}{11.22}\right] = [0.659,\, 0.242,\, 0.099]. $$ The loss is $L_{\mathrm{CE}} = -\log \hat{y}_1 = -\log(0.659) = 0.417$ nats. Had the model been certain and correct ($\hat{y}_1 = 1$) the loss would be $0$; had it put all mass on a wrong word the loss would diverge. The gradient of this loss with respect to the scores has a clean closed form, $\partial L_{\mathrm{CE}} / \partial \mathbf{z} = \hat{\mathbf{y}} - \mathbf{y}$ — the prediction minus the one-hot gold vector. Here $\mathbf{y} = [1, 0, 0]$, so $$ \frac{\partial L_{\mathrm{CE}}}{\partial \mathbf{z}} = [0.659 - 1,\; 0.242 - 0,\; 0.099 - 0] = [-0.341,\, 0.242,\, 0.099]. $$ The negative component on the true word raises its score next step; the positive components on the two wrong words lower theirs. That single vector, $\hat{\mathbf{y}} - \mathbf{y}$, is the error signal backpropagation carries into every earlier layer, which is why the same subtraction appears at the output of every softmax classifier. The gradient of this loss with respect to every parameter is computed by backpropagation — backward differentiation over the network's computation graph, applying the chain rule from the loss back to each weight. In outline: the loss's derivative at the output simplifies to $\hat{\mathbf{y}} - \mathbf{y}$ (prediction minus gold), and that error signal is multiplied by each layer's local derivative on the way back, giving $\partial L / \partial \theta$ for every parameter including the embeddings. The full derivation belongs to the [deep-learning treatment of backpropagation](/deep-learning/neural-networks/backpropagation); here it is enough that the gradient is exact and cheap to compute. With the gradient in hand, SGD updates each parameter by a small step against it. Over a long training text the model slides a window across the corpus, predicting each word from its predecessors and taking one step per position. ```algorithm caption: $\textsc{Train-NLM}(\text{corpus } w_1 \ldots w_T,\ \eta)$ — SGD for the neural language model initialize $\theta = \{\mathbf{E}, \mathbf{W}, \mathbf{U}, \mathbf{b}\}$ with small random values repeat for $t = N, N+1, \ldots, T$ do $\mathbf{e} \gets [\mathbf{E}\mathbf{x}_{t-N+1}; \ldots; \mathbf{E}\mathbf{x}_{t-1}]$ // look up and concatenate context embeddings $\mathbf{h} \gets \sigma(\mathbf{W}\mathbf{e} + \mathbf{b})$ // hidden layer $\hat{\mathbf{y}} \gets \mathrm{softmax}(\mathbf{U}\mathbf{h})$ // next-word distribution $L \gets -\log \hat{y}_{i}$ // $i$ = index of the true word $w_t$ $\mathbf{g} \gets \nabla_\theta L$ // backpropagation $\theta \gets \theta - \eta\,\mathbf{g}$ // gradient-descent step until converged return $\theta$ ``` Because the gradient reaches $\mathbf{E}$, the update tunes the embeddings _while_ learning to predict, so a good next-word predictor and a good set of word vectors fall out of the same objective. The learned $\mathbf{E}$ is a byproduct usable as word representations elsewhere — the same idea that underlies [vector semantics](/natural-language-processing/semantics/vector-semantics-and-embeddings). Optimization here is non-convex, so weights start from small random values (not zero), inputs are normalized, and regularizers such as dropout curb overfitting; these are the standard neural-net training practices, treated in full in the [deep-learning course](/deep-learning/foundations/what-is-deep-learning). ### Measuring a language model: perplexity A language model is scored on held-out text by **perplexity**, the exponential of the average per-word cross-entropy. For a test corpus $w_1 \ldots w_T$, $$ \mathrm{PP} = \exp\!\left(-\frac{1}{T}\sum_{t=1}^{T} \log P(w_t \mid w_{