Neural Networks and Neural Language Models

An n-gram language model predicts the next word by counting: it stores how often the cat gets was followed by each word and normalizes. That works until the context is one it never saw, and in a large vocabulary almost every long context is unseen — the counts are sparse and the estimate collapses. A neural language model replaces the count table with a function. It represents each context word as a dense vector, feeds those vectors through a small network, and reads a next-word distribution off a softmax. Because similar words get similar vectors, a context the model never saw can borrow strength from one it did. The rest of this lesson builds this model.

The model is a neural network: a stack of simple computing units, each a logistic regression classifier in its own right. We start with that single unit.

The unit

The building block is one unit. It takes real-valued inputs $x_1,\dots ,x_n$, forms a weighted sum with a set of weights $w_1,\dots ,w_n$ and a bias $b$, and passes the result through a non-linear function. Written with the dot product, the weighted sum is

a single real number. The unit then applies a non-linear activation function $f$ to get its output, the activation value $a=f(z)$. For a lone unit that activation is the whole network's output, which we write $y$:

The bias is a weight on a dummy input clamped at $+1$. That convention lets us fold $b$ into $\mathbf{w}$ and write every layer uniformly as a matrix multiply, which matters once we stack units.

Activation functions

The activation $f$ is what makes a network more than linear algebra. Three choices dominate.¹

The sigmoid maps any real value into $(0,1)$, squashing outliers toward $0$ or $1$ and staying nearly linear near the origin:

It is differentiable everywhere — handy for learning — and it is the same function that turns a logistic regression score into a probability. Substituting the weighted sum gives the full output of a sigmoid unit, $y=\sigma (\mathbf{w}\cdot \mathbf{x}+b)=1/(1+e^{-(\mathbf{w}\cdot \mathbf{x}+b)})$.

The tanh is a rescaled sigmoid ranging over $(-1,1)$:

Being centered at zero, it usually trains better than the sigmoid at internal layers.

The rectified linear unit or ReLU is the simplest and most common, the identity for positive inputs and zero otherwise:

The differences matter for learning. The sigmoid and tanh saturate: for large $|z|$ their outputs flatten and their derivatives fall near zero. Since training propagates an error signal backward by multiplying local derivatives, a chain of near-zero derivatives shrinks the signal until it vanishes — the vanishing gradient problem. ReLU has derivative $1$ for all positive $z$, so it avoids this failure, which is why it is the default at internal layers.

The XOR problem

A single unit is a linear classifier, and linear classifiers have a hard limit. The classic demonstration is that no single unit can compute the logical XOR of two binary inputs.² AND and OR are easy — a perceptron (a unit with a hard threshold and no non-linearity, outputting $1$ when $\mathbf{w}\cdot \mathbf{x}+b>0$ and $0$ otherwise) computes each with the right weights. XOR it cannot.

The reason is geometric. A unit's decision boundary is the set where $\mathbf{w}\cdot \mathbf{x}+b=0$, which for two inputs is a straight line. It splits the plane into a positive side and a negative side. For AND and OR you can draw such a line; for XOR the two positive points $(0,1)$ and $(1,0)$ sit on one diagonal and the two negative points $(0,0)$ and $(1,1)$ on the other, so no single straight line separates the classes. XOR is not linearly separable.

The fix is a hidden layer. Stack a second layer of units on top of the first and XOR becomes solvable — Goodfellow's construction uses two ReLU units in a middle layer feeding one output unit.² The hidden units transform the input into new coordinates, an $\mathbf{h}$ space, in which the two positive input points collapse together and the classes become linearly separable. Forming a representation of the input in which the final linear readout can succeed is the hidden layer's whole job. That is the recurring theme of neural networks, and it only works because the units are non-linear — a stack of purely linear units collapses back to a single linear map, as we show below.

Worked example: the XOR network, unit by unit

Goodfellow's XOR network is small enough to run by hand, and doing so shows the hidden layer building the separable representation. Two ReLU hidden units feed one output unit, with these weights and biases:²

The hidden layer computes $\mathbf{h}=\mathrm{ReLU}(\mathbf{Wx}+\mathbf{b})$, and the output $y=\mathbf{u}\cdot \mathbf{h}+b_y$. Take $\mathbf{x}=[0,0]$. The pre-activation is $\mathbf{Wx}+\mathbf{b}=[0,0]+[0,-1]=[0,-1]$, and ReLU clips the negative coordinate, so $\mathbf{h}=[0,0]$. Then $y=1(0)+(-2)(0)=0$ — correct, since $0\oplus 0=0$. Run all four inputs:

The decisive row pair is the middle two. In the input space, $[0,1]$ and $[1,0]$ sit on opposite corners; the hidden layer maps both to the same point $\mathbf{h}=[1,0]$. The two positive cases have been collapsed onto one location, and now a single line in $\mathbf{h}$-space — the readout $y=h_1-2h_2$ — separates the positives (at $[1,0]$, where $y=1$) from the negatives (at $[0,0]$ and $[2,1]$, where $y=0$). The hidden layer did not classify; it mapped the input into a space where the final linear unit could.

Feedforward networks

A feedforward network is a multilayer network with no cycles: each layer's outputs feed the next, never backward. It has three kinds of layers — an input layer, one or more hidden layers, and an output layer — and in the standard architecture each layer is fully connected, so every unit reads every output of the layer below.³ (Historically these are also called multi-layer perceptrons or MLPs, though modern units are not perceptrons.)

Collecting each hidden unit's weight vector as a row of a matrix $\mathbf{W}$ and its bias into a vector $\mathbf{b}$, the entire hidden layer is one matrix operation followed by an element-wise activation. For an input $\mathbf{x}\in {\mathbb{R}}^{n_0}$ and a hidden layer of $n_1$ units,

where $\sigma$ (or ReLU, or tanh) is applied to each coordinate. Element ${\mathbf{W}}_{ji}$ is the weight from input $x_i$ to hidden unit $h_j$, so the $j$-th coordinate is just the familiar $h_j=\sigma ({\sum }_i{\mathbf{W}}_{ji}{x}_i+b_j)$.

The output layer applies a second weight matrix $\mathbf{U}\in {\mathbb{R}}^{n_2\times n_1}$ to the hidden vector, producing an intermediate score vector $\mathbf{z}$, and then a softmax normalizes those scores into a probability distribution over the $n_2$ output classes:

Put together, a two-layer network — one hidden layer, one output layer — computes

By the convention of counting only layers with weights (not the input), this is a 2-layer network, and logistic regression — one weight layer, one softmax — is a 1-layer network. A neural classifier is thus logistic regression run on features $\mathbf{h}$ that the earlier layers learned, rather than features a human designed by hand.

The forward pass in matrix form

Deeper networks reuse the same two operations at every layer. Writing bracketed superscripts for layer index and ${\mathbf{a}}^{[0]}=\mathbf{x}$ for the input, an $n$-layer network computes its output by a single loop, each layer a matrix multiply plus an activation.³

The activation $g^{[i]}$ differs by layer: an internal layer uses ReLU or tanh, and the final layer uses softmax for multiclass output or the sigmoid for a binary decision.

Why the non-linearity is not optional

Without activations, the stack collapses to one linear map. Suppose two layers were purely linear, ${\mathbf{z}}^{[1]}={\mathbf{W}}^{[1]}\mathbf{x}+{\mathbf{b}}^{[1]}$ and ${\mathbf{z}}^{[2]}={\mathbf{W}}^{[2]}{\mathbf{z}}^{[1]}+{\mathbf{b}}^{[2]}$. Substituting,

The composition is a single affine map ${\mathbf{W}}^{\prime }\mathbf{x}+{\mathbf{b}}^{\prime }$. This generalizes to any depth: without non-linear activations a deep network is just a notational variant of one linear layer, and all the representational power of depth is lost. The non-linearity is what lets each hidden layer bend the space into a form the next layer can use.

The feedforward neural language model

Now the target application. A neural language model predicts the next word from the previous ones, exactly the task of an n-gram model, but it represents the context by embeddings rather than word identity.⁴ Like the n-gram model it makes a Markov approximation, using only the last $N-1$ words:

The architecture is the feedforward network of Bengio and colleagues.⁴ Take a window of the $N-1$ previous words. Each word is first a one-hot vector of length $|V|$ — all zeros except a single $1$ at the word's index in the vocabulary. Multiplying a one-hot vector by an embedding matrix $\mathbf{E}\in {\mathbb{R}}^{d\times |V|}$ selects one column, the $d$-dimensional embedding ${\mathbf{e}}_w$ of that word. The $N-1$ embeddings are concatenated into the embedding layer $\mathbf{e}$, passed through a hidden layer, and finished with a softmax over the whole vocabulary.

Writing the semicolon for concatenation, the model with a window of $3$ computes

where each ${\mathbf{x}}_{t-k}$ is the one-hot vector of a context word, so $\mathbf{E}{\mathbf{x}}_{t-k}={\mathbf{e}}_{w_{t-k}}$ is its embedding. Output node ${\hat{y}}_i$ gives $P(w_t=V_i\mid w_{t-3},w_{t-2},w_{t-1})$, the probability that the next word is the $i$-th vocabulary item.

Worked example: shapes and the forward pass

Fix the sizes to trace the forward pass end to end. Say the vocabulary is $|V|=20000$, the embedding dimension $d=50$, the window $N-1=3$, and the hidden layer $d_h=200$. The one-hot of each context word is a $20000$-vector with a single $1$. Multiplying by $\mathbf{E}\in {\mathbb{R}}^{50\times 20000}$ selects one column, a $50$-vector — no arithmetic actually happens, it is a table lookup. Concatenating three gives $\mathbf{e}\in {\mathbb{R}}^{150}$. Then $\mathbf{W}\in {\mathbb{R}}^{200\times 150}$ maps that to the hidden layer $\mathbf{h}\in {\mathbb{R}}^{200}$, and $\mathbf{U}\in {\mathbb{R}}^{20000\times 200}$ maps $\mathbf{h}$ to the score vector $\mathbf{z}\in {\mathbb{R}}^{20000}$, one score per vocabulary word, which softmax normalizes.

The parameter count is dominated by the two matrices touching the vocabulary. The embedding matrix has $50\times 20000=1,000,000$ weights; the output matrix has $200\times 20000=4,000,000$; the hidden matrix only $200\times 150=30,000$. The softmax over $20000$ classes is the expensive step at both training and inference, which is why later models economize on it — and why the byproduct $\mathbf{E}$, cheap to store, is the piece most often reused elsewhere.

One matrix $\mathbf{E}$ serves all $N-1$ positions rather than a separate matrix per slot: over a long text every word turns up in every position, and we want a single vector per word regardless of where it lands. The embeddings can be pretrained (initialized from word2vec or GloVe and either frozen or fine-tuned) or learned from scratch during language-model training. Either way the parameter set is $\theta =\{\mathbf{E},\mathbf{W},\mathbf{U},\mathbf{b}\}$.

Training

Training is the same procedure as logistic regression, scaled up: a cross-entropy loss minimized by stochastic gradient descent, with the gradient supplied by backpropagation.⁵

For language modeling the classes are the vocabulary words, and only one is correct at each step — the actual next word $w_t$. With a one-hot gold vector the multiclass cross-entropy collapses to the negative log probability the model assigns to that one correct word:

This is the negative log likelihood loss. Minimizing it pushes probability mass onto the word that actually occurred. Substituting the softmax makes the score $z_i$ of the correct word explicit, $L_{\mathrm{CE}}=-\log \frac{\exp (z_i)}{{\sum }_j\exp (z_j)}$, so the loss rises whenever a wrong word's score competes with the right one's.

Worked example: softmax, loss, and its gradient

Take a toy vocabulary of three words $\{\text{fish},\text{do},\text{zebra}\}$ and suppose the output scores at one step are $\mathbf{z}=[2.0,1.0,0.1]$, with the true next word fish (index $1$). Exponentiate: $e^{2.0}=7.39$, $e^{1.0}=2.72$, $e^{0.1}=1.11$, summing to $11.22$. The softmax is

The loss is $L_{\mathrm{CE}}=-\log {\hat{y}}_1=-\log (0.659)=0.417$ nats. Had the model been certain and correct (${\hat{y}}_1=1$) the loss would be $0$; had it put all mass on a wrong word the loss would diverge. The gradient of this loss with respect to the scores has a clean closed form, $\partial L_{\mathrm{CE}}/\partial \mathbf{z}=\hat{\mathbf{y}}-\mathbf{y}$ — the prediction minus the one-hot gold vector. Here $\mathbf{y}=[1,0,0]$, so

The negative component on the true word raises its score next step; the positive components on the two wrong words lower theirs. That single vector, $\hat{\mathbf{y}}-\mathbf{y}$, is the error signal backpropagation carries into every earlier layer, which is why the same subtraction appears at the output of every softmax classifier.

The gradient of this loss with respect to every parameter is computed by backpropagation — backward differentiation over the network's computation graph, applying the chain rule from the loss back to each weight. In outline: the loss's derivative at the output simplifies to $\hat{\mathbf{y}}-\mathbf{y}$ (prediction minus gold), and that error signal is multiplied by each layer's local derivative on the way back, giving $\partial L/\partial \theta$ for every parameter including the embeddings. The full derivation belongs to the deep-learning treatment of backpropagation; here it is enough that the gradient is exact and cheap to compute.

With the gradient in hand, SGD updates each parameter by a small step against it. Over a long training text the model slides a window across the corpus, predicting each word from its predecessors and taking one step per position.

Because the gradient reaches $\mathbf{E}$, the update tunes the embeddings while learning to predict, so a good next-word predictor and a good set of word vectors fall out of the same objective. The learned $\mathbf{E}$ is a byproduct usable as word representations elsewhere — the same idea that underlies vector semantics. Optimization here is non-convex, so weights start from small random values (not zero), inputs are normalized, and regularizers such as dropout curb overfitting; these are the standard neural-net training practices, treated in full in the deep-learning course.

Measuring a language model: perplexity

A language model is scored on held-out text by perplexity, the exponential of the average per-word cross-entropy. For a test corpus $w_1\dots w_T$,

Perplexity is the model's average branching factor — a perplexity of $100$ means the model is, on average, as uncertain as if choosing uniformly among $100$ words. And it is nothing but the training loss, averaged and exponentiated, so minimizing cross-entropy minimizes perplexity. On the same corpus a neural language model reaches lower perplexity than an n-gram model of comparable order, a quantitative consequence of the statistical-sharing argument below.

From Bengio to the transformer

The feedforward language model of this lesson is the direct ancestor of every model that followed. Three public developments trace the line from it to the modern era.

The original neural LM (Bengio et al., 2003).⁶ The architecture here is Bengio and colleagues' A Neural Probabilistic Language Model. Their central claim was that fighting the curse of dimensionality in language required learning a distributed representation for words jointly with the probability function, so that a sentence similar to one in training — similar in the embedding space — inherits its probability. They reported perplexity improvements of $20$–$35\%$ over the best smoothed trigram models on the Brown and AP News corpora, the first clear win of a neural LM over n-grams, and the paper that established learned word embeddings a decade before word2vec.

Weight tying (Press and Wolf, 2017; Inan et al., 2017).⁷ The model has two big vocabulary-sized matrices: the input embedding $\mathbf{E}$ and the output matrix $\mathbf{U}$. Independent work showed that sharing them — using ${\mathbf{E}}^{\top }$ as the output projection — cuts parameters by up to half and consistently lowers perplexity, because the two matrices learn compatible geometry anyway. Tied embeddings became standard in later language models and remain common in transformer LMs.

Scaling the same objective. The feedforward LM's fixed window is its main limitation. Removing it, while keeping the two commitments made here — words as learned vectors, training by cross-entropy — produced the recurrent language model, which carries a state across the whole sequence, and then the transformer, which lets every position attend to every other. Both are, at the loss level, this lesson's classifier: a softmax over the vocabulary trained by negative log likelihood, with a richer function feeding the scores. The GPT family (Radford et al., 2018 onward; Brown et al., 2020) is a decoder-only transformer trained on precisely this objective at scale — the same next-word cross-entropy, over a much larger context and corpus.⁸

Why embeddings cure sparsity

Return to the opening problem. An n-gram model treats words as atomic symbols with nothing in common: cat and dog are as unrelated as cat and the. So a model that saw I have to make sure that the cat gets fed learns a probability for fed after the cat gets and nothing about the dog gets — a distinct context, unseen, estimated from a back-off or a zero.⁴

The neural model shares strength through the embedding space. cat and dog land near each other because they appear in similar contexts, so the hidden layer sees nearly the same input whether the context word is cat or dog. Having learned to assign fed high probability after the cat gets, the model generalizes the same prediction to the dog gets — a context it never observed — because the two contexts are close in the space it built. The count table could not do this; the embedding does it automatically.

This is why neural language models generalize better and handle longer histories than n-gram models, at the cost of being slower and less interpretable — for small tasks an n-gram model is still a reasonable choice. The feedforward model here is the simplest of the family: its window is fixed, so it cannot see beyond $N-1$ words back. Lifting that limit is the subject of the next lessons. A recurrent network carries a running state across the whole sequence, and the transformer lets every position attend to every other — but both keep the two commitments made here: represent words as learned vectors, and train by gradient descent on a cross-entropy loss.