Applications

A trained network is a function $f_{\theta }$; an application is the engineering that makes $f_{\theta }$ fast enough to run, large enough to be accurate, and shaped to the structure of a particular kind of data. Goodfellow's twelfth chapter applies the abstractions: the same five-line training loop, specialized four ways. The specialization is never in the optimizer; it is in the prior each domain builds into the architecture and the preprocessing.¹

Every application in this chapter runs the same recipe, and it pays to name the steps once so the four case studies below read as instances of one procedure.

The steps that vary across domains are the first four. The optimizer, the backpropagation, and the training loop are shared. Each section below walks the recipe once, carrying explicit tensor shapes so the arithmetic of a forward pass is never left implicit.

Large-scale deep learning

The dominant empirical fact of the field is that scale works: test loss falls as a smooth power law in model size, dataset size, and compute, provided all three grow together. The cost is that a modern model neither fits nor trains on one device, so the practical question becomes how to spread one gradient step across many processors.²

The averaged gradient is exact. Averaging the shard-gradients reproduces the full-batch gradient, so data parallelism is mathematically transparent. Writing $g_k={\nabla }_{\theta }\frac{1}{|B_k|}{\sum }_{i\in B_k}\ell (f_{\theta }(x_i),y_i)$ for the gradient on shard $k$,

the synchronized update is $\theta \leftarrow \theta -\eta \frac{1}{K}{\sum }_k{g}_k$, with the sum realized by an all-reduce across devices.

The two axes compose: a large model is sharded by model parallelism across a small group of devices, and that group is replicated by data parallelism across many groups. Orthogonal to both is reducing the cost of each arithmetic operation.

Because half-precision gradients can underflow to zero, the loss is multiplied by a loss-scaling factor $S$ before backpropagation and the resulting gradients divided by $S$ afterward, a units-preserving trick that shifts small gradients into the representable range:

Training cost is one lever; inference cost is another, and it is paid every time the model runs. Three families of techniques shrink a trained model.

The student objective blends the soft-target term with the ordinary hard-label loss,

where the $T^2$ factor restores the gradient magnitude scaled down by softening. The full set of scaling techniques, training-side and inference-side:

Computer vision

Vision is the domain where deep learning first overturned a hand-engineered pipeline.³ The architectural prior is the convolutional network; the data prior is preprocessing and augmentation.

Fix the input tensor first. A batch of RGB images is a rank-4 tensor. PyTorch stores it channels-first as $(N,C,H,W)$; the NHWC convention used by some frameworks stores it as $(N,H,W,C)$. A batch of 64 color images at $224\times 224$ is therefore $(64,3,224,224)$ under PyTorch, holding $64\cdot 3\cdot 224\cdot 224\approx 9.6$ million floats.⁴ Pixels arrive in $[0,255]$ and must be normalized before they reach the network. Channel-wise standardization keeps each input coordinate at unit scale so the first layer's gradients are well-conditioned:

with the statistics ${\mu }_c,{\sigma }_c$ computed once over the training set per channel $c$ and frozen for inference. Data augmentation then manufactures extra training examples by applying label-preserving transformations such as random crops, horizontal flips, color jitter, and small rotations. These encode the invariances the task demands and act as a regularizer; we treat augmentation formally under dropout and data augmentation.⁵

The vision tasks form a spectrum of increasing output structure: from a single label, to a label plus a box, to many boxes, to a label for every pixel.

Box regression typically minimizes a smooth $L_1$ loss on the coordinates, quadratic near zero and linear in the tail so large errors do not dominate the gradient, while overlap is scored by intersection over union, $\mathrm{IoU}=\left|A\cap B\right|/\left|A\cup B\right|$, the fraction of the union two boxes share.

The classification path is the one to trace end to end, because every other vision task grows a different head onto the same convolutional trunk. A ResNet-style network takes the normalized $(N,3,224,224)$ tensor and passes it through successive stages that halve the spatial resolution and double the channel count, so the representation trades pixels for features. The figure carries the shapes.

Read the shapes as a budget. The input holds $3\cdot 224\cdot 224\approx 150000$ numbers with no abstraction. By stage 3 the map is $256\cdot 14\cdot 14\approx 50000$ numbers, each a learned feature summarizing a wide receptive field. Global average pooling then collapses every $14\times 14$ channel map to its mean, turning $(256,14,14)$ into a length-256 vector that no longer depends on where in the frame an object sat. A single linear layer maps that vector to 1000 class logits, and a softmax normalizes them to a distribution. The output head is thus a matrix of shape $256\times 1000$, the loss is cross-entropy against the one-hot label, and the metric reported at test time is top-1 or top-5 accuracy, which the smooth loss only approximates.

The other three tasks reattach a different head to the same trunk. Localization adds a four-number regression head for one box. Detection runs a classification-plus-box head at many spatial locations. Segmentation replaces global pooling with an upsampling decoder that returns to full resolution, emitting a $(K,H,W)$ map of per-pixel class logits scored by pixel-wise cross-entropy and evaluated by mean IoU. The trunk is shared; the head is task-specific.

The characteristic failure modes are worth naming. A classifier trained on clean, centered images degrades on the domain shift of real photographs (odd lighting, occlusion, unusual crops); the fix is heavier augmentation matched to the deployment distribution. A detector floods the frame with near-duplicate boxes unless non-maximum suppression prunes overlapping predictions. A segmenter smears object boundaries because pooling discarded the fine spatial detail the decoder must reconstruct, which is why skip connections from encoder to decoder (as in U-Net) carry the lost resolution forward.

Natural language processing

Language presents a fundamentally discrete, high-cardinality input: a vocabulary of $V$ tokens, often $V\sim 10^5$. The classical approach, the $n$-gram language model, estimates the probability of the next word from the previous $n-1$ by counting:

This is exact arithmetic on a lookup table, and it collapses under the curse of dimensionality. The table has one entry per possible context, so its size grows exponentially in $n$:

almost all of which are never observed, leaving the count zero and the probability estimate undefined without smoothing. The deeper problem is that the table treats every word as an isolated symbol: cat and dog are as unrelated as cat and thermodynamics, because one-hot codes are mutually orthogonal and share no statistics.⁶

To address this, replace the one-hot symbol with a dense, learned vector, a word embedding, so that statistically similar words occupy nearby points and the model can generalize across them.

Before the embedding, the raw text must become integers. Tokenization splits a string into units drawn from the vocabulary and maps each to its index, so a sentence of $T$ tokens becomes an integer vector of shape $(T,)$ with entries in $\{0,\dots ,V-1\}$. A batch of $N$ sentences padded to length $T$ is the integer tensor $(N,T)$. The embedding layer is a lookup into the matrix $E\in {\mathbb{R}}^{V\times d}$: row $E[j]$ is the vector for token $j$, so indexing $(N,T)$ into $E$ produces the float tensor $(N,T,d)$.⁷ For $V=10^5$, $d=300$, a 12-token sentence therefore expands from 12 integers to a $12\times 300=3600$-number matrix that a sequence model can consume.

Embeddings are learned from the distributional hypothesis, that a word is known by the company it keeps. word2vec trains $E$ by predicting context words from a center word (skip-gram); GloVe factorizes the global co-occurrence matrix. Both yield the same emergent property: semantic relations become vector arithmetic.

The parallelogram is the geometry of analogy: a single offset vector encodes royalty, another encodes gender, and they add independently, so $v_{\text{king}}-v_{\text{man}}+v_{\text{woman}}$ lands near $v_{\text{queen}}$. With dense vectors in hand, the neural language model replaces the count table with a network: look up each context word's embedding, concatenate, and feed a multilayer perceptron that outputs a softmax over the vocabulary.

The neural model's parameter budget is $Vd+(\text{MLP weights})$, additive rather than exponential in context length, and it generalizes to unseen $n$-grams because nearby embeddings produce nearby predictions. Fixed-width context still limits it. Lifting that limit, so that every token can attend to every other, leads to the Transformer architecture, which dropped the Markov window entirely and now underpins every large language model.

The full sequence pipeline is the same recipe carried through with different heads depending on whether the target is one label for the whole sequence or one label per position. The figure traces the shapes.

The output head is the only branch point. Sentiment classification pools the $(T,h)$ state sequence to one vector (mean-pool or the final state) and applies a linear layer to $K$ class logits, scored by cross-entropy and evaluated by accuracy. Sequence tagging (part-of-speech, named entities) keeps the time axis and applies the same linear layer at every position, producing $(T,K)$ logits scored by per-position cross-entropy and evaluated by token-level F1. Language modeling sets $K=V$ and predicts the next token at each step.

The failure modes of a text model trace back to the vocabulary. A token unseen at training time hits the out-of-vocabulary slot and loses all its content, which is why subword tokenization (splitting rare words into known pieces) is standard. Padding a short sequence to length $T$ injects meaningless positions that a naive mean-pool averages into the answer, so the pooling must mask them out. A model trained on one genre transfers poorly to another because the embedding geometry it learned reflects the training corpus, not the target domain.

Speech recognition

Speech recognition maps a waveform to text. The raw signal is first reduced to a sequence of acoustic feature frames, typically log-mel spectrograms or MFCCs computed on overlapping short windows, turning audio into a $T\times F$ matrix that a sequence model can consume. A 3-second utterance windowed every 10 ms with 80 mel bands is the tensor $(300,80)$: 300 time frames, each an 80-dimensional spectral summary.

The central difficulty is alignment: the input has $T$ frames but the output has $L\ll T$ characters, and the frame-to-character correspondence is unknown. Connectionist temporal classification (CTC) solves this without per-frame labels by introducing a blank symbol and summing over all alignments that collapse to the target.⁸

Collapsing removes blanks and merges adjacent repeats, so many frame paths map to the same string. The figure shows two of them landing on the target cat.

The sum over exponentially many alignments is tractable by a forward-backward recursion identical in spirit to the one for hidden Markov models, making the whole acoustic-model-to-transcript path differentiable end to end. The metric at test time is word error rate, the edit distance between prediction and reference over the reference length. The failure modes are homophones the acoustic signal cannot disambiguate (fixed by a language-model rescoring pass) and rare words absent from the training transcripts.

Recommender systems

A recommender predicts how much user $u$ will like item $i$ from a sparse matrix $R\in {\mathbb{R}}^{m\times n}$ of observed ratings, most of whose entries are missing. The standard method is matrix factorization: approximate $R$ by the product of a thin user-factor matrix and a thin item-factor matrix.⁹

The factors are fit by minimizing squared error over the observed entries only. The missing entries contribute nothing to the loss, so the model is never asked to predict a rating that was never given. An $L_2$ penalty controls the low-rank capacity:

solvable by stochastic gradient descent or alternating least squares (fix $P$, solve a least-squares for $Q$, alternate). The model has a structural limitation.

The metric a recommender optimizes at test time is rarely raw squared error. What matters is the ranking of the top few items shown to the user, so systems report ranking scores such as precision-at-$k$ or normalized discounted cumulative gain, and the squared-error training loss is only a differentiable stand-in for them.

Deep recommenders extend the bilinear score $p_u^T{q}_i$ by passing the learned embeddings through a neural network and concatenating side features, which both lifts the linear ceiling and softens cold-start by sharing structure through the content features. This is the same embedding-plus-MLP construction used in neural language models, transplanted to the user-item domain.

One architecture eats the four domains

Goodfellow's chapter presents four domains with four distinct priors — convolution for pixels, recurrence for language, spectral front-ends for audio, factorization for the user-item matrix. The decade since collapsed much of that diversity onto a single architecture: the Transformer (Vaswani et al., 2017, NeurIPS).

The consolidation ran domain by domain. Language went first, as attention replaced the gated RNNs of the NLP section outright. Vision followed with the Vision Transformer (Dosovitskiy et al., 2021, ICLR), which cuts an image into patches, treats them as a token sequence, and matches or beats convolutional networks once the training set is large enough — the convolutional prior turns out to be replaceable by data. Speech followed with wav2vec 2.0 and Whisper (Radford et al., 2022, arXiv), which run Transformer encoders on the spectral frames from the speech section. Even recommenders adopted self-attention over a user's interaction history. The prior that each domain once hard-coded into its architecture is increasingly learned instead, given enough data and compute — the same lesson the Vision Transformer taught first.

The hardware story kept pace. The $O(T^2)$ attention cost that would have made long sequences impractical was addressed by FlashAttention (Dao et al., 2022, NeurIPS), an IO-aware kernel that tiles the attention computation to keep it in fast on-chip memory, turning a memory-bound operation into a compute-bound one without changing the math. The two axes of parallelism from the large-scale section became production frameworks — Megatron-LM for tensor parallelism, ZeRO / FSDP for sharding optimizer state across devices — that make trillion-parameter training routine. The recipe is unchanged from the opening of this lesson: one gradient loop, one increasingly universal architecture, specialized now by the data it is fed rather than the prior baked into it.

Takeaways

• One recipe, four domains. Fix the input tensor and its shape, choose the architecture whose bias matches that shape, attach the output head for the target, pick the loss, then read the residual for domain-specific failure modes. Only the first four steps vary; the optimizer and training loop are shared.
• One loop, four priors. Vision, language, speech, and recommendation reuse the same optimizer; they differ only in the architectural and preprocessing prior fitted to the data's structure.
• Scaling has two axes. Data parallelism splits the batch and averages exact gradients; model parallelism splits the parameters when one model exceeds one device. Mixed precision, quantization, pruning, and distillation cut compute and memory on top.
• Vision is a task spectrum. Classification $\to$ localization $\to$ detection $\to$ segmentation adds output structure at each step; normalization and augmentation supply the data prior.
• Embeddings beat the curse of dimensionality. $n$-gram tables grow as $V^{n-1}$ and treat words as orthogonal symbols; a shared embedding matrix is $Vd$ parameters, generalizes across words, and makes analogy literal vector arithmetic, the road to the Transformer.
• CTC sums over alignments so speech models train without frame-level labels; matrix factorization of the user-item matrix is a low-rank inner product $p_u^T{q}_i$ that fails on the cold-start problem until content features fill the gap.