---
title: Applications
module: Practical Deep Learning
moduleNumber: 9
lessonNumber: 5
order: 905
summary: >
  We survey large-scale training (the
  hardware, the two axes of parallelism, mixed precision, and the compression
  tricks that shrink a model after it is trained), then specialize the same
  gradient loop to vision, language, speech, and recommendation. Each domain is a
  different prior bolted onto one optimizer: convolutional invariance for pixels,
  distributed word vectors for tokens, sequence transduction for audio, low-rank
  factorization for the user–item matrix.
topics: [Practical Deep Learning]
sources:
  - book: Goodfellow
    ref: "Ch. 12 — Applications; §12.1 Large-Scale Deep Learning"
  - book: Goodfellow
    ref: "§12.2 Computer Vision; §12.4 Natural Language Processing; §12.3 Speech Recognition; §12.5 Other Applications"
  - book: Chollet
    ref: "Ch. 5 — Deep Learning for Computer Vision; Ch. 6 — Sequence Models"
  - book: Stevens
    ref: "Ch. 4 — Real-world data representation: images and text as tensors"
---

A trained network is a function $f_\theta$; an _application_ is the engineering
that makes $f_\theta$ fast enough to run, large enough to be accurate, and shaped
to the structure of a particular kind of data. Goodfellow's twelfth chapter applies
the abstractions: the same five-line training loop, specialized
four ways. The specialization is never in the optimizer; it is in the **prior**
each domain builds into the architecture and the preprocessing.[^gf-apps]

Every application in this chapter runs the same recipe, and it pays to name the
steps once so the four case studies below read as instances of one procedure.

> **Definition (End-to-end application recipe).** Fix the data and its input-tensor
> shape; choose the architecture whose inductive bias matches that shape; attach the
> output head that emits the task's target; pick the loss that scores that target;
> train with the standard gradient loop; measure a task metric that the loss only
> approximates; then read the residual errors for the domain's characteristic
> failure modes.

The steps that vary across domains are the first four. The optimizer, the
backpropagation, and the training loop are shared. Each section below walks the
recipe once, carrying explicit tensor shapes so the arithmetic of a forward pass is
never left implicit.

| Domain | Native input | Prior baked in | Output structure |
| --- | --- | --- | --- |
| Vision | pixel grid $x \in \mathbb{R}^{H\times W\times C}$ | translation equivariance (convolution) | label / box / mask |
| Language | token sequence $x_1,\dots,x_T$ | distributional semantics (embeddings) | next token / sequence |
| Speech | spectrogram frames | temporal locality + monotonic alignment | transcript |
| Recommendation | sparse user–item matrix $R$ | low-rank structure | predicted ratings |

## Large-scale deep learning

The dominant empirical fact of the field is that **scale works**: test loss falls
as a smooth power law in model size, dataset size, and compute, provided all three
grow together. The cost is that a modern model neither fits nor trains on one
device, so the practical question becomes _how to spread one gradient step across
many processors_.[^gf-largescale]

> **Definition (Data parallelism).** Replicate the full model on each of $K$
> devices, split a minibatch of size $B$ into $K$ shards of size $B/K$, compute a
> local gradient on each shard, and average the local gradients into one update.
> Throughput scales with $K$; the per-device memory footprint is unchanged.

The averaged gradient is exact. Averaging the shard-gradients reproduces the
full-batch gradient, so data parallelism is mathematically transparent. Writing
$g_k = \nabla_\theta \tfrac{1}{\abs{B_k}}\sum_{i\in B_k}\ell(f_\theta(x_i),y_i)$ for the
gradient on shard $k$,

$$
g \;=\; \nabla_\theta\,\frac{1}{B}\sum_{i\in B}\ell_i
\;=\; \frac{1}{K}\sum_{k=1}^{K} g_k,
$$

the synchronized update is $\theta \gets \theta - \eta\,\tfrac1K\sum_k g_k$, with
the sum realized by an _all-reduce_ across devices.

> **Definition (Model parallelism).** Partition the parameters of a single model
> across $K$ devices, either by layer (_pipeline_ parallelism) or within a layer
> (_tensor_ parallelism), so each device holds and computes only its slice.
> Required when $\theta$ itself exceeds one device's memory; activations must be
> communicated across the partition boundary.

$$
% caption: Data parallelism (left) replicates the model and splits the batch; model
% parallelism (right) splits one model across devices.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  dev/.style={draw, minimum width=20mm, minimum height=20mm, align=center, font=\scriptsize},
  blk/.style={draw, minimum width=15mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{accmid}{HTML}{6A82F6}
  % --- DATA PARALLELISM (left) ---
  \node[font=\small] at (1.9,3.0) {data parallelism};
  \node[blk, fill=acc!12] (b1) at (0,1.9) {\texttt{batch 1}};
  \node[blk, fill=acc!12] (b2) at (3.8,1.9) {\texttt{batch 2}};
  \node[dev, draw=acc] (m1) at (0,0)   {\texttt{model}\\\texttt{(full copy)}};
  \node[dev, draw=acc] (m2) at (3.8,0) {\texttt{model}\\\texttt{(full copy)}};
  \draw[->, acc, thick] (b1) -- (m1);
  \draw[->, acc, thick] (b2) -- (m2);
  \draw[<->, black, thick] (m1) -- (m2) node[midway, below=1mm, black] {\texttt{all-reduce}};
  % --- MODEL PARALLELISM (right) ---
  \begin{scope}[xshift=8.3cm]
    \node[font=\small] at (1.9,3.0) {model parallelism};
    \node[blk, fill=acc!12] (bb) at (1.9,1.9) {\texttt{one batch}};
    \node[dev, draw=acc] (d1) at (0,0)   {\texttt{layers}\\\texttt{1 to 6}};
    \node[dev, draw=acc] (d2) at (3.8,0) {\texttt{layers}\\\texttt{7 to 12}};
    \draw[->, acc, thick] (bb) -- (d1.north);
    \draw[->, acc, thick] (d1) -- (d2) node[midway, below=1mm, black] {\texttt{activations}};
  \end{scope}
\end{tikzpicture}
$$

The two axes compose: a large model is sharded by model parallelism across a small
group of devices, and that group is replicated by data parallelism across many
groups. Orthogonal to both is reducing the cost of each arithmetic operation.

> **Definition (Mixed-precision training).** Store and multiply most tensors in
> half precision (16-bit) while keeping a master copy of the weights and the
> gradient accumulation in single precision (32-bit). Half-precision matrix
> multiplies run roughly twice as fast and halve memory; the 32-bit master copy
> prevents small updates from being lost to rounding.

Because half-precision gradients can underflow to zero, the loss is multiplied by a
**loss-scaling** factor $S$ before backpropagation and the resulting gradients
divided by $S$ afterward, a units-preserving trick that shifts small gradients
into the representable range:

$$
g \;=\; \frac{1}{S}\,\nabla_\theta\parens{S\cdot\mathcal{L}},
\qquad S = 2^{k}\ \text{chosen so } \min_i \abs{g_i} \ \text{stays representable}.
$$

Training cost is one lever; **inference** cost is another, and it is paid every
time the model runs. Three families of techniques shrink a trained model.

> **Definition (Knowledge distillation).** Train a small _student_ network to match
> the output distribution of a large _teacher_. The student minimizes a divergence
> to the teacher's softened logits $p^{T}_i = \mathrm{softmax}(z_i / T)$ at
> temperature $T>1$, which exposes the teacher's relative confidence across
> classes, the "dark knowledge" a hard label discards.

The student objective blends the soft-target term with the ordinary hard-label loss,

$$
\mathcal{L}_{\text{student}}
\;=\; (1-\alpha)\,\ell\parens{f_{\text{student}}(x), y}
\;+\; \alpha\,T^2\,\mathrm{KL}\!\parens{p^{T}_{\text{teacher}} \,\|\, p^{T}_{\text{student}}},
$$

where the $T^2$ factor restores the gradient magnitude scaled down by softening.
The full set of scaling techniques, training-side and inference-side:

| Technique | Axis | Mechanism | Cost / caveat |
| --- | --- | --- | --- |
| Data parallelism | training throughput | split batch, average gradients | all-reduce bandwidth |
| Model parallelism | training memory | split parameters across devices | activation-transfer latency |
| Pipeline parallelism | training memory | split by layer, micro-batch the pipeline | "bubble" idle time at fill/drain |
| Mixed precision | compute + memory | 16-bit math, 32-bit master copy | needs loss scaling |
| Quantization | inference memory + speed | store/compute weights in int8/int4 | accuracy loss without calibration |
| Pruning | inference memory | zero out low-magnitude weights | needs sparse kernels to pay off |
| Distillation | inference compute | small student mimics large teacher | student caps below teacher accuracy |

> **Definition (Quantization).** Replace 32-bit floating-point weights and
> activations with low-bit integers under an affine map $r = s\,(q - z)$, where $q$
> is the integer code, $s$ a per-tensor scale, and $z$ a zero-point. Integer matrix
> multiplies are cheaper and the model is $4\times$ to $8\times$ smaller; the cost
> is rounding error, mitigated by calibrating $s$ on representative data.

## Computer vision

Vision is the domain where deep learning first overturned a hand-engineered
pipeline.[^gf-vision] The architectural prior is the [convolutional
network](/deep-learning/architectures/convolutional-networks); the data prior is
**preprocessing and augmentation**.

Fix the input tensor first. A batch of RGB images is a rank-4 tensor. PyTorch
stores it channels-first as $(N, C, H, W)$; the NHWC convention used by some
frameworks stores it as $(N, H, W, C)$. A batch of 64 color images at $224\times224$
is therefore $(64, 3, 224, 224)$ under PyTorch, holding
$64 \cdot 3 \cdot 224 \cdot 224 \approx 9.6$ million floats.[^stevens-vision]
Pixels arrive in $[0,255]$ and must be
normalized before they reach the network. Channel-wise standardization keeps each
input coordinate at unit scale so the first layer's gradients are well-conditioned:

$$
\tilde{x}_{c} \;=\; \frac{x_{c} - \mu_{c}}{\sigma_{c}},
\qquad
\mu_{c} = \frac1N\sum_{i} x_{c}^{(i)},
\quad
\sigma_{c}^2 = \frac1N\sum_{i}\parens{x_{c}^{(i)} - \mu_{c}}^2,
$$

with the statistics $\mu_c,\sigma_c$ computed once over the training set per channel
$c$ and frozen for inference. **Data augmentation** then manufactures extra training
examples by applying label-preserving transformations such as random crops,
horizontal flips, color jitter, and small rotations. These encode the invariances the
task demands and act as a regularizer; we treat augmentation formally under
[dropout and data augmentation](/deep-learning/regularization/dropout-and-data-augmentation).[^chollet-vision]

> **Definition (Label-preserving transformation).** A map $\tau$ on inputs such
> that the true label is unchanged, $y(\tau(x)) = y(x)$. Augmenting the training set
> with $\{\tau(x) : \tau \in \mathcal{T}\}$ teaches the model invariance to
> $\mathcal{T}$ without new annotation.

The vision _tasks_ form a spectrum of increasing output structure: from a single
label, to a label plus a box, to many boxes, to a label for every pixel.

$$
% caption: The vision task spectrum: classification emits one label, detection
% labeled boxes, segmentation a per-pixel map; output structure grows rightward.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % --- panel 1: classification ---
  \draw[black, thick] (0,0) rectangle (3,3);
  \draw[acc, thick, fill=acc!15] (1.5,1.4) ellipse (0.9 and 1.0);
  \node[font=\footnotesize, anchor=south] at (1.5,3.05) {\texttt{classification}};
  \node[acc, font=\footnotesize, fill=white, inner sep=1pt] at (1.5,1.4) {\texttt{"cat"}};
  \node[black, font=\footnotesize, anchor=north] at (1.5,-0.15) {\texttt{one label}};
  % --- panel 2: detection ---
  \begin{scope}[xshift=4.2cm]
    \draw[black, thick] (0,0) rectangle (3,3);
    \draw[acc, thick, fill=acc!15] (1.0,1.5) ellipse (0.6 and 0.8);
    \draw[acc, thick, fill=acc!15] (2.2,1.0) ellipse (0.45 and 0.6);
    \draw[green, thick] (0.35,0.6) rectangle (1.65,2.4);
    \draw[red, thick] (1.7,0.35) rectangle (2.7,1.7);
    \node[font=\scriptsize, anchor=south] at (1.5,3.05) {detection};
    \node[green, font=\scriptsize, anchor=south west] at (0.35,2.42) {cat};
    \node[red, font=\scriptsize, anchor=south west] at (1.72,1.72) {dog};
    \node[black, font=\footnotesize, anchor=north] at (1.5,-0.15) {\texttt{labeled boxes}};
  \end{scope}
  % --- panel 3: segmentation ---
  \begin{scope}[xshift=8.4cm]
    \draw[black, thick] (0,0) rectangle (3,3);
    \draw[green, thick, fill=green!15] (0,0) -- (3,0) -- (3,0.7) -- (0,1.1) -- cycle;
    \draw[acc, thick, fill=acc!15] (1.5,1.5) ellipse (0.95 and 1.05);
    \node[font=\scriptsize, anchor=south] at (1.5,3.05) {segmentation};
    \node[acc, font=\scriptsize, fill=white, inner sep=1pt] at (1.5,1.5) {cat};
    \node[green, font=\scriptsize] at (2.3,0.45) {grass};
    \node[black, font=\footnotesize, anchor=north] at (1.5,-0.15) {\texttt{per-pixel label}};
  \end{scope}
  % progression arrow
  \draw[->, acc, thick] (0,-0.95) -- (11.4,-0.95)
     node[midway, below, black, font=\footnotesize] {\texttt{increasing output structure}};
\end{tikzpicture}
$$

| Task | Output | Loss | Representative architecture |
| --- | --- | --- | --- |
| Classification | one label $\hat y$ | cross-entropy | ResNet, ViT |
| Localization | one label + one box $(x,y,w,h)$ | cross-entropy $+$ box regression | overfeat-style heads |
| Detection | $\{(\text{label}, \text{box})_k\}$ | classification $+$ localization, per object | Faster R-CNN, YOLO |
| Segmentation | per-pixel label map | pixel-wise cross-entropy | U-Net, Mask R-CNN |

Box regression typically minimizes a smooth $L_1$ loss on the coordinates,
quadratic near zero and linear in the tail so large errors do not dominate the
gradient, while overlap is scored by **intersection over union**,
$\mathrm{IoU} = \abs{A \cap B} / \abs{A \cup B}$, the fraction of the union two
boxes share.

The classification path is the one to trace end to end, because every other vision
task grows a different head onto the same convolutional trunk. A ResNet-style
network takes the normalized $(N, 3, 224, 224)$ tensor and passes it through
successive stages that halve the spatial resolution and double the channel count,
so the representation trades pixels for features. The figure carries the shapes.

$$
% caption: Vision dataflow. A convolutional trunk halves spatial size and doubles
% channels stage by stage; global pooling flattens the map and a linear head emits
% one logit per class. Shapes are (channels, height, width) for a single image.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  st/.style={draw, minimum width=20mm, minimum height=11mm, align=center, font=\scriptsize},
  sh/.style={font=\scriptsize, text=black, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[st, fill=acc!8, draw=acc] (in)   at (0,0)    {\texttt{input}};
  \node[st, fill=acc!12, draw=acc] (s1)  at (2.6,0)  {\texttt{stage 1}};
  \node[st, fill=acc!16, draw=acc] (s2)  at (5.2,0)  {\texttt{stage 2}};
  \node[st, fill=acc!22, draw=acc] (s3)  at (7.8,0)  {\texttt{stage 3}};
  \node[st, fill=acc!12, draw=acc] (gp)  at (10.4,0) {\texttt{global}\\\texttt{pool}};
  \node[st, fill=acc!8, draw=acc] (fc)   at (13.0,0) {\texttt{linear}\\\texttt{head}};
  \draw[->, acc, thick] (in) -- (s1);
  \draw[->, acc, thick] (s1) -- (s2);
  \draw[->, acc, thick] (s2) -- (s3);
  \draw[->, acc, thick] (s3) -- (gp);
  \draw[->, acc, thick] (gp) -- (fc);
  \node[sh, anchor=north] at (in.south)  {\texttt{(3, 224, 224)}};
  \node[sh, anchor=north] at (s1.south)  {\texttt{(64, 56, 56)}};
  \node[sh, anchor=north] at (s2.south)  {\texttt{(128, 28, 28)}};
  \node[sh, anchor=north] at (s3.south)  {\texttt{(256, 14, 14)}};
  \node[sh, anchor=north] at (gp.south)  {\texttt{(256,)}};
  \node[sh, anchor=north] at (fc.south)  {\texttt{(1000,)}};
\end{tikzpicture}
$$

Read the shapes as a budget. The input holds $3 \cdot 224 \cdot 224 \approx 150\,000$
numbers with no abstraction. By stage 3 the map is $256 \cdot 14 \cdot 14 \approx
50\,000$ numbers, each a learned feature summarizing a wide receptive field. **Global
average pooling** then collapses every $14\times14$ channel map to its mean, turning
$(256, 14, 14)$ into a length-256 vector that no longer depends on where in the frame
an object sat. A single linear layer maps that vector to 1000 class logits, and a
softmax normalizes them to a distribution. The **output head** is thus a matrix of
shape $256 \times 1000$, the **loss** is cross-entropy against the one-hot label, and
the **metric** reported at test time is top-1 or top-5 accuracy, which the smooth loss
only approximates.

The other three tasks reattach a different head to the same trunk. Localization adds a
four-number regression head for one box. Detection runs a classification-plus-box head
at many spatial locations. Segmentation replaces global pooling with an upsampling
decoder that returns to full resolution, emitting a $(K, H, W)$ map of per-pixel class
logits scored by pixel-wise cross-entropy and evaluated by mean IoU. The trunk is
shared; the head is task-specific.

The characteristic **failure modes** are worth naming. A classifier trained on clean,
centered images degrades on the domain shift of real photographs (odd lighting,
occlusion, unusual crops); the fix is heavier augmentation matched to the deployment
distribution. A detector floods the frame with near-duplicate boxes unless
non-maximum suppression prunes overlapping predictions. A segmenter smears object
boundaries because pooling discarded the fine spatial detail the decoder must
reconstruct, which is why skip connections from encoder to decoder (as in U-Net)
carry the lost resolution forward.

## Natural language processing

Language presents a fundamentally _discrete_, high-cardinality
input: a vocabulary of $V$ tokens, often $V \sim 10^5$. The classical approach,
the **$n$-gram language model**, estimates the probability of the next word from
the previous $n-1$ by counting:

$$
P(w_t \mid w_{1:t-1}) \;\approx\; P(w_t \mid w_{t-n+1:t-1})
\;=\; \frac{\mathrm{count}(w_{t-n+1:t})}{\mathrm{count}(w_{t-n+1:t-1})}.
$$

This is exact arithmetic on a lookup table, and it collapses under the **curse of
dimensionality**. The table has one entry per possible context, so its size grows
exponentially in $n$:

$$
\#\{\text{contexts}\} \;=\; V^{\,n-1},
\qquad V = 10^5,\ n = 4
\;\Longrightarrow\;
10^{15}\ \text{entries},
$$

almost all of which are never observed, leaving the count zero and the probability
estimate undefined without smoothing. The deeper problem is that the table treats
every word as an isolated symbol: "cat" and "dog" are as unrelated as "cat" and
"thermodynamics," because one-hot codes are mutually orthogonal and share no
statistics.[^gf-nlp]

> **Definition ($n$-gram model).** A language model that approximates
> $P(w_t \mid w_{1:t-1})$ by the Markov assumption that only the previous $n-1$
> words matter, estimating the conditional from corpus counts. Its parameter count
> $V^{\,n-1}$ grows exponentially in the context length $n$, the curse of
> dimensionality for discrete sequences.

To address this, replace the one-hot symbol with a dense, learned vector, a **word
embedding**, so that statistically similar words occupy nearby points and the
model can _generalize_ across them.

Before the embedding, the raw text must become integers. **Tokenization** splits a
string into units drawn from the vocabulary and maps each to its index, so a
sentence of $T$ tokens becomes an integer vector of shape $(T,)$ with entries in
$\{0,\dots,V-1\}$. A batch of $N$ sentences padded to length $T$ is the integer
tensor $(N, T)$. The embedding layer is a lookup into the matrix
$E \in \mathbb{R}^{V\times d}$: row $E[j]$ is the vector for token $j$, so indexing
$(N, T)$ into $E$ produces the float tensor $(N, T, d)$.[^stevens-text] For
$V = 10^5$, $d = 300$, a 12-token sentence therefore expands from 12 integers to a
$12 \times 300 = 3600$-number matrix that a sequence model can consume.

> **Definition (Word embedding).** A learned map $E : \{1,\dots,V\} \to
> \mathbb{R}^{d}$ from token index to a dense vector, $d \ll V$, trained so that
> distributional similarity (appearing in similar contexts) becomes geometric
> proximity. The matrix $E \in \mathbb{R}^{V\times d}$ is shared across all
> positions, so $Vd$ parameters replace the $V^{\,n-1}$ of the count table.

Embeddings are learned from the distributional hypothesis, that _a word is known by
the company it keeps_. **word2vec** trains $E$ by predicting context words from a
center word (skip-gram); **GloVe** factorizes the global co-occurrence matrix. Both
yield the same emergent property: semantic relations become **vector arithmetic**.

$$
% caption: Word embeddings place related words so semantic relations become
% parallel vectors, closing the king $-$ man $+$ woman $\approx$ queen parallelogram.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % four embedding points forming a parallelogram
  \coordinate (man)   at (0,0);
  \coordinate (woman) at (2.6,0);
  \coordinate (king)  at (0.9,2.4);
  \coordinate (queen) at (3.5,2.4);
  % gender axis (bottom + via king-queen)
  \draw[->, black, thick] (man) -- (woman) node[midway, below, black] {\texttt{"gender" offset}};
  \draw[->, black, thick, dashed] (king) -- (queen);
  % royalty axis
  \draw[->, acc, thick] (man) -- (king) node[midway, left, text=acc] {\texttt{"royalty" offset}};
  \draw[->, acc, thick, dashed] (woman) -- (queen);
  % predicted point: king - man + woman lands at queen
  \draw[->, green, very thick] (woman) -- (queen)
     node[midway, right=1mm, text=green, align=left] {\texttt{lands at}\\\texttt{queen}};
  % points
  \fill[acc] (man)   circle (2.4pt) node[anchor=north east] {man};
  \fill[acc] (woman) circle (2.4pt) node[anchor=north west] {woman};
  \fill[acc] (king)  circle (2.4pt) node[anchor=south east] {king};
  \fill[green] (queen) circle (3.0pt) node[anchor=south west, text=green] {queen};
\end{tikzpicture}
$$

The parallelogram is the geometry of analogy: a single offset vector encodes
"royalty," another encodes "gender," and they add independently, so
$v_{\text{king}} - v_{\text{man}} + v_{\text{woman}}$ lands near $v_{\text{queen}}$.
With dense vectors in hand, the **neural language model** replaces the count table
with a network: look up each context word's embedding, concatenate, and feed a
multilayer perceptron that outputs a softmax over the vocabulary.

$$
% caption: The $n$-gram table's size $V^{n-1}$ explodes; the neural model instead
% shares one embedding matrix across positions and learns a smooth function.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  cell/.style={draw, minimum width=8mm, minimum height=6mm, font=\scriptsize},
  blk/.style={draw, minimum width=16mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % --- n-gram table (left) ---
  \node[font=\small] at (1.6,3.2) {$n$-gram table};
  \foreach \i in {0,1,2,3,4,5}
    \node[cell] (c\i) at (0.4, 2.4-0.6*\i) {};
  \node[cell, draw=red, thick] at (0.4, 2.4-0.6*2) {};
  \node[red, font=\footnotesize, anchor=west] at (1.0, 2.4-0.6*2) {\texttt{one row per context}};
  \node[red, font=\footnotesize, align=center] at (1.7,-1.35) {\texttt{size grows as}\\\texttt{V to the power n}};
  % --- neural model (right) ---
  \begin{scope}[xshift=6.2cm]
    \node[font=\small] at (2.2,3.2) {neural model};
    \node[blk] (w1) at (0,2.3)   {\texttt{"the"}};
    \node[blk] (w2) at (0,1.1)   {\texttt{"black"}};
    \node[blk] (w3) at (0,-0.1)  {\texttt{"cat"}};
    \node[blk, draw=acc, thick] (E) at (2.4,1.1) {\texttt{shared}\\\texttt{embedding E}};
    \node[blk] (mlp) at (4.6,1.1) {\texttt{MLP}\\\texttt{softmax}};
    \draw[->, acc, thick] (w1) -- (E);
    \draw[->, acc, thick] (w2) -- (E);
    \draw[->, acc, thick] (w3) -- (E);
    \draw[->, acc, thick] (E) -- (mlp);
    \node[acc, font=\scriptsize, anchor=north, align=center] at (2.4,-0.9)
        {\texttt{Vd parameters,}\\\texttt{reused at every position}};
  \end{scope}
\end{tikzpicture}
$$

The neural model's parameter budget is $Vd + (\text{MLP weights})$, _additive_
rather than exponential in context length, and it generalizes to unseen $n$-grams
because nearby embeddings produce nearby predictions. Fixed-width context still
limits it. Lifting that limit, so that every token can attend to every other, leads
to the
[Transformer architecture](/deep-learning/architectures/the-transformer-architecture),
which dropped the Markov window entirely and now underpins every large language
model.

| Model | Context | Parameters | Generalizes across words? |
| --- | --- | --- | --- |
| $n$-gram count table | fixed $n-1$ | $V^{\,n-1}$ | no (one-hot, orthogonal) |
| Neural LM (MLP) | fixed $n-1$ | $Vd + $ MLP | yes (shared embeddings) |
| Recurrent LM | unbounded (decaying) | $O(d^2)$ | yes |
| Transformer LM | full sequence (attention) | $O(d^2)$ per layer | yes |

The full sequence pipeline is the same recipe carried through with different heads
depending on whether the target is one label for the whole sequence or one label per
position. The figure traces the shapes.

$$
% caption: Sequence pipeline. Token ids index the shared embedding into a
% (T, d) matrix; an encoder produces a (T, h) state sequence; a classification head
% pools to one label, while a tagging head emits one label per position.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  st/.style={draw, minimum width=19mm, minimum height=11mm, align=center, font=\scriptsize},
  sh/.style={font=\scriptsize, text=black, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \node[st, fill=acc!8, draw=acc]  (tok) at (0,0)   {\texttt{token ids}};
  \node[st, fill=acc!12, draw=acc] (emb) at (2.7,0)  {\texttt{embedding}};
  \node[st, fill=acc!16, draw=acc] (enc) at (5.4,0)  {\texttt{encoder}};
  \node[st, fill=acc!22, draw=green] (cls) at (8.4,1.1) {\texttt{pool +}\\\texttt{linear}};
  \node[st, fill=acc!22, draw=green] (tag) at (8.4,-1.1) {\texttt{per-token}\\\texttt{linear}};
  \draw[->, acc, thick] (tok) -- (emb);
  \draw[->, acc, thick] (emb) -- (enc);
  \draw[->, green, thick] (enc) -- (cls);
  \draw[->, green, thick] (enc) -- (tag);
  \node[sh, anchor=north] at (tok.south) {\texttt{(T,)}};
  \node[sh, anchor=north] at (emb.south) {\texttt{(T, d)}};
  \node[sh, anchor=north] at (enc.south) {\texttt{(T, h)}};
  \node[sh, anchor=west]  at (cls.east)  {\texttt{(K,)}};
  \node[sh, anchor=west]  at (tag.east)  {\texttt{(T, K)}};
  \node[green, font=\footnotesize, anchor=south] at (8.4,2.0) {\texttt{sequence label}};
  \node[green, font=\footnotesize, anchor=north] at (8.4,-2.0) {\texttt{tag per token}};
\end{tikzpicture}
$$

The **output head** is the only branch point. Sentiment classification pools the
$(T, h)$ state sequence to one vector (mean-pool or the final state) and applies a
linear layer to $K$ class logits, scored by cross-entropy and evaluated by accuracy.
Sequence tagging (part-of-speech, named entities) keeps the time axis and applies the
same linear layer at every position, producing $(T, K)$ logits scored by per-position
cross-entropy and evaluated by token-level F1. Language modeling sets $K = V$ and
predicts the next token at each step.

The **failure modes** of a text model trace back to the vocabulary. A token unseen at
training time hits the out-of-vocabulary slot and loses all its content, which is why
subword tokenization (splitting rare words into known pieces) is standard. Padding a
short sequence to length $T$ injects meaningless positions that a naive mean-pool
averages into the answer, so the pooling must mask them out. A model trained on one
genre transfers poorly to another because the embedding geometry it learned reflects
the training corpus, not the target domain.

## Speech recognition

Speech recognition maps a waveform to text. The raw signal is first reduced to a
sequence of **acoustic feature** frames, typically log-mel spectrograms or MFCCs
computed on overlapping short windows, turning audio into a $T \times F$ matrix
that a sequence model can consume. A 3-second utterance windowed every 10 ms with
80 mel bands is the tensor $(300, 80)$: 300 time frames, each an 80-dimensional
spectral summary.

> **Definition (Acoustic features).** A frame-level summary of the speech spectrum,
> e.g. the log-energies in mel-scaled frequency bands over a $\sim$25 ms window
> stepped every $\sim$10 ms, giving a feature sequence of length $T$ proportional
> to the utterance's duration.

The central difficulty is **alignment**: the input has $T$ frames but the output
has $L \ll T$ characters, and the frame-to-character correspondence is unknown.
**Connectionist temporal classification** (CTC) solves this without per-frame
labels by introducing a blank symbol and summing over all alignments that collapse
to the target.[^gf-speech]

> **Definition (CTC loss).** Let $\mathcal{B}$ collapse a frame-level path
> (over the alphabet plus a blank) to a label string by removing blanks and merging
> repeats. The CTC objective maximizes the total probability of the target $y$
> summed over every path $\pi$ that collapses to it,
> $P(y\mid x) = \sum_{\pi\,:\,\mathcal{B}(\pi)=y} \prod_{t} p_t(\pi_t \mid x)$,
> computed in $O(TL)$ by dynamic programming.

Collapsing removes blanks and merges adjacent repeats, so many frame paths map to
the same string. The figure shows two of them landing on the target "cat".

$$
% caption: CTC collapsing. Two frame-level paths over the alphabet plus a blank
% (the empty cell) reduce to the same string once blanks are dropped and adjacent
% repeats are merged; CTC sums the probability of every such path.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  fr/.style={draw, minimum width=8mm, minimum height=8mm, font=\scriptsize, align=center},
  bl/.style={draw=acc, fill=black!6, minimum width=8mm, minimum height=8mm}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % path A: c c [blank] a t t  (blank cells drawn empty, lettered cells tinted)
  \node[font=\footnotesize, anchor=east] at (-0.4,1.1) {\texttt{path A}};
  \node[fr, fill=acc!10, draw=acc] at (0,1.1)    {c};
  \node[fr, fill=acc!10, draw=acc] at (0.95,1.1) {c};
  \node[bl]                        at (1.90,1.1) {};
  \node[fr, fill=acc!10, draw=acc] at (2.85,1.1) {a};
  \node[fr, fill=acc!10, draw=acc] at (3.80,1.1) {t};
  \node[fr, fill=acc!10, draw=acc] at (4.75,1.1) {t};
  % path B: [blank] c a a [blank] t
  \node[font=\footnotesize, anchor=east] at (-0.4,0) {\texttt{path B}};
  \node[bl]                        at (0,0)    {};
  \node[fr, fill=acc!10, draw=acc] at (0.95,0) {c};
  \node[fr, fill=acc!10, draw=acc] at (1.90,0) {a};
  \node[fr, fill=acc!10, draw=acc] at (2.85,0) {a};
  \node[bl]                        at (3.80,0) {};
  \node[fr, fill=acc!10, draw=acc] at (4.75,0) {t};
  % legend for blank
  \node[bl] (leg) at (0,-1.15) {};
  \node[font=\footnotesize, anchor=west, black] at (0.45,-1.15) {\texttt{= blank}};
  % collapse arrow
  \draw[->, green, thick] (5.5,0.55) -- (6.6,0.55)
     node[midway, above, text=green, font=\footnotesize] {\texttt{collapse}};
  \node[fr, fill=green!14, draw=green, minimum width=22mm] at (8.0,0.55) {\texttt{"cat"}};
\end{tikzpicture}
$$

The sum over exponentially many alignments is tractable by a forward-backward
recursion identical in spirit to the one for hidden Markov models, making the whole
acoustic-model-to-transcript path differentiable end to end. The **metric** at test
time is word error rate, the edit distance between prediction and reference over the
reference length. The **failure modes** are homophones the acoustic signal cannot
disambiguate (fixed by a language-model rescoring pass) and rare words absent from
the training transcripts.

## Recommender systems

A recommender predicts how much user $u$ will like item $i$ from a sparse matrix
$R \in \mathbb{R}^{m\times n}$ of observed ratings, most of whose entries are
missing. The standard method is **matrix factorization**: approximate $R$ by the product
of a thin user-factor matrix and a thin item-factor matrix.[^gf-recsys]

> **Definition (Matrix factorization).** Model the rating matrix as a low-rank
> product $R \approx P\,Q^{T}$ with $P \in \mathbb{R}^{m\times k}$,
> $Q \in \mathbb{R}^{n\times k}$, and $k \ll \min(m,n)$. Row $p_u$ is user $u$'s
> latent taste vector, row $q_i$ is item $i$'s latent attribute vector, and the
> predicted rating is their inner product $\hat r_{ui} = p_u^{T} q_i$.

$$
% caption: Matrix factorization approximates the sparse user–item matrix $R$ by a
% thin user-factor matrix $P$ times an item-factor matrix $Q^{T}$.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{accmid}{HTML}{6A82F6}
  % --- R matrix (sparse) ---
  \foreach \r in {0,1,2,3} \foreach \c in {0,1,2,3,4}
     \node[draw=black, minimum size=6mm, inner sep=0pt] at (\c*0.62, -\r*0.62) {};
  % observed entries: light tint + crisp accent outline (col c, row r, on the grid pitch)
  \foreach \c/\r in {0/0, 2/0, 3/1, 1/2, 4/2, 0/3, 2/3}
     \node[fill=acc!15, draw=acc, thick, minimum size=6mm, inner sep=0pt] at (\c*0.62, -\r*0.62) {};
  \node[font=\small] at (1.24,0.95) {R};
  \node[black, font=\scriptsize, anchor=north] at (1.24,-2.3) {users by items};
  % approx sign
  \node[font=\footnotesize] at (4.1,-0.95) {approx};
  % --- P (user factors) ---
  \foreach \r in {0,1,2,3} \foreach \c in {0,1}
     \node[fill=accmid!28, draw=black, minimum size=6mm, inner sep=0pt] at (5.2+\c*0.62, -\r*0.62) {};
  \node[font=\small] at (5.5,0.95) {P};
  \node[black, font=\scriptsize, anchor=north] at (5.5,-2.3) {users by k};
  % times
  \node[font=\footnotesize] at (6.85,-0.95) {times};
  % --- Q^T (item factors) ---
  \foreach \r in {0,1} \foreach \c in {0,1,2,3,4}
     \node[fill=accmid!28, draw=black, minimum size=6mm, inner sep=0pt] at (7.6+\c*0.62, -\r*0.62) {};
  \node[font=\small] at (8.84,0.95) {$Q^{T}$};
  \node[black, font=\scriptsize, anchor=north] at (8.84,-1.7) {k by items};
\end{tikzpicture}
$$

The factors are fit by minimizing squared error over the _observed_ entries only.
The missing entries contribute nothing to the loss, so the model is never asked to
predict a rating that was never given. An $L_2$ penalty controls the low-rank
capacity:

$$
\min_{P,Q}\;\sum_{(u,i)\in\Omega}\parens{r_{ui} - p_u^{T} q_i}^2
\;+\; \lambda\parens{\norm{p_u}^2 + \norm{q_i}^2},
\qquad \Omega = \{\text{observed }(u,i)\},
$$

solvable by stochastic gradient descent or alternating least squares (fix $P$,
solve a least-squares for $Q$, alternate). The model has a structural limitation.

> **Definition (Cold-start problem).** A new user or item has no observed ratings,
> so its latent factor $p_u$ or $q_i$ is unconstrained by data and matrix
> factorization cannot place it. The remedy is to fall back on _content_ features,
> such as user demographics and item metadata, until interactions accumulate.

The **metric** a recommender optimizes at test time is rarely raw squared error. What
matters is the ranking of the top few items shown to the user, so systems report
ranking scores such as precision-at-$k$ or normalized discounted cumulative gain, and
the squared-error training loss is only a differentiable stand-in for them.

Deep recommenders extend the bilinear score $p_u^{T} q_i$ by passing the learned
embeddings through a neural network and concatenating side features, which both
lifts the linear ceiling and softens cold-start by sharing structure through the
content features. This is the same embedding-plus-MLP construction used in neural
language models, transplanted to the user-item domain.

## One architecture eats the four domains

Goodfellow's chapter presents four domains with four distinct priors — convolution for
pixels, recurrence for language, spectral front-ends for audio, factorization for the
user-item matrix. The decade since collapsed much of that diversity onto a single
architecture: the [Transformer](/deep-learning/architectures/the-transformer-architecture)
(Vaswani et al., 2017, _NeurIPS_).

The consolidation ran domain by domain. Language went first, as attention replaced the
gated RNNs of the NLP section outright. Vision followed with the **Vision Transformer**
(Dosovitskiy et al., 2021, _ICLR_), which cuts an image into patches, treats them as a
token sequence, and matches or beats convolutional networks once the training set is
large enough — the convolutional prior turns out to be replaceable by data. Speech
followed with **wav2vec 2.0** and **Whisper** (Radford et al., 2022, arXiv), which run
Transformer encoders on the spectral frames from the speech section. Even recommenders
adopted self-attention over a user's interaction history. The prior that each domain once
hard-coded into its architecture is increasingly learned instead, given enough data and
compute — the same lesson the Vision Transformer taught first.

The hardware story kept pace. The $O(T^2)$ attention cost that would have made long
sequences impractical was addressed by **FlashAttention** (Dao et al., 2022, _NeurIPS_), an
IO-aware kernel that tiles the attention computation to keep it in fast on-chip memory,
turning a memory-bound operation into a compute-bound one without changing the math. The
two axes of parallelism from the large-scale section became production frameworks —
Megatron-LM for tensor parallelism, ZeRO / FSDP for sharding optimizer state across
devices — that make trillion-parameter training routine. The recipe is unchanged from the
opening of this lesson: one gradient loop, one increasingly universal architecture,
specialized now by the _data_ it is fed rather than the prior baked into it.

## Takeaways

- **One recipe, four domains.** Fix the input tensor and its shape, choose the
  architecture whose bias matches that shape, attach the output head for the target,
  pick the loss, then read the residual for domain-specific failure modes. Only the
  first four steps vary; the optimizer and training loop are shared.
- **One loop, four priors.** Vision, language, speech, and recommendation reuse the
  same optimizer; they differ only in the architectural and preprocessing prior
  fitted to the data's structure.
- **Scaling has two axes.** _Data parallelism_ splits the batch and averages exact
  gradients; _model parallelism_ splits the parameters when one model exceeds one
  device. Mixed precision, quantization, pruning, and distillation cut compute and
  memory on top.
- **Vision is a task spectrum.** Classification $\to$ localization $\to$ detection
  $\to$ segmentation adds output structure at each step; normalization and
  [augmentation](/deep-learning/regularization/dropout-and-data-augmentation) supply
  the data prior.
- **Embeddings beat the curse of dimensionality.** $n$-gram tables grow as
  $V^{\,n-1}$ and treat words as orthogonal symbols; a shared embedding matrix is
  $Vd$ parameters, generalizes across words, and makes analogy literal vector
  arithmetic, the road to the
  [Transformer](/deep-learning/architectures/the-transformer-architecture).
- **CTC** sums over alignments so speech models train without frame-level labels;
  **matrix factorization** of the user-item matrix is a low-rank inner product
  $p_u^{T} q_i$ that fails on the **cold-start** problem until content features
  fill the gap.

[^gf-apps]: **Goodfellow**, _Deep Learning_, Ch. 12 — Applications: one optimizer specialized four ways, the domain-specific prior being the only thing that changes across vision, language, speech, and recommendation.
[^gf-largescale]: **Goodfellow**, _Deep Learning_, §12.1 — Large-Scale Deep Learning: spreading a gradient step across devices via data and model parallelism, plus precision and inference-time compression.
[^gf-vision]: **Goodfellow**, _Deep Learning_, §12.2 — Computer Vision: contrast normalization and dataset augmentation as the data-side prior, convolution as the architectural one.
[^chollet-vision]: **Chollet**, _Deep Learning with Python_, Ch. 5 — Deep Learning for Computer Vision: data augmentation as label-preserving transformation and a primary regularizer on small image datasets.
[^gf-nlp]: **Goodfellow**, _Deep Learning_, §12.4 — Natural Language Processing: $n$-gram tables and the curse of dimensionality, cured by dense word embeddings that make distributional similarity geometric.
[^gf-speech]: **Goodfellow**, _Deep Learning_, §12.3 — Speech Recognition: acoustic features and the alignment problem that CTC dissolves by summing over collapsing paths.
[^gf-recsys]: **Goodfellow**, _Deep Learning_, §12.5 — Other Applications (Recommender Systems): low-rank matrix factorization of the user–item matrix and the cold-start limitation.
[^stevens-vision]: **Stevens et al.**, _Deep Learning with PyTorch_, Ch. 4 — Real-world data representation: images load as a $(N, C, H, W)$ tensor of channels-first floats, normalized channel-wise before training.
[^stevens-text]: **Stevens et al.**, _Deep Learning with PyTorch_, Ch. 4 — Representing text: tokenization to integer indices, then an embedding lookup that turns a $(N, T)$ index tensor into a $(N, T, d)$ float tensor.