---
title: What Is Deep Learning?
module: Foundations
moduleNumber: 1
lessonNumber: 1
order: 101
summary: >
  Deep learning is representation learning by composition: stack simple
  differentiable layers, define a loss, and let gradient descent discover the
  features a human would otherwise have to engineer by hand. We set up the whole
  vocabulary (model, loss, optimizer, data), the training loop that ties them
  together, and the three reasons the approach became practical.
topics: [Foundations]
sources:
  - book: Goodfellow
    ref: "Ch. 1 — Introduction; §5.1 Learning Algorithms"
  - book: Goodfellow
    ref: "§5.1.3 Unsupervised vs. Supervised; §5.2 Capacity, Overfitting, Underfitting"
  - book: Chollet
    ref: "Ch. 1 — What Is Deep Learning?; §1.1.4 Learning Representations"
---

Classical programming is a human writing the rules. You want to detect spam, so
you sit down and enumerate conditions: contains "free money," sender not in
contacts, more than three exclamation marks. **Machine learning** inverts that
arrangement. Instead of rules, you supply _examples_ (ten thousand emails, each
labelled spam or not) and a _measure of how wrong_ a guess is, and the machine
searches for rules that fit the examples. You no longer write the classifier; you
write the thing that _finds_ the classifier.

**Deep learning** is the special case of machine learning where the rules are a
deep stack of simple, differentiable transformations, and the search is gradient
descent. That is the entire definition.[^gf-intro] Everything else (convolutions, attention,
batch normalization, and the rest) is engineering on top of those two
commitments: _compose simple differentiable pieces_, and _learn them by following
the gradient_.

## Representation learning

The deepest reason the field is called "deep" is not the number of layers; it is
**representation learning**. To see why it matters, look at what the alternative
costs.

A [linear model](/deep-learning/foundations/linear-models-and-the-perceptron)
can only separate classes with a flat boundary drawn through its raw inputs. If
the inputs are pixel intensities, that is hopeless — no hyperplane through
raw-pixel space distinguishes cats from dogs, because the property "cat" is not a
linear function of individual pixels. For decades the fix was **feature
engineering**: a human expert designed a transformation $\phi(x)$ (edge
detectors, SIFT descriptors, hand-tuned filters) that mapped raw pixels into a
space where a linear model _could_ work. The learning algorithm was the easy part;
the labor and the bottleneck were in $\phi$.

Deep learning automates $\phi$. The network _learns_ the representation, one layer
at a time, each layer re-describing the output of the one below it. Trained on
images, the early layers reliably discover oriented edges; the middle layers
compose edges into textures and motifs; the late layers compose those into object
parts and whole objects — a hierarchy nobody programmed, recovered purely from the
pressure to reduce the loss.[^chollet-repr]

$$
% caption: A deep network learns a hierarchy of representations, each layer
% re-describing the one below, from raw pixels up to a class.
\begin{tikzpicture}[>=stealth, font=\small,
  stage/.style={draw, minimum width=20mm, minimum height=13mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[stage] (px)  at (0,0)    {pixels\\(raw input)};
  \node[stage] (ed)  at (2.9,0)  {edges};
  \node[stage] (pt)  at (5.8,0)  {parts};
  \node[stage] (ob)  at (8.7,0)  {objects};
  \node[stage, draw=acc, text=acc] (lb) at (11.6,0) {label};
  \draw[->, acc, thick] (px) -- (ed);
  \draw[->, acc, thick] (ed) -- (pt);
  \draw[->, acc, thick] (pt) -- (ob);
  \draw[->, acc, thick] (ob) -- (lb);
  \draw[->, black, thick] (3.5,-1.2) -- (8.1,-1.2);
  \node[font=\footnotesize, text=black, anchor=south] at (5.8,-1.15) {increasing abstraction};
\end{tikzpicture}
$$

> **Definition (Representation learning).** Learning not just a mapping from
> features to outputs, but the **features themselves** — a transformation of the
> raw input into a form in which the task becomes easy. Depth makes this practical
> by _composing_ representations: each layer's output is the next layer's input.

This is why "feature engineering," once the central task of applied machine
learning, has all but vanished from deep-learning practice. The representation is
learned rather than built.

### What raw pixels look like to a linear model

A $28 \times 28$ grayscale image is a vector $x \in \mathbb{R}^{784}$; a linear
classifier's only verdict is a weighted sum $w^\top x + b$, one fixed weight per
pixel. The failure is visible in the metric. Let $x$ be a clean $7$ and $T_\delta x$
the same $7$ shifted by $\delta$ pixels. A small shift changes hundreds of
coordinates, so Euclidean distance in pixel-space is dominated by alignment, not
identity:

$$
\norm{x - T_\delta x}_2 \;\gg\; \norm{x - x'}_2,
\qquad x' = \text{a different digit overlapping } x,
$$

i.e. the same digit shifted can land _farther_ from itself than from a different
class. A single hyperplane cannot track a digit across the regions of pixel-space
that translation, slant, and stroke-width scatter it into. The fix is to change
coordinates: learn a map $h = \phi_\theta(x)$ under which

$$
\norm{\phi_\theta(x) - \phi_\theta(T_\delta x)} \;\approx\; 0,
\qquad
\norm{\phi_\theta(x) - \phi_\theta(x')} \;\text{large},
$$

so a linear readout $w^\top h + b$, the operation that failed on raw pixels, now
succeeds on the learned ones. The pixels did not change; the coordinates did.

$$
% caption: In raw-pixel space (left) the two classes interleave and no line
% separates them; the learned map $\phi_\theta$ warps the space (right) so a
% straight boundary suffices.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % --- left: tangled in pixel space ---
  \draw[black] (-0.2,-0.2) rectangle (3.4,3.0);
  \node[font=\footnotesize, anchor=south] at (1.6,3.05) {raw pixels};
  % two interleaved classes (no linear separator)
  \foreach \p in {(0.4,0.6),(1.1,1.9),(0.7,2.4),(1.9,0.7),(2.6,2.1),(1.5,1.3)}
    \fill[acc] \p circle (2.2pt);
  \foreach \p in {(0.6,1.4),(1.4,2.5),(2.2,1.4),(2.9,0.9),(0.9,0.4),(2.4,2.6)}
    {\draw[red, thick] \p ++(-2pt,-2pt) -- ++(4pt,4pt); \draw[red, thick] \p ++(-2pt,2pt) -- ++(4pt,-4pt);}
  % --- arrow ---
  \draw[->, acc, very thick] (3.9,1.4) -- (5.1,1.4) node[midway, above] {learned map};
  % --- right: separated in feature space ---
  \begin{scope}[xshift=5.6cm]
    \draw[black] (-0.2,-0.2) rectangle (3.4,3.0);
    \node[font=\footnotesize, anchor=south] at (1.6,3.05) {learned features};
    % class 1 clustered low, class 2 clustered high
    \foreach \p in {(0.5,0.6),(1.0,0.9),(0.8,0.4),(1.4,0.7),(0.6,1.1),(1.2,0.5)}
      \fill[acc] \p circle (2.2pt);
    \foreach \p in {(2.1,2.2),(2.6,2.5),(2.3,1.9),(2.8,2.1),(2.0,2.6),(2.5,2.7)}
      {\draw[red, thick] \p ++(-2pt,-2pt) -- ++(4pt,4pt); \draw[red, thick] \p ++(-2pt,2pt) -- ++(4pt,-4pt);}
    % linear separator
    \draw[green!60!black, very thick] (0.1,2.2) -- (3.1,0.6);
  \end{scope}
\end{tikzpicture}
$$

Cats versus dogs is the same statement at scale: cat-ness survives translation,
rotation, lighting, and occlusion, and a deep net discovers that invariance because
staying invariant to nuisance variation is what drives the loss down across a
varied training set.

## A taxonomy of learning

Deep learning is a _method_ for fitting differentiable models; it is not itself a
_kind_ of learning problem. The same network and the same training loop slot into
several quite different problem settings, distinguished by what kind of supervision
the data carries.

$$
% caption: One differentiable model, trained under whatever supervision the
% setting provides: full labels, none, self-mined labels, or a reward signal.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  root/.style={draw, thick, minimum width=30mm, minimum height=10mm, align=center},
  leaf/.style={draw, minimum width=27mm, minimum height=14mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[root, draw=acc, text=acc] (r) at (0,0) {learning problems};
  \node[leaf] (sup)  at (-5.4,-2.6) {\texttt{supervised}\\(input, \texttt{label})};
  \node[leaf] (uns)  at (-1.8,-2.6) {\texttt{unsupervised}\\(input only)};
  \node[leaf] (self) at (1.8,-2.6)  {\texttt{self-supervised}\\(\texttt{label} from input)};
  \node[leaf] (rl)   at (5.4,-2.6)  {\texttt{reinforcement}\\(\texttt{reward} signal)};
  \draw[->, acc, thick] (r) -- (sup);
  \draw[->, acc, thick] (r) -- (uns);
  \draw[->, acc, thick] (r) -- (self);
  \draw[->, acc, thick] (r) -- (rl);
\end{tikzpicture}
$$

The four settings differ only in where the loss signal comes from. The model
$f_\theta$ and the training loop are shared; only the data and the loss target change.

| Paradigm | Data | Loss signal | Typical objective | Example |
| --- | --- | --- | --- | --- |
| Supervised | $(x, y)$ | given label $y$ | $\ell(f_\theta(x), y)$ | image classification |
| Unsupervised | $x$ only | none | reconstruction / density $-\log p_\theta(x)$ | clustering, autoencoding |
| Self-supervised | $x$, target mined from $x$ | label from a held-out part of $x$ | predict masked $x_{\text{hidden}}$ | masked-language modeling |
| Reinforcement | states, actions | scalar reward $r$, often delayed | maximize $\mathbb{E}[\sum_t \gamma^t r_t]$ | game-playing, control |

Self-supervised learning is the notable middle row: no human annotates anything, yet
a label is manufactured from the input by hiding part of it and predicting it back.
This is the basis of large language models.

> **Definition (Supervision).** The kind of target signal the data provides:
> explicit labels (supervised), nothing (unsupervised), labels synthesized from
> the input (self-supervised), or a scalar reward from acting (reinforcement).
> Deep learning supplies the _function approximator_ used inside any of these.

The line between them is softer than the taxonomy suggests (self-supervised
pretraining followed by supervised fine-tuning is now the dominant approach), but
the distinction in _where the loss signal comes from_ is real, and it is the first
question to ask about any new system.

## The four ingredients

Strip away the architecture jargon and every deep-learning system, from a
two-line perceptron to a four-hundred-billion-parameter language model, is
assembled from the same four parts.

| Ingredient | Symbol | Role | Example |
| --- | --- | --- | --- |
| Model | $f_\theta : \mathcal{X} \to \mathcal{Y}$ | maps input to prediction | a 50-layer CNN |
| Loss | $\ell(\hat{y}, y)$ | scores a single prediction | cross-entropy |
| Optimizer | $\theta \gets \theta - \eta\,\nabla_\theta \mathcal{L}$ | updates parameters | Adam |
| Data | $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^n$ | defines the problem | ImageNet |

> **Definition (Model).** A parameterized function $f_\theta : \mathcal{X} \to
> \mathcal{Y}$ mapping inputs to predictions, where the parameter vector $\theta$
> collects every tunable weight and bias. Choosing the _form_ of $f_\theta$,
> how the layers connect, is choosing an **architecture**.

> **Definition (Loss).** A function $\ell(f_\theta(x), y)$ scoring how wrong a
> single prediction is. Averaged over the data it gives the **empirical risk**
> $\mathcal{L}(\theta) = \tfrac{1}{n}\sum_{i=1}^n \ell(f_\theta(x_i), y_i)$, the
> single number training drives down.

> **Definition (Optimizer).** The rule that updates $\theta$ to reduce
> $\mathcal{L}$. In deep learning it is almost always a variant of gradient
> descent: nudge each parameter in the direction that most steeply decreases the
> loss.

These four are exhaustive: reading any new paper, the first useful question is
_which slot did they change?_ — and there are no other slots.

| Slot | Symbol changed | What a paper contributes | Example |
| --- | --- | --- | --- |
| Architecture | $f_\theta$ | a new way to connect layers | ResNet, Transformer |
| Objective | $\ell$ | a new thing to minimize | contrastive loss, GAN loss |
| Optimizer | update rule | a better descent step | Adam, RMSProp |
| Data | $\mathcal{D}$ | a bigger / cleaner sample | ImageNet, web-scale text |

### Parametric function approximators

A **parametric** model commits in advance to a fixed-size $\theta$ and discards the
data after training; a **non-parametric** model keeps the data and lets complexity
grow with the sample (e.g. $k$-nearest-neighbors stores every training point).

> **Definition (Parametric model).** A model whose capacity is fixed by a
> parameter vector $\theta$ of predetermined size, independent of how much data
> it is trained on. Prediction depends on the data only through $\theta$.

A deep network is squarely parametric: the architecture fixes the weight count
before a single example is seen, and training adjusts only their _values_. Two facts
justify the fixed-size commitment:[^gf-capacity]

- **Expressivity.** With enough units a net approximates any reasonable target
  function (the [universal approximation](/deep-learning/neural-networks/universal-approximation)
  result), so $\theta$ being fixed-size is not a real limitation.
- **Capacity scales with parameter count.** A wider or deeper net has more weights,
  represents more intricate functions, and, given enough data, fits more of the
  structure in it.

This is why simply making the model bigger works; we return to its limits
under [the bias–variance tradeoff](/deep-learning/foundations/machine-learning-refresher).

## The training loop

The four ingredients interact through the training loop. **Training** is a loop: draw a
batch of data, push it through the model, score the predictions with the loss,
measure how the loss responds to each parameter (the gradient), and step the
parameters downhill. Repeat until the loss stops falling.

$$
% caption: The training loop: a minibatch flows forward to a loss, the gradient
% flows back, and the optimizer updates the parameters before repeating.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=24mm, minimum height=12mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box] (data)  at (0,1.9)   {training\\data};
  \node[box, draw=acc, text=acc, thick] (model) at (4.6,1.9) {model};
  \node[box] (loss)  at (4.6,-1.1) {loss};
  \node[box] (grad)  at (0,-1.1)   {gradient};
  \draw[->, acc, thick] (data) -- (model) node[midway, above, font=\footnotesize] {\texttt{minibatch}};
  \draw[->, acc, thick] (model) -- (loss) node[midway, right, font=\footnotesize] {\texttt{forward} pass};
  \draw[->, acc, thick] (loss) -- (grad) node[midway, below, font=\scriptsize] {backprop};
  \draw[->, thick] (grad.north) .. controls (0,0.95) and (3.2,1.05) .. (model.west);
  \node[font=\footnotesize] at (1.55,0.3) {\texttt{update} weights};
\end{tikzpicture}
$$

Because every layer is differentiable, the gradient $\nabla_\theta \mathcal{L}$
can be computed exactly and cheaply by
[backpropagation](/deep-learning/neural-networks/backpropagation), and the update
is a single subtraction.[^chollet-loop] Written out, the loop is short:

```algorithm
caption: $\textsc{Train}(f_\theta, \mathcal{D}, \eta)$ — minibatch gradient descent
initialize $\theta$ randomly
repeat
  sample a minibatch $(X, y) \sim \mathcal{D}$
  $\hat{y} \gets f_\theta(X)$ // forward pass
  $L \gets \text{loss}(\hat{y}, y)$ // how wrong are we?
  $g \gets \nabla_\theta L$ // backward pass (autodiff)
  $\theta \gets \theta - \eta \cdot g$ // gradient-descent step
until converged
return $\theta$
```

Every later lesson refines one line of this skeleton. Better
[architectures](/deep-learning/architectures/convolutional-networks) change line 4;
better [losses](/deep-learning/neural-networks/loss-functions-and-output-units)
change line 5; better
[optimizers](/deep-learning/optimization/gradient-descent-and-sgd) change line 7;
[regularization](/deep-learning/regularization/regularization-overview) changes
what "converged" should mean. The loop itself never changes.

### Differentiability and the end-to-end bet

Line 6 of that algorithm is possible because every operation between
the input and the loss is **differentiable**, so the loss has a gradient with
respect to _every_ parameter, no matter how deep. This is
the central assumption of the field. Differentiability is what lets a
correction at the output propagate, by the chain rule, all the way back to the
first layer's weights, the layers that compute the _representation_. A single
gradient signal tunes the features and the classifier _at the same time_, each in
service of the final loss.[^gf-e2e]

> **Definition (End-to-end training).** Optimizing every stage of a system jointly
> against one final loss, rather than training hand-designed stages separately.
> It requires that the whole pipeline be differentiable so the gradient can reach
> every parameter.

Contrast the two ways to build a multi-stage system:

| | Classical pipeline | End-to-end |
| --- | --- | --- |
| Stage objective | a human-chosen _proxy_ per stage | the one final loss, for all stages |
| Coupling | stages frozen, optimized in isolation | stages co-adapt via shared gradient |
| Requirement | none (stages can be non-differentiable) | whole pipeline differentiable |
| Failure mode | early stage tuned for the wrong target | needs more data and compute |

The classical pipeline hand-builds stage one (an edge detector), freezes it, builds
stage two on top, and so on — but no early stage knows what the final task will ask
of it. End-to-end training removes those seams: with $g = \nabla_\theta \mathcal{L}$
reaching every weight, every stage is optimized for the only thing that matters. On
perception and language this has worked so well that the hand-built
pipeline has all but disappeared.

## Learning is descent on a surface

Line 7 has a geometric reading. The loss $\mathcal{L}(\theta)$ is a
function of millions of parameters; fix all but two and you can draw it as a
surface over a plane. Training is a ball rolling downhill on that surface, the
gradient pointing in the direction of steepest _ascent_ so we step the other way.

$$
% caption: Gradient descent on the loss surface. From a random start the optimizer
% steps against the gradient, tracing a path down the contours toward a minimum.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % concentric contour ellipses (a loss "bowl")
  \foreach \r in {0.6,1.2,1.8,2.4,3.0}
    \draw[black] (0,0) ellipse ({\r*1.25} and \r);
  % minimum
  \fill[green] (0,0) circle (2.8pt);
  \node[green, anchor=north] at (0,-3.25) {\texttt{minimum}};
  % descent path from a start point
  \coordinate (p0) at (-3.4,2.4);
  \coordinate (p1) at (-2.3,1.4);
  \coordinate (p2) at (-1.3,0.95);
  \coordinate (p3) at (-0.6,0.45);
  \coordinate (p4) at (-0.2,0.18);
  \draw[acc, very thick, ->] (p0) -- (p1);
  \draw[acc, very thick, ->] (p1) -- (p2);
  \draw[acc, very thick, ->] (p2) -- (p3);
  \draw[acc, very thick, ->] (p3) -- (p4);
  \fill[acc] (p0) circle (2.4pt);
  \node[acc, anchor=south] at (-3.4,2.55) {start};
\end{tikzpicture}
$$

The picture is useful but simplified. In two dimensions the surface is
a clean bowl with one minimum. In a million dimensions it is nothing of the kind —
riddled with saddle points, flat plateaus, and ravines. That geometry is why so
much of the subject is about _how_ to descend it well: momentum, adaptive learning
rates, normalization, good initialization. We return to this in the
[optimization](/deep-learning/optimization/gradient-descent-and-sgd) chapter.

## Why depth

If a single hidden layer is already a universal approximator — and
[it is](/deep-learning/neural-networks/universal-approximation) — why stack many?
Because the theorem is about _possibility_, not _cost_. A shallow network can
represent a given function only by paying in width, sometimes exponentially many
units; a deep network represents the same function compactly by _reusing_
intermediate features.[^gf-depth]

$$
% caption: A shallow net enumerates cases in one enormous layer; a deep net reuses
% features across layers, computing the same function with far fewer units.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  u/.style={circle, draw, minimum size=4.5mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  % --- shallow & wide (left) ---
  \node[u] (sx) at (0,0) {};
  \foreach \i/\y in {1/1.8, 2/1.2, 3/0.6, 4/0, 5/-0.6, 6/-1.2, 7/-1.8} \node[u] (sh\i) at (1.6,\y) {};
  \node[u] (so) at (3.2,0) {};
  \foreach \i in {1,...,7} {
    \draw[acc] (sx) -- (sh\i);
    \draw[acc] (sh\i) -- (so);
  }
  \node[align=center, font=\footnotesize] at (1.6,-2.7) {shallow \& wide\\(\texttt{many} units, one layer)};
  % --- deep & narrow (right) ---
  \begin{scope}[xshift=7cm]
    \node[u] (dx) at (0,0) {};
    \foreach \L in {1,2,3,4} {
      \node[u] (d\L t) at (\L*1.1,0.6) {};
      \node[u] (d\L b) at (\L*1.1,-0.6) {};
    }
    \node[u] (do) at (5.5,0) {};
    \draw[acc] (dx) -- (d1t);   \draw[acc] (dx) -- (d1b);
    \draw[acc] (d1t) -- (d2t); \draw[acc] (d1t) -- (d2b);
    \draw[acc] (d1b) -- (d2t); \draw[acc] (d1b) -- (d2b);
    \draw[acc] (d2t) -- (d3t); \draw[acc] (d2t) -- (d3b);
    \draw[acc] (d2b) -- (d3t); \draw[acc] (d2b) -- (d3b);
    \draw[acc] (d3t) -- (d4t); \draw[acc] (d3t) -- (d4b);
    \draw[acc] (d3b) -- (d4t); \draw[acc] (d3b) -- (d4b);
    \draw[acc] (d4t) -- (do);  \draw[acc] (d4b) -- (do);
    \node[align=center, font=\footnotesize] at (2.75,-2.7) {deep \& \texttt{narrow}\\(few units, many layers)};
  \end{scope}
\end{tikzpicture}
$$

> **Definition (Depth & width).** The **depth** of a network is the number of
> layers in its longest input-to-output path; the **width** of a layer is its
> number of units. Depth lets a network express _compositional_ structure ("an
> object is an arrangement of parts, a part an arrangement of edges") with
> parameter count growing additively rather than multiplicatively.

Real data is overwhelmingly compositional. A face is eyes-and-nose-and-mouth in an
arrangement; a sentence is clauses of phrases of words. Depth is the architectural
assumption that the world is built in layers, so the model should be too.

## Why now

The ideas are old — backpropagation dates to the 1980s, convolutional networks to
1989. Deep learning became dominant in the 2010s not because of a conceptual
breakthrough but because three enabling curves crossed at once.[^gf-history]

$$
% caption: Three curves had to cross for deep learning to work: large labelled
% datasets, parallel compute (GPUs), and a handful of training tricks. None alone
% was enough.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=26mm, minimum height=11mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box] (data) at (0,1.5)  {data\\(ImageNet, the web)};
  \node[box] (comp) at (0,0)    {compute\\(GPUs)};
  \node[box] (alg)  at (0,-1.5) {algorithms\\(ReLU, \texttt{dropout}, Adam)};
  \node[box, draw=acc, text=acc, thick] (dl) at (5.4,0) {deep learning\\works};
  \draw[->, acc, thick] (data) -- (dl);
  \draw[->, acc, thick] (comp) -- (dl);
  \draw[->, acc, thick] (alg)  -- (dl);
\end{tikzpicture}
$$

Each curve removed leaves the other two insufficient:

| Curve | What it supplied | Without it |
| --- | --- | --- |
| Data (ImageNet, the web) | signal to fit without memorizing | the model overfits |
| Compute (GPUs) | tractable billions of multiply-adds | training never finishes |
| Algorithms (ReLU, dropout, BatchNorm, Adam) | deep stacks that actually optimize | gradients vanish or explode |

The three are not independent; they _multiply_, and this observation has been
sharpened into **scaling laws**: test loss falls as a smooth power law in model
size $N$, dataset size $D$, and compute $C$, provided all three grow together,

$$
\mathcal{L}(N, D) \;\approx\; \mathcal{L}_\infty + \parens{\frac{N_c}{N}}^{\alpha_N} + \parens{\frac{D_c}{D}}^{\alpha_D},
\qquad \alpha_N, \alpha_D \in (0, 1).
$$

Starve any one term (too little $D$ for a huge $N$, or vice versa) and the
corresponding power-law term dominates, bending the curve flat early. Within the
regime the laws describe, a predictable drop in loss can be _bought_ by spending
proportionally more on all three at once; much of the last decade is that
observation followed to its expensive conclusion.

## What deep learning is bad at

Every failure mode is a thread picked up later; none is fatal, but a confident
practitioner knows exactly where the method is strong and where it is not.

| Failure mode | Cause | Symptom | Picked up in |
| --- | --- | --- | --- |
| Data-hungry | flexibility needs many examples to fit the _right_ thing | poor in the small-data regime | [generalization theory](/deep-learning/theory/generalization-theory) |
| Brittle | flat decision boundaries near the data | $x + \varepsilon$, $\norm{\varepsilon}$ tiny, flips the label; degrades under distribution shift | [adversarial robustness](/deep-learning/theory/adversarial-robustness) |
| Opaque | a billion weights, no symbolic account | no human-readable _why_ for a decision | [Bayesian & ensemble methods](/deep-learning/theory/bayesian-and-ensemble-methods) |

## Scaling laws and compute budgets

The three-curves story predates the era that made it a quantitative law. Since
Goodfellow and Chollet were written, the "why now" has hardened into numbers.

**Scaling laws, and how to spend a compute budget.** Kaplan et al. (2020) measured
the power-law drop in loss sketched above across model size, data, and compute.
Hoffmann et al. (2022) — the **Chinchilla** result — then corrected the recipe:
for a fixed compute budget, earlier large models were badly _undertrained_, and
loss is minimized by scaling parameters and training tokens in roughly equal
proportion, about 20 tokens per parameter. The practical upshot inverted an
industry: a smaller model trained on more data beats a larger model trained on
less, at the same cost.

**Emergent abilities.** Some capabilities appear absent in small models and then
turn on sharply past a scale threshold (Wei et al., 2022) — arithmetic,
multi-step reasoning, instruction following. Whether these are genuine phase
transitions or artifacts of discontinuous metrics is actively debated, but the
observation reshaped how the field reads a scaling curve: not every gain is smooth.

**The foundation-model shift.** The self-supervised middle row of the taxonomy
above became the field's center of gravity. A single model pretrained on
web-scale unlabeled data (Bommasani et al., 2021, name these _foundation models_)
is adapted to hundreds of downstream tasks, collapsing the per-task pipeline into
pretrain-then-adapt — the "which slot did they change?" question answered, for a
while, with _the data, at unprecedented scale_.[^scaling][^foundation]

[^gf-intro]: **Goodfellow**, _Deep Learning_, Ch. 1 — Introduction: deep learning as nested representation learning, distinguished from classical AI by composing learned simple functions rather than encoding hand-written rules.
[^chollet-repr]: **Chollet**, _Deep Learning with Python_, §1.1.4 — Learning Representations: each layer transforms its input into an increasingly abstract code, the edges-to-parts-to-objects hierarchy recovered from the loss alone.
[^gf-capacity]: **Goodfellow**, _Deep Learning_, §5.2 — Capacity, Overfitting and Underfitting: parametric models fix capacity through a finite parameter vector, with representational power growing as the architecture widens or deepens.
[^chollet-loop]: **Chollet**, _Deep Learning with Python_, §2.4 — The Engine of Neural Networks: the forward-pass / loss / gradient / update training loop, and why differentiability makes the gradient step a single subtraction.
[^gf-e2e]: **Goodfellow**, _Deep Learning_, §1.2; §6.5 — Back-Propagation and end-to-end learning: one final loss whose gradient reaches every parameter, co-adapting feature extractor and classifier instead of training hand-built stages in isolation.
[^gf-history]: **Goodfellow**, _Deep Learning_, §1.2.1 — historical trends: the method is old, but large labelled datasets, GPU compute, and a handful of optimization tricks had to arrive together for deep stacks to train.
[^gf-depth]: **Goodfellow**, _Deep Learning_, §6.4 — Architectural Design / depth: deep compositions express functions that a shallow net can match only at exponentially greater width, the formal case for stacking layers.
[^scaling]: Kaplan, McCandlish, Henighan, Brown, Chess, Child, Gray, Radford, Wu, Amodei (2020), _Scaling Laws for Neural Language Models_, arXiv:2001.08361; and Hoffmann et al. (2022), _Training Compute-Optimal Large Language Models_ (Chinchilla), arXiv:2203.15556 — the compute-optimal balance of parameters and training tokens. Emergence: Wei et al. (2022), _Emergent Abilities of Large Language Models_, TMLR.
[^foundation]: Bommasani et al. (2021), _On the Opportunities and Risks of Foundation Models_, arXiv:2108.07258 — the pretrain-once, adapt-many paradigm built on the self-supervised setting.