What Is Deep Learning?

Classical programming is a human writing the rules. You want to detect spam, so you sit down and enumerate conditions: contains free money, sender not in contacts, more than three exclamation marks. Machine learning inverts that arrangement. Instead of rules, you supply examples (ten thousand emails, each labelled spam or not) and a measure of how wrong a guess is, and the machine searches for rules that fit the examples. You no longer write the classifier; you write the thing that finds the classifier.

Deep learning is the special case of machine learning where the rules are a deep stack of simple, differentiable transformations, and the search is gradient descent. That is the entire definition.¹ Everything else (convolutions, attention, batch normalization, and the rest) is engineering on top of those two commitments: compose simple differentiable pieces, and learn them by following the gradient.

Representation learning

The deepest reason the field is called deep is not the number of layers; it is representation learning. To see why it matters, look at what the alternative costs.

A linear model can only separate classes with a flat boundary drawn through its raw inputs. If the inputs are pixel intensities, that is hopeless — no hyperplane through raw-pixel space distinguishes cats from dogs, because the property cat is not a linear function of individual pixels. For decades the fix was feature engineering: a human expert designed a transformation $ϕ (x)$ (edge detectors, SIFT descriptors, hand-tuned filters) that mapped raw pixels into a space where a linear model could work. The learning algorithm was the easy part; the labor and the bottleneck were in $ϕ$ .

Deep learning automates $ϕ$ . The network learns the representation, one layer at a time, each layer re-describing the output of the one below it. Trained on images, the early layers reliably discover oriented edges; the middle layers compose edges into textures and motifs; the late layers compose those into object parts and whole objects — a hierarchy nobody programmed, recovered purely from the pressure to reduce the loss.²

A deep network learns a hierarchy of representations, each layer re-describing the one below, from raw pixels up to a class.

This is why feature engineering, once the central task of applied machine learning, has all but vanished from deep-learning practice. The representation is learned rather than built.

What raw pixels look like to a linear model

A $28 \times 28$ grayscale image is a vector $x \in R^{784}$ ; a linear classifier's only verdict is a weighted sum $w^{⊤} x + b$ , one fixed weight per pixel. The failure is visible in the metric. Let $x$ be a clean $7$ and $T_{δ} x$ the same $7$ shifted by $δ$ pixels. A small shift changes hundreds of coordinates, so Euclidean distance in pixel-space is dominated by alignment, not identity:

∥ x - T_{δ} x ∥_{2} ≫ ∥ x - x^{'} ∥_{2}, x^{'} = a different digit overlapping x,

i.e. the same digit shifted can land farther from itself than from a different class. A single hyperplane cannot track a digit across the regions of pixel-space that translation, slant, and stroke-width scatter it into. The fix is to change coordinates: learn a map $h = ϕ_{θ} (x)$ under which

∥ ϕ_{θ} (x) - ϕ_{θ} (T_{δ} x)∥ \approx 0, ∥ ϕ_{θ} (x) - ϕ_{θ} (x^{'})∥ large,

so a linear readout $w^{⊤} h + b$ , the operation that failed on raw pixels, now succeeds on the learned ones. The pixels did not change; the coordinates did.

In raw-pixel space (left) the two classes interleave and no line separates them; the learned map

ϕ_{θ}

warps the space (right) so a straight boundary suffices.

Cats versus dogs is the same statement at scale: cat-ness survives translation, rotation, lighting, and occlusion, and a deep net discovers that invariance because staying invariant to nuisance variation is what drives the loss down across a varied training set.

A taxonomy of learning

Deep learning is a method for fitting differentiable models; it is not itself a kind of learning problem. The same network and the same training loop slot into several quite different problem settings, distinguished by what kind of supervision the data carries.

One differentiable model, trained under whatever supervision the setting provides: full labels, none, self-mined labels, or a reward signal.

The four settings differ only in where the loss signal comes from. The model $f_{θ}$ and the training loop are shared; only the data and the loss target change.

Paradigm	Data	Loss signal	Typical objective	Example
Supervised	$(x, y)$	given label $y$	$ℓ (f_{θ} (x), y)$	image classification
Unsupervised	$x$ only	none	reconstruction / density $- log p_{θ} (x)$	clustering, autoencoding
Self-supervised	$x$ , target mined from $x$	label from a held-out part of $x$	predict masked $x_{hidden}$	masked-language modeling
Reinforcement	states, actions	scalar reward $r$ , often delayed	maximize $E [\sum_{t} γ^{t} r_{t}]$	game-playing, control

Self-supervised learning is the notable middle row: no human annotates anything, yet a label is manufactured from the input by hiding part of it and predicting it back. This is the basis of large language models.

The line between them is softer than the taxonomy suggests (self-supervised pretraining followed by supervised fine-tuning is now the dominant approach), but the distinction in where the loss signal comes from is real, and it is the first question to ask about any new system.

The four ingredients

Strip away the architecture jargon and every deep-learning system, from a two-line perceptron to a four-hundred-billion-parameter language model, is assembled from the same four parts.

Ingredient	Symbol	Role	Example
Model	$f_{θ} : X \to Y$	maps input to prediction	a 50-layer CNN
Loss	$ℓ (\overset{y}{^}, y)$	scores a single prediction	cross-entropy
Optimizer	$θ \leftarrow θ - η \nabla_{θ} L$	updates parameters	Adam
Data	$D = {(x_{i}, y_{i})}_{i = 1}^{n}$	defines the problem	ImageNet

These four are exhaustive: reading any new paper, the first useful question is which slot did they change? — and there are no other slots.

Slot	Symbol changed	What a paper contributes	Example
Architecture	$f_{θ}$	a new way to connect layers	ResNet, Transformer
Objective	$ℓ$	a new thing to minimize	contrastive loss, GAN loss
Optimizer	update rule	a better descent step	Adam, RMSProp
Data	$D$	a bigger / cleaner sample	ImageNet, web-scale text

Parametric function approximators

A parametric model commits in advance to a fixed-size $θ$ and discards the data after training; a non-parametric model keeps the data and lets complexity grow with the sample (e.g. $k$ -nearest-neighbors stores every training point).

A deep network is squarely parametric: the architecture fixes the weight count before a single example is seen, and training adjusts only their values. Two facts justify the fixed-size commitment:³

Expressivity. With enough units a net approximates any reasonable target function (the universal approximation result), so $θ$ being fixed-size is not a real limitation.
Capacity scales with parameter count. A wider or deeper net has more weights, represents more intricate functions, and, given enough data, fits more of the structure in it.

This is why simply making the model bigger works; we return to its limits under the bias–variance tradeoff.

The training loop

The four ingredients interact through the training loop. Training is a loop: draw a batch of data, push it through the model, score the predictions with the loss, measure how the loss responds to each parameter (the gradient), and step the parameters downhill. Repeat until the loss stops falling.

The training loop: a minibatch flows forward to a loss, the gradient flows back, and the optimizer updates the parameters before repeating.

Because every layer is differentiable, the gradient $\nabla_{θ} L$ can be computed exactly and cheaply by backpropagation, and the update is a single subtraction.⁴ Written out, the loop is short:

Algorithm:

\textsc{Train}(f_\theta, \mathcal{D}, \eta)

— minibatch gradient descent

1
initialize $\theta$ randomly
2
repeat
3
sample a minibatch $(X, y) \sim \mathcal{D}$
4
$\hat{y} \gets f_\theta(X)$
forward pass
5
$L \gets \text{loss}(\hat{y}, y)$
how wrong are we?
6
$g \gets \nabla_\theta L$
backward pass (autodiff)
7
$\theta \gets \theta - \eta \cdot g$
gradient-descent step
8
until converged
9
return $\theta$

Every later lesson refines one line of this skeleton. Better architectures change line 4; better losses change line 5; better optimizers change line 7; regularization changes what converged should mean. The loop itself never changes.

Differentiability and the end-to-end bet

Line 6 of that algorithm is possible because every operation between the input and the loss is differentiable, so the loss has a gradient with respect to every parameter, no matter how deep. This is the central assumption of the field. Differentiability is what lets a correction at the output propagate, by the chain rule, all the way back to the first layer's weights, the layers that compute the representation. A single gradient signal tunes the features and the classifier at the same time, each in service of the final loss.⁵

Contrast the two ways to build a multi-stage system:

	Classical pipeline	End-to-end
Stage objective	a human-chosen proxy per stage	the one final loss, for all stages
Coupling	stages frozen, optimized in isolation	stages co-adapt via shared gradient
Requirement	none (stages can be non-differentiable)	whole pipeline differentiable
Failure mode	early stage tuned for the wrong target	needs more data and compute

The classical pipeline hand-builds stage one (an edge detector), freezes it, builds stage two on top, and so on — but no early stage knows what the final task will ask of it. End-to-end training removes those seams: with $g = \nabla_{θ} L$ reaching every weight, every stage is optimized for the only thing that matters. On perception and language this has worked so well that the hand-built pipeline has all but disappeared.

Learning is descent on a surface

Line 7 has a geometric reading. The loss $L (θ)$ is a function of millions of parameters; fix all but two and you can draw it as a surface over a plane. Training is a ball rolling downhill on that surface, the gradient pointing in the direction of steepest ascent so we step the other way.

Gradient descent on the loss surface. From a random start the optimizer steps against the gradient, tracing a path down the contours toward a minimum.

The picture is useful but simplified. In two dimensions the surface is a clean bowl with one minimum. In a million dimensions it is nothing of the kind — riddled with saddle points, flat plateaus, and ravines. That geometry is why so much of the subject is about how to descend it well: momentum, adaptive learning rates, normalization, good initialization. We return to this in the optimization chapter.

Why depth

If a single hidden layer is already a universal approximator — and it is — why stack many? Because the theorem is about possibility, not cost. A shallow network can represent a given function only by paying in width, sometimes exponentially many units; a deep network represents the same function compactly by reusing intermediate features.⁶

A shallow net enumerates cases in one enormous layer; a deep net reuses features across layers, computing the same function with far fewer units.

Real data is overwhelmingly compositional. A face is eyes-and-nose-and-mouth in an arrangement; a sentence is clauses of phrases of words. Depth is the architectural assumption that the world is built in layers, so the model should be too.

Why now

The ideas are old — backpropagation dates to the 1980s, convolutional networks to 1989. Deep learning became dominant in the 2010s not because of a conceptual breakthrough but because three enabling curves crossed at once.⁷

Three curves had to cross for deep learning to work: large labelled datasets, parallel compute (GPUs), and a handful of training tricks. None alone was enough.

Each curve removed leaves the other two insufficient:

Curve	What it supplied	Without it
Data (ImageNet, the web)	signal to fit without memorizing	the model overfits
Compute (GPUs)	tractable billions of multiply-adds	training never finishes
Algorithms (ReLU, dropout, BatchNorm, Adam)	deep stacks that actually optimize	gradients vanish or explode

The three are not independent; they multiply, and this observation has been sharpened into scaling laws: test loss falls as a smooth power law in model size $N$ , dataset size $D$ , and compute $C$ , provided all three grow together,

L (N, D) \approx L_{\infty} + (\frac{N _{c}}{N})^{α_{N}} + (\frac{D _{c}}{D})^{α_{D}}, α_{N}, α_{D} \in (0, 1) .

Starve any one term (too little $D$ for a huge $N$ , or vice versa) and the corresponding power-law term dominates, bending the curve flat early. Within the regime the laws describe, a predictable drop in loss can be bought by spending proportionally more on all three at once; much of the last decade is that observation followed to its expensive conclusion.

What deep learning is bad at

Every failure mode is a thread picked up later; none is fatal, but a confident practitioner knows exactly where the method is strong and where it is not.

Failure mode	Cause	Symptom	Picked up in
Data-hungry	flexibility needs many examples to fit the right thing	poor in the small-data regime	generalization theory
Brittle	flat decision boundaries near the data	$x + ε$ , $∥ ε ∥$ tiny, flips the label; degrades under distribution shift	adversarial robustness
Opaque	a billion weights, no symbolic account	no human-readable why for a decision	Bayesian & ensemble methods

Scaling laws and compute budgets

The three-curves story predates the era that made it a quantitative law. Since Goodfellow and Chollet were written, the why now has hardened into numbers.

Scaling laws, and how to spend a compute budget. Kaplan et al. (2020) measured the power-law drop in loss sketched above across model size, data, and compute. Hoffmann et al. (2022) — the Chinchilla result — then corrected the recipe: for a fixed compute budget, earlier large models were badly undertrained, and loss is minimized by scaling parameters and training tokens in roughly equal proportion, about 20 tokens per parameter. The practical upshot inverted an industry: a smaller model trained on more data beats a larger model trained on less, at the same cost.

Emergent abilities. Some capabilities appear absent in small models and then turn on sharply past a scale threshold (Wei et al., 2022) — arithmetic, multi-step reasoning, instruction following. Whether these are genuine phase transitions or artifacts of discontinuous metrics is actively debated, but the observation reshaped how the field reads a scaling curve: not every gain is smooth.

The foundation-model shift. The self-supervised middle row of the taxonomy above became the field's center of gravity. A single model pretrained on web-scale unlabeled data (Bommasani et al., 2021, name these foundation models) is adapted to hundreds of downstream tasks, collapsing the per-task pipeline into pretrain-then-adapt — the which slot did they change? question answered, for a while, with the data, at unprecedented scale.⁸⁹

Goodfellow, Deep Learning, Ch. 1 — Introduction: deep learning as nested representation learning, distinguished from classical AI by composing learned simple functions rather than encoding hand-written rules. ↩
Chollet, Deep Learning with Python, §1.1.4 — Learning Representations: each layer transforms its input into an increasingly abstract code, the edges-to-parts-to-objects hierarchy recovered from the loss alone. ↩
Goodfellow, Deep Learning, §5.2 — Capacity, Overfitting and Underfitting: parametric models fix capacity through a finite parameter vector, with representational power growing as the architecture widens or deepens. ↩
Chollet, Deep Learning with Python, §2.4 — The Engine of Neural Networks: the forward-pass / loss / gradient / update training loop, and why differentiability makes the gradient step a single subtraction. ↩
Goodfellow, Deep Learning, §1.2; §6.5 — Back-Propagation and end-to-end learning: one final loss whose gradient reaches every parameter, co-adapting feature extractor and classifier instead of training hand-built stages in isolation. ↩
Goodfellow, Deep Learning, §6.4 — Architectural Design / depth: deep compositions express functions that a shallow net can match only at exponentially greater width, the formal case for stacking layers. ↩
Goodfellow, Deep Learning, §1.2.1 — historical trends: the method is old, but large labelled datasets, GPU compute, and a handful of optimization tricks had to arrive together for deep stacks to train. ↩
Kaplan, McCandlish, Henighan, Brown, Chess, Child, Gray, Radford, Wu, Amodei (2020), Scaling Laws for Neural Language Models, arXiv:2001.08361; and Hoffmann et al. (2022), Training Compute-Optimal Large Language Models (Chinchilla), arXiv:2203.15556 — the compute-optimal balance of parameters and training tokens. Emergence: Wei et al. (2022), Emergent Abilities of Large Language Models, TMLR. ↩
Bommasani et al. (2021), On the Opportunities and Risks of Foundation Models, arXiv:2108.07258 — the pretrain-once, adapt-many paradigm built on the self-supervised setting. ↩