Practical Methodology

A practitioner who knows every optimizer and architecture in this course can still ship a model that fails because of a wrong decision about which equation to change next. Goodfellow's central claim is that correctly applying a commonplace algorithm usually beats sloppily applying an exotic one.¹ The skill is methodological: pick the right metric, build an end-to-end system fast, instrument it, and let a diagnosis, not a hunch, pick the next move. This lesson is that decision procedure, made explicit.

The reason discipline matters so much is that deep-learning failures are ambiguous. A training run that plateaus at $40%$ error could be starved of capacity, starved of data, poorly optimized, or silently corrupted by a data-loader bug, and the loss curve alone does not say which. Every one of those causes has a different remedy, and applying the wrong remedy costs days. The methodology exists to turn that ambiguity into a sequence of cheap measurements that each rule out one cause.

The design process

The recommended workflow is a loop, not a waterfall. Each pass through it changes exactly one thing, chosen by what the instrumentation says is the bottleneck. Changing several things at once is the classic error: if error drops, you cannot attribute the win, and if it rises, you cannot attribute the loss, so the next iteration starts blind.

The methodology loop. Measure a baseline, let the train/validation gap diagnose bias versus variance, apply the one remedy the diagnosis names, and re-measure. The loop exits only when the target metric is met.

The single most common mistake is to skip step 2, to tune an idea in the abstract before any complete system runs. The baseline is what converts vague worry into a measured number, and the number is what makes step 3 possible.²

Chollet frames step 1 more sharply: before you trust any learned model, it must beat a common-sense baseline that uses no learning at all. For a balanced two-class problem that baseline is random guessing at $50%$ accuracy; for a temperature forecaster it is tomorrow equals today; for an imbalanced detector it is the majority-class constant. A model that fails to clear this floor is not learning anything useful from its inputs, and no amount of tuning will rescue it until the data pipeline or the target itself is fixed. The floor is also a sanity check on the whole harness: if a model that should be able to beat it does not, the bug is usually in the plumbing, not the network.

Performance metrics

Before building anything, decide what good means as a single number. The default choice, accuracy, silently fails under class imbalance, the regime where one class dominates the data.

To address this, score the two error types separately. Fix a positive class and count the four outcomes of a binary decision.

The confusion matrix. Precision reads down the predicted-positive column; recall reads across the true-positive row.

Using these counts, we can compute three metrics that are robust against imbalance:³

Precision: how often are positive labels correct?
Recall: how many of the actual positives are correctly identified?
$F_{1}$ : the harmonic mean of precision and recall, giving a more holistic metric that punishes models that sacrifice one for the other

precision = \frac{T P}{T P + F P}, recall = \frac{T P}{T P + F N}, F_{1} = \frac{2 P R}{P + R} .

The harmonic mean is deliberate: with precision $P$ and recall $R$ ,

F_{1} = 2 (\frac{1}{P} + \frac{1}{R})^{- 1},

which collapses toward the smaller of the two: a model at $P = 0.99, R = 0.01$ scores $F_{1} \approx 0.02$ , not the $0.50$ an arithmetic mean would award.

For example, take a disease screen run on $N = 10, 000$ patients of whom $50$ actually have the disease ( $p = 0.005$ ). A model flags $80$ patients as positive; of those, $40$ are true cases and $40$ are false alarms, so it misses $10$ real cases:

T P = 40, F P = 40, F N = 10, T N = 9, 910.

Its accuracy is $(T P + T N) / N = 9, 950/10, 000 = 99.5%$ , which sounds excellent, yet the always-negative constant classifier scores $99.5%$ too and catches nobody. The error-type metrics show the problem:

P = \frac{40}{40 + 40} = 0.50, R = \frac{40}{40 + 10} = 0.80, F_{1} = \frac{2 ( 0.50 ) ( 0.80 )}{0.50 + 0.80} \approx 0.615.

Recall of $0.80$ says the model catches four of every five true cases; precision of $0.50$ says half its alarms are false. Whether that trade is acceptable depends on the downstream cost, a missed case versus a needless follow-up test, and no single scalar decides it for you. The right metric is dictated by the cost structure of the goal.

Metric	Definition	When it is the right choice
Accuracy	$(T P + T N) / N$	balanced classes, symmetric error costs
Precision	$T P / (T P + F P)$	false positives are expensive (spam filter flags real mail)
Recall	$T P / (T P + F N)$	false negatives are expensive (missed tumor, fraud)
$F_{1}$	$2 P R / (P + R)$	imbalanced data, both errors matter
PR-AUC	area under precision–recall curve	imbalanced data, threshold not yet fixed
ROC-AUC	area under TPR–FPR curve	ranking quality, roughly balanced classes
Coverage	fraction the model is allowed to answer	a system may decline to predict when unsure

Precision and recall trade off against each other as the decision threshold $t$ on the score $\overset{y}{^} = p (positive ∣ x)$ moves. Sweeping $t$ from $1$ down to $0$ traces a curve; its summary, the area underneath, is threshold-free.

The precision–recall curve, swept by the decision threshold

t

: raising

t

trades recall for precision, and PR-AUC summarizes the whole curve.

Default baseline models

With a metric chosen, build the simplest system that could plausibly work. A strong baseline both establishes the number to beat and tells you whether the problem is even tractable with the data on hand. The sensible first choice is dictated by the input type.

Data type	Default architecture	Default optimizer
Fixed-size vectors	feedforward MLP, fully connected	SGD + momentum, or Adam
Images / grid topology	convolutional network with residual connections	SGD + momentum, or Adam
Sequences / time series	gated RNN (LSTM/GRU) or a Transformer	Adam
Text / language	Transformer with pretrained embeddings	Adam with warmup
Tabular, small $n$	regularized linear model or gradient-boosted trees	closed-form / coordinate descent

Resist the urge to start exotic. If a known result exists for a similar task, copy its architecture and optimizer wholesale; reproducing a published baseline before innovating is the fastest path to a trustworthy number.

Whether to gather more data

After the baseline runs, the most consequential decision is whether to collect more data (often expensive) or to change the model. The diagnosis is read directly off two numbers: training error and validation error. This is the bias–variance decomposition applied in practice.

The two failure modes call for opposite remedies, and confusing them wastes the most resources. More data cannot fix underfitting; a bigger model cannot fix overfitting.⁴

Symptom	Diagnosis	Remedy
$E_{train}$ high	underfitting (bias)	bigger model, train longer, better optimizer, fewer constraints
$E_{train}$ low, gap large	overfitting (variance)	more data, regularize, data augmentation, smaller model
both acceptable, $E_{val}$ still short	irreducible / framing	new features, new metric, reconsider the goal

Plotting error against the size of the training set separates the two regimes: high-bias curves converge high and close together; high-variance curves leave a persistent gap that more data would narrow.

Learning-curve diagnosis. Left: high bias, curves converge high and more data will not help. Right: high variance, a wide gap that more data closes.

The extrapolation is worth doing arithmetically before spending a data-collection budget. Suppose the error follows $E (m) = a m^{- α}$ , and you measure $E = 0.20$ at $m = 10, 000$ and $E = 0.14$ at $m = 40, 000$ . The exponent comes from the ratio,

\frac{0.14}{0.20} = (\frac{40 , 000}{10 , 000})^{- α} ⟹ α = - \frac{log ( 0.70 )}{log 4} \approx 0.257.

To hit a target $E^{⋆} = 0.10$ from the $m = 10, 000$ anchor you need

m^{⋆} = 10, 000 (\frac{0.20}{0.10})^{1/ α} = 10, 000 \cdot 2^{3.9} \approx 1.5 \times 1 0^{5},

so reaching $10%$ error costs roughly fifteen times the current data. That number is what decides whether to label more examples or to change the model instead, and it is far cheaper to estimate than to discover after a failed collection round.

The data pipeline

More data helps only if the path from raw examples to the network delivers them correctly, and this path is where a large share of real bugs hide. A subtle preprocessing error, normalizing with the wrong statistics, leaking test labels into training, shuffling images but not their labels, corrupts every gradient the model ever sees, yet training loss may still fall because the network fits the corrupted signal anyway. The pipeline deserves the same scrutiny as the model.

A standard training data pipeline. Normalization statistics

(μ, σ)

are computed on the training split ONLY and reused for validation and test; augmentation is applied to training batches only.

Two rules govern this pipeline, and both prevent a specific and common failure. Normalization statistics, the per-feature mean $μ$ and standard deviation $σ$ used to standardize inputs to zero mean and unit variance, must be estimated on the training split alone and then reused verbatim on validation and test. Fitting them on the full dataset lets information about the test distribution leak backward into training, inflating the reported score above what deployment will deliver. Second, augmentation is a training-only transformation: random crops, flips, color jitter, and additive noise expand the effective training set and act as a regularizer, but validation and test must see clean, canonical inputs so the metric measures real generalization rather than luck on a random crop.

The transformation must preserve the label, and that is domain-specific. A horizontal flip is safe for a cat photo but destroys a photo of a printed digit, where it turns a $2$ into a mirror image that is no digit at all, and it flips left and right in a self-driving scene. Choose augmentations that respect the invariances the task actually has, not a generic recipe.

Scaling laws turn the gap diagnosis quantitative

Goodfellow's power-law remark — that generalization error falls predictably with data — was sharpened into a design tool by the neural scaling laws literature. Hestness et al. (2017, arXiv) and then Kaplan et al. (2020, arXiv, Scaling Laws for Neural Language Models) measured loss across orders of magnitude of data, parameters, and compute and found it follows power laws over a wide range,

L (N) \approx (\frac{N _{c}}{N})^{α_{N}}, L (D) \approx (\frac{D _{c}}{D})^{α_{D}},

where $N$ is parameter count, $D$ dataset size, and the exponents $α$ are small (often $\approx 0.05$ – $0.1$ for language). Straight lines on a log–log plot mean the extrapolation from the estimating the data needed remark is a fit rather than a heuristic: measure two points, read the slope, predict the third.

The refinement that matters for the gather-more-data decision is compute-optimal allocation. Hoffmann et al. (2022, NeurIPS, the Chinchilla paper) showed that, for a fixed compute budget, most large models were badly over-parameterized and under-trained: the optimum scales $N$ and $D$ together in roughly equal proportion, so a model half the size trained on twice the data beats it at the same cost. That is the gap diagnosis at industrial scale — the bottleneck was data, not capacity, exactly the reading the train/validation gap prescribes, now made quantitative enough to size a training run before launching it. The practitioner's lesson is unchanged from the loop above: measure, diagnose, then spend the next unit of budget where the diagnosis points.

Takeaways

Methodology beats novelty. A standard algorithm applied with discipline (fix a metric, build an end-to-end baseline, instrument, iterate on a diagnosis) outperforms an exotic algorithm applied carelessly. Every learned model must first clear a common-sense baseline that uses no learning at all.
Choose the metric to match the goal. Accuracy collapses under class imbalance; precision, recall, $F_{1}$ , and the PR/ROC curves separate the error types so the threshold can be set by the real cost structure. The $F_{1}$ harmonic mean collapses toward the smaller of precision and recall, punishing a model that sacrifices one.
Start with a strong, type-appropriate baseline (MLP for vectors, CNN for images, gated RNN or Transformer for sequences) using reasonable defaults (ReLU, Adam or SGD+momentum, early stopping). If a tuned exotic model cannot beat it, the bottleneck is the data or the framing, not the architecture.
Let the train/validation gap pick the next move. High training error means underfitting (bigger model, better optimization); a large gap means overfitting (more data, regularization). More data never fixes bias. Fit the learning curve's power law to estimate how many examples reach the target before paying to collect them.
Guard the data pipeline. Fit normalization statistics on the training split only, apply augmentation to training batches only, and pick label-preserving transforms that match the task's real invariances — a horizontal flip is safe for a cat but destroys a printed digit.

This continues in Hyperparameters and Debugging, which picks up the tuning half of the loop: the learning rate as the dominant hyperparameter, random search over grid, an ordered debugging playbook, and monitoring for drift after deployment.

Goodfellow, Deep Learning, Ch. 11 — Practical Methodology: the recommended iterate-to-target design loop, and the thesis that disciplined application of a standard method beats careless use of a novel one. ↩
Chollet, Deep Learning with Python, §4.5 — The Universal Workflow of Machine Learning: stand up an end-to-end baseline that beats a common-sense reference before tuning anything in the abstract. ↩
Goodfellow, Deep Learning, §11.1 — Performance Metrics: accuracy's failure under class imbalance and the precision/recall/ $F_{1}$ family that scores the two error types separately. ↩
Goodfellow, Deep Learning, §11.3 / §5.2 — Determining Whether to Gather More Data: read the train/validation gap as bias versus variance, since more data fixes overfitting but never underfitting. ↩