--- title: Practical Methodology module: Practical Deep Learning moduleNumber: 9 lessonNumber: 1 order: 901 summary: > Knowing the algorithms is half the job; the other half is a disciplined loop. Fix a goal and a metric, stand up an end-to-end baseline, then read the train/validation gap to decide whether the next move is more data or a bigger model. We detail that loop: choosing metrics under class imbalance, default baselines by data type, extrapolating the data a target needs, and guarding the data pipeline against the leaks and label bugs that corrupt every gradient. Hyperparameter tuning, debugging, and deployment continue in the sequel. topics: [Practical Deep Learning] sources: - book: Goodfellow ref: "Ch. 11 — Practical Methodology; §11.1 Performance Metrics" - book: Goodfellow ref: "§5.2 Capacity, Overfitting, Underfitting; §11.3 Whether to Gather More Data" - book: Chollet ref: "Ch. 4 — Fundamentals of Machine Learning; §4.5 The Universal Workflow" --- A practitioner who knows every optimizer and architecture in this course can still ship a model that fails because of a wrong _decision_ about which equation to change next. Goodfellow's central claim is that **correctly applying a commonplace algorithm usually beats sloppily applying an exotic one**.[^gf-method] The skill is methodological: pick the right metric, build an end-to-end system fast, instrument it, and let a _diagnosis_, not a hunch, pick the next move. This lesson is that decision procedure, made explicit. The reason discipline matters so much is that deep-learning failures are _ambiguous_. A training run that plateaus at $40\%$ error could be starved of capacity, starved of data, poorly optimized, or silently corrupted by a data-loader bug, and the loss curve alone does not say which. Every one of those causes has a different remedy, and applying the wrong remedy costs days. The methodology exists to turn that ambiguity into a sequence of cheap measurements that each rule out one cause. ## The design process The recommended workflow is a loop, not a waterfall. Each pass through it changes exactly one thing, chosen by what the instrumentation says is the bottleneck. Changing several things at once is the classic error: if error drops, you cannot attribute the win, and if it rises, you cannot attribute the loss, so the next iteration starts blind. > **Definition (Practical design loop).** A four-step iterate-to-target cycle: (1) > fix the goal as an _error metric_ and a _target value_; (2) build a working > end-to-end pipeline as fast as possible; (3) instrument it to find the > bottleneck (too little capacity, too little data, a bug, or bad > hyperparameters); (4) make one change indicated by the diagnosis, and repeat. $$ % caption: The methodology loop. Measure a baseline, let the train/validation gap % diagnose bias versus variance, apply the one remedy the diagnosis names, and % re-measure. The loop exits only when the target metric is met. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=27mm, minimum height=11mm, align=center}, dec/.style={draw, diamond, aspect=1.7, inner sep=1pt, align=center}, act/.style={draw, minimum width=26mm, minimum height=9mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \definecolor{red}{HTML}{C0392B} % central spine (top to bottom) \node[box, draw=acc, text=acc, thick] (goal) at (0,3.0) {goal:\\metric + target}; \node[box] (base) at (0,0.9) {end-to-end\\baseline}; \node[box] (meas) at (0,-1.2) {measure\\train + val error}; \node[dec] (diag) at (0,-3.8) {target\\met?}; \node[green] (done) at (4.4,-3.8) {ship}; % left column: gap decision at bottom, two remedies stacked above \node[dec] (gap) at (-5.4,-3.8) {large\\gap?}; \node[act] (data) at (-5.4,0.9) {more data /\\regularize}; \node[act] (cap) at (-5.4,-1.2) {bigger model /\\better optim}; % spine arrows \draw[->, acc, thick] (goal) -- (base); \draw[->, acc, thick] (base) -- (meas); \draw[->, acc, thick] (meas) -- (diag); \draw[->, green, thick] (diag) -- (done) node[midway, above, font=\scriptsize] {yes}; % target not met: go diagnose the gap (straight left, no crossings) \draw[->, acc, thick] (diag) -- (gap) node[midway, above, font=\scriptsize] {no}; % large gap? yes = variance = more data / regularize (long left branch) \draw[->, red, thick] (gap.west) .. controls (-7.4,-1.8) and (-7.4,0.2) .. (data.west) node[pos=0.08, above right, font=\scriptsize] {yes}; % large gap? no = bias = bigger model / better optim \draw[->, red, thick] (gap.north) -- (cap.south) node[midway, right, font=\scriptsize] {no}; % remedies feed back into the spine \draw[->, acc, thick] (data.north) .. controls (-2.6,2.6) .. (base.west); \draw[->, acc, thick] (cap.east) -- (meas.west); \end{tikzpicture} $$ The single most common mistake is to skip step 2, to tune an idea in the abstract before any complete system runs. The baseline is what converts vague worry into a measured number, and the number is what makes step 3 possible.[^chollet-workflow] Chollet frames step 1 more sharply: before you trust any learned model, it must beat a **common-sense baseline** that uses no learning at all. For a balanced two-class problem that baseline is random guessing at $50\%$ accuracy; for a temperature forecaster it is "tomorrow equals today"; for an imbalanced detector it is the majority-class constant. A model that fails to clear this floor is not learning anything useful from its inputs, and no amount of tuning will rescue it until the data pipeline or the target itself is fixed. The floor is also a sanity check on the whole harness: if a model that _should_ be able to beat it does not, the bug is usually in the plumbing, not the network. ## Performance metrics Before building anything, decide what "good" means as a single number. The default choice, accuracy, silently fails under **class imbalance**, the regime where one class dominates the data. > **Definition (Accuracy's failure mode).** On a dataset where a fraction > $1-p$ of examples are negative, the constant classifier "always predict > negative" attains accuracy $1-p$. With $p = 0.001$ (a rare disease) it scores > $99.9\%$ while detecting nothing: accuracy rewards ignoring the class that > matters. To address this, score the two error types separately. Fix a positive class and count the four outcomes of a binary decision. $$ % caption: The confusion matrix. Precision reads down the predicted-positive % column; recall reads across the true-positive row. \begin{tikzpicture}[font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \definecolor{red}{HTML}{C0392B} \def\s{2.0} % cells: light tint fill + crisp colored outline (correct = green, error = red) \draw[green, very thick, fill=green!15] (0,0) rectangle (\s,\s); % TN bottom-left \draw[green, very thick, fill=green!15] (\s,\s) rectangle (2*\s,2*\s); % TP top-right \draw[red, very thick, fill=red!15] (\s,0) rectangle (2*\s,\s); % FP bottom-right \draw[red, very thick, fill=red!15] (0,\s) rectangle (\s,2*\s); % FN top-left \draw[black] (0,0) rectangle (2*\s,2*\s); % cell labels (correct = green, error = red) \node[text=red] at (0.5*\s,1.5*\s) {FN}; \node[text=green] at (1.5*\s,1.5*\s) {TP}; \node[text=green] at (0.5*\s,0.5*\s) {TN}; \node[text=red] at (1.5*\s,0.5*\s) {FP}; % axis labels (short forms; full words in the caption to dodge ligature breaks) \node[align=center] at (0.5*\s,2*\s+0.5) {pred.\\neg}; \node[align=center] at (1.5*\s,2*\s+0.5) {pred.\\pos}; \node[align=center, rotate=90] at (-0.55,1.5*\s) {true\\pos}; \node[align=center, rotate=90] at (-0.55,0.5*\s) {true\\neg}; \end{tikzpicture} $$ Using these counts, we can compute three metrics that are robust against imbalance:[^gf-metrics] - **Precision**: how often are positive labels correct? - **Recall**: how many of the actual positives are correctly identified? - **$F_1$**: the harmonic mean of precision and recall, giving a more holistic metric that punishes models that sacrifice one for the other $$ \text{precision} = \frac{TP}{TP + FP}, \qquad \text{recall} = \frac{TP}{TP + FN}, \qquad F_1 = \frac{2\,PR}{P + R}. $$ The harmonic mean is deliberate: with precision $P$ and recall $R$, $$ F_1 = 2\parens{\frac1P + \frac1R}^{-1}, $$ which collapses toward the _smaller_ of the two: a model at $P = 0.99,\ R = 0.01$ scores $F_1 \approx 0.02$, not the $0.50$ an arithmetic mean would award. For example, take a disease screen run on $N = 10{,}000$ patients of whom $50$ actually have the disease ($p = 0.005$). A model flags $80$ patients as positive; of those, $40$ are true cases and $40$ are false alarms, so it misses $10$ real cases: $$ TP = 40, \quad FP = 40, \quad FN = 10, \quad TN = 9{,}910. $$ Its accuracy is $(TP + TN)/N = 9{,}950/10{,}000 = 99.5\%$, which sounds excellent, yet the always-negative constant classifier scores $99.5\%$ too and catches nobody. The error-type metrics show the problem: $$ P = \frac{40}{40 + 40} = 0.50, \qquad R = \frac{40}{40 + 10} = 0.80, \qquad F_1 = \frac{2(0.50)(0.80)}{0.50 + 0.80} \approx 0.615. $$ Recall of $0.80$ says the model catches four of every five true cases; precision of $0.50$ says half its alarms are false. Whether that trade is acceptable depends on the downstream cost, a missed case versus a needless follow-up test, and no single scalar decides it for you. The right metric is dictated by the cost structure of the goal. | Metric | Definition | When it is the right choice | | --- | --- | --- | | Accuracy | $(TP+TN)/N$ | balanced classes, symmetric error costs | | Precision | $TP/(TP+FP)$ | false positives are expensive (spam filter flags real mail) | | Recall | $TP/(TP+FN)$ | false negatives are expensive (missed tumor, fraud) | | $F_1$ | $2PR/(P+R)$ | imbalanced data, both errors matter | | PR-AUC | area under precision–recall curve | imbalanced data, threshold not yet fixed | | ROC-AUC | area under TPR–FPR curve | ranking quality, roughly balanced classes | | Coverage | fraction the model is allowed to answer | a system may _decline_ to predict when unsure | Precision and recall trade off against each other as the decision **threshold** $t$ on the score $\hat y = p(\text{positive} \mid x)$ moves. Sweeping $t$ from $1$ down to $0$ traces a curve; its summary, the area underneath, is threshold-free. $$ % caption: The precision–recall curve, swept by the decision threshold $t$: raising % $t$ trades recall for precision, and PR-AUC summarizes the whole curve. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \draw[->, thick] (0,0) -- (5.4,0) node[right, font=\footnotesize] {recall}; \draw[->, thick] (0,0) -- (0,4.0) node[above, font=\footnotesize] {precision}; \node[font=\footnotesize, anchor=east] at (-0.05,3.5) {1}; \node[font=\footnotesize, anchor=north] at (5.0,-0.05) {1}; \draw[black, dashed] (0,3.5) -- (5.0,3.5); \draw[black, dashed] (5.0,0) -- (5.0,3.5); % PR curve: high precision at low recall, decays as recall rises \draw[acc, very thick] plot[domain=0.1:5.0, samples=60] (\x, {3.5*(1.02 - 0.16*(\x/5.0) - 0.78*(\x/5.0)^4)}); \node[acc, anchor=south west, font=\footnotesize] at (1.4,2.55) {PR curve}; % operating point (label placed in open area below-right, short leader to the dot) \fill[red] (3.0,{3.5*(1.02 - 0.16*0.6 - 0.78*0.6^4)}) circle (2.6pt); \draw[red] (3.0,{3.5*(1.02 - 0.16*0.6 - 0.78*0.6^4)}) -- (3.6,1.7); \node[red, anchor=west, font=\scriptsize] at (3.6,1.55) {operating point}; % threshold arrows (labels placed clear of the arrow segments) \draw[->, black] (1.0,0.4) -- (0.55,1.25); \node[font=\scriptsize, text=black, anchor=west] at (1.05,0.45) {raise $t$}; \draw[->, black] (4.0,0.95) -- (4.5,0.35); \node[font=\scriptsize, text=black, anchor=east] at (3.95,0.95) {lower $t$}; \end{tikzpicture} $$ ## Default baseline models With a metric chosen, build the simplest system that could plausibly work. A strong baseline both establishes the number to beat and tells you whether the problem is even tractable with the data on hand. The sensible first choice is dictated by the input type. | Data type | Default architecture | Default optimizer | | --- | --- | --- | | Fixed-size vectors | feedforward MLP, fully connected | SGD + momentum, or Adam | | Images / grid topology | convolutional network with residual connections | SGD + momentum, or Adam | | Sequences / time series | gated RNN (LSTM/GRU) or a Transformer | Adam | | Text / language | Transformer with pretrained embeddings | Adam with warmup | | Tabular, small $n$ | regularized linear model or gradient-boosted trees | closed-form / coordinate descent | > **Remark (Reasonable defaults).** Goodfellow's standing recommendations: > piecewise-linear units (ReLU and its variants) as the activation; SGD with > momentum and a decaying learning rate, or Adam, as the optimizer; batch > normalization for convolutional nets; and mild regularization: early stopping > almost always, dropout where the model overfits. > **Definition (Strong baseline).** A simple, well-understood model run with sane > defaults, serving as the reference the metric must beat. Its value is diagnostic: > if even a tuned exotic model cannot improve on it, the bottleneck is the _data_ > or the _problem framing_, not the architecture. Resist the urge to start exotic. If a known result exists for a similar task, copy its architecture and optimizer wholesale; reproducing a published baseline before innovating is the fastest path to a trustworthy number. ## Whether to gather more data After the baseline runs, the most consequential decision is whether to collect more data (often expensive) or to change the model. The diagnosis is read directly off two numbers: training error and validation error. This is the [bias–variance decomposition](/deep-learning/foundations/machine-learning-refresher) applied in practice. > **Definition (The gap diagnosis).** Let $E_{\text{train}}$ and $E_{\text{val}}$ > be the training and validation errors, and $E^\star$ the target. **High training > error** ($E_{\text{train}} \gg E^\star$) is _underfitting_: the model lacks > capacity or is poorly optimized. A **large gap** ($E_{\text{val}} \gg > E_{\text{train}}$) is _overfitting_: the model memorizes and needs more data > or stronger regularization. The two failure modes call for opposite remedies, and confusing them wastes the most resources. More data cannot fix underfitting; a bigger model cannot fix overfitting.[^gf-gap] | Symptom | Diagnosis | Remedy | | --- | --- | --- | | $E_{\text{train}}$ high | underfitting (bias) | bigger model, train longer, better optimizer, fewer constraints | | $E_{\text{train}}$ low, gap large | overfitting (variance) | more data, regularize, data augmentation, smaller model | | both acceptable, $E_{\text{val}}$ still short | irreducible / framing | new features, new metric, reconsider the goal | Plotting error against the size of the training set separates the two regimes: high-bias curves converge high and close together; high-variance curves leave a persistent gap that more data would narrow. $$ % caption: Learning-curve diagnosis. Left: high bias, curves converge high and more % data will not help. Right: high variance, a wide gap that more data closes. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % ===== left panel: high bias ===== \begin{scope} \draw[->, thick] (0,0) -- (4.4,0) node[right, font=\scriptsize] {train size}; \draw[->, thick] (0,0) -- (0,3.4) node[above, font=\scriptsize] {error}; % validation: decreasing toward a high plateau \draw[red, very thick] plot[domain=0.4:4.0, samples=40] (\x, {1.7 + 1.3*exp(-1.4*\x)}); % train: increasing toward the same high plateau \draw[acc, very thick] plot[domain=0.4:4.0, samples=40] (\x, {1.7 - 0.9*exp(-1.4*\x)}); \draw[black, dashed] (0,0.7) -- (4.0,0.7) node[right, font=\scriptsize, black] {target}; \node[red, font=\scriptsize, anchor=west] at (2.2,2.0) {val}; \node[acc, font=\scriptsize, anchor=west] at (2.2,1.35) {train}; \node[font=\scriptsize, anchor=south] at (2.0,3.0) {high bias}; \end{scope} % ===== right panel: high variance ===== \begin{scope}[xshift=6.4cm] \draw[->, thick] (0,0) -- (4.4,0) node[right, font=\scriptsize] {train size}; \draw[->, thick] (0,0) -- (0,3.4) node[above, font=\scriptsize] {error}; % validation: high, slowly decreasing \draw[red, very thick] plot[domain=0.4:4.0, samples=40] (\x, {0.9 + 1.7*exp(-0.7*\x)}); % train: very low, slowly increasing \draw[acc, very thick] plot[domain=0.4:4.0, samples=40] (\x, {0.35 + 0.3*exp(-1.0*\x)}); \draw[black, dashed] (0,0.7) -- (4.0,0.7) node[right, font=\scriptsize, black] {target}; \node[red, font=\scriptsize, anchor=west] at (2.3,2.1) {val}; \node[acc, font=\scriptsize, anchor=north west] at (2.6,0.35) {train}; \node[font=\scriptsize, anchor=south] at (2.0,3.0) {high variance}; % gap brace \draw[<->, black] (1.2,0.45) -- (1.2,2.05); \node[font=\scriptsize, text=black, anchor=east] at (1.15,1.25) {gap}; \end{scope} \end{tikzpicture} $$ > **Remark (Estimating the data needed).** Generalization error typically falls as > a power law in the number of examples $m$, so doubling data buys a predictable, > diminishing gain. Plot validation error against $\log m$ on the data you _have_, > fit the trend, and extrapolate to estimate how many examples reach $E^\star$ > before committing to an expensive collection effort. The extrapolation is worth doing arithmetically before spending a data-collection budget. Suppose the error follows $E(m) = a\, m^{-\alpha}$, and you measure $E = 0.20$ at $m = 10{,}000$ and $E = 0.14$ at $m = 40{,}000$. The exponent comes from the ratio, $$ \frac{0.14}{0.20} = \parens{\frac{40{,}000}{10{,}000}}^{-\alpha} \;\Longrightarrow\; \alpha = -\frac{\log(0.70)}{\log 4} \approx 0.257. $$ To hit a target $E^\star = 0.10$ from the $m = 10{,}000$ anchor you need $$ m^\star = 10{,}000 \parens{\frac{0.20}{0.10}}^{1/\alpha} = 10{,}000 \cdot 2^{3.9} \approx 1.5 \times 10^5, $$ so reaching $10\%$ error costs roughly fifteen times the current data. That number is what decides whether to label more examples or to change the model instead, and it is far cheaper to estimate than to discover after a failed collection round. ## The data pipeline More data helps only if the path from raw examples to the network delivers them correctly, and this path is where a large share of real bugs hide. A subtle preprocessing error, normalizing with the wrong statistics, leaking test labels into training, shuffling images but not their labels, corrupts every gradient the model ever sees, yet training loss may still fall because the network fits the corrupted signal anyway. The pipeline deserves the same scrutiny as the model. $$ % caption: A standard training data pipeline. Normalization statistics % $(\mu, \sigma)$ are computed on the training split ONLY and reused for validation % and test; augmentation is applied to training batches only. \begin{tikzpicture}[>=stealth, font=\footnotesize, st/.style={draw, minimum width=20mm, minimum height=9mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[st] (raw) at (0,0) {raw data}; \node[st] (split) at (2.7,0) {\texttt{split}\\train/\texttt{val}/test}; \node[st, draw=acc, text=acc] (norm) at (5.7,0) {\texttt{normalize}\\(mu, sig)}; \node[st] (aug) at (8.6,0) {\texttt{augment}\\(train only)}; \node[st] (batch) at (11.3,0) {\texttt{batch} +\\\texttt{shuffle}}; \node[st, draw=acc, text=acc] (model) at (11.3,-2.2) {\texttt{model}}; \draw[->, acc, thick] (raw) -- (split); \draw[->, acc, thick] (split) -- (norm); \draw[->, acc, thick] (norm) -- (aug); \draw[->, acc, thick] (aug) -- (batch); \draw[->, acc, thick] (batch) -- (model); % stats fit on train, reused downstream \node[st, fill=black!5] (fit) at (5.7,2.0) {\texttt{fit} stats\\on train}; \draw[->, red, thick] (split.north) .. controls (3.4,2.0) .. (fit.west) node[pos=0.5, above, font=\footnotesize] {train \texttt{split}}; \draw[->, red, thick] (fit) -- (norm) node[midway, right, font=\scriptsize] {reuse}; \end{tikzpicture} $$ Two rules govern this pipeline, and both prevent a specific and common failure. Normalization statistics, the per-feature mean $\mu$ and standard deviation $\sigma$ used to standardize inputs to zero mean and unit variance, must be estimated on the **training split alone** and then reused verbatim on validation and test. Fitting them on the full dataset lets information about the test distribution leak backward into training, inflating the reported score above what deployment will deliver. Second, **augmentation is a training-only transformation**: random crops, flips, color jitter, and additive noise expand the effective training set and act as a regularizer, but validation and test must see clean, canonical inputs so the metric measures real generalization rather than luck on a random crop. > **Definition (Data augmentation).** Label-preserving transformations applied to > training inputs to enlarge the effective dataset. For an image classifier, a > horizontal flip, a small rotation, or a random crop yields a new example the model > has not seen while leaving the label unchanged. Augmentation is the cheapest > regularizer available when data is scarce, and it directly addresses the variance > half of the gap diagnosis. The transformation must preserve the label, and that is domain-specific. A horizontal flip is safe for a cat photo but destroys a photo of a printed digit, where it turns a $2$ into a mirror image that is no digit at all, and it flips left and right in a self-driving scene. Choose augmentations that respect the invariances the task actually has, not a generic recipe. ## Scaling laws turn the gap diagnosis quantitative Goodfellow's power-law remark — that generalization error falls predictably with data — was sharpened into a design tool by the **neural scaling laws** literature. Hestness et al. (2017, arXiv) and then Kaplan et al. (2020, arXiv, _Scaling Laws for Neural Language Models_) measured loss across orders of magnitude of data, parameters, and compute and found it follows power laws over a wide range, $$ L(N) \approx \parens{\frac{N_c}{N}}^{\alpha_N}, \qquad L(D) \approx \parens{\frac{D_c}{D}}^{\alpha_D}, $$ where $N$ is parameter count, $D$ dataset size, and the exponents $\alpha$ are small (often $\approx 0.05$–$0.1$ for language). Straight lines on a log–log plot mean the extrapolation from the "estimating the data needed" remark is a fit rather than a heuristic: measure two points, read the slope, predict the third. The refinement that matters for the gather-more-data decision is **compute-optimal allocation**. Hoffmann et al. (2022, _NeurIPS_, the "Chinchilla" paper) showed that, for a fixed compute budget, most large models were badly _over-parameterized and under-trained_: the optimum scales $N$ and $D$ together in roughly equal proportion, so a model half the size trained on twice the data beats it at the same cost. That is the gap diagnosis at industrial scale — the bottleneck was data, not capacity, exactly the reading the train/validation gap prescribes, now made quantitative enough to size a training run before launching it. The practitioner's lesson is unchanged from the loop above: measure, diagnose, then spend the next unit of budget where the diagnosis points. ## Takeaways - **Methodology beats novelty.** A standard algorithm applied with discipline (fix a metric, build an end-to-end baseline, instrument, iterate on a diagnosis) outperforms an exotic algorithm applied carelessly. Every learned model must first clear a common-sense baseline that uses no learning at all. - **Choose the metric to match the goal.** Accuracy collapses under class imbalance; precision, recall, $F_1$, and the PR/ROC curves separate the error types so the threshold can be set by the real cost structure. The $F_1$ harmonic mean collapses toward the smaller of precision and recall, punishing a model that sacrifices one. - **Start with a strong, type-appropriate baseline** (MLP for vectors, CNN for images, gated RNN or Transformer for sequences) using reasonable defaults (ReLU, Adam or SGD+momentum, early stopping). If a tuned exotic model cannot beat it, the bottleneck is the data or the framing, not the architecture. - **Let the train/validation gap pick the next move.** High training error means underfitting (bigger model, better optimization); a large gap means overfitting (more data, regularization). More data never fixes bias. Fit the learning curve's power law to estimate how many examples reach the target before paying to collect them. - **Guard the data pipeline.** Fit normalization statistics on the training split only, apply augmentation to training batches only, and pick label-preserving transforms that match the task's real invariances — a horizontal flip is safe for a cat but destroys a printed digit. This continues in [Hyperparameters and Debugging](/deep-learning/practical/hyperparameters-and-debugging), which picks up the tuning half of the loop: the learning rate as the dominant hyperparameter, random search over grid, an ordered debugging playbook, and monitoring for drift after deployment. [^gf-method]: **Goodfellow**, _Deep Learning_, Ch. 11 — Practical Methodology: the recommended iterate-to-target design loop, and the thesis that disciplined application of a standard method beats careless use of a novel one. [^chollet-workflow]: **Chollet**, _Deep Learning with Python_, §4.5 — The Universal Workflow of Machine Learning: stand up an end-to-end baseline that beats a common-sense reference before tuning anything in the abstract. [^gf-metrics]: **Goodfellow**, _Deep Learning_, §11.1 — Performance Metrics: accuracy's failure under class imbalance and the precision/recall/$F_1$ family that scores the two error types separately. [^gf-gap]: **Goodfellow**, _Deep Learning_, §11.3 / §5.2 — Determining Whether to Gather More Data: read the train/validation gap as bias versus variance, since more data fixes overfitting but never underfitting.