Hyperparameters & Debugging

This builds on Practical Methodology, which fixed the goal and metric, stood up an end-to-end baseline, read the train/validation gap to choose between more data and more capacity, and guarded the data pipeline. With the baseline running and the gap diagnosed, the next moves are tuning the hyperparameters, debugging failures, and — once the model ships — monitoring it in production.

Hyperparameter selection

Hyperparameters are the settings not chosen by gradient descent: learning rate, layer widths, regularization strengths, batch size. They can be tuned manually (which demands understanding what each one does) or automatically.

Most hyperparameters control the model's effective capacity, and the relationship to generalization error is U-shaped: too little capacity underfits, too much overfits, and the optimum sits between. The learning rate is the exception that deserves first priority.¹

Hyperparameter	Increasing it raises capacity?	Effect when too high	Effect when too low
Learning rate	non-monotone (U-shaped)	loss diverges / oscillates	training stalls, slow, stuck high
Number of hidden units	yes	overfit, more compute	underfit
Depth (layers)	yes	overfit, harder to optimize	underfit
Weight decay ( $L^{2}$ )	no (lowers capacity)	underfit, over-smoothed	overfit
Dropout rate	no (lowers capacity)	underfit, noisy training	overfit
Batch size	no (not a capacity dial)	wastes memory, gradient noise too low	slow throughput, noisy gradients
Convolution kernel size	yes	overfit, more compute	underfit, misses structure

Batch size deserves a note because it is not a capacity dial and is often misread as one. It sets how many examples contribute to each gradient estimate, so it trades hardware throughput against gradient noise. A larger batch gives a lower-variance gradient and uses the accelerator more fully, but the small noise of a smaller batch is itself a mild regularizer, and very large batches can generalize worse unless the learning rate is raised to compensate. The common heuristic is the linear scaling rule: when the batch size grows by a factor $k$ , scale the learning rate by $k$ as well, so the expected parameter update per epoch is preserved. Pick the largest batch that fits in memory, then re-tune the learning rate around it rather than treating the old rate as fixed.

When tuning is automated, the choice of search strategy matters more than it looks. The naive choice, grid search, scales exponentially: $k$ values on each of $d$ hyperparameters costs $k^{d}$ trials. Worse, it wastes those trials.²

With one important axis and one irrelevant axis, a $3 \times 3$ grid probes the important axis at only $3$ values; $9$ random points probe it at $9$ .

Grid versus random search over one important and one unimportant hyperparameter: random search resolves the important axis at far more values.

Beyond random search lies Bayesian optimization, which models the validation-error surface as a function of the hyperparameters and chooses each next trial to balance exploring uncertain regions against exploiting promising ones. It spends fewer trials but adds its own overhead and tuning; for most problems, random search over a sensible range is the high-value default.

Strategy	Cost	Strength	Weakness
Manual	human time	uses understanding; cheap in trials	needs expertise, hard to reproduce
Grid search	$k^{d}$ trials	exhaustive, simple	exponential; wastes trials on unimportant axes
Random search	fixed budget, any size	resolves important axes; trivially parallel	no memory across trials
Bayesian optimization	fewer trials, more overhead	sample-efficient	sequential, sensitive to its own priors

The reason is a matter of counting. A linear grid of $100$ points over $[1 0^{- 5}, 1 0^{- 1}]$ places $90$ of them in $[1 0^{- 2}, 1 0^{- 1}]$ and only one below $1 0^{- 3}$ , so almost the entire budget probes learning rates too large to be useful, and the good region near $1 0^{- 3}$ is sampled once. Drawing the exponent uniformly instead, $η = 1 0^{u}$ with $u \sim Uniform (- 5, - 1)$ , spends equal effort in each decade, where the equal-sized improvements are.

Coarse-to-fine search

The efficient practical approach combines the two ideas: search coarse-to-fine. Run a wide, cheap, low-resolution random search over the full log range first, read off the decade where the best trials cluster, then run a second, denser random search zoomed into that narrowed range. Two rounds of ten trials each localize a good learning rate far more reliably than twenty trials spread thin over five decades.

Coarse-to-fine learning-rate search on a log axis. Round 1 samples the full range

[1 0^{- 5}, 1 0^{- 1}]

coarsely; the winners cluster near

1 0^{- 3}

; round 2 re-samples densely inside the shaded band around that basin.

For example, round 1 draws six learning rates spanning $1 0^{- 5}$ to $1 0^{- 1}$ ; the runs at $1 0^{- 1}$ and $1 0^{- 2}$ diverge, the run at $1 0^{- 5}$ barely moves, and the best validation loss lands at $η \approx 1 0^{- 3}$ . Round 2 then samples seven rates in $[3 \times 1 0^{- 4}, 3 \times 1 0^{- 3}]$ and settles on $η = 1.2 \times 1 0^{- 3}$ . Thirteen total runs beat the resolution a single $13$ -point sweep of the full range would have given near the basin.

Debugging strategies

When a model underperforms, the question is whether the algorithm is wrong or the implementation is buggy, and the two are hard to distinguish, because a buggy network still trains to something. These tests isolate the cause.³

The order in which you run the debugging tests matters, because each one isolates a different layer of the stack and the cheap ones rule out whole categories first. A workable sequence:

Overfit one batch. Take a single mini-batch of, say, eight examples and train on only that batch for a few hundred steps. Training loss must fall to near zero. If it does not, the model, loss, or optimizer is broken, and no data-side fix will help. This is the fastest end-to-end test that the gradient actually flows.
Sanity-check the loss at initialization. For a $C$ -class classifier with random weights and softmax output, the expected cross-entropy before any training is $ln C$ : a $10$ -class problem should start near $ln 10 \approx 2.30$ . A wildly different starting loss means the output layer, the label encoding, or the loss reduction is wrong.
Check gradient magnitudes. Log the global gradient norm $∥ g ∥ = \sum_{i} g_{i}^{2}$ each step. A norm that is $0$ (or NaN) at step one points to dead units or a detached graph; a norm that grows without bound points to an exploding-gradient regime that needs a lower learning rate or clipping. Healthy training shows a norm that is finite, nonzero, and trending down.
Verify the data pipeline visually. Pull a batch after all preprocessing and look at it: display the images with their labels, print a few tokenized sequences with their targets. A surprising fraction of model bugs are shuffled labels, wrong normalization, or an off-by-one in the target that only inspection reveals.
Ablate. Once the model trains, remove one component at a time, an augmentation, a normalization layer, a loss term, and re-measure. The change in the metric attributes credit to that component and exposes any part that is silently doing nothing or actively hurting.

The centered difference is far more accurate than the one-sided form, with error $O (ϵ^{2})$ rather than $O (ϵ)$ :

\overset{g}{^}_{i} = \frac{L ( θ + ϵ e _{i} ) - L ( θ - ϵ e _{i} )}{2 ϵ} = g_{i} + O (ϵ^{2}),

and the test is the relative error, which is scale-free and the right thing to threshold (typically $< 1 0^{- 7}$ for double precision):

rel-err = \frac{∣ g ^ _{i} - g _{i} ∣}{max ( ∣ g ^ _{i} ∣ , ∣ g _{i} ∣ )} < 1 0^{- 7} .

Algorithm:

\textsc{GradCheck}(\mathcal{L}, \theta, \epsilon)

— compare backprop to finite differences

1
$g \gets \nabla_\theta \mathcal{L}(\theta)$
analytic gradient from backprop
2
for each coordinate $i$ do
3
$\hat g_i \gets \dfrac{\mathcal{L}(\theta + \epsilon\, e_i) - \mathcal{L}(\theta - \epsilon\, e_i)}{2\epsilon}$
centered difference
4
$r_i \gets \dfrac{\abs{\hat g_i - g_i}}{\max\parens{\abs{\hat g_i},\, \abs{g_i}}}$
relative error
5
if $r_i > 10^{-5}$ then
6
report coordinate $i$ as suspect
7
return $\max_i r_i$

Beyond gradient checks, the most informative diagnostics visualize the model in action rather than reading summary scalars.

Symptom in the diagnostic	Likely cause	Action
Loss is `NaN` / `Inf`	exploding gradients, bad learning rate, log of zero	lower learning rate, clip gradients, add $ϵ$ to logs
Training loss flat from step 0	dead units, broken backprop, learning rate $\approx 0$	gradient-check; raise learning rate; switch init
Activation histograms collapse to 0	dead ReLU / vanishing signal	better initialization, normalization, leaky ReLU
Activation histograms saturate at extremes	saturating nonlinearity, learning rate too high	normalize inputs, lower learning rate
Weight histograms drift unbounded	missing or too-weak weight decay	add $L^{2}$ , clip gradients
Train loss falls, val loss rises	overfitting	regularize, early-stop, more data

Deployment and monitoring

A model that clears its offline target is not finished. Two gaps open between the validation score and the live one, and both need instrumentation rather than a single pre-launch number.

The first gap is train-test skew: the preprocessing applied at serving time must match training exactly, including the normalization statistics $(μ, σ)$ fit on the training split. A serving path that recomputes statistics from live traffic, or tokenizes text with a different vocabulary, degrades the model in ways the offline metric never saw. Freeze the transform and ship it with the weights.

The second gap is distribution shift: the input distribution at serving time drifts away from the training distribution as the world changes, and a model that was accurate at launch decays silently because the loss is not observable without labels. The defense is monitoring proxies that are observable: track the distribution of the model's inputs and its output scores over time, alert when either moves, and sample a small stream of live predictions for human labeling to estimate the live error directly.

Monitoring closes the methodology loop at production scale: the live metric becomes a new measurement, distribution shift becomes a new diagnosis, and fresh labeled data or a retrained model becomes the next single change. The discipline that stood up the baseline is the same discipline that keeps it working.

Schedules, warm restarts, and one-shot LR finding

The standard references treat the learning rate as one number to tune. Modern practice tunes a schedule — how the rate changes over training — and the schedule often matters as much as the peak value.

Warmup then decay is now standard for training Transformers. The original Transformer (Vaswani et al., 2017, NeurIPS) ramps the rate linearly for the first few thousand steps, then decays it as $1/ t$ ; the warmup keeps early, high-variance updates from destabilizing a freshly initialized model before the statistics settle. Cosine annealing (Loshchilov & Hutter, 2017, ICLR, SGDR) instead decays the rate smoothly along a half-cosine from the peak to near zero, optionally with warm restarts that periodically jump it back up to escape sharp minima and explore a new basin. Each restart is a fresh short anneal, and the ensemble of snapshots taken at the restart points is itself a cheap way to average several models.

The LR range test (Smith, 2017, WACV, cyclical learning rates) turns the coarse-to-fine search from the previous lesson into a single run. Start training with an absurdly small rate and increase it geometrically every batch while logging the loss; the loss falls, bottoms out, then diverges. The largest rate before divergence is the top of the usable band, and a good peak sits a notch below it. One forward-backward sweep across a few hundred batches replaces a whole grid of full training runs — the same spend the budget where the signal is discipline, applied to the one hyperparameter that dominates. One-cycle training then rides that band up and back down once over the whole run, which often trains faster than any fixed rate.

This matches the methodology loop: the learning rate is worth the most attention, so it gets the most machinery (a schedule, a restart policy, and a cheap one-shot way to find its ceiling) while the other hyperparameters stay on sane defaults until the gap diagnosis says otherwise.

Takeaways

Tune the learning rate first. Its effect is the largest and least substitutable, with a sharp U-shaped basin: too high diverges, too low stalls. Sample it on a log scale, search coarse-to-fine, and prefer a schedule (warmup, cosine decay) over a single fixed value.
Prefer random search to grid search when only a few of many hyperparameters matter: a $k^{d}$ grid resolves each axis at only $k$ values however many trials it runs, while random search resolves the important axes at full resolution and parallelizes trivially. Bayesian optimization spends fewer trials at the cost of its own overhead.
Batch size is not a capacity dial; it trades hardware throughput against gradient noise. Pick the largest that fits memory, then re-tune the learning rate around it (the linear scaling rule: grow the rate with the batch).
Debug in a fixed order, cheap tests first: overfit one batch to prove the gradient flows, check the initial loss against $ln C$ , watch the global gradient norm for zeros/NaNs/blowups, inspect a preprocessed batch by eye, then ablate one component at a time.
Gradient-check with the centered difference ( $O (ϵ^{2})$ error) and threshold the relative error, and prefer visualizing activation and weight histograms over trusting a single scalar loss — the histogram shows the bug the scalar hides.
Monitor after launch. Freeze the serving transform to prevent train-test skew, watch input and output distributions for drift, sample live predictions for labeling, and use confidence-based abstention to trade coverage for accuracy where a wrong automated answer is costly.

Goodfellow, Deep Learning, §11.4.1 — The learning rate is the single most important hyperparameter, with a U-shaped, sharply-peaked error curve; tune it first. ↩
Goodfellow, Deep Learning, §11.4.3–4 — Grid and Random Search: random search resolves the few important axes at full resolution where grid search wastes its exponential budget. ↩
Goodfellow, Deep Learning, §11.5 — Debugging Strategies: fit a tiny dataset, gradient-check against centered finite differences, and visualize activations/weights instead of trusting a scalar loss. ↩