---
title: Hyperparameters & Debugging
module: Practical Deep Learning
moduleNumber: 9
lessonNumber: 2
order: 902
summary: >
  The tuning half of the methodology loop. The learning rate is the one
  hyperparameter that dominates, so we tune it first, on a log scale, coarse to
  fine, and prefer random search to grid when only a few dials matter. Then an
  ordered debugging playbook — overfit one batch, check the loss at
  initialization against ln C, watch the gradient norm, gradient-check against
  centered finite differences — and, after launch, monitoring for train-test skew
  and distribution drift with confidence-based abstention.
topics: [Practical Deep Learning]
sources:
  - book: Goodfellow
    ref: "§11.4 — Selecting Hyperparameters (learning rate, grid vs. random search)"
  - book: Goodfellow
    ref: "§11.5 — Debugging Strategies (fit a tiny dataset, gradient checking)"
  - book: Goodfellow
    ref: "§11.6 — deployment, monitoring, and abstention"
---

This builds on [Practical Methodology](/deep-learning/practical/practical-methodology),
which fixed the goal and metric, stood up an end-to-end baseline, read the
train/validation gap to choose between more data and more capacity, and guarded the
data pipeline. With the baseline running and the gap diagnosed, the next moves are
tuning the hyperparameters, debugging failures, and — once the model ships —
monitoring it in production.

## Hyperparameter selection

Hyperparameters are the settings not chosen by gradient descent: learning rate, layer
widths, regularization strengths, batch size. They can be tuned **manually** (which
demands understanding what each one does) or **automatically**.

Most hyperparameters control the model's _effective capacity_, and the relationship
to generalization error is U-shaped: too little capacity underfits, too much
overfits, and the optimum sits between. The learning rate is the exception that
deserves first priority.[^gf-lr]

> **Theorem (Learning rate dominates).** Among all hyperparameters, the learning
> rate has the largest and least substitutable effect on performance. Too high and
> the loss diverges or oscillates; too low and training stalls in a region of high
> loss. Its own error-versus-value curve is U-shaped with a sharp, narrow basin, so
> tune it first and tune it finely.

| Hyperparameter | Increasing it raises capacity? | Effect when too high | Effect when too low |
| --- | --- | --- | --- |
| Learning rate | non-monotone (U-shaped) | loss diverges / oscillates | training stalls, slow, stuck high |
| Number of hidden units | yes | overfit, more compute | underfit |
| Depth (layers) | yes | overfit, harder to optimize | underfit |
| Weight decay ($L^2$) | no (lowers capacity) | underfit, over-smoothed | overfit |
| Dropout rate | no (lowers capacity) | underfit, noisy training | overfit |
| Batch size | no (not a capacity dial) | wastes memory, gradient noise too low | slow throughput, noisy gradients |
| Convolution kernel size | yes | overfit, more compute | underfit, misses structure |

**Batch size** deserves a note because it is not a capacity dial and is often
misread as one. It sets how many examples contribute to each gradient estimate, so it
trades hardware throughput against gradient noise. A larger batch gives a lower-variance
gradient and uses the accelerator more fully, but the small noise of a smaller batch is
itself a mild regularizer, and very large batches can generalize worse unless the
learning rate is raised to compensate. The common heuristic is the **linear scaling
rule**: when the batch size grows by a factor $k$, scale the learning rate by $k$ as
well, so the expected parameter update per epoch is preserved. Pick the largest batch
that fits in memory, then re-tune the learning rate around it rather than treating the
old rate as fixed.

When tuning is automated, the choice of _search strategy_ matters more than it
looks. The naive choice, **grid search**, scales exponentially: $k$ values on
each of $d$ hyperparameters costs $k^d$ trials. Worse, it wastes those trials.[^gf-random]

> **Theorem (Random beats grid).** When only a few of the $d$ hyperparameters
> meaningfully affect the loss, **random search** finds good values far more
> efficiently than grid search. A grid of $k^d$ points tries only $k$ distinct
> values of any single hyperparameter, however many trials it runs; random search
> tries a distinct value of every hyperparameter on every trial, so it explores the
> _important_ axis at full resolution regardless of the unimportant ones.

With one important axis and one irrelevant axis, a
$3\times 3$ grid probes the important axis at only $3$ values; $9$ random points
probe it at $9$.

$$
% caption: Grid versus random search over one important and one unimportant
% hyperparameter: random search resolves the important axis at far more values.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % ---- grid panel ----
  \begin{scope}
    \draw[black] (0,0) rectangle (3.2,3.2);
    \node[font=\scriptsize, anchor=south] at (1.6,3.25) {grid search};
    \draw[->, thick] (0,-0.15) -- (3.2,-0.15) node[right, font=\scriptsize] {key knob};
    \draw[->, thick] (-0.15,0) -- (-0.15,3.2) node[above, font=\scriptsize] {idle knob};
    \foreach \x in {0.53,1.6,2.67}
      \foreach \y in {0.53,1.6,2.67}
        \fill[acc] (\x,\y) circle (2.4pt);
    % projection onto important axis: only 3 distinct
    \foreach \x in {0.53,1.6,2.67}
      \fill[green] (\x,-0.5) circle (2.0pt);
    \node[green, font=\scriptsize, anchor=north] at (1.6,-0.6) {3 values};
  \end{scope}
  % ---- random panel ----
  \begin{scope}[xshift=5.4cm]
    \draw[black] (0,0) rectangle (3.2,3.2);
    \node[font=\scriptsize, anchor=south] at (1.6,3.25) {random search};
    \draw[->, thick] (0,-0.15) -- (3.2,-0.15) node[right, font=\scriptsize] {key knob};
    \draw[->, thick] (-0.15,0) -- (-0.15,3.2) node[above, font=\scriptsize] {idle knob};
    \def\rx{0.31,1.05,0.74,2.35,1.62,2.88,0.52,2.05,1.31}
    \def\ry{2.71,0.42,1.85,2.55,1.12,2.05,0.71,3.0,2.25}
    \foreach \x/\y in {0.31/2.71,1.05/0.42,0.74/1.85,2.35/2.55,1.62/1.12,2.88/2.05,0.52/0.71,2.05/3.0,1.31/2.25}
      \fill[acc] (\x,\y) circle (2.4pt);
    \foreach \x in {0.31,1.05,0.74,2.35,1.62,2.88,0.52,2.05,1.31}
      \fill[green] (\x,-0.5) circle (2.0pt);
    \node[green, font=\scriptsize, anchor=north] at (1.6,-0.6) {9 values};
  \end{scope}
\end{tikzpicture}
$$

Beyond random search lies **Bayesian optimization**, which models the
validation-error surface as a function of the hyperparameters and chooses each next
trial to balance exploring uncertain regions against exploiting promising ones. It
spends fewer trials but adds its own overhead and tuning; for most problems, random
search over a sensible range is the high-value default.

| Strategy | Cost | Strength | Weakness |
| --- | --- | --- | --- |
| Manual | human time | uses understanding; cheap in trials | needs expertise, hard to reproduce |
| Grid search | $k^d$ trials | exhaustive, simple | exponential; wastes trials on unimportant axes |
| Random search | fixed budget, any size | resolves important axes; trivially parallel | no memory across trials |
| Bayesian optimization | fewer trials, more overhead | sample-efficient | sequential, sensitive to its own priors |

> **Remark (Search on a log scale).** Multiplicative hyperparameters (learning
> rate, weight decay, hidden-unit counts) should be sampled in _log space_ (e.g.
> uniform over $[10^{-5}, 10^{-1}]$ for the learning rate). A linear grid wastes
> nearly all its points in the wrong order of magnitude.

The reason is a matter of counting. A linear grid of $100$ points over
$[10^{-5}, 10^{-1}]$ places $90$ of them in $[10^{-2}, 10^{-1}]$ and only one below
$10^{-3}$, so almost the entire budget probes learning rates too large to be useful,
and the good region near $10^{-3}$ is sampled once. Drawing the exponent uniformly
instead, $\eta = 10^{u}$ with $u \sim \mathrm{Uniform}(-5, -1)$, spends equal effort in
each decade, where the equal-sized improvements are.

### Coarse-to-fine search

The efficient practical approach combines the two ideas: search **coarse-to-fine**.
Run a wide, cheap, low-resolution random search over the full log range first, read
off the decade where the best trials cluster, then run a second, denser random search
zoomed into that narrowed range. Two rounds of ten trials each localize a good
learning rate far more reliably than twenty trials spread thin over five decades.

$$
% caption: Coarse-to-fine learning-rate search on a log axis. Round 1 samples the
% full range $[10^{-5}, 10^{-1}]$ coarsely; the winners cluster near $10^{-3}$;
% round 2 re-samples densely inside the shaded band around that basin.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % log axis from 10^-5 to 10^-1 mapped to x in [0,8]
  \draw[->, thick] (-0.3,0) -- (8.6,0) node[right, font=\scriptsize] {log learn rate};
  \foreach \i/\lab in {0/{1e-5},2/{1e-4},4/{1e-3},6/{1e-2},8/{1e-1}}
    \draw (\i,0.1) -- (\i,-0.1) node[below, font=\scriptsize] {\lab};
  % best band near 10^-3 (x=4), drawn first so dots sit on top
  \fill[green!18] (3.2,-0.6) rectangle (4.8,2.2);
  \node[green, font=\scriptsize, anchor=south] at (4.0,2.2) {best basin};
  % round 1: coarse, one winner lands inside the basin
  \node[acc, font=\scriptsize, anchor=east] at (-0.5,1.3) {round 1};
  \foreach \x in {0.6,1.5,2.4,4.0,5.9,7.4}
    \fill[acc] (\x,1.3) circle (2.4pt);
  % round 2: dense inside the band
  \node[red, font=\scriptsize, anchor=east] at (-0.5,0.7) {round 2};
  \foreach \x in {3.35,3.6,3.85,4.05,4.3,4.55,4.75}
    \fill[red] (\x,0.7) circle (2.2pt);
\end{tikzpicture}
$$

For example, round 1 draws six learning rates spanning
$10^{-5}$ to $10^{-1}$; the runs at $10^{-1}$ and $10^{-2}$ diverge, the run at
$10^{-5}$ barely moves, and the best validation loss lands at $\eta \approx 10^{-3}$.
Round 2 then samples seven rates in $[3\times 10^{-4}, 3\times 10^{-3}]$ and settles on
$\eta = 1.2 \times 10^{-3}$. Thirteen total runs beat the resolution a single $13$-point
sweep of the full range would have given near the basin.

## Debugging strategies

When a model underperforms, the question is whether the _algorithm_ is wrong or the
_implementation_ is buggy, and the two are hard to distinguish, because a buggy
network still trains to _something_. These tests isolate the cause.[^gf-debug]

> **Remark (Fit a tiny dataset).** The single most useful test: train on a handful
> of examples, even one, and confirm the model drives training loss to (near) zero. A
> model that _cannot_ overfit four examples has a bug or a capacity defect, not a
> generalization problem. This separates "the optimizer cannot reduce loss at all"
> from "the model reduces loss but does not generalize."

The order in which you run the debugging tests matters, because each one isolates a
different layer of the stack and the cheap ones rule out whole categories first. A
workable sequence:

1. **Overfit one batch.** Take a single mini-batch of, say, eight examples and train
   on _only_ that batch for a few hundred steps. Training loss must fall to near zero.
   If it does not, the model, loss, or optimizer is broken, and no data-side fix will
   help. This is the fastest end-to-end test that the gradient actually flows.
2. **Sanity-check the loss at initialization.** For a $C$-class classifier with random
   weights and softmax output, the expected cross-entropy before any training is
   $\ln C$: a $10$-class problem should start near $\ln 10 \approx 2.30$. A wildly
   different starting loss means the output layer, the label encoding, or the loss
   reduction is wrong.
3. **Check gradient magnitudes.** Log the global gradient norm
   $\lVert g \rVert = \sqrt{\sum_i g_i^2}$ each step. A norm that is $0$ (or `NaN`) at
   step one points to dead units or a detached graph; a norm that grows without bound
   points to an exploding-gradient regime that needs a lower learning rate or
   clipping. Healthy training shows a norm that is finite, nonzero, and trending down.
4. **Verify the data pipeline visually.** Pull a batch _after_ all preprocessing and
   look at it: display the images with their labels, print a few tokenized sequences
   with their targets. A surprising fraction of "model" bugs are shuffled labels,
   wrong normalization, or an off-by-one in the target that only inspection reveals.
5. **Ablate.** Once the model trains, remove one component at a time, an augmentation,
   a normalization layer, a loss term, and re-measure. The change in the metric
   attributes credit to that component and exposes any part that is silently doing
   nothing or actively hurting.

> **Definition (Gradient checking).** Verify back-propagated gradients against a
> finite-difference estimate. For each parameter $\theta_i$, compare the analytic
> gradient $g_i = \partial \mathcal{L} / \partial \theta_i$ to the centered
> difference, and flag any coordinate whose relative error is large.

The centered difference is far more accurate than the one-sided form, with error
$O(\epsilon^2)$ rather than $O(\epsilon)$:

$$
\hat g_i = \frac{\mathcal{L}(\theta + \epsilon\, e_i) - \mathcal{L}(\theta - \epsilon\, e_i)}{2\epsilon}
\;=\; g_i + O(\epsilon^2),
$$

and the test is the _relative_ error, which is scale-free and the right thing to
threshold (typically $< 10^{-7}$ for double precision):

$$
\text{rel-err} = \frac{\abs{\hat g_i - g_i}}{\max\parens{\abs{\hat g_i},\, \abs{g_i}}} \;<\; 10^{-7}.
$$

```algorithm
caption: $\textsc{GradCheck}(\mathcal{L}, \theta, \epsilon)$ — compare backprop to finite differences
$g \gets \nabla_\theta \mathcal{L}(\theta)$ // analytic gradient from backprop
for each coordinate $i$ do
  $\hat g_i \gets \dfrac{\mathcal{L}(\theta + \epsilon\, e_i) - \mathcal{L}(\theta - \epsilon\, e_i)}{2\epsilon}$ // centered difference
  $r_i \gets \dfrac{\abs{\hat g_i - g_i}}{\max\parens{\abs{\hat g_i},\, \abs{g_i}}}$ // relative error
  if $r_i > 10^{-5}$ then
    report coordinate $i$ as suspect
return $\max_i r_i$
```

Beyond gradient checks, the most informative diagnostics _visualize the model in
action_ rather than reading summary scalars.

| Symptom in the diagnostic | Likely cause | Action |
| --- | --- | --- |
| Loss is `NaN` / `Inf` | exploding gradients, bad learning rate, log of zero | lower learning rate, clip gradients, add $\epsilon$ to logs |
| Training loss flat from step 0 | dead units, broken backprop, learning rate $\approx 0$ | gradient-check; raise learning rate; switch init |
| Activation histograms collapse to 0 | dead ReLU / vanishing signal | better initialization, normalization, leaky ReLU |
| Activation histograms saturate at extremes | saturating nonlinearity, learning rate too high | normalize inputs, lower learning rate |
| Weight histograms drift unbounded | missing or too-weak weight decay | add $L^2$, clip gradients |
| Train loss falls, val loss rises | overfitting | regularize, early-stop, more data |

> **Remark (Visualize, do not summarize).** Inspect the model directly: look at its
> actual outputs on real inputs (a generated sample, the detections drawn on an
> image), histogram the activations and the weights layer by layer to confirm they
> neither vanish nor explode, and watch the gradient norm over training. A single
> scalar loss hides the bug; the histogram shows it.

## Deployment and monitoring

A model that clears its offline target is not finished. Two gaps open between the
validation score and the live one, and both need instrumentation rather than a single
pre-launch number.

The first gap is **train-test skew**: the preprocessing applied at serving time must
match training exactly, including the normalization statistics $(\mu, \sigma)$ fit on
the training split. A serving path that recomputes statistics from live traffic, or
tokenizes text with a different vocabulary, degrades the model in ways the offline
metric never saw. Freeze the transform and ship it with the weights.

The second gap is **distribution shift**: the input distribution at serving time
drifts away from the training distribution as the world changes, and a model that was
accurate at launch decays silently because the loss is not observable without labels.
The defense is monitoring proxies that _are_ observable: track the distribution of the
model's inputs and its output scores over time, alert when either moves, and sample a
small stream of live predictions for human labeling to estimate the live error
directly.

> **Remark (Confidence and abstention).** A calibrated model reports how sure it is,
> and a deployed system can use that. Route low-confidence inputs to a human or to a
> fallback rule instead of forcing a guess. The **coverage** metric, the fraction of
> inputs the model is allowed to answer, then trades against accuracy on the answered
> subset: a model that abstains on its hardest $10\%$ can be far more accurate on the
> remaining $90\%$, which is the right operating point when a wrong automated answer is
> costly.

Monitoring closes the methodology loop at production scale: the live metric becomes a
new measurement, distribution shift becomes a new diagnosis, and fresh labeled data or
a retrained model becomes the next single change. The discipline that stood up the
baseline is the same discipline that keeps it working.

## Schedules, warm restarts, and one-shot LR finding

The standard references treat the learning rate as one number to tune. Modern practice tunes a
_schedule_ — how the rate changes over training — and the schedule often matters as
much as the peak value.

**Warmup then decay** is now standard for training Transformers. The original
Transformer (Vaswani et al., 2017, _NeurIPS_) ramps the rate linearly for the first few
thousand steps, then decays it as $1/\sqrt{t}$; the warmup keeps early, high-variance
updates from destabilizing a freshly initialized model before the statistics settle.
**Cosine annealing** (Loshchilov & Hutter, 2017, _ICLR_, _SGDR_) instead decays the rate
smoothly along a half-cosine from the peak to near zero, optionally with **warm
restarts** that periodically jump it back up to escape sharp minima and explore a new
basin. Each restart is a fresh short anneal, and the ensemble of snapshots taken at the
restart points is itself a cheap way to average several models.

**The LR range test** (Smith, 2017, _WACV_, "cyclical learning rates") turns the
coarse-to-fine search from the previous lesson into a single run. Start training with an
absurdly small rate and increase it geometrically every batch while logging the loss;
the loss falls, bottoms out, then diverges. The largest rate before divergence is the
top of the usable band, and a good peak sits a notch below it. One forward-backward
sweep across a few hundred batches replaces a whole grid of full training runs — the
same "spend the budget where the signal is" discipline, applied to the one hyperparameter
that dominates. **One-cycle** training then rides that band up and back down once over the
whole run, which often trains faster than any fixed rate.

This matches the methodology loop: the learning rate is worth the
most attention, so it gets the most machinery (a schedule, a restart policy, and a
cheap one-shot way to find its ceiling) while the other hyperparameters stay on sane
defaults until the gap diagnosis says otherwise.

## Takeaways

- **Tune the learning rate first.** Its effect is the largest and least
  substitutable, with a sharp U-shaped basin: too high diverges, too low stalls.
  Sample it on a log scale, search coarse-to-fine, and prefer a schedule (warmup,
  cosine decay) over a single fixed value.
- **Prefer random search to grid search** when only a few of many hyperparameters
  matter: a $k^d$ grid resolves each axis at only $k$ values however many trials it
  runs, while random search resolves the important axes at full resolution and
  parallelizes trivially. Bayesian optimization spends fewer trials at the cost of
  its own overhead.
- **Batch size is not a capacity dial;** it trades hardware throughput against
  gradient noise. Pick the largest that fits memory, then re-tune the learning rate
  around it (the linear scaling rule: grow the rate with the batch).
- **Debug in a fixed order,** cheap tests first: overfit one batch to prove the
  gradient flows, check the initial loss against $\ln C$, watch the global gradient
  norm for zeros/`NaN`s/blowups, inspect a preprocessed batch by eye, then ablate one
  component at a time.
- **Gradient-check with the centered difference** ($O(\epsilon^2)$ error) and
  threshold the _relative_ error, and prefer visualizing activation and weight
  histograms over trusting a single scalar loss — the histogram shows the bug the
  scalar hides.
- **Monitor after launch.** Freeze the serving transform to prevent train-test skew,
  watch input and output distributions for drift, sample live predictions for
  labeling, and use confidence-based abstention to trade coverage for accuracy where
  a wrong automated answer is costly.

[^gf-lr]: **Goodfellow**, _Deep Learning_, §11.4.1 — The learning rate is the single most important hyperparameter, with a U-shaped, sharply-peaked error curve; tune it first.
[^gf-random]: **Goodfellow**, _Deep Learning_, §11.4.3–4 — Grid and Random Search: random search resolves the few important axes at full resolution where grid search wastes its exponential budget.
[^gf-debug]: **Goodfellow**, _Deep Learning_, §11.5 — Debugging Strategies: fit a tiny dataset, gradient-check against centered finite differences, and visualize activations/weights instead of trusting a scalar loss.