---
title: "Model Compression and Distillation"
module: Practical Deep Learning
moduleNumber: 9
lessonNumber: 6
order: 906
summary: >
  A trained network and a deployable one are rarely the same object. This lesson
  is the toolkit for closing that gap: knowledge distillation transfers a large
  teacher's soft, information-rich logits into a small student; pruning deletes
  the weights that contribute least; quantization swaps 32-bit floats for
  8- or 4-bit integers; and low-rank factorization replaces a fat matrix with two
  thin ones. We derive each method, show what it costs in accuracy, and lay out
  which combinations win on which hardware.
topics: [Practical Deep Learning]
sources:
  - book: Stevens
    ref: "Deploying models — size, latency, and quantization"
  - book: Goodfellow
    ref: "Ch. 12 — practical considerations for deployment"
---

A model that scores well on a held-out set has met only the first requirement. The
second is deployment, where the binding constraints are memory, latency, and
energy, not validation loss. A network with a billion parameters in `fp32`
occupies four gigabytes before a single activation is allocated, and that footprint
decides whether it runs on a phone, fits a GPU's memory, or meets a millisecond
service-level target. **Model compression** is the set of techniques that shrink a
trained network while giving back as little accuracy as possible. This lesson treats
the four that matter in practice and the order in which to combine them. It assumes
the [practical methodology](/deep-learning/practical/practical-methodology) loop and
the regularization view from the [regularization overview](/deep-learning/regularization/regularization-overview);
the largest payoffs land on the [large language models](/deep-learning/large-models-and-agents/large-language-models)
where the `fp32` checkpoint will not even load.

## Why compress: the accuracy–cost frontier

Three resources bind at inference, and they scale differently. Memory scales with
the parameter count $N$ times bytes-per-weight $b$. Compute scales with the
multiply-accumulate count, roughly $2N$ FLOPs per token for a dense layer.
Energy is dominated not by arithmetic but by data movement: reading a weight from
DRAM costs orders of magnitude more than the multiply it feeds.[^stevens-deploy]

> **Definition (Model compression).** Any transformation of a trained model
> $f_\theta$ into a smaller or cheaper model $f_{\theta'}$ that approximately
> preserves its input–output behavior, $f_{\theta'} \approx f_\theta$, while
> reducing at least one of: parameter memory, inference FLOPs, or activation
> memory.

The methods do not move a model _along_ a fixed curve; each defines its own
accuracy-versus-cost frontier, and the practitioner picks the point. The goal is the
Pareto frontier: for a target accuracy, the cheapest model; for a cost budget, the
most accurate.

Compression works at all because a trained network is not a tight fit to its task. It
carries far more capacity than the task's information content requires, and that slack
is what every method removes.[^gf-generalize] Removing it trades a small increase in
bias for a large reduction in the effective parameter count; the frontier collects
the points where that trade is favorable. A method that pushed past the frontier
would raise bias faster than it cut cost, which is the over-compressed regime marked in
red below.

$$
% caption: The accuracy--cost frontier. Each method pushes the achievable curve
% down and left; the dashed line is the dense $\textsf{fp32}$ baseline.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, thick] (0,0) -- (6.6,0) node[right, font=\footnotesize] {cost: memory, latency};
  \draw[->, thick] (0,0) -- (0,4.2) node[above, font=\footnotesize] {accuracy};
  % baseline accuracy (dense fp32)
  \draw[black, dashed] (0,3.5) -- (6.2,3.5);
  \node[black, anchor=south east, font=\scriptsize] at (6.2,3.5) {dense fp32};
  % frontier curve: accuracy falls as cost falls, knee in the middle
  \draw[acc, very thick] plot[domain=0.5:6.0, samples=60]
    (\x, {3.5 - 2.6*exp(-0.95*\x)});
  \node[acc, anchor=north west, font=\footnotesize] at (2.2,2.55) {\texttt{achievable frontier}};
  % a few operating points
  \fill[green] (5.6,{3.5 - 2.6*exp(-0.95*5.6)}) circle (2.6pt);
  \node[green, anchor=west, font=\footnotesize] at (4.4,2.55) {\texttt{good trade}};
  \fill[red] (0.85,{3.5 - 2.6*exp(-0.95*0.85)}) circle (2.6pt);
  \draw[red] (0.85,{3.5 - 2.6*exp(-0.95*0.85)}) -- (1.7,1.05);
  \node[red, anchor=west, font=\footnotesize] at (1.7,0.92) {\texttt{over-compressed}};
\end{tikzpicture}
$$

| Resource | Scales with | Dominant cost | Compression lever |
| --- | --- | --- | --- |
| Parameter memory | $N \cdot b$ bytes | DRAM capacity | quantization (cut $b$), pruning (cut $N$) |
| Compute | $\approx 2N$ FLOPs / token | matrix multiply | distillation, structured pruning, low-rank |
| Activation memory | batch $\cdot$ width $\cdot$ depth | on-chip SRAM | quantization, smaller student |
| Energy | weight reads $\gg$ FLOPs | DRAM bandwidth | quantization, pruning (sparse skips) |

> **Remark (Latency is not FLOPs).** Cutting FLOPs by half rarely halves latency.
> Unstructured sparsity removes multiplies the hardware still schedules; an `int8`
> kernel speeds up only where the chip has integer tensor units. Always measure
> wall-clock latency on the target device, never a FLOP count.

## Knowledge distillation

The cheapest way to get a small, accurate model is often not to train a small model
from scratch but to train it to imitate a large one. The large model is the
**teacher**; the small one is the **student**. The student learns from the teacher's
full output distribution, not just the hard label.[^gf-ensemble]

> **Definition (Knowledge distillation).** Train a student $f_s$ to match the
> _softened output distribution_ of a fixed teacher $f_t$, rather than (or in
> addition to) the one-hot ground-truth label. The student's supervision is the
> teacher's probability vector, which carries more information per example than a
> single class index.

### Soft targets and dark knowledge

A network's final logits $z \in \mathbb{R}^K$ become a distribution through the
softmax. Introduce a **temperature** $T \ge 1$ that flattens it:

$$
p_i(z; T) \;=\; \frac{\exp\!\parens{z_i / T}}{\sum_{j=1}^{K} \exp\!\parens{z_j / T}}.
$$

At $T = 1$ this is the ordinary softmax. As $T$ grows the distribution spreads
toward uniform, amplifying the small probabilities the teacher assigns to the
_wrong_ classes. Those small probabilities are the signal.

> **Definition (Dark knowledge).** The information in a teacher's relative
> probabilities over incorrect classes — that a "7" is more like a "1" than a "8",
> that a husky is closer to a wolf than to a car. A one-hot label discards this
> entirely; the softened teacher distribution preserves it, giving the student a
> richer target than the label alone.

Temperature is what exposes dark knowledge. A confident teacher puts $0.99$ on the
true class and $10^{-6}$ on the rest, so at $T = 1$ the soft target is
indistinguishable from one-hot. Raising $T$ rescales the logit gaps and lifts those
tiny probabilities into a usable gradient.

$$
% caption: Temperature flattens the teacher softmax, lifting wrong-class
% probabilities (dark knowledge) into a learnable signal for the student.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % ---- panel: T = 1 (peaked) ----
  \begin{scope}
    \draw[->, thick] (-0.2,0) -- (3.4,0) node[right, font=\scriptsize] {class};
    \draw[->, thick] (0,0) -- (0,3.0) node[above, font=\scriptsize] {prob};
    \foreach \x/\h in {0.4/0.06,1.0/0.06,1.6/2.6,2.2/0.1,2.8/0.06}
      \draw[acc, thick, fill=acc!15] (\x,0) rectangle ++(0.34,\h);
    \node[anchor=north, font=\scriptsize] at (1.6,-0.2) {$T = 1$};
    \node[acc, anchor=west, font=\footnotesize] at (0.9,2.7) {\texttt{peaked}};
  \end{scope}
  % ---- panel: T = 4 (flattened) ----
  \begin{scope}[xshift=5.0cm]
    \draw[->, thick] (-0.2,0) -- (3.4,0) node[right, font=\scriptsize] {class};
    \draw[->, thick] (0,0) -- (0,3.0) node[above, font=\scriptsize] {prob};
    \foreach \x/\h in {0.4/0.55,1.0/0.7,1.6/1.7,2.2/0.95,2.8/0.5}
      \draw[green, thick, fill=green!15] (\x,0) rectangle ++(0.34,\h);
    \node[anchor=north, font=\scriptsize] at (1.6,-0.2) {$T = 4$};
    \node[green, anchor=west, font=\footnotesize] at (0.55,2.7) {\texttt{dark knowledge}};
  \end{scope}
\end{tikzpicture}
$$

### The distillation loss

The student is trained on a convex combination of two terms: a **soft** term that
matches the teacher at temperature $T$, and a **hard** term that matches the true
label at $T = 1$. With teacher logits $z_t$, student logits $z_s$, true label $y$,
and mixing weight $\alpha \in [0,1]$,

$$
\mathcal{L}_{\text{KD}}
\;=\;
\alpha\, T^2 \cdot \mathrm{KL}\!\parens{\,p(z_t; T)\;\big\|\;p(z_s; T)\,}
\;+\;
(1 - \alpha)\cdot \mathrm{CE}\!\parens{\,y,\; p(z_s; 1)\,}.
$$

Each term has a role.

- The **KL term** $\mathrm{KL}\parens{p(z_t;T) \,\|\, p(z_s;T)}$ measures how far
  the student's softened distribution sits from the teacher's. Minimizing it over
  $z_s$ (the teacher is fixed) pulls the student's _whole_ logit vector toward the
  teacher's, dark knowledge included.

- The **$T^2$ factor** rescales the soft loss. The gradient of the softened
  cross-entropy with respect to a logit carries a $1/T$ from the softmax argument
  and another $1/T$ from the target probabilities, so the soft gradient shrinks like
  $1/T^2$. Multiplying by $T^2$ restores it to the same scale as the hard gradient,
  keeping $\alpha$ meaningful as $T$ changes.

- The **CE term** anchors the student to the ground truth, so it does not inherit
  the teacher's mistakes wholesale. Setting $\alpha = 1$ trains purely on the
  teacher; $\alpha = 0$ ignores it.

> **Theorem (Soft-target gradient $\approx$ logit matching).** In the limit of
> large $T$ and zero-mean logits, minimizing the softened cross-entropy is
> equivalent to matching student and teacher logits directly. The gradient of the
> soft loss with respect to student logit $z_{s,i}$ is
> $\frac{1}{T}\parens{p_i(z_s;T) - p_i(z_t;T)}$, and a first-order expansion of the
> softmax for large $T$ gives
> $\frac{\partial \mathcal{L}_{\text{soft}}}{\partial z_{s,i}}
> \approx \frac{1}{K T^2}\parens{z_{s,i} - z_{t,i}}$.

> **Proof.** For a single softened cross-entropy term
> $\mathcal{L} = -\sum_i p_i(z_t;T)\log p_i(z_s;T)$, the standard softmax gradient
> is $\partial \mathcal{L} / \partial z_{s,i} = \tfrac{1}{T}\parens{p_i(z_s;T) -
> p_i(z_t;T)}$. Expand each softmax for large $T$ using $\exp(z/T) \approx 1 + z/T$:
> $p_i(z;T) \approx \frac{1 + z_i/T}{K + \sum_j z_j/T}$. With zero-mean logits
> $\sum_j z_j = 0$, this is $\approx \tfrac{1}{K}\parens{1 + z_i/T}$. Substituting,
> $p_i(z_s;T) - p_i(z_t;T) \approx \tfrac{1}{KT}\parens{z_{s,i} - z_{t,i}}$, so the
> gradient is $\tfrac{1}{KT^2}\parens{z_{s,i} - z_{t,i}}$. The $T^2$ multiplier in
> $\mathcal{L}_{\text{KD}}$ exactly cancels this, leaving a logit-matching
> objective. $\qed$

The picture is a teacher feeding two channels into the student loss: a softened
distribution and the hard label.

$$
% caption: Distillation: a frozen teacher's softened logits and the true label both
% supervise the student through the combined KD loss.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=22mm, minimum height=10mm, align=center},
  sm/.style={draw, minimum width=20mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \node[box, draw=black] (x) at (0,0) {\texttt{input} $x$};
  \node[box, draw=acc, text=acc, thick] (teach) at (3.2,1.5) {\texttt{teacher}\\\texttt{(frozen)}};
  \node[box, draw=green, text=green, thick] (stud) at (3.2,-1.5) {\texttt{student}\\\texttt{(train)}};
  \node[sm] (soft) at (6.8,1.5) {\texttt{soft logits}\\\texttt{at} $T$};
  \node[sm] (hard) at (6.8,-2.6) {\texttt{true label} $y$};
  \node[box, draw=black] (loss) at (10.2,-0.4) {\texttt{KD loss}};
  \draw[->, acc, thick] (x) |- (teach);
  \draw[->, green, thick] (x) |- (stud);
  \draw[->, acc, thick] (teach) -- (soft);
  \draw[->, acc, thick] (soft) -- (loss) node[midway, above, font=\scriptsize] {KL};
  \draw[->, green, thick] (stud) -- (10.2,-1.5) -- (loss) node[pos=0.25, below, font=\footnotesize] {\texttt{match}};
  \draw[->, black] (hard) -- (loss) node[midway, below, font=\scriptsize] {CE};
\end{tikzpicture}
$$

```algorithm
caption: $\textsc{Distill}(f_t, f_s, \mathcal{D}, T, \alpha)$ — train a student to match a frozen teacher
freeze teacher $f_t$ // teacher weights never updated
for each minibatch $(x, y) \in \mathcal{D}$ do
  $z_t \gets f_t(x)$ // teacher logits (no gradient)
  $z_s \gets f_s(x)$ // student logits
  $L_{\text{soft}} \gets T^2 \cdot \mathrm{KL}\parens{p(z_t; T) \,\|\, p(z_s; T)}$ // dark knowledge
  $L_{\text{hard}} \gets \mathrm{CE}\parens{y,\, p(z_s; 1)}$ // ground truth
  $L \gets \alpha\, L_{\text{soft}} + (1 - \alpha)\, L_{\text{hard}}$
  $\theta_s \gets \theta_s - \eta\, \nabla_{\theta_s} L$ // student step only
return $f_s$
```

> **Remark (Why distillation predates deep nets).** The idea is older than the
> temperature trick: compressing an expensive ensemble into one fast model by
> training the small model on the ensemble's predictions was already a known move.
> An ensemble averages many models to lower variance;[^gf-ensemble] distillation then
> folds that averaged behavior back into a single fast network. The temperature-softened
> loss and the "dark knowledge" framing add the explanation for _why_ the soft targets
> help: they expose the teacher's relative confidence over wrong classes.

> **Remark (Beyond logits).** Modern variants match more than the final
> distribution: feature distillation aligns intermediate activations, and
> attention-transfer matches attention maps. The student gets a denser signal,
> at the cost of architecture-specific plumbing between teacher and student layers.

## Pruning

Distillation builds a small model; pruning carves one out of a large model by
deleting weights. The premise is that trained networks are heavily over-parameterized
and most weights contribute little.

> **Definition (Pruning).** Setting a subset of a trained network's weights to zero
> (and never restoring them), producing a sparse weight tensor. The pruned model has
> fewer effective parameters; whether it also runs faster depends on whether the
> sparsity is _structured_.

### Magnitude pruning

The simplest and strongest baseline ranks weights by absolute value and zeros the
smallest. A weight near zero contributes little to any activation, so removing it
perturbs the function least. That such weights exist in bulk is the same
over-parameterization that lets large networks fit and still generalize.[^gf-capacity]

$$
\text{prune } w_j \;\iff\; \abs{w_j} \;<\; \tau,
\qquad
\tau \;=\; \text{the } s\text{-th percentile of } \braces{\abs{w_j}},
$$

where $s$ is the target sparsity. Pruning in one shot to high sparsity destroys
accuracy; to address this, alternate pruning with retraining so the surviving weights
absorb the slack.

```algorithm
caption: $\textsc{IterativePrune}(f_\theta, s, k)$ — reach sparsity $s$ over $k$ prune--retrain rounds
train $f_\theta$ to convergence // dense model first
for round $r = 1$ to $k$ do
  $s_r \gets s \cdot r / k$ // ramp sparsity gradually
  $\tau \gets$ the $s_r$-th percentile of $\braces{\abs{\theta_j}}$
  for each weight $\theta_j$ do
    if $\abs{\theta_j} < \tau$ then
      $\theta_j \gets 0$ and freeze it // mask out, never updated again
  retrain unmasked weights for a few epochs // recover accuracy
return the sparse $f_\theta$
```

The loop matters more than the pruning rule. One-shot pruning to sparsity $s$ moves the
weights far off the trained optimum in a single jump; the surviving weights never had a
chance to compensate. Ramping $s_r$ from $0$ to $s$ over $k$ rounds, each followed by a
short retrain, keeps the model near a good basin the whole way down.

$$
% caption: Iterative prune--retrain. Each round tightens the sparsity target,
% masks the smallest survivors, and retrains the rest to recover accuracy.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  node/.style={draw, minimum width=20mm, minimum height=9mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \node[node, draw=black] (train) at (0,0) {train dense};
  \node[node, draw=acc, text=acc, thick] (rank) at (3.2,0) {rank by\\magnitude};
  \node[node, draw=acc, text=acc, thick] (mask) at (6.4,0) {mask\\smallest};
  \node[node, draw=green, text=green, thick] (retrain) at (9.6,0) {retrain\\\texttt{survivors}};
  \draw[->, thick] (train) -- (rank);
  \draw[->, acc, thick] (rank) -- (mask);
  \draw[->, thick] (mask) -- (retrain);
  % feedback loop: retrain back up to rank, next round raises sparsity
  \draw[->, green, thick] (retrain) -- (9.6,-1.4) -- (3.2,-1.4) -- (rank);
  \node[green, anchor=north, font=\footnotesize] at (6.4,-1.4) {\texttt{next round: raise sparsity} $s_r$};
  \node[anchor=south, font=\footnotesize] at (6.4,1.0) {\texttt{repeat} $k$ \texttt{times, then stop at target} $s$};
\end{tikzpicture}
$$

### Structured versus unstructured

_Where_ the zeros land decides whether the chip can exploit them.

| Granularity | What is removed | Sparsity pattern | Speedup on dense hardware |
| --- | --- | --- | --- |
| Unstructured | individual weights | scattered | none (needs sparse kernels) |
| Structured: channel | whole conv filters / channels | dense sub-tensor | yes, shrinks the matrix |
| Structured: head | whole attention heads | dense sub-tensor | yes |
| Structured: block | $N{:}M$ blocks (e.g. 2:4) | semi-structured | yes (tensor-core support) |

Unstructured pruning reaches the highest sparsity at a given accuracy because it can
remove any weight, but the leftover dense tensor with scattered zeros runs no faster
on a GPU that schedules every multiply regardless. Structured pruning removes entire
rows, columns, channels, or heads, leaving a smaller _dense_ matrix that standard
kernels run faster.

$$
% caption: Unstructured pruning scatters zeros (left); structured pruning deletes
% whole rows/columns, leaving a smaller dense matrix (right).
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \def\c{0.42}
  % ---- left: unstructured ----
  \begin{scope}
    \foreach \i in {0,...,4}
      \foreach \j in {0,...,4} {
        \draw[black] (\j*\c,\i*\c) rectangle ++(\c,\c);
      }
    % keep a scattered subset filled (acc), rest empty (pruned)
    \foreach \i/\j in {0/1,0/3,1/0,1/2,1/4,2/1,2/3,3/0,3/2,3/4,4/1,4/3}
      \fill[acc!30] (\j*\c+0.04,\i*\c+0.04) rectangle ++(\c-0.08,\c-0.08);
    \node[anchor=north, font=\footnotesize] at (2.5*\c,-0.25) {\texttt{unstructured}};
    \node[red, anchor=south, font=\footnotesize] at (2.5*\c,5*\c+0.1) {\texttt{scattered zeros}};
  \end{scope}
  % ---- right: structured (whole rows/cols removed) ----
  \begin{scope}[xshift=5.0cm]
    \foreach \i in {0,...,4}
      \foreach \j in {0,...,4} {
        \draw[black] (\j*\c,\i*\c) rectangle ++(\c,\c);
      }
    % keep full rows 0,2,4 and full cols 0,2,4 -> dense survivors
    \foreach \i in {0,2,4}
      \foreach \j in {0,2,4}
        \fill[green!30] (\j*\c+0.04,\i*\c+0.04) rectangle ++(\c-0.08,\c-0.08);
    \node[anchor=north, font=\footnotesize] at (2.5*\c,-0.25) {\texttt{structured}};
    \node[green, anchor=south, font=\footnotesize] at (2.5*\c,5*\c+0.1) {\texttt{dense survivor}};
  \end{scope}
\end{tikzpicture}
$$

### The Lottery Ticket Hypothesis

Iterative magnitude pruning raises a question: could the pruned subnetwork
have been trained alone from the start? The answer is yes, but only
with the original initialization.

> **Theorem (Lottery Ticket Hypothesis).** A randomly-initialized dense network
> contains a sparse subnetwork — a _winning ticket_ — that, when trained in
> isolation from the _same initial weights_, matches the full network's accuracy in
> at most the same number of iterations. Resetting the winning ticket's weights to
> fresh random values instead destroys it: the initialization, not just the
> connectivity pattern, is what wins the lottery.

The practical procedure to find the ticket is iterative: train, prune the smallest
magnitudes, _rewind_ the survivors to their original initial values, and retrain.
The lesson is that sparsity is a property of the trainable subnetwork, not a
post-hoc cleanup. It reframes pruning as discovering structure the dense network
already contained.

> **Remark (Deep Compression).** Pruning composes with the other methods. The
> classic pipeline prunes to sparse, quantizes the survivors to a small set of
> shared values, then entropy-codes the result, reaching an order-of-magnitude
> size reduction with negligible accuracy loss on the vision nets of its day.

## Quantization

Quantization attacks the bytes-per-weight factor $b$ directly: store and compute in
low-precision integers instead of 32-bit floats. The standard first step is `fp32` $\to$
`int8`, a $4\times$ memory cut, with integer matrix-multiply units that are faster
and far more energy-efficient than their floating-point counterparts.[^stevens-deploy]

### The affine quantization map

Map a real value $x$ in a range $[\beta_{\min}, \beta_{\max}]$ to an 8-bit integer
$x_q \in \{0, \dots, 255\}$ through a **scale** $s$ and a **zero-point** $z$:

$$
x_q \;=\; \round\!\parens{\frac{x}{s}} + z,
\qquad
\hat x \;=\; s\,(x_q - z),
$$

where $\hat x$ is the dequantized value and the round-trip error $\abs{x - \hat x}$
is bounded by half a quantization step, $s/2$. The scale and zero-point are fixed by
the range:

$$
s \;=\; \frac{\beta_{\max} - \beta_{\min}}{q_{\max} - q_{\min}},
\qquad
z \;=\; q_{\min} - \round\!\parens{\frac{\beta_{\min}}{s}}.
$$

The zero-point ensures the real value $0$ maps to an exact integer, which keeps
zero-padding and ReLU outputs exact. A symmetric variant fixes $z = 0$ and uses a
range $[-\beta, \beta]$, trading one representable level for simpler integer
arithmetic.

For example, take a weight tensor whose observed range is
$[\beta_{\min}, \beta_{\max}] = [-0.8, 1.2]$ quantized to unsigned `int8`, so
$q_{\min} = 0$ and $q_{\max} = 255$. The scale and zero-point are

$$
s = \frac{1.2 - (-0.8)}{255 - 0} = \frac{2.0}{255} \approx 0.00784,
\qquad
z = 0 - \round\!\parens{\frac{-0.8}{0.00784}} = \round(102) = 102.
$$

The weight $x = 0.5$ then quantizes to
$x_q = \round(0.5 / 0.00784) + 102 = \round(63.75) + 102 = 166$,
and dequantizes to $\hat x = 0.00784\,(166 - 102) = 0.502$, an error of $0.002$, safely
under the half-step bound $s/2 \approx 0.0039$.

The reason integer inference is fast is that the multiply itself never leaves the
integer domain. For a matrix product $y = W x$ with weight scale $s_w$, zero-point
$z_w$, and activation scale $s_a$, zero-point $z_a$, substitute the affine maps and
the real product factors into an integer accumulation times a single float rescale:

$$
y \;=\; s_w s_a \sum_j (W_{q,j} - z_w)(x_{q,j} - z_a).
$$

The inner sum runs entirely in `int8`-times-`int8`-accumulated-in-`int32`, which is
what the integer tensor units execute; the scalar $s_w s_a$ multiplies the accumulated
result once at the end. All the arithmetic that scales with the matrix size is integer;
the lone floating-point operation is $O(1)$ per output.

$$
% caption: The affine map: a continuous range is rounded onto evenly spaced integer
% levels; each real value snaps to its nearest grid point.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % real axis (continuous)
  \draw[->, thick] (0,2.4) -- (8.4,2.4) node[right, font=\scriptsize] {real $x$};
  \draw[black] (0.6,2.5) -- (0.6,2.3) node[below, font=\scriptsize] {min};
  \draw[black] (7.8,2.5) -- (7.8,2.3) node[below, font=\scriptsize] {max};
  % a real value to be quantized
  \fill[acc] (4.3,2.4) circle (2.4pt);
  \node[acc, anchor=south, font=\scriptsize] at (4.3,2.55) {$x$};
  % integer grid (quantized levels)
  \draw[->, thick] (0,0) -- (8.4,0) node[right, font=\scriptsize] {int level $x_q$};
  \foreach \k in {0,1,...,8}
    \draw[black] (0.6+\k*0.9,0.12) -- (0.6+\k*0.9,-0.12);
  \foreach \k/\lab in {0/0,4/4,8/8}
    \node[anchor=north, font=\scriptsize] at (0.6+\k*0.9,-0.18) {\lab};
  % the chosen grid point (nearest)
  \fill[green] (4.2,0) circle (2.6pt);
  \node[green, anchor=north, font=\scriptsize] at (4.2,-0.5) {round};
  % snap arrow
  \draw[->, red, thick] (4.3,2.25) -- (4.2,0.2);
  \node[red, anchor=west, font=\scriptsize] at (4.4,1.2) {snap to nearest};
\end{tikzpicture}
$$

### PTQ versus QAT

Two regimes set the scales. **Post-training quantization (PTQ)** quantizes a
finished model and calibrates ranges on a small unlabeled set, with no retraining.
It is fast and free of the training loop, but at low bit-widths the rounding error
can reduce accuracy. **Quantization-aware training (QAT)** simulates quantization
_during_ training so the weights adapt to the rounding.

The obstacle in QAT is that the rounding function has zero gradient almost
everywhere. The solution is the **straight-through estimator (STE)**: in the forward pass
apply real rounding; in the backward pass pretend rounding was the identity, passing
the gradient through unchanged.

$$
\text{forward: } x_q = \round(x),
\qquad
\text{backward: } \frac{\partial x_q}{\partial x} \;:=\; 1
\;\text{ (clipped to the quantized range)}.
$$

$$
% caption: The straight-through estimator. The forward pass rounds; the backward
% pass replaces the flat rounding gradient with the identity so a signal reaches $x$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  node/.style={draw, minimum width=16mm, minimum height=8mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \node[node, draw=black] (x) at (0,0) {$x$};
  \node[node, draw=acc, text=acc, thick] (r) at (3.0,0) {round};
  \node[node, draw=black] (xq) at (6.0,0) {$x_q$};
  \node[node, draw=green, text=green, thick] (loss) at (9.0,0) {loss};
  % forward (top)
  \draw[->, acc, thick] (x) -- (r);
  \draw[->, acc, thick] (r) -- (xq);
  \draw[->, acc, thick] (xq) -- (loss);
  \node[acc, anchor=south, font=\footnotesize] at (4.5,0.55) {\texttt{forward: real rounding}};
  % backward (bottom): gradient flows straight through as if identity
  \node[red, anchor=north, font=\scriptsize] at (4.5,-1.15)
    {\texttt{backward: treat round as identity, gradient passes through}};
  \draw[->, red, thick] (loss) to[out=-90,in=-90,looseness=1.6] (x);
\end{tikzpicture}
$$

| Aspect | PTQ | QAT |
| --- | --- | --- |
| Retraining | none (calibrate only) | full fine-tune with simulated quant |
| Cost | minutes | hours to days |
| Accuracy at `int8` | good for robust models | better, near-lossless |
| Accuracy at `int4` | often poor | recovers much of the gap |
| Gradient trick | not needed | straight-through estimator |

> **Definition (Per-channel scales).** Rather than one scale $s$ for an entire
> weight tensor, assign a separate scale $s_c$ to each output channel (or row) of
> the matrix. Channels with very different weight ranges no longer share a grid, so
> a few wide-range channels stop forcing a coarse step on the rest. Per-channel
> quantization is near-mandatory below `int8`.

### Quantizing large language models

LLMs break naive `int8` quantization for one reason: a small number of
**outlier features** in the activations carry magnitudes orders larger than the
rest, and a single tensor-wide scale stretched to cover them quantizes everything
else to near-zero.

> **Definition (Outlier features).** In a trained transformer, a handful of hidden
> dimensions develop activation magnitudes far larger than the bulk. They are
> systematic, appearing in the same dimensions across tokens, and they dominate the
> quantization range; ignoring them collapses LLM accuracy.

The main LLM quantization methods each handle this differently.

| Method | Bits | Idea | Handles outliers by |
| --- | --- | --- | --- |
| `LLM.int8()` | 8 | mixed-precision decomposition | keeping outlier columns in `fp16`, rest in `int8` |
| GPTQ | 3–4 | second-order weight rounding | layer-wise error compensation via the Hessian |
| AWQ | 4 | activation-aware scaling | protecting salient weight channels by activation scale |
| QLoRA / NF4 | 4 | normal-float storage + LoRA | a quantile-optimal 4-bit grid for normal weights |

- **`LLM.int8()`** splits each matrix multiply: the few outlier feature columns run
  in `fp16`, the rest in `int8`, recombined exactly. It is lossless at `int8` for
  models past the scale where outliers emerge.

- **GPTQ** quantizes weights one layer at a time, after each rounding step adjusting
  the still-unquantized weights to compensate for the error introduced, using
  curvature from the layer's Hessian. It reaches 3–4 bits with small loss.

- **AWQ** observes that not all weights matter equally; the salient ones are those
  multiplying large-activation channels. It scales those channels up before
  quantizing so they keep precision, then folds the scale back.

- **QLoRA** stores the frozen base model in **NF4**, a 4-bit floating format whose
  levels are the quantiles of a normal distribution (matching the empirical weight
  histogram), and fine-tunes only small LoRA adapters on top. This makes 4-bit
  fine-tuning of very large models fit on a single GPU.

> **Remark (Where quantization pays).** At `int8`, accuracy loss is usually under a
> point and the $4\times$ memory and bandwidth cut is immediate, so `int8` is the
> default first move. At 4 bits the win doubles but the methods above (or QAT)
> become necessary to hold accuracy. Below 4 bits, returns shrink fast.

## Low-rank factorization

A linear layer's weight matrix $W \in \mathbb{R}^{m \times n}$ holds $mn$
parameters. If $W$ is approximately low-rank, replacing it with a product of two thin
matrices is a direct compression.

$$
W \;\approx\; U V^{\top},
\qquad
U \in \mathbb{R}^{m \times r},\;\; V \in \mathbb{R}^{n \times r},
\qquad r \ll \min(m, n).
$$

The parameter count drops from $mn$ to $r(m + n)$, a win whenever $r < mn/(m+n)$.
The best rank-$r$ approximation in Frobenius norm is the truncated SVD: keep the top
$r$ singular values.

> **Theorem (Eckart–Young).** Among all matrices of rank at most $r$, the truncated
> SVD $W_r = \sum_{i=1}^{r} \sigma_i u_i v_i^{\top}$ minimizes
> $\norm{W - W_r}_F$, and the residual error is
> $\norm{W - W_r}_F^2 = \sum_{i > r} \sigma_i^2$. A weight matrix whose singular
> values decay fast compresses well; one with a flat spectrum does not.

$$
% caption: A fat $m \times n$ matrix factored into two thin matrices of inner
% dimension $r$, cutting parameters from $mn$ to $r(m + n)$.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % big W
  \draw[acc, very thick, fill=acc!12] (0,0) rectangle (2.4,2.4);
  \node[acc] at (1.2,1.2) {$W$};
  \node[anchor=north, font=\footnotesize] at (1.2,-0.15) {$m$ \texttt{by} $n$};
  \node[font=\large] at (3.0,1.2) {$=$};
  \node[anchor=south, font=\footnotesize] at (3.0,1.35) {\texttt{approx}};
  % U (tall thin)
  \draw[green, very thick, fill=green!12] (3.7,0) rectangle (4.3,2.4);
  \node[green] at (4.0,1.2) {$U$};
  \node[anchor=north, font=\footnotesize] at (4.0,-0.15) {$m$ \texttt{by} $r$};
  % V^T (wide short)
  \draw[green, very thick, fill=green!12] (5.4,0.9) rectangle (7.8,1.5);
  \node[green] at (6.6,1.2) {$V^{T}$};
  \node[anchor=north, font=\footnotesize] at (6.6,0.75) {$r$ \texttt{by} $n$};
\end{tikzpicture}
$$

### The tie to LoRA

Low-rank structure also drives the most common parameter-efficient fine-tuning
method. **LoRA** freezes the pretrained weight $W_0$ and learns only a low-rank
_update_, so the adapted weight is

$$
W \;=\; W_0 \;+\; \Delta W,
\qquad
\Delta W \;=\; B A,
\qquad
B \in \mathbb{R}^{m \times r},\;\; A \in \mathbb{R}^{r \times n},\;\; r \ll \min(m,n).
$$

Only $A$ and $B$ are trained, a tiny fraction of the parameters, and at inference
$\Delta W$ folds into $W_0$ at no added latency. The premise is that the _fine-tuning
update_ is intrinsically low-rank even when $W_0$ is not. Combined with a 4-bit
frozen base, this recovers the QLoRA method above: quantize $W_0$ to NF4, train
$A, B$ in higher precision.

## Combining methods

The four levers are largely orthogonal and routinely stacked. Distillation produces
a small dense student; pruning sparsifies it; quantization shrinks each surviving
weight to `int8` or `int4`; low-rank factorization or LoRA handles the fine-tuning.
What trades off against what:

| Method | Memory cut | Latency win | Accuracy hit | Hardware need |
| --- | --- | --- | --- | --- |
| Distillation | large (small student) | large | small–moderate | none (trains a normal net) |
| Unstructured pruning | moderate | none on dense HW | small at moderate sparsity | sparse kernels for speedup |
| Structured pruning | moderate | yes | moderate (coarser deletions) | none |
| Quantization (`int8`) | $4\times$ | yes | minimal | integer tensor units |
| Quantization (`int4`) | $8\times$ | yes | small with GPTQ/AWQ/QAT | 4-bit kernels |
| Low-rank / LoRA | matrix-dependent | yes if rank low | small if spectrum decays | none |

> **Remark (A sensible order).** Distill first to fix the architecture and
> recover accuracy, then prune (structured, to get real speedups), then quantize
> last (it is cheapest and composes with everything). Quantizing before pruning
> wastes effort, since pruned weights are discarded anyway, and pruning a quantized
> model is awkward because magnitudes are already coarsened.

> **Remark (Measure, do not assume).** Every compression method's accuracy hit is
> dataset- and model-dependent, and every speedup is hardware-dependent. The
> [practical methodology](/deep-learning/practical/practical-methodology) loop
> applies unchanged: fix the deployment metric (latency at fixed accuracy, or
> accuracy at fixed budget), measure on the target device, and change one lever at
> a time.

## Compression in the large-model era

The four levers above predate large language models; scaling to billions of parameters
sharpened each into a specialized method that Goodfellow does not cover.

**Post-training quantization** grew precise. Naively rounding an LLM's weights to `int4`
sharply degrades accuracy, but **GPTQ** (Frantar et al., 2023, _ICLR_) quantizes one weight column
at a time and adjusts the not-yet-quantized weights to compensate for the error each
rounding introduces, using second-order (Hessian) information. **AWQ** (Lin et al., 2024,
_MLSys_) observes that a small fraction of "salient" weight channels — identifiable from
the activation magnitudes — carry most of the model's quality, and protects those while
aggressively quantizing the rest. Both recover near-full-precision accuracy at 4 bits with
no retraining, exactly the `int4`-with-GPTQ/AWQ row in the table above.

**QLoRA** (Dettmers et al., 2023, _NeurIPS_) fuses quantization with the parameter-efficient
fine-tuning from the [transfer-learning lesson](/deep-learning/practical/transfer-learning):
freeze the base model at 4-bit precision and train only the LoRA adapters in higher
precision on top, which lets a single consumer GPU fine-tune a 65-billion-parameter model
that would not otherwise fit in memory. Compression and adaptation, treated separately in
this lesson, become one step.

**Pruning** got a theory and a one-shot method. The **lottery ticket hypothesis** (Frankle
& Carbin, 2019, _ICLR_) argues that a dense network contains a sparse subnetwork — a
"winning ticket" — that, trained in isolation from the original initialization, matches the
full model, which reframes pruning as _finding_ that subnetwork rather than degrading the
dense one. **SparseGPT** (Frantar & Alistarh, 2023, _ICML_) then made pruning practical at
LLM scale, removing half the weights of a 175-billion-parameter model in one pass with
negligible accuracy loss, using the same error-compensation idea as GPTQ. The principle
is unchanged from this lesson — trade capacity for cost, measure the hit on the deployment
metric — but the methods now assume a frozen, pretrained giant rather than a model trained
from scratch.

## Takeaways

- **Compression is a deployment problem, not a training one.** The binding
  constraints are memory, latency, and energy on the target device, and energy is
  dominated by weight movement, not arithmetic. Each method defines its own
  accuracy–cost frontier; pick the operating point.
- **Distillation transfers dark knowledge.** A student trained on a teacher's
  temperature-softened logits learns the teacher's relative confidence over wrong
  classes, a richer signal than one-hot labels. The KD loss mixes a $T^2$-scaled KL
  term with a hard cross-entropy; the $T^2$ keeps the soft gradient on scale.
- **Pruning deletes weights; structure decides the speedup.** Magnitude pruning with
  iterative retraining reaches high sparsity, but only _structured_ pruning (channels,
  heads, blocks) speeds up dense hardware. The Lottery Ticket Hypothesis says the
  winning subnetwork was already present at initialization.
- **Quantization cuts bytes-per-weight.** The affine map $x_q = \round(x/s) + z$
  takes `fp32` to `int8` for a $4\times$ cut; QAT uses the straight-through
  estimator to train through rounding, and per-channel scales are essential below
  `int8`. LLMs need outlier-aware methods (`LLM.int8()`, GPTQ, AWQ, NF4/QLoRA).
- **Low-rank factorization replaces a fat matrix with two thin ones** when the
  spectrum decays (Eckart–Young), and the same structure powers LoRA's low-rank
  fine-tuning update.
- **Stack the methods in order:** distill, then prune, then quantize, measuring
  wall-clock latency on the real hardware at each step.

[^stevens-deploy]: **Stevens**, _Deep Learning with PyTorch_, Part III — deployment: model size, latency, and the memory and bandwidth costs that bind at inference, with reduced-precision (`int8`) storage and compute as the first lever.
[^gf-ensemble]: **Goodfellow**, _Deep Learning_, §7.11 — ensemble methods average many models to reduce variance; the distillation setup trains one fast model to reproduce that averaged output distribution.
[^gf-capacity]: **Goodfellow**, _Deep Learning_, §5.2 — capacity, over-parameterization, and why a large model can fit its task while most of its representational budget is redundant, leaving weights that can be removed.
[^gf-generalize]: **Goodfellow**, _Deep Learning_, §5.4–5.5 — the bias--variance view of generalization: a compressed model trades a small increase in bias for a large cut in the effective number of parameters.