Model Compression and Distillation

A model that scores well on a held-out set has met only the first requirement. The second is deployment, where the binding constraints are memory, latency, and energy, not validation loss. A network with a billion parameters in fp32 occupies four gigabytes before a single activation is allocated, and that footprint decides whether it runs on a phone, fits a GPU's memory, or meets a millisecond service-level target. Model compression is the set of techniques that shrink a trained network while giving back as little accuracy as possible. This lesson treats the four that matter in practice and the order in which to combine them. It assumes the practical methodology loop and the regularization view from the regularization overview; the largest payoffs land on the large language models where the fp32 checkpoint will not even load.

Why compress: the accuracy–cost frontier

Three resources bind at inference, and they scale differently. Memory scales with the parameter count $N$ times bytes-per-weight $b$. Compute scales with the multiply-accumulate count, roughly $2N$ FLOPs per token for a dense layer. Energy is dominated not by arithmetic but by data movement: reading a weight from DRAM costs orders of magnitude more than the multiply it feeds.¹

The methods do not move a model along a fixed curve; each defines its own accuracy-versus-cost frontier, and the practitioner picks the point. The goal is the Pareto frontier: for a target accuracy, the cheapest model; for a cost budget, the most accurate.

Compression works at all because a trained network is not a tight fit to its task. It carries far more capacity than the task's information content requires, and that slack is what every method removes.² Removing it trades a small increase in bias for a large reduction in the effective parameter count; the frontier collects the points where that trade is favorable. A method that pushed past the frontier would raise bias faster than it cut cost, which is the over-compressed regime marked in red below.

Knowledge distillation

The cheapest way to get a small, accurate model is often not to train a small model from scratch but to train it to imitate a large one. The large model is the teacher; the small one is the student. The student learns from the teacher's full output distribution, not just the hard label.³

Soft targets and dark knowledge

A network's final logits $z\in {\mathbb{R}}^K$ become a distribution through the softmax. Introduce a temperature $T\ge 1$ that flattens it:

At $T=1$ this is the ordinary softmax. As $T$ grows the distribution spreads toward uniform, amplifying the small probabilities the teacher assigns to the wrong classes. Those small probabilities are the signal.

Temperature is what exposes dark knowledge. A confident teacher puts $0.99$ on the true class and $10^{-6}$ on the rest, so at $T=1$ the soft target is indistinguishable from one-hot. Raising $T$ rescales the logit gaps and lifts those tiny probabilities into a usable gradient.

The distillation loss

The student is trained on a convex combination of two terms: a soft term that matches the teacher at temperature $T$, and a hard term that matches the true label at $T=1$. With teacher logits $z_t$, student logits $z_s$, true label $y$, and mixing weight $\alpha \in [0,1]$,

Each term has a role.

• The KL term $\mathrm{KL}\left(p(z_t;T)\Vert p(z_s;T)\right)$ measures how far the student's softened distribution sits from the teacher's. Minimizing it over $z_s$ (the teacher is fixed) pulls the student's whole logit vector toward the teacher's, dark knowledge included.
• The $T^2$ factor rescales the soft loss. The gradient of the softened cross-entropy with respect to a logit carries a $1/T$ from the softmax argument and another $1/T$ from the target probabilities, so the soft gradient shrinks like $1/T^2$. Multiplying by $T^2$ restores it to the same scale as the hard gradient, keeping $\alpha$ meaningful as $T$ changes.
• The CE term anchors the student to the ground truth, so it does not inherit the teacher's mistakes wholesale. Setting $\alpha =1$ trains purely on the teacher; $\alpha =0$ ignores it.

The picture is a teacher feeding two channels into the student loss: a softened distribution and the hard label.

Pruning

Distillation builds a small model; pruning carves one out of a large model by deleting weights. The premise is that trained networks are heavily over-parameterized and most weights contribute little.

Magnitude pruning

The simplest and strongest baseline ranks weights by absolute value and zeros the smallest. A weight near zero contributes little to any activation, so removing it perturbs the function least. That such weights exist in bulk is the same over-parameterization that lets large networks fit and still generalize.⁴

where $s$ is the target sparsity. Pruning in one shot to high sparsity destroys accuracy; to address this, alternate pruning with retraining so the surviving weights absorb the slack.

The loop matters more than the pruning rule. One-shot pruning to sparsity $s$ moves the weights far off the trained optimum in a single jump; the surviving weights never had a chance to compensate. Ramping $s_r$ from $0$ to $s$ over $k$ rounds, each followed by a short retrain, keeps the model near a good basin the whole way down.

Structured versus unstructured

Where the zeros land decides whether the chip can exploit them.

Unstructured pruning reaches the highest sparsity at a given accuracy because it can remove any weight, but the leftover dense tensor with scattered zeros runs no faster on a GPU that schedules every multiply regardless. Structured pruning removes entire rows, columns, channels, or heads, leaving a smaller dense matrix that standard kernels run faster.

The Lottery Ticket Hypothesis

Iterative magnitude pruning raises a question: could the pruned subnetwork have been trained alone from the start? The answer is yes, but only with the original initialization.

The practical procedure to find the ticket is iterative: train, prune the smallest magnitudes, rewind the survivors to their original initial values, and retrain. The lesson is that sparsity is a property of the trainable subnetwork, not a post-hoc cleanup. It reframes pruning as discovering structure the dense network already contained.

Quantization

Quantization attacks the bytes-per-weight factor $b$ directly: store and compute in low-precision integers instead of 32-bit floats. The standard first step is fp32 $\to$int8, a $4\times$ memory cut, with integer matrix-multiply units that are faster and far more energy-efficient than their floating-point counterparts.¹

The affine quantization map

Map a real value $x$ in a range $[{\beta }_{\min },{\beta }_{\max }]$ to an 8-bit integer $x_q\in \{0,\dots ,255\}$ through a scale $s$ and a zero-point $z$:

where $\hat{x}$ is the dequantized value and the round-trip error $\left|x-\hat{x}\right|$ is bounded by half a quantization step, $s/2$. The scale and zero-point are fixed by the range:

The zero-point ensures the real value $0$ maps to an exact integer, which keeps zero-padding and ReLU outputs exact. A symmetric variant fixes $z=0$ and uses a range $[-\beta ,\beta ]$, trading one representable level for simpler integer arithmetic.

For example, take a weight tensor whose observed range is $[{\beta }_{\min },{\beta }_{\max }]=[-0.8,1.2]$ quantized to unsigned int8, so $q_{\min }=0$ and $q_{\max }=255$. The scale and zero-point are

The weight $x=0.5$ then quantizes to $x_q=\mathrm{round}(0.5/0.00784)+102=\mathrm{round}(63.75)+102=166$, and dequantizes to $\hat{x}=0.00784(166-102)=0.502$, an error of $0.002$, safely under the half-step bound $s/2\approx 0.0039$.

The reason integer inference is fast is that the multiply itself never leaves the integer domain. For a matrix product $y=Wx$ with weight scale $s_w$, zero-point $z_w$, and activation scale $s_a$, zero-point $z_a$, substitute the affine maps and the real product factors into an integer accumulation times a single float rescale:

The inner sum runs entirely in int8-times-int8-accumulated-in-int32, which is what the integer tensor units execute; the scalar $s_w{s}_a$ multiplies the accumulated result once at the end. All the arithmetic that scales with the matrix size is integer; the lone floating-point operation is $O(1)$ per output.

PTQ versus QAT

Two regimes set the scales. Post-training quantization (PTQ) quantizes a finished model and calibrates ranges on a small unlabeled set, with no retraining. It is fast and free of the training loop, but at low bit-widths the rounding error can reduce accuracy. Quantization-aware training (QAT) simulates quantization during training so the weights adapt to the rounding.

The obstacle in QAT is that the rounding function has zero gradient almost everywhere. The solution is the straight-through estimator (STE): in the forward pass apply real rounding; in the backward pass pretend rounding was the identity, passing the gradient through unchanged.

Quantizing large language models

LLMs break naive int8 quantization for one reason: a small number of outlier features in the activations carry magnitudes orders larger than the rest, and a single tensor-wide scale stretched to cover them quantizes everything else to near-zero.

The main LLM quantization methods each handle this differently.

• LLM.int8() splits each matrix multiply: the few outlier feature columns run in fp16, the rest in int8, recombined exactly. It is lossless at int8 for models past the scale where outliers emerge.
• GPTQ quantizes weights one layer at a time, after each rounding step adjusting the still-unquantized weights to compensate for the error introduced, using curvature from the layer's Hessian. It reaches 3–4 bits with small loss.
• AWQ observes that not all weights matter equally; the salient ones are those multiplying large-activation channels. It scales those channels up before quantizing so they keep precision, then folds the scale back.
• QLoRA stores the frozen base model in NF4, a 4-bit floating format whose levels are the quantiles of a normal distribution (matching the empirical weight histogram), and fine-tunes only small LoRA adapters on top. This makes 4-bit fine-tuning of very large models fit on a single GPU.

Low-rank factorization

A linear layer's weight matrix $W\in {\mathbb{R}}^{m\times n}$ holds $mn$ parameters. If $W$ is approximately low-rank, replacing it with a product of two thin matrices is a direct compression.

The parameter count drops from $mn$ to $r(m+n)$, a win whenever $r<mn/(m+n)$. The best rank-$r$ approximation in Frobenius norm is the truncated SVD: keep the top $r$ singular values.

The tie to LoRA

Low-rank structure also drives the most common parameter-efficient fine-tuning method. LoRA freezes the pretrained weight $W_0$ and learns only a low-rank update, so the adapted weight is

Only $A$ and $B$ are trained, a tiny fraction of the parameters, and at inference $\Delta W$ folds into $W_0$ at no added latency. The premise is that the fine-tuning update is intrinsically low-rank even when $W_0$ is not. Combined with a 4-bit frozen base, this recovers the QLoRA method above: quantize $W_0$ to NF4, train $A,B$ in higher precision.

Combining methods

The four levers are largely orthogonal and routinely stacked. Distillation produces a small dense student; pruning sparsifies it; quantization shrinks each surviving weight to int8 or int4; low-rank factorization or LoRA handles the fine-tuning. What trades off against what:

Compression in the large-model era

The four levers above predate large language models; scaling to billions of parameters sharpened each into a specialized method that Goodfellow does not cover.

Post-training quantization grew precise. Naively rounding an LLM's weights to int4 sharply degrades accuracy, but GPTQ (Frantar et al., 2023, ICLR) quantizes one weight column at a time and adjusts the not-yet-quantized weights to compensate for the error each rounding introduces, using second-order (Hessian) information. AWQ (Lin et al., 2024, MLSys) observes that a small fraction of salient weight channels — identifiable from the activation magnitudes — carry most of the model's quality, and protects those while aggressively quantizing the rest. Both recover near-full-precision accuracy at 4 bits with no retraining, exactly the int4-with-GPTQ/AWQ row in the table above.

QLoRA (Dettmers et al., 2023, NeurIPS) fuses quantization with the parameter-efficient fine-tuning from the transfer-learning lesson: freeze the base model at 4-bit precision and train only the LoRA adapters in higher precision on top, which lets a single consumer GPU fine-tune a 65-billion-parameter model that would not otherwise fit in memory. Compression and adaptation, treated separately in this lesson, become one step.

Pruning got a theory and a one-shot method. The lottery ticket hypothesis (Frankle & Carbin, 2019, ICLR) argues that a dense network contains a sparse subnetwork — a winning ticket — that, trained in isolation from the original initialization, matches the full model, which reframes pruning as finding that subnetwork rather than degrading the dense one. SparseGPT (Frantar & Alistarh, 2023, ICML) then made pruning practical at LLM scale, removing half the weights of a 175-billion-parameter model in one pass with negligible accuracy loss, using the same error-compensation idea as GPTQ. The principle is unchanged from this lesson — trade capacity for cost, measure the hit on the deployment metric — but the methods now assume a frozen, pretrained giant rather than a model trained from scratch.

Takeaways

• Compression is a deployment problem, not a training one. The binding constraints are memory, latency, and energy on the target device, and energy is dominated by weight movement, not arithmetic. Each method defines its own accuracy–cost frontier; pick the operating point.
• Distillation transfers dark knowledge. A student trained on a teacher's temperature-softened logits learns the teacher's relative confidence over wrong classes, a richer signal than one-hot labels. The KD loss mixes a $T^2$-scaled KL term with a hard cross-entropy; the $T^2$ keeps the soft gradient on scale.
• Pruning deletes weights; structure decides the speedup. Magnitude pruning with iterative retraining reaches high sparsity, but only structured pruning (channels, heads, blocks) speeds up dense hardware. The Lottery Ticket Hypothesis says the winning subnetwork was already present at initialization.
• Quantization cuts bytes-per-weight. The affine map $x_q=\mathrm{round}(x/s)+z$ takes fp32 to int8 for a $4\times$ cut; QAT uses the straight-through estimator to train through rounding, and per-channel scales are essential below int8. LLMs need outlier-aware methods (LLM.int8(), GPTQ, AWQ, NF4/QLoRA).
• Low-rank factorization replaces a fat matrix with two thin ones when the spectrum decays (Eckart–Young), and the same structure powers LoRA's low-rank fine-tuning update.
• Stack the methods in order: distill, then prune, then quantize, measuring wall-clock latency on the real hardware at each step.