--- title: Adversarial Defenses module: Theory & Frontiers moduleNumber: 6 lessonNumber: 3 order: 603 summary: > Defending a network against an adversary is far harder than attacking one. This lesson covers the defense side: certified guarantees via randomized smoothing, the transferability that makes black-box attacks possible, and the recurring failure of gradient masking, where a defense hides the attacker's gradient instead of moving the decision boundary. It ends with the adaptive-attack discipline (BPDA, EOT, transfer) that every robustness claim must be tested against. topics: [Theory & Frontiers] sources: - book: Goodfellow ref: "§7.13 — Adversarial Training; Ch. 7 Regularization" - book: Chollet ref: "Ch. 5 — Fundamentals of ML (generalization, the manifold view)" --- This builds on [Adversarial Robustness](/deep-learning/theory/adversarial-robustness), which set up the threat model, the fast gradient sign method, projected gradient descent, and adversarial training as a min-max problem. Those tools tell an attacker how to break a network and give the defender one honest response: train on the worst case. This lesson is about everything on the defense side that the first pass left open — how to _prove_ robustness rather than merely measure it, why attacks crafted on one model fool another, and the long history of defenses that looked strong and were not. ## Why a small ball matters Start from the geometry, because it explains what a defense must actually accomplish. A clean point can sit a vanishing distance from the decision boundary while still being classified correctly. Standard training only requires that $x$ land on the right side; it does not enforce any _margin_,[^chollet-manifold] so the boundary is free to pass arbitrarily close. Robustness requires more: every point in the $\epsilon$-ball around $x$ must stay on the correct side, which forces the boundary out beyond radius $\epsilon$. $$ % caption: Why robustness needs margin. The clean point is correct, but the boundary passes % within $\epsilon$, so a perturbation inside the ball crosses into the wrong region. \begin{tikzpicture}[>=stealth, font=\small] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \definecolor{red}{HTML}{C0392B} % decision boundary \draw[red, very thick] (-0.4,-1.8) .. controls (1.0,-0.2) and (1.0,0.2) .. (0.0,2.0) node[anchor=south, font=\footnotesize, text=red] {boundary}; % regions \node[green, font=\footnotesize] at (-1.7,1.4) {correct}; \node[red, font=\footnotesize] at (2.2,-1.0) {wrong}; % clean point near the boundary \fill[black] (-0.7,0.2) circle (2.6pt); \node[anchor=north, font=\footnotesize] at (-0.7,0.05) {clean}; % epsilon ball crossing the boundary \draw[black, very thick] (-0.7,0.2) circle (0.95); \node[black, font=\footnotesize, anchor=north] at (-0.7,-0.85) {eps-ball}; % an adversarial point inside the ball, across the boundary \fill[red] (0.2,0.45) circle (2.4pt); \node[red, font=\footnotesize, anchor=west] at (0.3,0.55) {adversarial}; \end{tikzpicture} $$ This is the same brittleness flagged for deep nets in [what is deep learning](/deep-learning/foundations/what-is-deep-learning), made quantitative: the failure is not that the model is wrong on $x$, but that the _correct_ region does not even contain a small ball around $x$. Adversarial training, from the previous lesson, attacks this by pushing the boundary out; the defenses below either _prove_ the boundary is far enough, or fail by only appearing to move it. ## Certified defenses: randomized smoothing Adversarial training gives an _empirical_ number: robustness against the strongest attack tried so far, with no guarantee a cleverer attack will not appear. A **certified** defense instead proves a radius $R$ such that _no_ perturbation with $\norm{\delta}_2 \le R$ can change the prediction. The most scalable construction is **randomized smoothing**, which turns any base classifier $f$ into a smoothed classifier $\hat f$ that reports the class $f$ predicts most often under Gaussian input noise: $$ \hat f(x) = \arg\max_{c}\; \Pr_{\eta \sim \mathcal{N}(0,\sigma^2 I)}\!\brackets{f(x + \eta) = c}. $$ Averaging over noise smooths the decision function, and a smooth function cannot change its majority vote quickly. Concretely, let $p_A$ be the probability the base classifier returns the top class $c_A$ under the noise and $p_B$ the probability of the runner-up. Then $\hat f(x + \delta) = c_A$ for every $\delta$ with $$ \norm{\delta}_2 \;<\; R \;=\; \frac{\sigma}{2}\parens{\Phi^{-1}(p_A) - \Phi^{-1}(p_B)}, $$ where $\Phi^{-1}$ is the inverse standard-normal CDF. The certified radius grows with the noise level $\sigma$ and with the confidence gap between the top two classes: a model that is barely sure ($p_A$ just above $p_B$) certifies nothing, while a model that is almost always right under noise ($p_A \to 1$, so $\Phi^{-1}(p_A) \to +\infty$) certifies a large ball. The cost is that a larger $\sigma$ blurs the input and lowers clean accuracy — the same robustness-accuracy tension, now controlled by the noise scale. In practice $p_A, p_B$ are unknown and estimated from $n$ noise samples with a confidence interval, so the certificate is _probabilistic_: it holds with probability $1 - \eta_{\text{err}}$ over the sampling. For example, suppose a smoothed classifier at noise scale $\sigma = 0.5$ returns the top class on $p_A = 0.99$ of the noise draws and the runner-up on $p_B = 0.01$. The standard-normal quantiles are $\Phi^{-1}(0.99) \approx 2.33$ and $\Phi^{-1}(0.01) \approx -2.33$, so the certified radius is $$ R = \frac{0.5}{2}\,(2.33 - (-2.33)) = 0.25 \times 4.66 \approx 1.16. $$ No $L_2$ perturbation of norm below $1.16$ can flip the smoothed prediction — a hard guarantee, not an empirical one. Halve the confidence gap (say $p_A = 0.9$, $\Phi^{-1}(0.9)\approx 1.28$, $p_B = 0.1$, $\Phi^{-1}(0.1)\approx -1.28$) and the radius collapses to $0.25 \times 2.56 \approx 0.64$: the certificate is only as wide as the model is confident under noise. Raising $\sigma$ widens $R$ for a fixed confidence gap, but a larger $\sigma$ also lowers $p_A$ by blurring the signal, so the two effects trade off; $\sigma$ is tuned, not maximized. $$ % caption: Randomized smoothing. Gaussian noise of scale $\sigma$ around $x$ blurs the base classifier into a smooth majority vote; the certified radius $R$ is the ball the prediction provably cannot leave. \begin{tikzpicture}[>=stealth, font=\small] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \definecolor{red}{HTML}{C0392B} % base decision boundary (wiggly) \draw[red, very thick] (1.7,-2.0) .. controls (1.1,-0.7) and (2.3,0.6) .. (1.6,2.0); \node[red, font=\footnotesize, anchor=south] at (1.7,2.0) {base boundary}; \node[green, font=\footnotesize] at (-1.6,1.6) {class A}; \node[red, font=\footnotesize] at (2.7,-1.6) {class B}; % noise cloud around x \foreach \a in {0,40,...,320} \fill[acc!55] ($ (0,0) + (\a:0.9) $) circle (1.2pt); \foreach \a in {20,70,...,340} \fill[acc!35] ($ (0,0) + (\a:1.45) $) circle (1.0pt); % clean point \fill[black] (0,0) circle (2.4pt); \node[anchor=north, font=\footnotesize] at (0,-0.18) {clean}; \node[acc, font=\footnotesize, anchor=north] at (0,-1.7) {noise scale sigma}; % certified radius \draw[green, very thick] (0,0) circle (1.05); \draw[->, green, very thick] (0,0) -- (0.68,0.68); \node[green, font=\footnotesize, anchor=south west] at (0.82,0.84) {radius R}; \end{tikzpicture} $$ > **Definition (Randomized smoothing).** Given a base classifier $f$ and noise scale > $\sigma$, the smoothed classifier $\hat f(x)$ returns the class $f$ predicts most > often under $\eta \sim \mathcal{N}(0, \sigma^2 I)$. It carries a certificate: the > prediction is provably constant on the $L_2$ ball of radius > $R = \tfrac{\sigma}{2}(\Phi^{-1}(p_A) - \Phi^{-1}(p_B))$, where $p_A, p_B$ are the > top-two class probabilities under the noise. The guarantee is $L_2$, probabilistic, > and costs clean accuracy through the blur. Randomized smoothing is not the only certification route, but it is the one that scales to large models, because it treats the base network as a black box and only needs its outputs under noise. The alternatives propagate a guaranteed range _through_ the network layer by layer; they are tighter but limited to small models. | Certification | Mechanism | Norm | Scale | | --- | --- | --- | --- | | Interval-bound propagation | push a box $[x-\epsilon, x+\epsilon]$ through each layer, tracking worst-case output | $L_\infty$ | small nets; bounds loosen with depth | | Lipschitz bounds | bound $\norm{f(x')-f(x)} \le L\norm{x'-x}$ via per-layer spectral norms | any | needs Lipschitz-constrained layers | | Randomized smoothing | majority vote under Gaussian noise, statistical certificate | $L_2$ | any base model; needs many samples | ## Transferability and black-box attacks The linearity hypothesis from the previous lesson has a corollary the attacker exploits when it cannot see $\theta$: adversarial examples **transfer**. Two models trained on the same data tend to learn similar near-linear functions, so their input-gradients point in similar directions, and a $\delta$ that raises the loss on one raises it on the other. An attacker with no gradient access trains its own **surrogate** model, runs white-box FGSM or PGD against the surrogate, and ships the resulting $x'$ at the target. $$ % caption: Transfer attack. The attacker cannot see the target's gradient, so it % trains a surrogate on similar data, attacks the surrogate white-box, and ships the % resulting perturbed input at the target, which shares enough structure to be fooled. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=22mm, minimum height=11mm, align=center}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \definecolor{green}{HTML}{1F9D4D} \node[box, draw=acc, text=acc, thick] (sur) at (0,0) {\texttt{surrogate}\\\texttt{model}}; \node[box] (att) at (3.6,0) {\texttt{white-box}\\\texttt{PGD}}; \node[box] (adv) at (7.0,0) {\texttt{adversarial}\\$x'$}; \node[box, draw=red, text=red, thick] (tgt) at (10.4,0) {\texttt{target}\\\texttt{model}}; \draw[->, acc, thick] (sur) -- (att) node[midway, above, font=\scriptsize] {\texttt{gradient}}; \draw[->, thick] (att) -- (adv); \draw[->, red, thick] (adv) -- (tgt) node[midway, above, font=\scriptsize] {\texttt{ship}}; \node[font=\footnotesize, anchor=north] at (5.2,-0.9) {\texttt{no target gradient ever queried}}; \end{tikzpicture} $$ The success rate falls off with how different the two models are, but for undefended targets it is high enough to make black-box robustness no easier to earn than white-box. Transfer is also the diagnostic tool of last resort: because it needs no gradient of the target at all, a transfer attack that beats a white-box attack is proof the white-box gradient was being masked. ## Defenses and their failure modes Beyond adversarial training and certified bounds, the persistent failure mode is **gradient masking**: a defense that lowers the _apparent_ attack success by obfuscating or zeroing the gradient the attacker differentiates, without actually flattening the loss surface. It collapses the moment the attacker estimates the gradient another way or transfers an example from a surrogate model. FGSM and PGD fix the budget $\epsilon$ and maximize the loss inside it. The **Carlini–Wagner (CW)** attack inverts the question: fix misclassification as a hard requirement and minimize the perturbation needed to achieve it. It solves $$ \min_{\delta}\; \norm{\delta}_2^2 \;+\; c\cdot h(x + \delta), \qquad h(x') = \max\!\parens{\max_{i \ne t} Z_i(x') - Z_t(x'),\; -\kappa}, $$ where $Z_i$ are the pre-softmax logits, $t$ is the target class, $c$ trades the two terms, and $\kappa \ge 0$ sets a confidence margin. The surrogate loss $h$ is zero exactly when the target logit exceeds every other logit by at least $\kappa$, so the optimizer pushes just past the boundary and then shrinks $\delta$. Because CW optimizes the _size_ of the perturbation rather than climbing a fixed-budget loss, it finds smaller perturbations than PGD and defeats defenses tuned to a specific $\epsilon$. | Attack | Order | Strength | Cost | Notes | | --- | --- | --- | --- | --- | | FGSM | first-order, 1 step | weak | one backward pass | the linearized optimum; fast but overshoots | | PGD | first-order, $T$ steps | strong | $T$ backward passes | iterated + projected; the standard benchmark | | CW (Carlini–Wagner) | optimization-based | strongest | many iterations | minimizes $\norm{\delta}$ subject to misclassification; breaks weak defenses | | Defense | Mechanism | Guarantee | Pitfall | | --- | --- | --- | --- | | Adversarial training | train on PGD examples (the min-max) | empirical only | costs clean accuracy; $\epsilon$-specific | | Certified bounds | propagate interval/Lipschitz bounds through layers | provable for the certified $\epsilon$ | loose bounds; limited to small models | | Randomized smoothing | classify under Gaussian noise, take the majority vote | probabilistic $L_2$ certificate | needs many samples; $L_2$ only | | Gradient masking | obfuscate or zero the input-gradient | **none** (false security) | broken by gradient estimation / transfer | ## Obfuscated gradients: a false sense of security For several years after FGSM, a steady stream of defenses reported high robust accuracy against PGD, and almost all of them were later broken.[^obf] The common fault was shared: the defense lowered the _measured_ attack success by corrupting the gradient the attacker reads, not by moving the decision boundary. PGD climbs the loss using $\nabla_x\ell$; if that gradient is shattered, randomized, or saturated, PGD stalls and reports the model as robust while the boundary still sits a perturbation away. This is **obfuscated gradients**, the most common form of gradient masking. It postdates Goodfellow's text, which already names the honest baseline.[^obf-postdate] Obfuscated gradients come in three forms, distinguished by _how_ they corrupt the gradient the attacker differentiates through. | Failure mode | What the defense does | Symptom in evaluation | Adaptive attack that defeats it | | --- | --- | --- | --- | | **Shattered gradient** | inserts a non-differentiable or numerically broken op (quantize, JPEG, discretize) | $\nabla_x\ell$ is zero, wrong, or undefined; PGD makes no progress | **BPDA** — replace the op with a differentiable surrogate on the backward pass | | **Stochastic gradient** | randomizes the input or network (random resize/pad, noise, random routing) | gradients differ every query; single-sample PGD is high-variance and weak | **EOT** — average the gradient over the randomization to attack the expectation | | **Vanishing / exploding** | stacks many sequential steps (deep purification, unrolled optimization) | gradient through the long chain vanishes or explodes; the loss looks flat | reparameterize / shorten the path, or **transfer** from a clean surrogate | > **Definition (Gradient masking).** A defense exhibits gradient masking when it > reduces the success of gradient-based attacks _without_ increasing the distance from > each input to the nearest decision boundary. Equivalently, the reported robustness > rests on the attacker's $\nabla_x\ell$ being uninformative (shattered, stochastic, > or saturated) rather than on the loss surface being genuinely flat inside the > $\epsilon$-ball. Obfuscated gradients are the differentiable-model special case. Three symptoms are diagnostic. A defense is masking the gradient when (i) a one-step attack like FGSM beats iterative PGD, the reverse of the true ordering; (ii) black-box or transfer attacks beat white-box ones, though white-box has strictly more information; or (iii) robust accuracy stays high at large $\epsilon$ where an unbounded adversary should win outright. Any of these signals that the gradient, not the model, is doing the defending. $$ % caption: True robustness versus masked gradient. Left: the boundary is genuinely far. Right: it stays close, but the masked gradient points away, so naive PGD misses. \begin{tikzpicture}[>=stealth, font=\small] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \definecolor{red}{HTML}{C0392B} % ===== LEFT: true robustness ===== \begin{scope} \node[font=\footnotesize, anchor=south] at (0,2.4) {true robustness}; % boundary far from the point \draw[red, very thick] (2.0,-1.9) .. controls (2.4,-0.4) and (2.4,0.4) .. (2.0,1.9); \node[red, font=\footnotesize, anchor=south west] at (2.05,1.6) {boundary}; \node[green, font=\footnotesize] at (-1.2,1.7) {correct}; \node[red, font=\footnotesize] at (3.0,-1.7) {wrong}; % point + ball, both inside correct region \fill[black] (0,0) circle (2.6pt); \node[anchor=north, font=\footnotesize] at (0,-0.2) {clean}; \draw[acc, very thick] (0,0) circle (0.95); \node[acc, font=\footnotesize, anchor=north] at (0,-0.98) {eps-ball}; % gradient points at boundary, attack still lands inside correct region \draw[->, acc, very thick] (0,0) -- (0.85,0.0); \fill[green] (0.85,0.0) circle (2.6pt); \node[green, font=\footnotesize, anchor=west] at (1.08,0.02) {safe}; \end{scope} % ===== RIGHT: masked gradient ===== \begin{scope}[xshift=7.6cm] \node[font=\footnotesize, anchor=south] at (0,2.4) {masked gradient}; % boundary still close \draw[red, very thick] (0.55,-1.9) .. controls (0.85,-0.4) and (0.85,0.4) .. (0.55,1.9); \node[red, font=\footnotesize, anchor=south west] at (0.6,1.6) {boundary}; \node[green, font=\footnotesize] at (-1.4,1.7) {correct}; \node[red, font=\footnotesize] at (2.2,-1.7) {wrong}; \fill[black] (0,0) circle (2.6pt); \node[anchor=north, font=\footnotesize] at (0,-0.2) {clean}; \draw[acc, very thick] (0,0) circle (0.95); % masked gradient points AWAY from the close boundary \draw[->, red, very thick] (0,0) -- (-0.85,0.0); \node[red, font=\footnotesize, anchor=south east] at (-0.6,0.05) {masked grad}; % the true exploit sits across the close boundary \fill[red] (0.8,0.0) circle (2.6pt); \node[red, font=\footnotesize, anchor=south west] at (0.85,0.1) {true exploit}; \end{scope} \end{tikzpicture} $$ ### Adaptive attacks that defeat it To address this, evaluation uses the **adaptive attack**: an attack rebuilt with full knowledge of the defense, so the gradient it follows is the gradient of the _defended_ model, not of a raw surrogate. Two constructions recover a usable gradient from the two ways defenses break it. **BPDA (backward-pass differentiable approximation)** handles a non-differentiable component $g(\cdot)$. Run the true $g$ on the forward pass to get the real prediction, but on the backward pass replace $g$ by a differentiable surrogate $\hat g \approx g$ whose Jacobian is computable. The common surrogate is the identity, $\hat g(x) = x$, valid whenever $g$ is an input purifier with $g(x) \approx x$: $$ \frac{\partial \ell}{\partial x} \;\approx\; \frac{\partial \ell}{\partial g(x)} \cdot \frac{\partial \hat g}{\partial x}, \qquad \frac{\partial \hat g}{\partial x} = I \;\;\text{for}\;\; \hat g = \mathrm{id}. $$ **EOT (expectation over transformation)** handles a stochastic defense that draws a transform $t \sim \mathcal{T}$ each query. A single sample of $\nabla_x \ell(t(x))$ is noisy, but its expectation recovers the gradient of the quantity the defense reports (the expected loss), and gradients commute with the expectation: $$ \nabla_x \,\mathbb{E}_{t \sim \mathcal{T}}\brackets{\ell(t(x), y)} \;=\; \mathbb{E}_{t \sim \mathcal{T}}\brackets{\nabla_x\, \ell(t(x), y)} \;\approx\; \frac{1}{m}\sum_{i=1}^{m} \nabla_x\, \ell(t_i(x), y). $$ Averaging $m$ samples drives down the variance and recovers a descent direction the randomization cannot hide. When neither applies cleanly, a **transfer attack** from a separately trained, fully differentiable surrogate sidesteps the masked gradient entirely, which is why transfer beating white-box is a red flag. ```algorithm caption: $\textsc{BpdaPgd}(x, y, \epsilon, \alpha, T)$ — adaptive attack on a non-diff. defense $x' \gets x + \text{rand}(-\epsilon, \epsilon)$ // random start inside the box for $t \gets 1$ to $T$ do $z \gets g(x')$ // forward pass through the TRUE defense op $h \gets \nabla_z\, \text{loss}(f(z), y)$ // gradient w.r.t. the op output $h \gets h \cdot \nabla_{x'}\hat g(x')$ // backward through the SURROGATE (identity) $x' \gets x' + \alpha \cdot \text{sign}(h)$ // ascent on the recovered gradient $x' \gets x + \text{clip}(x' - x, -\epsilon, \epsilon)$ // project onto eps-box return $x'$ ``` > **Remark (Adaptive evaluation).** A robustness number is only as strong as the > _strongest_ attack it survived, so the attack must be adaptive: built against the > specific defense, with BPDA for shattered gradients, EOT for stochastic ones, and a > transfer baseline as a sanity check. Running stock FGSM or PGD on the raw model > measures nothing about a defense that masks the gradient. The exception is > adversarial training, which raises robust accuracy by genuinely pushing the boundary > out of the $\epsilon$-ball, not by corrupting $\nabla_x\ell$; that is why it remains > the honest baseline adaptive attacks fail to break for free.[^gf-honest] ## The state of the robustness arms race Goodfellow's §7.13 predates almost the entire defense literature, so the modern picture rests on public papers. Two results anchor it. First, the **obfuscated gradients** audit (Athalye, Carlini & Wagner, 2018) took seven defenses accepted at a single conference and broke six of them with BPDA and EOT within weeks — the study that made adaptive evaluation a requirement. Second, the **RobustBench** benchmark (Croce et al., 2021) standardized robustness measurement around AutoAttack, an ensemble of four parameter-free attacks (two PGD variants, the FAB minimum-norm attack, and the gradient-free Square attack) whose diversity makes gradient masking hard to hide behind. Under that yardstick, PGD-based **adversarial training** (Madry et al., 2018) and its refinement **TRADES** (Zhang et al., 2019), which splits the loss into a clean-accuracy term and an explicit robustness regularizer to tune the tradeoff directly, remain the defenses that survive. The honest summary is that no scalable defense yet closes the gap on large-$\epsilon$ $L_\infty$ threat models: certified radii from smoothing stay small, and empirical robust accuracy on hard benchmarks sits well below clean accuracy. The lasting lesson is methodological — robustness is a claim that must survive the strongest _adaptive_ attack, and any number reported against a fixed non-adaptive attack means nothing about a defense engineered to mask its gradient. ## Takeaways - Robustness is a **margin** requirement: the correct region must contain the whole $\epsilon$-ball around $x$, so a defense must push the boundary out, not merely classify $x$ correctly. - **Certified** defenses prove a radius rather than measure one. **Randomized smoothing** classifies under Gaussian noise and certifies an $L_2$ ball of radius $R = \tfrac{\sigma}{2}(\Phi^{-1}(p_A) - \Phi^{-1}(p_B))$ — probabilistic, $L_2$, and paid for in clean accuracy through the blur. - Adversarial examples **transfer** across models with similar near-linear functions, so an attacker with no gradient access attacks a **surrogate** and ships the result; transfer beating white-box is a diagnostic for a masked gradient. - Beware **gradient masking** / **obfuscated gradients** (shattered, stochastic, vanishing): they break the attacker's gradient, not the boundary, and collapse under **adaptive attacks** — BPDA for non-differentiable ops, EOT for randomization, transfer as a baseline. - Every robustness claim needs **adaptive evaluation**; PGD-based adversarial training and TRADES are the defenses that survive standardized benchmarks, and no scalable method yet closes the large-$\epsilon$ gap. [^chollet-manifold]: **Chollet**, _Deep Learning with Python_, Ch. 5 — the manifold view of generalization: a boundary that merely separates the training samples may pass arbitrarily close to them, leaving no robust margin. [^gf-honest]: **Goodfellow**, _Deep Learning_, §7.13 — adversarial training as the honest robustness baseline: it lowers worst-case loss by reshaping the boundary, not by hiding the input-gradient. [^obf]: The obfuscated-gradients result (Athalye, Carlini & Wagner, 2018) postdates **Goodfellow**'s 2016 text: of seven ICLR defenses, six masked the gradient and fell to BPDA/EOT, leaving adversarial training (§7.13) standing. [^obf-postdate]: Gradient masking and adaptive attacks are not in **Goodfellow** (2016), which predates them but already frames adversarial training (§7.13) as the principled defense the later work confirmed.