Adversarial Examples and the Threat Model
Why an imperceptible perturbation flips a confident classifier, what a norm ball is actually assuming, and why stating the threat model precisely is the first substantive step in any robustness claim.
A classifier assigns 99.7 percent confidence to "panda". Add a perturbation with maximum per-pixel change of 2 out of 255, invisible on any screen, and it assigns 99.3 percent to "gibbon". The image did not meaningfully change; the model's decision did. This has been reproducible since 2014 and it remains largely unsolved, which makes it a useful case study in the difference between high accuracy and reliable behaviour.
Why it happens
The early explanation was that models are too non-linear and land in strange pockets. The more durable account is the opposite: models are locally too linear (Goodfellow et al., 2015, Explaining and Harnessing Adversarial Examples, arXiv:1412.6572). In a high-dimensional input space, a small perturbation applied coherently across thousands of dimensions produces a large change in a linear function's output, because the effects add. A per-pixel change of 2 across 150,000 pixels, aligned with the gradient, moves a logit a long way.
A complementary account is that adversarial examples arise from non-robust features: patterns in the data that genuinely predict the label but are imperceptible to humans and are brittle to small changes (Ilyas et al., 2019, arXiv:1905.02175). Under this view the model is not malfunctioning; it is using real signal that happens to be fragile, which reframes robustness as a decision to restrict the model to human-aligned features at some cost in accuracy.
The threat model is the claim
A robustness result without a threat model is not a result. Four axes have to be stated.
The perturbation set. Usually an \(\ell_p\) ball: \(\ell_\infty\) bounds the maximum change to any pixel, \(\ell_2\) bounds the total energy, \(\ell_0\) bounds how many pixels change. These are mathematically convenient proxies for imperceptibility, not definitions of it, and they exclude rotations, translations, colour shifts, compression and physical-world transformations, all of which are realistic attacks that a norm ball says nothing about.
Attacker knowledge. White-box means full access to weights and gradients. Black-box means query access only. Transfer attacks assume neither, relying on adversarial examples generalising across models, which they do to a substantial degree.
Attacker goal. Untargeted means any misclassification. Targeted means a specific wrong class, which is harder and often the one that matters operationally.
The budget. Number of queries, compute, and the size of the perturbation, since a claim of robustness at \(\epsilon = 2/255\) says nothing about \(\epsilon = 8/255\).
When it breaks
Norm balls are a poor model of a real attacker. Nobody defending a face recognition system faces an adversary constrained to \(\ell_\infty\) perturbations. They face printed patterns, adversarial patches, lighting, and physical objects. Robustness within a norm ball is a tractable proxy that the field optimised heavily and it does not transfer to unconstrained attacks.
Most published defences were broken. A large fraction of proposed defences were shown to work only because they obfuscated gradients, making standard attacks fail while stronger adaptive attacks succeeded (Athalye et al., 2018, arXiv:1802.00420). A defence evaluated only against fixed standard attacks has not been evaluated.
Robustness costs clean accuracy. The tradeoff is real and, under the non-robust features account, expected: discarding fragile-but-predictive features loses genuine signal. A defence that claims both is worth examining closely.
The concept transfers to text with different mechanics. Language input is discrete, so imperceptible perturbation is not available in the same form, and the analogous attacks are paraphrases, character substitutions and suffix optimisation. The threat-model discipline transfers even where the norm-ball formalism does not.
12 flashcards for this concept
Click a card to reveal the answer.