Robustness Evaluation That Means Something

The history of adversarial defences is a history of evaluation failure. Defences were published with strong numbers, and a later paper with an attack designed specifically against them reduced those numbers to near zero. The methods were not fraudulent; the evaluations were incomplete in a way the field took years to characterise.

Gradient obfuscation

The common failure mode has a name. Many defences do not remove adversarial examples; they make them hard to find by gradient descent, by introducing non-differentiable operations, randomisation, or gradients that vanish or explode. Standard attacks, which rely on gradients, then fail, and the defence appears to work.

Athalye and colleagues identified the signatures (2018, arXiv:1802.00420), and they remain the right checklist. One-step attacks outperforming iterative ones is a red flag, since iterative attacks are strictly stronger when gradients are informative. Black-box attacks outperforming white-box ones is a red flag, since white-box has strictly more information. Unbounded perturbation not reaching 100 percent attack success is a red flag, since a large enough perturbation should turn any image into any class. Increasing the perturbation budget not increasing attack success is a red flag.

Any of these means the attack is failing rather than the defence succeeding.

What an adaptive evaluation requires

The evaluator must know how the defence works and design an attack against it: differentiating through the defence where possible, using backward pass differentiable approximation where the forward pass is non-differentiable, using expectation over transformation where the defence is randomised, and running enough restarts and steps that the attack has genuinely converged.

Standardised ensembles such as AutoAttack, which combine several parameter-free attacks including targeted and black-box variants, exist because bespoke adaptive evaluation is hard to do well and easy to do badly. They are a floor rather than a ceiling: passing AutoAttack is necessary and a determined adaptive attacker can still succeed.

Reporting that permits scrutiny

A robustness claim needs the threat model stated in full, the exact attack configuration including steps, restarts and step size, clean accuracy alongside robust accuracy, results across several perturbation budgets rather than one, and enough detail to reproduce. Leaderboards that fix the threat model and run standardised attacks against submitted models exist precisely because self-reported robustness proved unreliable.

When it breaks

Robustness numbers do not compose. A model robust to \(\ell_\infty\) at one budget is not robust to \(\ell_2\), to a larger budget, to rotations, or to physical patches. Reporting one number invites a general reading of a specific result.

Certified and empirical numbers are not comparable. Certified accuracy is a proven lower bound and empirical robust accuracy is an upper bound on true robustness given the attacks tried. Placing them in the same column of a table compares quantities that differ in kind.

The evaluation itself has a compute budget. A stronger attack with more restarts finds more adversarial examples, so reported robustness depends on how much the evaluator spent attacking. Comparing two papers' numbers without comparing their attack budgets compares two different measurements.

Real deployments face a different distribution. Benchmark robustness on curated test sets says little about behaviour under naturally occurring distribution shift, which is a distinct and often more consequential failure mode than adversarial perturbation.