A Decade of Adversarial Examples: Why Robustness Never Came Free

The perturbation had a maximum size of 0.007 per pixel, the magnitude of the smallest bit of an 8-bit image after GoogLeNet's input conversion. Added to a photograph GoogLeNet labelled "panda" with 57.7 percent confidence, it produced an image labelled "gibbon" with 99.3 percent confidence (Goodfellow et al., 2014, arXiv:1412.6572). No human could see the difference. The network could not see anything else.

As of September 2026, the top entry on the RobustBench CIFAR-10 \(\ell_\infty\) leaderboard classifies 93.68 percent of clean test images correctly and 73.71 percent of images attacked at \(\epsilon = 8/255\) (RobustBench). Its authors trained a WideResNet-94-16 on 300 million unique synthetic images, using more than \(10^{21}\) training FLOPs, and their own scaling laws predict a plateau near 90 percent however much more compute is spent (Bartoldson et al., 2024, arXiv:2404.09349). A standardly trained WideResNet-28-10 on the same leaderboard scores 94.78 percent clean and 0.0 percent robust.

Why this matters: Adversarial robustness is the field's longest-running experiment on what it costs to make a learned system behave under worst-case inputs. The answer is that robustness is always paid for: in compute, clean accuracy, data, and evaluation effort. Every defence that seemed to avoid the bill was later shown to have hidden it, and the same pattern is now replaying for jailbreaks against aligned language models.

TL;DR

• The vulnerability comes from linearity in high dimensions. A sign-aligned perturbation moves a linear score by \(\epsilon \lVert w \rVert_1\), growing with \(d\); random noise moves it by about \(\epsilon \lVert w \rVert_2\), growing with \(\sqrt{d}\). On a CIFAR-10 input that is a 55x amplification.
• Adversarial training is the one empirical defence that survived, and it multiplies training compute. Madry et al.'s wide CIFAR-10 model went from 95.2 percent clean and 3.5 percent under 20-step PGD to 87.3 percent clean and 45.8 percent robust.
• Most "free" defences were measuring their own gradient masking. Seven of nine non-certified white-box defences at ICLR 2018 relied on obfuscated gradients; six were completely circumvented (Athalye et al., 2018).
• Evaluation is a cost too. AutoAttack lowered the reported robust accuracy of all but two of more than 50 published models, by over 10 points for 13 of them.
• The accuracy-robustness tradeoff is provable in a toy model and disputed on real data. Tsipras et al. cap robust accuracy at \(19\delta\) for standard error \(\delta\); Yang et al. find CIFAR-10 classes 0.212 apart in \(\ell_\infty\), seven times the attack radius.
• Certification pays in samples. Randomised smoothing certified 49 percent ImageNet top-1 at \(\ell_2\) radius 0.5 against 67 percent clean, at 110 seconds per image.
• LLM jailbreaks are the same game with a discrete optimiser. GCG suffixes optimised on Vicuna transferred to GPT-3.5 at 87.9 percent success; simple adaptive attacks later hit 100 percent on a model adversarially trained against GCG (Andriushchenko et al., 2024).

At a Glance

flowchart LR
    X["Clean input x"] --> ATK["Attacker maximises loss in the ball"]
    ATK --> XA["Adversarial input"]
    XA --> M["Model under test"]
    ATK -.->|"training data"| AT["Adversarial training"]
    AT --> M
    M --> EV["Adaptive evaluation"]
    EV --> BILL["Cost ledger"]
    BILL --> C1["Compute times steps"]
    BILL --> C2["Clean accuracy lost"]
    BILL --> C3["Extra or synthetic data"]
    BILL --> C4["Attack effort to verify"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    class X blue
    class ATK,XA rose
    class M,AT purple
    class EV teal
    class BILL,C1,C2,C3,C4 amber

Each amber box has, at some point in the past decade, been claimed to be zero. None stayed zero.

Before Adversarial Training

The discovery was a side note. In December 2013 Szegedy and colleagues reported that networks "learn input-output mappings that are fairly discontinuous to a significant extent" (Szegedy et al., 2013, arXiv:1312.6199). Their perturbations, found by a slow per-image L-BFGS search, frequently fooled networks trained on different data. That transfer was the ominous part: an attacker did not need your model.

Szegedy et al. suspected non-linearity. Goodfellow et al. argued the opposite: models are too linear. If the loss is locally linear, the worst \(\ell_\infty\) perturbation is the gradient's sign. The fast gradient sign method (FGSM) drove a maxout network on MNIST to 89.4 percent error at \(\epsilon = 0.25\); training on FGSM examples cut that to 17.9 percent, though its remaining errors carried 81.4 percent average confidence.

Defensive distillation cut the success of the attacks of the day from 95 percent to 0.5 percent; Carlini and Wagner's optimisation attacks then succeeded on distilled networks with 100 percent probability (Carlini & Wagner, 2017, arXiv:1608.04644). A defence is only as good as the strongest attack run against it: the field's organising principle, and its most ignored.

timeline
    title Adversarial robustness, 2013 to 2024
    2013 : Szegedy et al. find imperceptible perturbations
         : Perturbations transfer across networks
    2014 : Goodfellow et al. propose FGSM and the linearity view
    2017 : Carlini and Wagner break defensive distillation
         : Madry et al. frame robustness as min-max with PGD
    2018 : Athalye et al. show 7 of 9 ICLR defences obfuscate gradients
    2019 : TRADES prices the accuracy-robustness exchange
         : Ilyas et al. recast adversarial examples as features
         : Cohen et al. certify ImageNet with randomised smoothing
    2020 : AutoAttack and RobustBench standardise evaluation
    2023 : GCG suffixes transfer to commercial chat models
    2024 : Scaling laws predict a CIFAR-10 plateau near 90 percent
         : Simple adaptive attacks reach 100 percent on aligned LLMs

[IMAGE: Three-panel panda figure. Left, original labelled "panda, 57.7 percent"; centre, the sign-of-gradient perturbation amplified 50x, labelled "epsilon = 0.007"; right, "gibbon, 99.3 percent". Caption: "Every pixel moved by the same tiny amount in the direction the gradient pointed."]

How Robustness Is Bought

Why linear models in high dimensions are fragile

Take a linear score \(z = w^\top x + b\) and a perturbation \(\eta\) with \(\lVert \eta \rVert_\infty \le \epsilon\). The score changes by \(w^\top \eta\), minimised by \(\eta = -\epsilon \, \text{sign}(w)\):

The operative word is sum: if each weight has magnitude around \(a\), the worst-case shift is \(\epsilon d a\), linear in dimension. Random signs of the same size give zero mean shift and standard deviation \(\epsilon \lVert w \rVert_2 = \epsilon a \sqrt{d}\), so the ratio is

For CIFAR-10, \(d = 3072\) and \(\sqrt{d} \approx 55\); for ImageNet at 224 pixels, about 388. A change far below perception accumulates across thousands of coordinates. ReLU networks are piecewise linear, so they inherit this. It also predicts transfer: models trained on the same data learn similar weight directions.

From one step to PGD, and robustness as a saddle point

FGSM takes one step, \(x' = x + \epsilon \, \text{sign}(\nabla_x \mathcal{L})\), optimal for a linear model. Projected gradient descent (PGD) iterates with a smaller step \(\alpha\) and projects back after each:

\(\mathcal{B}_\epsilon(x)\) is the \(\ell_\infty\) ball around the clean input, and \(\Pi\) is coordinate-wise clipping to the ball and the pixel range. A random start inside the ball matters, because the gradient at the clean point can be deliberately uninformative.

Madry et al. turned a bag of tricks into a research programme by writing training as robust optimisation (Madry et al., 2018, arXiv:1706.06083):

The inner max is the attacker, the outer min the defender. Danskin's theorem gives the recipe: at a maximiser \(\delta^*\), the gradient of the inner max with respect to \(\theta\) is the loss gradient at \(x + \delta^*\). So find \(\delta^*\) with PGD and take an ordinary SGD step on the perturbed example.

Now the bill. Madry's CIFAR-10 models trained against 7 PGD steps, so each example costs about eight forward-backward passes instead of one. Robustness also needed capacity, the 10x-wide ResNet, and clean accuracy, as their Table 5 shows:

The middle row is the most instructive line in the field's history. The FGSM-trained model scored higher against FGSM than on clean data, and zero against PGD. It had learned to bend its loss surface so the gradient at \(x\) points somewhere harmless while a few steps away the loss is still enormous. Cheap robustness had been measured with a cheap attack.

Gradient masking

Athalye, Carlini and Wagner named this obfuscated gradients (Athalye et al., 2018, arXiv:1802.00420): shattered gradients from non-differentiable preprocessing such as quantisation, stochastic gradients from randomised inference-time transforms, and exploding or vanishing gradients from unrolled purification loops. Each has a countermeasure. Backward Pass Differentiable Approximation (BPDA) swaps a non-differentiable component for a differentiable surrogate, often the identity, on the backward pass only; Expectation over Transformation (EOT) attacks the average gradient over the defence's randomness.

Of nine non-certified white-box defences at ICLR 2018, seven depended on obfuscated gradients; six fell completely and one partially. Madry's adversarial training, attacked with the same tools at \(\epsilon = 0.031\), held at 47 percent. The defence that did not claim to be free survived. The paper's symptoms remain the fastest sanity check: if one-step attacks beat iterative ones, or black-box beats white-box, the attack is broken, not the model.

TRADES: pricing the exchange rate

Zhang et al. split robust error into natural error plus boundary error, the mass of correctly classified points within \(\epsilon\) of the decision boundary, and optimised a surrogate (Zhang et al., 2019, arXiv:1901.08573):

The second term penalises the prediction changing inside the ball, whatever the label. \(\beta\) (written \(1/\lambda\) in the paper) is the exchange rate, and their CIFAR-10 sweep shows it paid both ways: 91.31 percent clean and 26.53 robust at \(1/\lambda = 0.1\), 87.01 and 44.68 at 1.0, 81.72 and 50.64 at 5.0. Past the knee, every robustness point costs more than a point of clean accuracy.

Why the tradeoff exists, and whether it must

Tsipras et al. built a distribution where the tension is provable (Tsipras et al., 2019, arXiv:1805.12152). One feature equals the label with probability \(p\); \(d\) more are drawn from \(\mathcal{N}(\eta y, 1)\) with small \(\eta\), individually useless but collectively enough for near-perfect accuracy. An adversary with \(\epsilon \ge 2\eta\) flips all the weak features at once. Any classifier with standard accuracy at least \(1 - \delta\) then has robust accuracy at most

With \(p = 0.95\) the bound is \(19\delta\): a 99 percent accurate classifier can be at most 19 percent robust, because accuracy requires the weak features the adversary controls.

Ilyas et al. tested this on real data (Ilyas et al., 2019, arXiv:1905.02175). They perturbed every training image toward a target class and relabelled it as that target. With targets from a fixed class permutation, so robust features pointed to the wrong label, a ResNet-50 trained on the set still reached 43.7 percent on the real CIFAR-10 test set. The perturbations carried features that generalise; a model that ignores them discards real signal.

Credible people disagree. In a Distill discussion of six commentaries (Engstrom et al., 2019), Preetum Nakkiran constructed adversarial examples that neither transfer nor carry generalisable features, and the authors accepted such "bugs" exist. Separately, Yang et al. measured real datasets (Yang et al., 2020, arXiv:2003.02460): the minimum \(\ell_\infty\) distance between CIFAR-10 training images of different classes is 0.212, against an attack radius of 0.031, so a classifier both perfectly accurate and perfectly robust at that radius exists, and they prove it. Both positions hold at once: a robust, accurate classifier exists, and nobody has found an affordable way to learn it.

[IMAGE: Two panels. Left, Tsipras toy model: robust feature on x, mean of weak features on y, an epsilon-box that crosses the weak-feature boundary only. Right, histogram of nearest other-class L-infinity distances on CIFAR-10, lines at 2 epsilon = 0.062 and minimum 0.212. Caption: "Provable in a constructed distribution, absent in principle on real data. The gap is the cost of learning."]

Certification: a proof, paid for in samples

Empirical robustness is always conditional on the attack. Randomised smoothing removes the condition (Cohen et al., 2019, arXiv:1902.02918). For any base classifier \(f\), define

If the top class has noisy probability at least \(\underline{p_A}\) and the runner-up at most \(\overline{p_B}\), then \(g\) is constant within an \(\ell_2\) ball of radius

with \(\Phi^{-1}\) the inverse standard normal CDF. It works for any \(f\), so it scaled to ImageNet. The costs: \(\underline{p_A}\) is a Monte Carlo confidence bound, so Cohen et al. used 100,000 samples per certification, 110 seconds per ImageNet image on a ResNet-50. Noise tolerance costs clean accuracy: 49 percent certified at radius 0.5 against 67 clean, 19 at radius 2.0 against 57. And the guarantee is \(\ell_2\) only.

Seeing It in Motion

flowchart TB
    subgraph Inner["Inner maximisation, per example"]
        R0["Random start in the ball"] --> G1["Input gradient of loss"]
        G1 --> S1["Signed step of size alpha"]
        S1 --> P1["Clip to ball and pixel range"]
        P1 -->|"repeat K times"| G1
    end
    subgraph Outer["Outer minimisation"]
        L["Loss at adversarial point"] --> U["SGD update of weights"]
    end
    D["Minibatch x, y"] --> R0
    P1 --> L
    U --> D
    U --> E["Evaluate with AutoAttack"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    class D blue
    class R0,G1,S1,P1 amber
    class L,U purple
    class E teal

The amber loop runs \(K\) times per example per epoch. Wong, Rice and Kolter attacked that cost in "Fast is better than free" (Wong et al., 2020, arXiv:2001.03994): FGSM training with a random start reached 45 percent PGD robustness on CIFAR-10 in 6 minutes, against 80 hours they cite for PGD training. The price was catastrophic overfitting: single-step training can look healthy for many epochs, then collapse to near-zero PGD accuracy within one, Madry's FGSM row again. Compute was cut; risk moved into training dynamics.

sequenceDiagram
    participant Def as Defence author
    participant AA as AutoAttack
    participant Ad as Adaptive attacker
    participant LB as Leaderboard
    Def->>LB: Submit PGD robust accuracy
    LB->>AA: Run APGD-CE, APGD-T, FAB-T, Square
    AA-->>LB: Lower number
    LB->>Ad: Flag possible gradient masking
    Ad->>Ad: BPDA on blocking parts, EOT on randomness
    Ad-->>LB: Best known robust accuracy
    Note over Def,LB: The published figure is the minimum over all attacks tried

Croce and Hein built AutoAttack because hand-tuned PGD fails predictably: fixed step sizes and a saturating cross-entropy loss (Croce & Hein, 2020, arXiv:2003.01690). Auto-PGD adapts its step size, a targeted variant uses a scale-invariant loss, and the black-box Square Attack covers what gradients miss. Madry's released model fell from a reported 47.04 to 44.04 percent. RobustBench made AutoAttack the leaderboard floor and invited adaptive evaluations (Croce et al., 2021, arXiv:2010.09670). The invitation is still used: the second-ranked CIFAR-10 entry, a post-training "MeanSparse" operator on the top model, lists 75.28 percent under AutoAttack but 73.10 under APGD with BPDA, below the base model's 73.71.

[IMAGE: Waterfall chart of robust accuracy as evaluation strengthens, "paper PGD" to "AutoAttack" to "adaptive BPDA". Madry's model steps 47.04 to 44.04; MeanSparse steps 75.28 to 73.10, with a dashed line at the base model's 73.71. Caption: "Robust accuracy only goes down as the evaluation improves."]

By the Numbers

Sources: Madry et al. Table 5; Cohen et al. Table 1; Croce & Hein; Bartoldson et al.; Zou et al.; Andriushchenko et al. TRADES, Gowal (arXiv:2010.03593) and Bartoldson figures are from the RobustBench CIFAR-10 \(\ell_\infty\) leaderboard, September 2026, at \(\epsilon = 8/255\). LLM attack success rates depend on the judge and are not comparable to image robust accuracy.

Robust accuracy climbs slowly, 45.8 to 53 to 66 to 74; cost climbs from one gradient step to 300 million synthetic images. Bartoldson et al. find the frontier inefficient (they match the prior best with 20 percent fewer FLOPs) yet not closable by scale: humans score near 90 percent on the AutoAttack images, because some \(8/255\) perturbations genuinely change what the image shows.

[IMAGE: Log-scale scatter, x-axis training FLOPs 10^17 to 10^30, y-axis CIFAR-10 AutoAttack accuracy. Points for Madry 2017, TRADES 2019, Gowal 2020, Bartoldson 2024; a saturating curve approaching a dashed line at 90 percent, "estimated human accuracy". Mark 10^30 "extrapolated" and non-Bartoldson x positions "approximate". Caption: "Each robustness point costs more compute than the last."]

A Concrete Example

A four-feature logistic regression shows every cost above with numbers you can check by hand.

Setup. Weights \(w = [3, -2, 1, -4]\), bias \(b = 0.5\), predict class 1 when \(z = w^\top x + b > 0\). Clean input \(x = [0.6, 0.2, 0.5, 0.1]\), label \(y = 1\), features in \([0, 1]\).

Step 1: Clean. \(z = 1.8 - 0.4 + 0.5 - 0.4 + 0.5 = 2.0\), so \(p = \sigma(2.0) = 0.881\). Correct.

Step 2: Gradient. \(\nabla_x \mathcal{L} = (p - y)\,w = -0.119 \times [3, -2, 1, -4] = [-0.358, 0.238, -0.119, 0.477]\). Its sign, \([-1, +1, -1, +1]\), is \(-\text{sign}(w)\). FGSM uses only the signs.

Step 3: FGSM at \(\epsilon = 0.1\). \(x' = [0.5, 0.3, 0.4, 0.2]\), \(z' = 1.5 - 0.6 + 0.4 - 0.8 + 0.5 = 1.0\), \(p' = 0.731\). The formula agrees: \(\Delta z = -0.1 \times \lVert w \rVert_1 = -0.1 \times 10 = -1.0\). Margin halved.

Step 4: FGSM at \(\epsilon = 0.25\). \(x' = [0.35, 0.45, 0.25, 0.35]\), \(z' = 1.05 - 0.9 + 0.25 - 1.4 + 0.5 = -0.5\), \(p' = 0.378\). Flipped. The smallest flipping budget is \(z / \lVert w \rVert_1 = 0.2\).

Step 5: PGD adds nothing, which is the point. With \(\alpha = 0.15\), step one reaches \([0.45, 0.35, 0.35, 0.25]\) and \(z = 0.5\). Step two's raw point \([0.30, 0.50, 0.20, 0.40]\) is \(\pm 0.30\) from \(x\); clipping to \(\pm 0.25\) lands on the FGSM corner. A deep network's loss surface bends, and can learn to bend on purpose: Madry's FGSM row.

Step 6: Random noise. Of the 16 sign patterns at \(\epsilon = 0.25\), only the FGSM one pushes \(z\) below zero. Scale up: \(d = 3072\), every \(|w_i| = 0.01\), \(\epsilon = 0.0314\). The adversarial shift is \(0.0314 \times 30.72 = 0.96\); the random shift's standard deviation is \(0.0314 \times 0.01 \times \sqrt{3072} = 0.017\), 55 times smaller.

Step 7: Buy robustness by dropping features. Zero the two weakest weights: \(w_r = [3, 0, 0, -4]\). Now \(z = 1.9\) and \(\lVert w_r \rVert_1 = 7\), so the worst case at \(\epsilon = 0.25\) is \(1.9 - 1.75 = 0.15\). Robust.

Step 8: Pay for it. A second class-1 input \(x_2 = [0.3, 0.0, 0.9, 0.38]\) keeps its evidence in feature 3. The original model gives \(z = 0.9 + 0.9 - 1.52 + 0.5 = 0.78\), correct. The robust model gives \(z = 0.9 - 1.52 + 0.5 = -0.12\), wrong. Robustness came from refusing weak, attacker-controllable evidence, and this example needed exactly that evidence: the Tsipras tradeoff in four dimensions.

[IMAGE: Four panels: bars of z for clean (2.0), FGSM 0.1 (1.0) and FGSM 0.25 (-0.5); a 2D slice of the box with PGD hitting the clipped corner; z shifts for all 16 sign patterns; z for x and x2 under w and w_r. Caption: "One linear model: the attack, the corner, the noise baseline, and the accuracy robustness costs."]

Where It Breaks

Robust accuracy describes the attack, not the model

Empirical robust accuracy is an upper bound, as tight as the strongest attack run. The usual failure is mundane: a step size tuned for another model, a saturating loss, a component returning zero gradients. Tramer et al. circumvented thirteen defences that had already performed adaptive evaluations (arXiv:2002.08347), and MeanSparse shows the failure still reaches the top of the leaderboard.

Robustness does not generalise the way accuracy does

Robust test accuracy peaks and then falls while robust training accuracy keeps climbing; Rice, Wong and Kolter found early stopping alone matched most algorithmic improvements on PGD training (arXiv:2002.11569). Carmon et al. showed 500,000 unlabelled images added over 5 points (arXiv:1905.13736). The leaderboard's upper half is a list of data strategies because robust generalisation is data-hungry.

The threat model is too narrow and too wide

Training at \(\ell_\infty\), \(8/255\) says nothing about \(\ell_2\), \(16/255\), rotations or patches. Yet at \(8/255\) some "adversarial" CIFAR-10 images have genuinely changed class, per Bartoldson et al.'s human study, so the ball also contains inputs the model should not be robust on.

The LLM version replays the vision playbook

Text is discrete, so PGD does not apply directly. Greedy Coordinate Gradient (GCG) takes the gradient with respect to one-hot token indicators, proposes the top 256 substitutions per position, evaluates 512 candidate swaps exactly, and keeps the best, for 500 steps over a 20-token suffix (Zou et al., 2023, arXiv:2307.15043). It is FGSM's logic in a combinatorial space, with Szegedy's transfer property intact.

Jain et al. found perplexity filters catch GCG's gibberish and that weak discrete optimisers make adaptive attacks harder than in vision, while leaving open whether better optimisers would erase the gap (Jain et al., 2023, arXiv:2309.00614). History predicted the answer: Andriushchenko et al. reached 100 percent attack success on Llama-2-Chat, GPT-4o and R2D2, a model adversarially trained against GCG, with a template plus random search on log-probabilities, and 100 percent on Claude models via transfer or prefilling. That is ICLR 2018 with different nouns.

[IMAGE: Two-column analogy diagram. "Images 2014-2020": FGSM, PGD, gradient masking, BPDA/EOT, AutoAttack, adversarial training. "LLMs 2023-2025": GCG, multi-restart GCG, perplexity filter, random-search attacks, adaptive suites, training against GCG. Arrows join rows. Caption: "Jailbreak research is replaying image robustness at roughly three times the speed."]

Alternative Designs

Cheap designs' limitations all say "fails under adaptive attack"; the durable ones' all say "expensive". Nothing here is both cheap and robust against a white-box adaptive attacker.

How It Is Used in Practice

Robustness matters most as an evaluation discipline. A team facing motivated adversaries (content moderation, identity verification, malware classification) gains more from running AutoAttack plus a small adaptive budget before launch than from any defence not evaluated that way. Teams that do adversarially train should budget several times standard compute, select checkpoints on robust accuracy to dodge robust overfitting, and report clean and robust accuracy per class, since the clean-accuracy loss is rarely spread evenly.

For language models, practice has converged on layers: adversarial safety data, input and output classifiers, perplexity filters, log-probability limits, and adaptive red-teaming. No layer alone is evidence of robustness. Andriushchenko et al. note that prefilling and log-probability exposure are attack surface, which makes product decisions part of the threat model.

Insights Worth Remembering

1. Adversarial vulnerability is a consequence of dimension, not a defect of deep learning. The \(\sqrt{d}\) gap applies to logistic regression as much as to a ResNet; no architecture tweak removes it.

2. A cheap defence that works is usually a cheap attack that failed. FGSM training at 95.1 percent against FGSM and 0.0 against PGD is the template. When robustness arrives without its usual costs, suspect the measurement.

3. Robust accuracy only ever goes down. It is a minimum over attacks, so treat any figure as provisional until an adaptive attacker has tried to lower it.

4. The tradeoff is real in practice and unproven as a law. Tsipras shows how it can be forced; Yang shows real data does not force it. Learning a robust and accurate classifier is currently too expensive, not impossible.

5. The last five years of gains came from data, not algorithms. The leaderboard, from 500,000 unlabelled images to 300 million synthetic ones, is a record of sample complexity.

6. Jailbreak defence is repeating image defence faster. Discrete optimisation delayed the adaptive attacker by about a year; a defence trained against GCG and broken by random search is obfuscated gradients again.

Open Questions

Is the roughly 90 percent CIFAR-10 ceiling a property of the task or the threat model? Bartoldson et al. measured human accuracy near 90 percent on a small sample of fooling images, matching their scaling-law plateau. Whether a perception-aligned threat model would raise the ceiling is untested at scale.

Can robust generalisation be made data-efficient? Robust learning is shown to need more samples. Whether an algorithm can close the gap without the data is unknown.

Will fluency-aware optimisers erase LLM filters? Jain et al. measured filters holding up better than in vision; the 2024 results suggest the gap is closing. Survival against attacks optimised for fluency is open.

Can certification reach radii that matter for deployed models? Smoothing reached ImageNet at modest \(\ell_2\) radii. Guarantees for billion-parameter models or token inputs are speculative; no established result exists.

Sources and Further Reading

1. Szegedy, C., et al. (2013). "Intriguing properties of neural networks." arXiv:1312.6199
2. Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014). "Explaining and Harnessing Adversarial Examples." ICLR 2015. arXiv:1412.6572
3. Carlini, N., & Wagner, D. (2017). "Towards Evaluating the Robustness of Neural Networks." IEEE S&P. arXiv:1608.04644
4. Madry, A., et al. (2018). "Towards Deep Learning Models Resistant to Adversarial Attacks." ICLR 2018. arXiv:1706.06083
5. Athalye, A., Carlini, N., & Wagner, D. (2018). "Obfuscated Gradients Give a False Sense of Security." ICML 2018. arXiv:1802.00420
6. Tsipras, D., et al. (2019). "Robustness May Be at Odds with Accuracy." ICLR 2019. arXiv:1805.12152
7. Zhang, H., et al. (2019). "Theoretically Principled Trade-off between Robustness and Accuracy." ICML 2019. arXiv:1901.08573
8. Ilyas, A., et al. (2019). "Adversarial Examples Are Not Bugs, They Are Features." NeurIPS 2019. arXiv:1905.02175
9. Engstrom, L., et al. (2019). "A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features'." Distill. doi:10.23915/distill.00019
10. Cohen, J. M., Rosenfeld, E., & Kolter, J. Z. (2019). "Certified Adversarial Robustness via Randomized Smoothing." ICML 2019. arXiv:1902.02918
11. Yang, Y.-Y., et al. (2020). "A Closer Look at Accuracy vs. Robustness." NeurIPS 2020. arXiv:2003.02460
12. Wong, E., Rice, L., & Kolter, J. Z. (2020). "Fast is better than free: Revisiting adversarial training." ICLR 2020. arXiv:2001.03994
13. Croce, F., & Hein, M. (2020). "Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks." ICML 2020. arXiv:2003.01690
14. Croce, F., et al. (2021). "RobustBench: a standardized adversarial robustness benchmark." NeurIPS Datasets and Benchmarks. arXiv:2010.09670; leaderboard
15. Bartoldson, B. R., Diffenderfer, J., Parasyris, K., & Kailkhura, B. (2024). "Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies." ICML 2024. arXiv:2404.09349
16. Zou, A., et al. (2023). "Universal and Transferable Adversarial Attacks on Aligned Language Models." arXiv:2307.15043
17. Jain, N., et al. (2023). "Baseline Defenses for Adversarial Attacks Against Aligned Language Models." arXiv:2309.00614
18. Andriushchenko, M., Croce, F., & Flammarion, N. (2024). "Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks." ICLR 2025. arXiv:2404.02151

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.