Black-Box and Transfer Attacks

In 2016 a team trained a small local network on synthetic inputs labelled by querying image classifiers hosted by MetaMind, Amazon and Google, never seeing a weight or gradient. Adversarial examples crafted against the local copy were then misclassified by the hosted models 84.24%, 96.19% and 88.94% of the time (Papernot et al., 2017, Practical Black-Box Attacks against Machine Learning, AsiaCCS, arXiv:1602.02697).

That result retired the idea that hiding a model behind an API is a defence. The white-box threat model in Adversarial Examples and the Threat Model grants the attacker gradients; black-box attacks show that most of the attack survives without them, at a price paid in queries.

Score-based attacks: estimate the gradient

If the API returns probabilities, the attacker treats the model as a function \(f: \mathbb{R}^d \to \mathbb{R}\), say the margin of the true class, and estimates its gradient by finite differences. The coordinate-wise estimator used by ZOO (Chen et al., 2017, ZOO, AISec, arXiv:1708.03999) is

where \(e_j\) is the \(j\)-th unit vector and \(h\) a small step. One full gradient costs \(2d\) queries. For a \(299 \times 299 \times 3\) ImageNet input, \(d = 268{,}203\), so a single gradient is 536,406 queries; ZOO is practical only through dimension reduction and importance sampling.

Random directions do better than coordinates. Natural evolution strategies estimate

with \(\sigma\) the smoothing radius and \(n\) the number of samples, usually drawn in antithetic pairs \(\pm u_i\) to cut variance. Ilyas et al. used this for a targeted attack on the Google Cloud Vision API under query limits and partial output (Ilyas et al., 2018, Black-box Adversarial Attacks with Limited Queries and Information, ICML, arXiv:1804.08598). Square Attack dropped gradient estimation entirely: random search over localised square updates, keeping any that lower the margin, improved query efficiency by 1.8x to 3x over prior untargeted methods on ImageNet (Andriushchenko et al., 2020, Square Attack, ECCV, arXiv:1912.00049).

Decision-based attacks: walk the boundary

Top-1 labels alone still suffice. The Boundary Attack starts from an input that is already misclassified, possibly by a wide margin, and takes random steps that shrink the distance to the original while staying adversarial (Brendel, Rauber & Bethge, 2018, Decision-Based Adversarial Attacks, ICLR, arXiv:1712.04248). HopSkipJump estimates the gradient direction at the boundary from binary answers alone, \(\widetilde{\nabla} = \frac{1}{B}\sum_{b=1}^{B} \phi(x_{\text{bd}} + \delta u_b)\, u_b\), where \(\phi \in \{-1, +1\}\) records which side each probe landed on (Chen, Jordan & Wainwright, 2019, HopSkipJumpAttack, arXiv:1904.02144).

Transfer: attack a model you never query

Transfer spends no queries on the target at attack time: craft on a surrogate \(\hat f\), submit to \(f\), and rely on both having learned similar non-robust features. Adversarial examples occupy a contiguous subspace of roughly 25 dimensions, and a significant fraction of it is shared between independently trained models (Tramèr et al., 2017, The Space of Transferable Adversarial Examples, arXiv:1704.03453). Untargeted transfer is easy. Targeted transfer, where the wrong label must also match, mostly failed until Liu et al. optimised against an ensemble of surrogates and landed targeted examples on Clarifai's commercial API (Liu et al., 2017, Delving into Transferable Adversarial Examples and Black-box Attacks, ICLR, arXiv:1611.02770).

The recipe crossed into language models. GCG optimises a 20-token suffix \(s\) to maximise the probability of an affirmative prefix \(y^\star\), minimising \(\mathcal{L}(s) = -\log p(y^\star \mid x \oplus s)\). At each of 500 steps it uses gradients with respect to one-hot tokens to shortlist the top 256 substitutions per position, evaluates 512 candidate swaps exactly, and keeps the best: at least 256,000 forward passes on open-weight surrogates, none on the target. Suffixes optimised on an ensemble reached attack success of 86.6% on GPT-3.5, 66.0% on PaLM-2, 47.9% on Claude-1 and 46.9% on GPT-4, but 2.1% on Claude-2 (Zou et al., 2023, Universal and Transferable Adversarial Attacks on Aligned Language Models, arXiv:2307.15043). Score-based search followed: random search on returned log-probabilities reached 100% success on several frontier models, with transfer or prefilling where log-probabilities were hidden (Andriushchenko, Croce & Flammarion, 2025, Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks, ICLR, arXiv:2404.02151). Jailbreaks and Refusal Robustness covers these families from the defender's side.

When it breaks

Transfer rates are not stable across targets. The spread from 86.6% to 2.1% is the rule, not noise. A vendor who blocks a published suffix string defeats that string without touching the weakness.

Defences that stop white-box attacks can leak through transfer. Standard adversarial training resisted single-step white-box attacks yet fell to perturbations transferred from undefended models. Ensemble adversarial training was built to close that gap, and its authors later noted that stronger black-box attacks cut its robustness substantially (Tramèr et al., 2018, Ensemble Adversarial Training, ICLR, arXiv:1705.07204).

Query budgets are a real control, and a partial one. Returning labels instead of scores, hiding log-probs, rate limits and detection of near-duplicate query streams all raise attack cost. None of them touches transfer, which spends its compute offline.

Researchers disagree about how much text changes the picture. Jain et al. argue that discrete optimisation over tokens is weak and expensive enough that simple defences such as perplexity filtering work better for LLMs than their analogues did in vision (Jain et al., 2023, Baseline Defenses for Adversarial Attacks Against Aligned Language Models, arXiv:2309.00614). The adaptive results cut the other way: a filter tuned to gibberish suffixes encodes one threat model, and attackers choose another.

A black-box attack that beats white-box signals a broken evaluation. Gradient access is strictly more information, so a higher black-box number means gradients are masked, the red flag that Robustness Evaluation That Means Something catalogues.