Adversarial Robustness
Perturbation attacks, adversarial training, certified defences, and the robustness-accuracy tradeoff.
12concepts
103flashcards
93minutes of reading
- 01 Adversarial Examples and the Threat Model Why an imperceptible perturbation flips a confident classifier, what a norm ball is actually assuming, and why stating the threat model precisely is the first substantive step in any robustness claim.
- 02 Attack Scaling and the Sampling Budget Why attack success against a language model is a function of how many times the attacker is allowed to try, how many-shot and best-of-N jailbreaking turn that into a predictable power law, and what it does to any robustness number reported at a single budget.
- 03 Layered Defences and the Cost of Safeguards What a classifier layer in front of and behind a model actually buys against universal jailbreaks, measured in success rate, false refusals and compute overhead, and why cascading cheap screens before expensive ones is the engineering that made it deployable.
- 04 Adversarial Training and Certified Defences The min-max formulation behind the only empirical defence that has held up, what a certificate actually guarantees, and the gap between certified and empirical robustness.
- 05 Attacker Cost as the Robustness Metric Why language-model defences cannot be certified and are therefore reported as attacker spend rather than as a bound, what evidence makes such a claim checkable, and the 2025 result that broke twelve defences which had each reported near-zero attack success.
- 06 Black-Box and Transfer Attacks How an attacker with only query access, or none, still produces adversarial inputs by estimating gradients from scores, walking the decision boundary from labels, or crafting on a surrogate and relying on transfer, and why the same logic carries GCG suffixes from open-weight models to closed LLMs.
- 07 Cross-Modal Attack Surfaces Why adding vision or audio to a language model hands back the continuous input space and the gradient attack that came with it, and why the resulting attacks are both stronger and cheaper than their text equivalents.
- 08 Data Poisoning and Backdoors How an attacker who controls a small fraction of training data can install a trigger, why backdoors are nearly invisible to standard evaluation, and what the web-scale training setting makes possible.
- 09 Embedding-Space Attacks and Representation-Level Defences How relaxing the discrete token constraint restores the full strength of gradient attacks, why circuit breakers and latent adversarial training operate on representations instead of inputs, and the evaluation failure that gave one of them a 100% attack success rate.
- 10 Model Extraction and Membership Inference What an attacker can learn from query access alone, why confident outputs leak training set membership, and the defences that trade utility for protection.
- 11 Robustness Evaluation That Means Something Why fixed-attack benchmarks systematically overstate robustness, what an adaptive evaluation requires, and the reporting practices that make a robustness claim checkable.
- 12 Shallow Safety Alignment and Generation-Interface Attacks Why refusal behaviour concentrated in a model's first few output tokens explains prefilling, decoding-parameter and fine-tuning attacks at once, and what deepening alignment past those tokens actually changes.