Concept library
1134 concepts across 20 domains and 101 tracks. Each track is a coherent sequence — read it top to bottom or dip in wherever the gap is.
All domains
01Foundations
02Transformer Internals
03Training & Fine-Tuning
04Reinforcement Learning
05Inference, Systems & Hardware
06Applied LLM Engineering
07Reasoning, Evaluation & Safety
08Multimodal & Applications
09Classical ML & Statistical Learning
10Causal Inference & Experimentation
11Time Series & Forecasting
12Graphs, Recommenders & Structured Data
13Generative Modelling Beyond Transformers
14Efficiency, Compression & Edge AI
15Search & Information Retrieval
16Data & Feature Engineering
17MLOps & Platform Engineering
18Security, Privacy & Adversarial ML
19Governance, Risk & Responsible AI
20Human-AI Interaction, Product & Economics
18
Security, Privacy & Adversarial ML
Attacks on models, data and the supply chain, and the defences that survive contact.
5tracks
53concepts
485cards
6.6hreading
Adversarial Robustness Perturbation attacks, adversarial training, certified defences, and the robustness-accuracy tradeoff. 12 concepts · 103 cards
- 01 Adversarial Examples and the Threat Model Why an imperceptible perturbation flips a confident classifier, what a norm ball is actually assuming, and why stating the threat model precisely is the first substantive step in any robustness claim.
- 02 Attack Scaling and the Sampling Budget Why attack success against a language model is a function of how many times the attacker is allowed to try, how many-shot and best-of-N jailbreaking turn that into a predictable power law, and what it does to any robustness number reported at a single budget.
- 03 Layered Defences and the Cost of Safeguards What a classifier layer in front of and behind a model actually buys against universal jailbreaks, measured in success rate, false refusals and compute overhead, and why cascading cheap screens before expensive ones is the engineering that made it deployable.
- 04 Adversarial Training and Certified Defences The min-max formulation behind the only empirical defence that has held up, what a certificate actually guarantees, and the gap between certified and empirical robustness.
- 05 Attacker Cost as the Robustness Metric Why language-model defences cannot be certified and are therefore reported as attacker spend rather than as a bound, what evidence makes such a claim checkable, and the 2025 result that broke twelve defences which had each reported near-zero attack success.
- 06 Black-Box and Transfer Attacks How an attacker with only query access, or none, still produces adversarial inputs by estimating gradients from scores, walking the decision boundary from labels, or crafting on a surrogate and relying on transfer, and why the same logic carries GCG suffixes from open-weight models to closed LLMs.
- 07 Cross-Modal Attack Surfaces Why adding vision or audio to a language model hands back the continuous input space and the gradient attack that came with it, and why the resulting attacks are both stronger and cheaper than their text equivalents.
- 08 Data Poisoning and Backdoors How an attacker who controls a small fraction of training data can install a trigger, why backdoors are nearly invisible to standard evaluation, and what the web-scale training setting makes possible.
- 09 Embedding-Space Attacks and Representation-Level Defences How relaxing the discrete token constraint restores the full strength of gradient attacks, why circuit breakers and latent adversarial training operate on representations instead of inputs, and the evaluation failure that gave one of them a 100% attack success rate.
- 10 Model Extraction and Membership Inference What an attacker can learn from query access alone, why confident outputs leak training set membership, and the defences that trade utility for protection.
- 11 Robustness Evaluation That Means Something Why fixed-attack benchmarks systematically overstate robustness, what an adaptive evaluation requires, and the reporting practices that make a robustness claim checkable.
- 12 Shallow Safety Alignment and Generation-Interface Attacks Why refusal behaviour concentrated in a model's first few output tokens explains prefilling, decoding-parameter and fine-tuning attacks at once, and what deepening alignment past those tokens actually changes.
Privacy-Preserving ML Differential privacy accounting, federated learning, secure aggregation, and the utility cost of each guarantee. 12 concepts · 110 cards
- 01 PII Detection and Redaction Pipelines Why regular expressions catch the easy half, how the recall-utility tradeoff differs between training corpora and live traffic, and the re-identification risk that survives field-level redaction.
- 02 k-Anonymity and Re-identification Why removing names does not anonymise a table, what k-anonymity guarantees and the attacks it does not stop, how the Netflix Prize data was de-anonymised from a handful of ratings, and why differential privacy replaced syntactic anonymity as the formal standard.
- 03 Attestation-Gated Key Release The deployment pattern that turns an attestation token into an enforceable control, where a key manager releases model weights or a data key only to an enclave whose measurements match policy, plus the revocation, upgrade and trust-concentration problems the pattern creates.
- 04 Confidential Computing for Model Inference How a CPU trusted execution environment paired with a GPU in confidential mode keeps prompts, KV cache and weights encrypted while they are being computed on, why the cost lands on the host-to-device transfer rather than the matmuls, and what the measured overhead is on Hopper and Blackwell.
- 05 Differential Privacy and the Epsilon You Actually Get What the differential privacy guarantee says precisely, how DP-SGD achieves it through clipping and noise, and why the epsilon values used in practice permit far more leakage than the formalism's reputation suggests.
- 06 Federated Learning and Secure Aggregation Why keeping data on device is not by itself a privacy guarantee, how gradient inversion recovers training inputs, and what secure aggregation does and does not prevent.
- 07 Machine Unlearning What it would mean to remove a training example's influence from a model, why retraining is the only exact method, and the approximate approaches and their verification problem.
- 08 Remote Attestation for AI Workloads What an attestation report actually contains, how a verifier turns raw evidence into a signed token by comparing measurements against reference values, why freshness and the composition of CPU and GPU evidence are where implementations go wrong, and the specific claims attestation cannot make.
- 09 Secure Multiparty Computation and Homomorphic Encryption for ML How secret sharing and homomorphic encryption let a server run a model on inputs it cannot read, why non-linear layers dominate the bill, and the measured overheads, from seconds for ResNet-50 to minutes per token for a 7B LLM, that decide where cryptographic inference is actually deployed.
- 10 Side Channels Against Confidential Inference The attack classes that survive memory encryption, from ciphertext side channels and single-stepping on confidential VMs to GPU power, interconnect and PCIe leakage, and what the repeated disclosure cycle implies for how much weight an attestation should carry.
- 11 Synthetic Data and What It Does Not Protect Why generating data from a model trained on real records does not by itself anonymise anything, the utility-privacy frontier that no generator escapes, and the evaluation that a synthetic dataset needs.
- 12 Verifiable Transparency and Non-Targetability Why attestation plus encryption still leaves a provider able to ship one bad build or single out one user, and how transparency logs, client-side enforcement, stateless nodes and anonymising relays turn a privacy promise into a property an outsider can check.
LLM Application Security Injection across trust boundaries, tool and sandbox escape, secret exposure and threat modelling for agents. 12 concepts · 108 cards
- 01 Agent Identity and Workload Credentials Why an agent is a third party rather than a proxy for the user, what goes wrong when it authenticates with a shared long-lived key, and how workload identity replaces that key with an attested, short-lived credential that names the running process.
- 02 Human-in-the-Loop Authorisation Design Where to put the human in an agent's action path so that approval remains a real decision, why per-action prompts decay into rubber-stamping, and how budgets, reversibility tiers and batched review preserve the signal.
- 03 Secrets and Credentials in LLM Context Why anything placed in a model's context should be treated as disclosed to whoever can influence its output, how credential brokering keeps secrets out of the prompt entirely, and where secrets arrive uninvited through users, logs and indexed tools.
- 04 Agent Permissions and Blast Radius How to design tool access for a component you must assume is compromised, why capability scoping beats identity-based permissions here, and the reversibility test that decides what needs confirmation.
- 05 Audience Binding and Token Exchange Why a bearer token that does not name its destination becomes a skeleton key the moment an agent passes it along, and the OAuth mechanisms — resource indicators, token exchange, protected resource metadata — that MCP now requires to stop it.
- 06 Delegated Authority and the Confused Deputy The 1988 result that a privileged intermediary acting on ambiguous instructions will misuse its own authority, why every tool-using agent is that intermediary, and the capability discipline that fixes it where filtering cannot.
- 07 Output Handling and Downstream Injection Why model output is untrusted input to everything it touches, the injection classes that follow from rendering or executing it, and the encoding discipline that prevents them.
- 08 Prompt Injection as a Trust Boundary Failure Why injection is architectural rather than a prompting problem, the distinction between direct and indirect injection that decides severity, and why no known technique closes it.
- 09 RAG Poisoning and Retrieval Trust How a handful of crafted passages in a corpus of millions can control what a RAG system says, why detection-based defences have fallen short, and why the durable controls are provenance on the write path and aggregation that bounds any single passage's influence.
- 10 Red-Teaming and Security Evaluation for LLM Apps What automated attack generation covers, why a pass rate is not a security property against an adversary who retries, and how to structure an evaluation that informs a decision.
- 11 Signed Mandates and Agent Accountability Why logs written by the agent cannot settle a dispute about what a user authorised, and how cryptographically signed mandates plus a hash-chained action record turn "the agent did it" into evidence someone can be held to.
- 12 Threat Modelling an LLM System The trust boundaries specific to model-based applications, an inventory of assets and adversaries worth enumerating, and how to turn that into controls rather than a document.
Model Provenance & Watermarking Output watermarking, content credentials, fingerprinting weights and detecting extraction. 6 concepts · 65 cards
- 01 Content Credentials and Provenance Metadata The complementary approach of signing assertions about how content was made, why cryptographic provenance is strong where watermarking is weak, and the stripping problem that limits it.
- 02 Disclosure Obligations for Generated Content What the emerging transparency rules actually require, why machine-readable marking and human-visible disclosure are separate obligations, and the design decisions a deployer has to make.
- 03 Detecting Synthetic Media Without Watermarks Why passive detectors work in the lab and fail in deployment, the base rate problem that makes accusation dangerous, and what the evidence supports doing instead.
- 04 Invisible Image Watermarking and Its Robustness How learned image watermarks hide a detectable signal in pixels, in decoder weights or in the initial diffusion noise, how the detection threshold sets the false-positive rate, and why regeneration attacks strip most of them at little cost to image quality.
- 05 Model Fingerprinting and Weight Attribution How to prove a deployed model was derived from yours, the difference between backdoor-style and intrinsic fingerprints, and why fine-tuning is the adversary that matters.
- 06 Text Watermarking and the Detectability Tradeoff How a statistical signal is embedded in generated text by biasing the sampler, the detection test that makes it verifiable, and the reasons the scheme survives paraphrase poorly.
ML Supply Chain Security Untrusted weights and datasets, deserialisation risk, dependency and registry attacks, and signing artefacts. 11 concepts · 99 cards
- 01 Dependency and Registry Attacks in ML Stacks Why ML environments are unusually exposed to package-level attacks, the specific techniques that keep working, and the controls that actually reduce exposure.
- 02 Hallucinated Dependencies and Slopsquatting Why a code-generating model invents package names that do not exist, why it invents the same ones repeatedly, and how that repeatability turns a model error into a pre-registerable attack surface that lockfiles and signatures cannot see.
- 03 Prompt Artefacts as Executable Dependencies Why rules files, skills, agent instructions and system prompts shipped in a repository are code by any operational definition, how invisible Unicode makes them reviewable in theory and not in practice, and what intake for a text dependency has to check.
- 04 Untrusted Weights and Deserialisation Risk Why loading a model file can execute code, what safetensors changed, and the checks that belong in any pipeline that downloads weights from a public hub.
- 05 Vetting Third-Party Models and Datasets A practical intake process for an artefact you did not produce, what each check can and cannot establish, and how to size the effort to the deployment's exposure.
- 06 Coding Agents as a Supply Chain Target Why an installed coding agent is a credential-discovery engine an attacker can borrow, what the Nx s1ngularity compromise demonstrated about that, and how agent autonomy turns a per-machine compromise into a self-propagating one.
- 07 Containing Install and Build Time Execution Why arbitrary code runs before any of your tests do, what disabling lifecycle scripts and constraining egress actually buys, and how to keep dependency resolution and dependency execution as separate, separately-governed events.
- 08 MCP Server Trust and Tool Poisoning Why a tool description is executable content inside the model's trust boundary, how shadowing and rug-pull attacks exploit the absence of any integrity check on that description, and what intake and pinning look like for a dependency whose interface is prose.
- 09 Securing Model Serving Infrastructure Why an inference server is an internet-facing distributed system with a large native attack surface, what recent CVEs in Triton, vLLM and Ray teach about where it breaks, and what GPU isolation and confidential computing do and do not protect.
- 10 Securing the Training Pipeline Why the training environment is a high-value target with unusually broad access, the specific credential and isolation failures that recur, and the controls proportionate to what a compromise would yield.
- 11 Signing, Attestation and SBOMs for Models How software supply chain frameworks map onto model artefacts, what a model bill of materials should contain, and why the interesting claims are about the training process rather than the file.