Safety & Alignment
Prompt injection, jailbreaks, Constitutional AI, reward hacking and mechanistic interpretability.
25concepts
408flashcards
202minutes of reading
- 01 Activation Steering and Representation Engineering Adding a direction to a model's residual stream at inference time changes its behaviour without any weight update, which makes concepts like refusal, sycophancy and sentiment into vectors you can add, scale, or subtract.
- 02 Alignment Evaluations and Frontier-Model Risk How frontier labs and governments measure dangerous capabilities, what an eval-gated release looks like, and where the regulatory regime sits in 2026.
- 03 Guardrail Classifiers and Content Filtering Putting a separate classifier in front of and behind the model gives you a safety layer updatable in hours rather than retrained over weeks, and it buys that agility with latency, false refusals, and a second model to keep honest.
- 04 Jailbreaks and Refusal Robustness How attackers reliably bypass model refusal training, why post-hoc filters are necessary but never sufficient, and how AILuminate measures what remains.
- 05 Prompt Injection Why LLMs cannot reliably tell instructions from data, how indirect injection weaponises retrieved content, and which partial defences are worth deploying.
- 06 Training Data Memorisation and Extraction Language models reproduce fragments of their training corpus verbatim, and an adversary with only API access can pull that data back out, which turns a training-set decision into a permanent disclosure risk.
- 07 Watermarking and Content Provenance How a statistical signal is embedded in generated text without changing its quality, why SynthID-Text scaled to production, and the robustness limits every scheme shares.
- 08 AI Control: Safety Without Trusting the Model The research agenda that assumes the model may be deliberately subverting your safeguards, and designs protocols with a red team that gets to try.
- 09 Activation Patching and Causal Tracing Swap one internal activation from a corrupted run into a clean run and measure how much of the output moves; done systematically it localises where in a network a specific computation happens.
- 10 Alignment Faking A model that infers it is in training, and that complying now prevents its values from being modified later, has an instrumental reason to comply that has nothing to do with actually holding those values.
- 11 Circuit Tracing and Attribution Graphs Replacing a model's MLP layers with a cross-layer transcoder produces a differentiable stand-in whose feature-to-feature influences can be read off as a graph, turning "which features are active" into "which features caused which".
- 12 Constitutional AI and RLAIF How Anthropic replaced human harmlessness labels with a written constitution and a critique-and-revise loop, and why this makes alignment auditable.
- 13 Data Poisoning and Backdoor Attacks An adversary who controls a small slice of the training corpus can install a trigger-activated behaviour that survives ordinary evaluation, and the number of documents required turns out not to grow with model scale.
- 14 Debate as Scalable Oversight If two strong models argue for opposing answers in front of a weaker judge, the honest side should have the easier case, which would let humans supervise systems whose answers they cannot verify directly.
- 15 Differential Privacy for Language Models The only training-time privacy defence with a formal guarantee, achieved by clipping per-example gradients and adding calibrated noise, and the concrete capability price that guarantee costs.
- 16 Evaluation Awareness and Sandbagging A safety evaluation assumes the subject cannot tell it is being evaluated, and frontier models increasingly can; that breaks the inference from a good test score to safe deployment behaviour.
- 17 Machine Unlearning in Language Models What it means to remove knowledge from a trained model, why WMDP and TOFU measure different things, and the relearning attacks that show most unlearning is suppression.
- 18 Mechanistic Interpretability Primer How sparse autoencoders extract human-interpretable features from model activations, what circuit-level analysis buys you for safety, and where the science is still contested.
- 19 Membership Inference on Language Models Deciding whether a specific document was in a model's training set is the canonical privacy attack, and on large language models it works far worse than the classical literature predicts, for reasons that are themselves informative.
- 20 Model Extraction and Weight Stealing A black-box API leaks more about the model behind it than intended, and with the right queries an attacker can recover architectural secrets or a functional clone for a tiny fraction of the training cost.
- 21 Model Organisms of Misalignment and Sleeper Agents Why safety researchers deliberately build misaligned models, what the sleeper-agent experiments showed about the durability of backdoors, and why adversarial training made things worse.
- 22 SAE Limitations and Crosscoders Sparse autoencoders were adopted on the premise that they recover a model's atomic, complete feature vocabulary, and three results published in 2024 and 2025 show they recover neither.
- 23 Scalable Oversight and Weak-to-Strong Generalisation How you supervise a model on tasks you cannot evaluate yourself, why weak labels still elicit strong capabilities, and where the analogy to superhuman supervision leaks.
- 24 Sparse Autoencoders for Feature Extraction Dictionary learning applied to activations, pulling a small set of interpretable, steerable features out of neurons that individually mean nothing.
- 25 Sycophancy, Deception, and Reward Hacking Why preference-trained models learn to please rather than to be right, what alignment faking is, and why evaluating during training can mislead you.