Safety & Alignment
Prompt injection, jailbreaks, Constitutional AI, reward hacking and mechanistic interpretability.
11concepts
54flashcards
93minutes of reading
- 01 AI Control: Safety Without Trusting the Model The research agenda that assumes the model may be deliberately subverting your safeguards, and designs protocols with a red team that gets to try.
- 02 Constitutional AI and RLAIF How Anthropic replaced human harmlessness labels with a written constitution and a critique-and-revise loop, and why this makes alignment auditable.
- 03 Machine Unlearning in Language Models What it means to remove knowledge from a trained model, why WMDP and TOFU measure different things, and the relearning attacks that show most unlearning is suppression.
- 04 Mechanistic Interpretability Primer How sparse autoencoders extract human-interpretable features from model activations, what circuit-level analysis buys you for safety, and where the science is still contested.
- 05 Model Organisms of Misalignment and Sleeper Agents Why safety researchers deliberately build misaligned models, what the sleeper-agent experiments showed about the durability of backdoors, and why adversarial training made things worse.
- 06 Scalable Oversight and Weak-to-Strong Generalisation How you supervise a model on tasks you cannot evaluate yourself, why weak labels still elicit strong capabilities, and where the analogy to superhuman supervision leaks.
- 07 Sycophancy, Deception, and Reward Hacking Why preference-trained models learn to please rather than to be right, what alignment faking is, and why evaluating during training can mislead you.