Safety & Alignment
Prompt injection, jailbreaks, Constitutional AI, reward hacking and mechanistic interpretability.
25concepts
408flashcards
202minutes of reading
- 01 Activation Steering and Representation Engineering Adding a direction to a model's residual stream at inference time changes its behaviour without any weight update, which makes concepts like refusal, sycophancy and sentiment into vectors you can add, scale, or subtract.
- 02 Alignment Evaluations and Frontier-Model Risk How frontier labs and governments measure dangerous capabilities, what an eval-gated release looks like, and where the regulatory regime sits in 2026.
- 03 Guardrail Classifiers and Content Filtering Putting a separate classifier in front of and behind the model gives you a safety layer updatable in hours rather than retrained over weeks, and it buys that agility with latency, false refusals, and a second model to keep honest.
- 04 Jailbreaks and Refusal Robustness How attackers reliably bypass model refusal training, why post-hoc filters are necessary but never sufficient, and how AILuminate measures what remains.
- 05 Prompt Injection Why LLMs cannot reliably tell instructions from data, how indirect injection weaponises retrieved content, and which partial defences are worth deploying.
- 06 Training Data Memorisation and Extraction Language models reproduce fragments of their training corpus verbatim, and an adversary with only API access can pull that data back out, which turns a training-set decision into a permanent disclosure risk.
- 07 Watermarking and Content Provenance How a statistical signal is embedded in generated text without changing its quality, why SynthID-Text scaled to production, and the robustness limits every scheme shares.