Safety Cases and Structured Assurance Arguments

A pile of evaluation results is not an argument. It is a set of measurements, and the step from "the model scored below the threshold on nine capability benchmarks" to "deploying this system will not cause a catastrophe" is a piece of reasoning that somebody has to write down and somebody else has to attack. A safety case is that reasoning, made explicit enough to be wrong.

The idea is not from AI. After Piper Alpha killed 167 people in 1988, Lord Cullen's inquiry recommended that operators be required to submit a safety case for each installation, and the Offshore Installations (Safety Case) Regulations came into force in 1992 (SPE, Process Safety Thirty Years After Piper Alpha). The regime spread into nuclear, rail, defence and, in 2020, into autonomous vehicles through UL 4600, which asks a developer to set out a safety case and demonstrate compliance with it rather than prescribing engineering methods.

Claim, argument, evidence

Every assurance case has the same three layers, and conflating them is the usual mistake.

The claim is a proposition about a specific system in a specific deployment over a specific period. "The model is safe" is not a claim; it has no truth conditions. "This model, deployed through the API with these filters, does not provide uplift to a novice attempting a chemical weapons synthesis" is a claim, because you can say what would falsify it.

The argument decomposes the claim into sub-claims whose conjunction implies it, and keeps going until each leaf is something evidence can settle. This is where the intellectual work lives, and it is also the layer that most AI documentation skips entirely, jumping from a top-level assurance to a results appendix.

The evidence sits at the leaves: evaluation scores, red-team transcripts, code review, formal proofs, operational telemetry. It is only as good as the sub-claim it attaches to, and a mountain of it under the wrong sub-claim buys nothing.

Two notations exist for writing this down. Goal Structuring Notation came out of the High Integrity Systems Engineering group at the University of York in the 1990s, combining Toulmin-style argumentation with goal-based requirements engineering, and is maintained as a community standard by the Safety-Critical Systems Club (SCSC, 2021, GSN Community Standard v3). Claims-Arguments-Evidence is the other, used by the UK AI Security Institute for its published frontier-AI template (Goemans et al., 2024, Safety Case Template for Frontier AI: A Cyber Inability Argument, arXiv:2411.08088).

The four arguments available for an AI system

The most-used taxonomy for AI sorts top-level arguments by what they claim about the system rather than by what evidence they use (Clymer et al., 2024, Safety Cases: How to Justify the Safety of Advanced AI Systems, arXiv:2403.10462).

Inability. The system cannot cause the harm in any environment it could plausibly reach. The cheapest argument to make and the first to expire, because capability is the thing that goes up every release.

Control. The system might be able to cause the harm, but the measures wrapped around it would stop it: monitoring, sandboxing, permission scoping, approval gates. The evidence is a measured interception rate under adversarial conditions, not a capability score.

Trustworthiness. The system is capable and unconstrained, and would not do it anyway. This is the argument the field is worst at supporting, because it requires evidence about dispositions rather than about behaviour on a test set.

Deference. The judgement that the system is safe comes from other AI systems whose judgement has itself been established. The authors treat this as applying only to a future with far more capable systems; it marks where the sequence ends.

A real case combines them: inability for the capabilities genuinely absent, control for the ones present but containable, trustworthiness for what is left. The discipline that matters is being explicit about which argument carries which risk, because each fails for a different reason.

When it breaks

Paper safety is the documented failure mode, not a hypothetical one. The Nimrod Review examined the safety case for an aircraft lost in 2006 with fourteen crew and found the work had been a lamentable job from start to finish, riddled with errors, and had missed the key dangers; around forty percent of hazards had been left open and thirty percent unclassified (Haddon-Cave, 2009, The Nimrod Review, HC 1025). The diagnosis was a culture in which the document was produced to be accepted rather than to be true. An AI programme that treats a safety case as a deliverable owed to a regulator inherits that failure directly.

The argument is only as strong as its weakest elicitation assumption. Nearly every leaf under an inability or control argument rests on a claim that testing surfaced what the system can do. If the red team under-elicited, every number above it overestimates safety, and extra evidence at other leaves does not repair it.

Defeaters are the part that gets dropped. A case without a written list of the conditions that would invalidate it is unfalsifiable and therefore worthless as a decision aid. The conditions belong in the document: a new fine-tune, a new tool in the agent's hands, a jailbreak class that defeats the filters, a capability eval that was gamed.

Cases decay on a schedule nobody sets. Fine-tuning, scaffolding changes and new integrations each void parts of the argument silently, which is why continuous rather than point-in-time assurance keeps being proposed (Buhl et al., 2024, Safety Cases for Frontier AI, arXiv:2410.21572). See /learn/assurance-for-continuously-changing-systems.

A safety case is not an audit. The developer writes the case; an auditor tests whether to believe it. Confusing them produces a document with the rhetorical form of independent assurance and the incentives of marketing.