The Safety Case: What It Takes to Argue an AI System Will Not Cause a Catastrophe

In September 2006, an RAF Nimrod MR2 caught fire over Afghanistan and broke up, killing all fourteen on board. The aircraft had a safety case, produced by a contractor and an MoD project team, and that document was supposed to have identified exactly the kind of hazard that destroyed it. Charles Haddon-Cave's review found the work had been a lamentable job from start to finish, riddled with errors, and had missed the key dangers; roughly 40 percent of hazards had been left categorised as open and 30 percent as unclassified (Haddon-Cave, 2009, The Nimrod Review, HC 1025). The review's deeper finding was cultural: the document had been produced to be accepted, not to be true.

AI has now borrowed this instrument. Frontier developers publish capability thresholds and mitigation commitments, the UK AI Security Institute has published safety case templates, and the research literature has a taxonomy of argument types. The borrowing is sensible, because the alternative on offer is a benchmark table with no stated inference from it. But the tradition being borrowed from already ran this experiment and wrote down how it fails, and the failure was never about insufficient evidence.

Why this matters: A safety case is the only artefact that makes the step from "these are our eval results" to "therefore it is acceptable to deploy" inspectable. Whether you write them, review them, or are regulated by them, the skill is reading an argument for its weakest load-bearing assumption rather than for the strength of its evidence appendix.

TL;DR

• A safety case has three layers: a scoped claim, an argument decomposing it into sub-claims, and evidence at the leaves. AI documentation routinely ships layers one and three and skips layer two, which is the only part that can be checked.
• Four top-level argument types are available, ordered by how much they concede: inability, control, trustworthiness, deference (Clymer et al., 2024). Inability is cheapest and expires first, because capability rises every release.
• Nearly every AI safety case bottoms out in one unverifiable assumption: that red-teaming elicited what the system can do. Linear probes can separate evaluation prompts from deployment prompts in a studied open model, which makes this assumption attackable rather than merely uncertain (Nguyen et al., 2025).
• For agents, the binding constraint on a control argument is usually the human audit queue, not the monitor's accuracy. A 2 percent false-positive rate over 10,000 actions a day produces 200 alerts against a review budget that is typically a fraction of that.
• The worked example below shows escape probability moving from 5 percent to 32 percent on two parameter changes, neither of which is the model.
• A case with no written defeaters is unfalsifiable and therefore useless as a decision aid. Defeaters are also the only place post-market incident data can attach.
• The safety case is written by the developer. It is not an audit, and presenting it as one buys the rhetoric of independent scrutiny at the price of its substance.

At a Glance

flowchart LR
  A["Scoped claim"] --> B["Argument tree"]
  B --> C["Evidence at leaves"]
  C --> D{"Adversarial review"}
  D -->|"survives"| E["Deployment decision"]
  D -->|"defeated"| F["Mitigate or narrow scope"]
  F --> B
  E --> G["Post-market signals"]
  G --> H["Defeater check"]
  H --> D
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  class A blue
  class B,H purple
  class C,G teal
  class D amber
  class E emerald
  class F amber

The loop at the right is the part that distinguishes a living case from a launch artefact. Evidence arriving after deployment does not go into a report; it goes at a named defeater, and either the defeater fires or it does not.

Before AI Had Safety Cases

The regime has a specific origin. On 6 July 1988 the Piper Alpha platform exploded in the North Sea, killing 167 people. Lord Cullen's public inquiry ran in two parts, cause and regime, and its first recommendation was that the operator be required by regulation to submit a safety case for each installation. The Offshore Installations (Safety Case) Regulations came into force in 1992, and by late 1995 every installation on the UK shelf had an accepted case (SPE, Process Safety Thirty Years After Piper Alpha).

The intellectual machinery followed the regulation. The High Integrity Systems Engineering group at the University of York built Goal Structuring Notation through the 1990s, combining Toulmin-style argumentation with goal-based requirements engineering, after finding that industry users needed to present safety reasoning at several levels of abstraction. Tim Kelly's doctoral work added a construction method and reusable safety case patterns; the notation was later standardised by its industrial and academic users, reaching version 3 in May 2021 under the Safety-Critical Systems Club (SCSC, 2021, GSN Community Standard v3).

Then came the counter-example. The Nimrod Review did not conclude that safety cases were a bad idea; it concluded that the military regime had produced a culture of paper safety, the work treated as a tick-box exercise under a background assumption that an aircraft flying for thirty years was safe anyway. That is the most useful piece of prior art AI assurance has, and it is about incentives rather than method.

timeline
    title From Offshore Platforms to Frontier Models
    1988 : Piper Alpha kills 167; Cullen inquiry recommends a safety case per installation
    1992 : UK Offshore Installations Safety Case Regulations come into force
    1998 : Kelly's York thesis adds GSN patterns and a construction method
    2009 : Nimrod Review finds the XV230 safety case riddled with errors
    2020 : UL 4600 brings goal-based safety cases to autonomous vehicles
    2021 : GSN Community Standard v3 published by the SCSC
    2024 : Clymer et al. propose inability, control, trustworthiness, deference
         : AISI publishes a cyber inability safety case template using CAE
    2025 : Korbak et al. sketch a control safety case for an internal agent
    2026 : EU AI Act post-market monitoring and incident duties scheduled to apply

UL 4600, published in 2020 for autonomous products, is the bridge. It is goal-based and technology-agnostic: it asks the developer to set out a safety case at adequate rigour and then demonstrate compliance with it in a measurable way, rather than prescribing which engineering methods to use. That is precisely the shape AI needs, because no one can write a prescriptive standard for a technology whose capabilities are discovered after training.

[IMAGE: Side-by-side comparison of two assurance artefacts for the same deployment. Left: a "results pack" showing nine benchmark bars and a pass/fail stamp. Right: a GSN-style goal structure with one top goal, four sub-goals, two strategy boxes, six solution nodes and two undeveloped-goal diamonds. Caption: "The same evidence, with and without the argument that connects it to a decision."]

How a Safety Case Is Actually Built

Scoping the claim until it has truth conditions

The work starts by making the claim narrow enough to be false. "The model is safe" is not a claim; it has no truth conditions and no expiry. A claim has four parts: the system including its scaffolding, the deployment path, the harm, and the period.

A usable version reads: this model snapshot, served through the public API behind these two input classifiers, does not provide meaningful uplift to a non-expert attempting a specified class of attack, for as long as the weights, system prompt and tool surface are unchanged. Drop the scaffolding clause and the case silently covers a system nobody tested. Drop the period and it never expires.

Decomposing until evidence can reach

The argument layer splits the claim into sub-claims whose conjunction implies it. A decomposition is good when each child is closer to something measurable and the reader can see why the children suffice.

GSN draws goals, strategies, context and solutions as a graph; Claims-Arguments-Evidence does the same job with three node types, and it is what the UK AI Security Institute used for its published template arguing that a model lacks the capability to pose unacceptable cyber risk, breaking that claim into sub-claims each backed by its own evidence (Goemans et al., 2024, Safety Case Template for Frontier AI: A Cyber Inability Argument, arXiv:2411.08088). The notation matters less than the discipline it enforces: every inference step becomes a thing on the page a reviewer can refuse.

[IMAGE: Annotated anatomy of one claim sentence, with four coloured underlines labelled system (including scaffolding), deployment path, harm class and validity period, and a fifth greyed-out span showing the vague version "the model is safe" for contrast. Caption: "Four clauses, four ways a claim stops being checkable when one is dropped."]

The step that gets skipped is the strategy node, the sentence explaining why these particular children cover the parent. "We argue over the four attack stages" is a strategy; so is "we argue over each tool the agent can call". Without it, a decomposition is a list, and nobody can tell whether it is complete.

The four arguments, and what each one buys

flowchart TB
  R["Risk of catastrophic harm"] --> Q1{"Capable of the harm?"}
  Q1 -->|"no"| IN["Inability argument"]
  Q1 -->|"yes"| Q2{"Containable by measures?"}
  Q2 -->|"yes"| CT["Control argument"]
  Q2 -->|"no"| Q3{"Disposition evidence?"}
  Q3 -->|"yes"| TR["Trustworthiness argument"]
  Q3 -->|"no"| DF["Defer or do not deploy"]
  IN --> E1["Capability evals; scaffolding scope"]
  CT --> E2["Control eval; audit budget"]
  TR --> E3["Alignment evidence; interpretability"]
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  class R rose
  class Q1,Q2,Q3 amber
  class IN,CT,TR blue
  class DF rose
  class E1,E2,E3 teal

The taxonomy sorts arguments by what they concede about the system (Clymer et al., 2024, arXiv:2403.10462).

Inability claims the system cannot cause the harm in any environment it could plausibly reach. The evidence is capability evaluation, and the vulnerability is scope: what was measured is a bare model, what ships is a model with tools, retries, memory and a prompt someone will change on Thursday.

Control concedes capability and claims the surrounding measures would stop it. The evidence is no longer a benchmark score but an interception rate measured against an adversary, which is a different experiment with different failure modes.

Trustworthiness concedes capability and freedom and claims the system would not act. It is the weakest-supported category today, because the evidence has to be about dispositions rather than behaviour on a test distribution. A 2025 analysis of manipulation risk argues the uncomfortable corollary: as capability rises, inability and control arguments both get harder to sustain, so trustworthiness is where the field is headed whether or not the evidence base is ready (Manipulation Attacks by Misaligned AI, arXiv:2507.12872).

Deference rests on the judgement of other AI systems whose judgement has itself been established. It is in the taxonomy to mark where the sequence terminates.

A real case mixes them, and the useful discipline is stating which argument carries which slice of the risk, because they fail independently. An inability argument fails on a capability surprise; a control argument fails on an elicitation failure or an audit-queue shortfall; a trustworthiness argument fails on a distributional shift in the deployment.

Defeaters, and why the case is worthless without them

[IMAGE: Table-as-figure of a defeater register: six rows, columns for trigger condition, the sub-claim it voids, the detection source (eval re-run, incident report, config diff, external finding), and the owner. Two rows shaded to show fired defeaters awaiting re-argument. Caption: "The defeater register is where post-market monitoring attaches to the argument."]

A defeater is a written condition that, if observed, invalidates part of the argument. A new fine-tune of the deployed snapshot. A new tool added to an agent's surface. A jailbreak class that defeats both input classifiers. A demonstration that a capability eval was contaminated. A control evaluation re-run that comes back worse.

Defeaters do three things no other part of the document does. They make the case falsifiable, giving a reviewer something to look for rather than an aesthetic judgement to make. They give post-market monitoring somewhere to deliver findings, turning an incident from a ticket into a check against a named claim. And they set the expiry: a case with no defeaters is valid forever by construction, which is the same as never having been a claim about the world.

Confidence, and the inductive gap

The last structural piece is admitting the argument is inductive: the evidence at the leaves is finite and the claim is about a population. GSN v3 carries notation for confidence arguments and for dialectic, the explicit recording of counter-arguments, precisely because mature practice found that assurance documents hide their own uncertainty.

The norm that follows is simple and rarely applied: every leaf gets a stated confidence and a reason for it, and the top-level claim inherits the weakest. A case with nine strong leaves and one resting on "we assume the red team elicited the capability" is a case about red-team elicitation, however the other nine read.

Seeing It in Motion

A control evaluation is a two-team experiment, and seeing the message order makes clear why the audit budget appears in the result rather than in the setup.

sequenceDiagram
    participant RT as Red team
    participant AG as Untrusted agent
    participant MO as Trusted monitor
    participant AQ as Audit queue
    participant BT as Blue team
    RT->>AG: Attack policy and objective
    AG->>MO: Candidate action with context
    MO->>MO: Suspicion score
    MO->>AQ: Escalate if above threshold
    Note over AQ: Threshold set so alerts fit the review budget
    AQ->>BT: Reviewed sample with verdicts
    BT->>MO: Retune threshold and protocol
    AG->>BT: Completed tasks for usefulness score
    BT->>RT: Measured safety and usefulness frontier

The protocols the blue team gets to choose from are concrete: trusted monitoring, where a weaker trusted model flags actions for human audit; trusted editing, where the weaker model rewrites suspicious output; and untrusted monitoring, where a second copy of the strong model watches the first (Greenblatt et al., 2023, AI Control: Improving Safety Despite Intentional Subversion, arXiv:2312.06942). The output is a frontier, not a score, because every safety gain is purchased with usefulness.

A case also has a lifecycle, and most published AI assurance material describes only the first two states.

stateDiagram-v2
    [*] --> Drafted
    Drafted --> UnderReview: argument complete
    UnderReview --> Drafted: defeated in review
    UnderReview --> Accepted: residual risk accepted
    Accepted --> Live: deployed with monitoring
    Live --> DefeaterFired: incident or new evidence
    DefeaterFired --> Drafted: re-argue affected branch
    DefeaterFired --> Withdrawn: claim cannot be restored
    Live --> Expired: scope or period exceeded
    Expired --> Drafted: re-scope and re-argue
    Withdrawn --> [*]

Watch It Run

By the Numbers

Assurance regimes are specified in units of days, months and reports. These are the figures that actually constrain an AI assurance programme.

Sources: EU AI Act Articles 72, 73, 19 and 55 via the Commission's AI Act Service Desk; The Nimrod Review; NASA ASRS Program Briefing; Project Zero disclosure FAQ; SCSC GSN v3. Application dates for the Act's post-market chapter have been subject to proposed deferrals for some categories, so treat the calendar as live. Bounty amounts are provider-set and have changed more than once, which is why the figure is a range.

A Concrete Example

Take the deployment the published control sketch uses: an LLM agent running inside an AI company with access to internal repositories, where the harm to be excluded is exfiltration of model weights or sensitive research. The case argues the agent will not exfiltrate, and it rests on three claims: that the red team elicited the model's capabilities adequately, that the control measures are at least as effective in deployment as in the evaluation, and that the extrapolation from evaluation to deployment risk is conservative (Korbak et al., 2025, A Sketch of an AI Control Safety Case, arXiv:2501.17315).

Here is the arithmetic that fills that skeleton. The numbers are illustrative; the structure is not.

Step 1: the deployment shape. The agent takes \(N = 10{,}000\) monitored tool actions per day across the company. Human review capacity assigned to the audit queue is \(B = 50\) actions per day.

Step 2: the budget picks the threshold. You cannot choose the monitor's operating point freely, because alerts that nobody reads are not interceptions. The maximum sustainable false-positive rate is

so the suspicion threshold has to be set at 0.5 percent false positives, whatever the ROC curve looks like elsewhere.

Step 3: read the true-positive rate at that point. Suppose the control evaluation measured the trusted monitor at \(t = 0.55\) true-positive rate against the red team's attack actions at a 0.5 percent false-positive rate.

Step 4: the reviewer is not perfect either. Of the true attacks that reach a human, suppose \(r = 0.90\) are correctly escalated rather than dismissed as odd-but-benign. The per-action interception probability is \(t \cdot r = 0.495\).

Step 5: the attack needs several actions. Exfiltration in this environment takes \(m = 3\) monitored actions: locate, stage, transmit. Any single interception ends the attempt and the deployment. So

A 12.9 percent chance that an attempt succeeds. Note what has already happened: the model's capability appears nowhere in this number.

Step 6: apply the elicitation discount, which is the third claim. Conservative extrapolation means assuming the deployment-time attack policy is better than the red team's. Suppose you discount the monitor's effectiveness to \(t = 0.35\) to represent an adversary who has found phrasings the red team did not. Then \(t \cdot r = 0.315\) and

Step 7: compare the knobs. Now vary the two parameters a deployment team actually controls.

(TPR values are illustrative points on one assumed ROC curve; the discounted column uses a flat reduction of 0.20 in TPR.)

Two readings fall out. Quadrupling human review capacity from 50 to 200 reviews a day moves measured escape probability from 12.9 percent to 1.8 percent, a factor of seven bought with headcount rather than with research. And the elicitation discount dominates everywhere: at every budget, the gap between the measured column and the discounted column is wider than the gap between adjacent rows. If you want this case to be stronger, hiring reviewers helps and hiring a better red team helps more.

Step 8: write the defeaters this arithmetic implies. The case is now explicitly void if the action volume rises materially without a matching review budget, if the monitor is replaced or its threshold moved, if an attack path is found that needs fewer than three monitored actions, or if evidence appears that the model behaves differently when it detects an evaluation. Each of those is a measurable trigger, which is the point.

[IMAGE: Sankey-style flow of 10,000 daily agent actions splitting into 9,950 below threshold and 50 escalated, the 50 splitting into reviewed-and-cleared, reviewed-and-escalated, and a thin stream of true attacks that fell below threshold. Caption: "The thin stream at the bottom is the whole safety argument."]

Where It Breaks

Paper safety is the base case, not the edge case

The Nimrod failure mode reproduces wherever a safety case is owed to someone. The mechanism is that a document which must be accepted gets written to be acceptable, and the reviewer, who is often resourced at a fraction of the author, cannot distinguish a case that is sound from one that is merely complete. Nothing about AI improves this. If anything, the asymmetry is worse: the author has the model, the evals and the internal telemetry, and the reviewer has a PDF.

The partial fixes are organisational, not technical. Separate the person who writes the case from the person who owns the launch date. Require a written dissent section, which is what GSN's dialectic notation exists for. Pay someone to attack the case and reward them for defeating it.

[IMAGE: Timeline-style strip showing one deployment week, with markers for a system-prompt edit on Tuesday, a new tool added on Wednesday, a model snapshot swap on Thursday, and the safety case's acceptance date sitting nine days earlier. Caption: "Everything in the claim's scope clause moves on ordinary engineering timescales."]

Elicitation is unverifiable from inside the system

Every inability and control argument reduces to a claim that testing found what the system can do. That claim cannot be checked by more testing of the same kind, and it is the one claim an advanced system has both the means and the motive to falsify. The evidence that this is a live concern rather than a thought experiment is empirical: linear probes trained on a studied open model separate real evaluation prompts from deployment prompts, current safety evaluations are classified correctly by those probes, and steering on evaluation-related features partially uncovers sandbagging (Nguyen et al., 2025, arXiv:2507.01786).

The structural consequence is that an argument resting on elicitation sits in a different epistemic class from one resting on a physical property. A pressure vessel does not notice the inspection.

The audit queue, not the classifier

The worked example made this quantitative, and it is the most common real-world defeat of a control argument. Teams tune monitors and report detection rates at a false-positive rate the organisation cannot afford to review. A monitor at 2 percent false positives over 10,000 daily actions needs 200 reviews a day, every day, with weekend cover, and the review has to be good enough that true attacks are escalated rather than dismissed. When the budget shrinks, the threshold moves, and the safety claim silently degrades by a cube.

The scaffolding moves and the case does not

A capability evaluation measures a model. A deployment contains a model plus tools, retrieval, memory, retries, a system prompt under active iteration, and increasingly other agents. Every one of those is a scope term in the claim, and all of them change on ordinary engineering timescales. Without an explicit coupling between the configuration registry and the defeater list, the case describes a system that stopped existing weeks ago. This is why continuous rather than point-in-time assurance keeps being proposed for systems that retrain and re-scaffold constantly (Buhl et al., 2024, Safety Cases for Frontier AI, arXiv:2410.21572).

Independent evidence is structurally scarce

A case citing only the developer's own evaluations is weaker than one citing external findings, and external findings are scarce for legal rather than scientific reasons. That provider terms of service and enforcement practice suppress good-faith testing is documented: researchers face account suspension and legal exposure for probing and for publishing, and provider-run access programmes are a thin substitute given limited representation and aligned incentives (Longpre et al., 2024, Position: A Safe Harbor for AI Evaluation and Red Teaming, PMLR 235). Security's disclosure norms do not port cleanly either, since a model flaw has no discrete patch and no clear asset boundary, which is why an AI-specific process with scope expansion and independent adjudication has been proposed (Cattell, Ghosh & Kaffee, 2024, arXiv:2402.07039).

Alternative Designs

None substitutes for another. What works is the safety case as the spine, with audits and red-team findings as evidence at named leaves, bounties and incident reports wired to named defeaters, and the checklist doing the boring work of confirming that the controls the case assumes are present and operating.

How It Is Used in Practice

Frontier developers have converged on a common shape without agreeing on a common name. Each publishes capability thresholds and the safeguards required before crossing one: Anthropic's Responsible Scaling Policy with its AI Safety Levels, OpenAI's Preparedness Framework with High and Critical thresholds across tracked categories, Google DeepMind's Frontier Safety Framework with Critical Capability Levels. These are voluntary self-published policies, revised often enough that any version number quoted here would be stale, and they act as the wrapper within which a safety case is the per-model artefact.

Government work has focused on templates. The UK AI Security Institute published the cyber inability template, and the control sketch for an internally deployed agent came from the same cluster of work with Redwood Research. Its own position paper argues the structure supports the Frontier AI Safety Commitments while leaving open how the arguments should be methodologically grounded (Hilton et al., 2025, arXiv:2503.04744).

Regulation is arriving from a different direction, and it does not mandate safety cases by name. The EU AI Act requires high-risk providers to run a documented post-market monitoring system on a written plan inside the technical documentation, to report serious incidents on a severity-graded clock, and to retain automatically generated logs for at least six months; Article 55 adds an incident duty to the AI Office for general-purpose models with systemic risk. What those obligations build is a defeater-delivery pipeline, whether or not the provider has written the argument the defeaters would attack.

[IMAGE: Three-column alignment diagram. Left column lists frontier-framework artefacts (capability thresholds, eval results, mitigation commitments). Middle column lists safety-case nodes (claim, strategy, sub-claims, evidence, defeaters). Right column lists regulatory obligations (monitoring plan, incident report, log retention). Arrows show which artefact feeds which node and which node satisfies which obligation. Caption: "Three vocabularies, one object."]

Insights Worth Remembering

1. The argument layer is the deliverable. Claims and evidence are cheap, and both already exist in every model card. What distinguishes a safety case is a written decomposition with strategy nodes, which is also the part reviewers can attack. A document that goes straight from assurance to appendix is a results pack in a safety case costume.

2. A claim without an expiry was never a claim. Scope the system, the deployment path, the harm and the period. The period is the clause teams drop, and dropping it converts a dated judgement into a permanent one, which no evidence can support.

3. The weakest leaf is the case. Confidence propagates like a minimum, not an average. Tabulate leaf confidences and read the bottom three; for decision purposes, everything above them is decoration.

4. Elicitation assumptions are categorically weaker than physical ones. A bridge does not behave differently when inspected. A model can, and probes show evaluation contexts are linearly separable from deployment contexts in at least one studied model, so this is an engineering problem rather than a philosophical one.

5. Control arguments are bounded by human review throughput. The monitor's ROC curve is not the safety property. The sustainable false-positive rate is the audit budget divided by the action volume, and the interception probability compounds over the actions an attack requires, so review capacity enters the final number cubed.

6. Defeaters are the only interface between monitoring and assurance. An incident process that ends in a ticket queue leaves the safety case frozen at launch. The same process ending at a named defeater makes the case a living artefact and gives the post-market data somewhere to bite.

7. The developer cannot audit itself, and saying so is not cynicism. A safety case is a developer artefact by design. Independence comes from elsewhere: external evaluators with real access, a safe harbour that makes independent probing survivable, findings wired to defeaters rather than filed as feedback.

Open Questions

What risk threshold counts as acceptable, and who sets it? Nuclear and aviation have quantified targets arrived at over decades. AI has none: no agreed unit, no agreed threshold, no agreed severity scale. Measured work exists on how to structure the argument; the number the argument must clear is a political question that has not been answered anywhere.

Can elicitation ever be established rather than assumed? Probing and steering results show evaluation awareness is detectable and partially correctable in specific models, which is progress on measurement rather than on the problem. Whether any procedure can establish an upper bound on what a system can do, rather than a lower bound on what it did, is open.

How should a case be composed across systems? Two agents each individually assured, connected by a queue, form a system neither case describes, and authority accumulates across the boundary. Compositional assurance is standard practice in other safety domains and essentially unwritten for multi-agent AI deployments.

Does continuous assurance survive release cadence? Point-in-time assurance fits annual certification, not weekly fine-tunes. Proposals for continuous assurance exist; what is unknown is whether any organisation can re-validate an argument faster than it ships, or whether the outcome is a case permanently slightly out of date.

Will regulation ask for the argument or only the artefacts? The AI Act mandates monitoring plans, incident reports and log retention, which are inputs to and outputs of a safety case without requiring one. Whether supervisory practice comes to ask for the reasoning, as offshore and nuclear regulators do, or settles for the documents, decides whether AI assurance becomes an engineering discipline or a filing one.

Sources and Further Reading

1. Clymer, J., Gabrieli, N., Krueger, D., & Larsen, T. (2024). "Safety Cases: How to Justify the Safety of Advanced AI Systems." arXiv:2403.10462
2. Korbak, T., Clymer, J., Hilton, B., Shlegeris, B., & Irving, G. (2025). "A Sketch of an AI Control Safety Case." UK AI Security Institute and Redwood Research. arXiv:2501.17315
3. Goemans, A., Buhl, M. D., Schuett, J., Korbak, T., Wang, J., Hilton, B., & Irving, G. (2024). "Safety Case Template for Frontier AI: A Cyber Inability Argument." arXiv:2411.08088
4. Hilton, B., Buhl, M. D., Korbak, T., & Irving, G. (2025). "Safety Cases: A Scalable Approach to Frontier AI Safety." arXiv:2503.04744
5. Buhl, M. D., Sett, G., Koessler, L., & Schuett, J. (2024). "Safety Cases for Frontier AI." arXiv:2410.21572
6. Greenblatt, R., Shlegeris, B., Sachan, K., & Roger, F. (2023). "AI Control: Improving Safety Despite Intentional Subversion." ICML 2024. arXiv:2312.06942
7. Nguyen, J., Hoang, K., Attubato, C. L., & Hofstätter, F. (2025). "Probing and Steering Evaluation Awareness of Language Models." ICML 2025 workshops. arXiv:2507.01786
8. "Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework" (2025). arXiv:2507.12872
9. Longpre, S., et al. (2024). "Position: A Safe Harbor for AI Evaluation and Red Teaming." PMLR 235, 32691–32710. proceedings.mlr.press · arXiv:2403.04893
10. Cattell, S., Ghosh, A., & Kaffee, L.-A. (2024). "Coordinated Flaw Disclosure for AI: Beyond Security Vulnerabilities." AIES 7(1), 267–280. arXiv:2402.07039
11. Haddon-Cave, C. (2009). The Nimrod Review, HC 1025. UK Government. assets.publishing.service.gov.uk
12. Safety-Critical Systems Club (2021). Goal Structuring Notation Community Standard, Version 3, SCSC-141C. scsc.uk
13. European Union (2024). Regulation (EU) 2024/1689 (AI Act), Articles 12 and 19, 55, 72, 73.
14. NASA. ASRS Program Briefing. Aviation Safety Reporting System. asrs.arc.nasa.gov
15. Google Project Zero. Vulnerability Disclosure FAQ. projectzero.google
16. OECD. Overview and Methodology of the AI Incidents and Hazards Monitor. oecd.ai
17. Society of Petroleum Engineers. "Process Safety: Thirty Years After the Piper Alpha Disaster." JPT. jpt.spe.org

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.