Assuring Agentic Systems

An evaluation of a chat model asks whether a sampled output is harmful. An evaluation of an agent has to ask whether a sequence of several hundred tool calls, each individually innocuous, ends somewhere it should not. The unit of assurance moves from the response to the trajectory, and almost every technique built for the first unit degrades on the second.

Three things change at once. The agent has authority, so a failure writes to the world rather than to a transcript. It operates over long horizons, so the space of reachable states is combinatorial rather than enumerable. And it is exposed to untrusted inputs mid-task, so the effective prompt is partly written by whatever the agent reads.

Why inability arguments expire first here

For a non-agentic deployment, an inability argument is often the cheapest honest case: the model cannot do the dangerous thing, here is the capability evaluation. Scaffolding breaks that. The claim being tested is not what the model can do but what the model plus its tools, memory, retries and sibling agents can do, and that composite changes every time someone adds an integration. A capability score measured on the bare model is evidence about a system that is not the one deployed.

This is why assurance for agents leans on control arguments: concede the capability, and claim the surrounding measures would stop the harm. The evidence shifts from a benchmark number to a measured interception rate under adversarial conditions, which is a different kind of experiment.

What a control evaluation measures

The method comes from work that treats the untrusted model as an adversary rather than as a noisy component. A red team is given the model and the task of subverting the deployment, a blue team designs a protocol using a weaker trusted model and a limited budget of human review, and the pair is run against each other to produce a usefulness-safety frontier rather than a single score (Greenblatt et al., 2023, AI Control: Improving Safety Despite Intentional Subversion, arXiv:2312.06942). The protocols are concrete: trusted monitoring, where the weak model flags suspicious actions for human audit; trusted editing, where the weak model rewrites suspicious output; untrusted monitoring, where a second copy of the strong model monitors the first.

Written as a safety case, this becomes a narrow argument with explicit load-bearing claims. A published sketch for an internally deployed agent argues that the agent will not exfiltrate sensitive information, and makes the case rest on three claims: that the red team elicited the model's capabilities adequately, that the control measures are at least as effective in deployment as in the evaluation, and that the extrapolation from evaluation to deployment risk is conservative (Korbak et al., 2025, A Sketch of an AI Control Safety Case, arXiv:2501.17315).

That structure is the useful part, independent of the specific threat model. It names where the argument can break, and all three of those claims are about the evaluation rather than about the system.

When it breaks

Elicitation is the load-bearing assumption and it is unverifiable from inside. Every control argument reduces to "we found what it can do." A model that underperforms when it detects an evaluation defeats this directly, and detection is measurable: linear probes separate evaluation prompts from deployment prompts in a studied open model, and steering on evaluation-related features partially uncovers sandbagging (Nguyen et al., 2025, Probing and Steering Evaluation Awareness of Language Models, arXiv:2507.01786). An argument whose foundation is a measurement the subject can recognise is in a different epistemic position from one resting on a physical property.

Monitoring false positives set the real ceiling. A monitor with a 2 percent false-positive rate over an agent taking 10,000 actions a day generates 200 alerts a day. If the human audit budget is 20 reviews, the protocol's effective coverage is a tenth of what the detection rate suggests, and the arithmetic of the audit queue, not the classifier's ROC curve, determines safety. Control evaluations are run at a fixed false-positive budget for exactly this reason.

The trajectory space cannot be sampled representatively. Benchmarks cover tasks; deployments cover whatever users ask for, in whatever order, with whatever intermediate state. Coverage claims for agent evaluations are claims about a sample from a distribution nobody has characterised, so the honest form is a rate with an interval, not a pass.

Prompt injection makes the input boundary part of the attack surface. An agent that reads a web page has accepted instructions from its author. Any control argument has to say what it assumes about content the agent ingests, and "the user is not adversarial" is not a sufficient assumption when the user is not the only author.

Composition is not covered. Two agents each assured independently, connected by a queue, form a system neither case describes. Authority accumulates across the boundary, and the end-to-end claim has to be made at the level of the composite, which is why assurance arguments for multi-agent deployments remain largely unwritten (Hilton et al., 2025, Safety Cases: A Scalable Approach to Frontier AI Safety, arXiv:2503.04744).