Plant something and see

Over August 2026, agents working inside Anthropic made more than a billion decisions. A monitor stood in front of all of them and blocked 0.002% of them, which the report renders as about one in 47,000. A second monitor, reading transcripts after the fact, flagged roughly 100,000 a week. Of those, about fifty reached a person.

The figures were published on 17 September by Marina Favaro and Phillie Wright, as one of three measurements Anthropic proposes that frontier labs should report in public: how much of the lab's own AI research is now led by AI, how its internal agents are overseen, and what share of research compute goes to safety work. The stated reason is unusually plain. "As the world considers pacing the frontier," the report says, "we should do everything possible to minimize the gap between what frontier labs know and what the public knows."

That is a real disclosure, and as far as I can establish no other lab has published its equivalent. It deserves to be read closely rather than applauded, and reading it closely turns up a property of the oversight numbers that matters before anyone starts comparing labs on them. Coverage, review latency and escalation rate cannot fall when oversight fails. Coverage is defined as "the share of an agent's actions that pass through a monitor before or after execution": it is already at its maximum, and it describes the plumbing. Latency measures speed. Escalation rate is a threshold the lab chooses, which makes it a setting reported as a finding. The quantity that would drop if the monitors stopped catching things is the share of genuinely bad actions detected, and it is not among the three.

What a monitor actually does

There are two kinds here and the difference matters. An online monitor sits in the path of the action. It sees the action before or after it executes, has to decide inside whatever delay the agent can tolerate, and can block or redirect. An offline monitor reads whole transcripts afterwards. It can spend far more computation and see far more context, and it cannot prevent anything, because whatever it is reading has already happened.

Both are classifiers, and a classifier does not emit the word "bad". It emits a score. ControlArena, the open-source AI control library from the UK AI Security Institute and Redwood Research, calls it a suspicion score, and the name is honest about what it is: a number expressing how much a particular trajectory resembles the kind of thing the classifier was built to notice. A threshold then turns the score into an action. Every figure downstream is a function of where that threshold sits. Move it down and the escalation rate rises. Move it up and it falls. Nothing about the agents needs to change for either to happen.

The threshold is not set arbitrarily, and the constraint that sets it is arithmetic. Suppose one action in a million is genuinely harmful. Across a billion decisions, that is a thousand of them. Now suppose a very good classifier: it catches nine in ten, and raises a false alarm on one action in a thousand. It will flag 900 real problems and about a million false ones. Fewer than one flag in a thousand is a true positive. The numbers in that calculation are invented, but the shape is not, and it is visible in the report's own funnel. Roughly 100,000 transcripts are flagged each week, then "reviewed by multiple classifiers for false-positives and initial oversight review", most are resolved by that process, and "the highest priority flags (approximately ~50 per week) are escalated to human review". That cascade is a precision filter. It exists because the raw flag stream is mostly noise, and it has to be, given the base rate.

Here is the part worth carrying away. The cascade tells you about precision and tells you nothing about recall. A false alarm can always be checked, because you have the flagged transcript in front of you. What the monitor never flagged leaves no trace, because nothing marks it. The denominator does not exist. This is the ordinary condition of monitoring without labels, and it is why a count of blocks or alerts is a numerator-only statistic. Of the three published measures, coverage says the action passed through a monitor, latency says how quickly the monitor answered, and escalation rate says where the threshold was set. None of them says how much got through.

Knowing what you do not know

Anthropic says a version of this itself, in a caveat that is easy to skim past: "At this scale of agent activity, it is hard to be certain we have a complete picture of all possible kinds of behaviors, or that we are making these patterns adequately legible to human oversight." Set that against the stated aim of narrowing the gap between what the lab knows and what the public knows. On the efficacy of oversight, that gap is narrow already, because the lab does not know either. Publishing the numbers transfers an uncertainty rather than a piece of knowledge. That is worth doing, so long as nobody reads the result as a grade.

The sibling metric in the same report shows the authors understand this class of problem well. The automation index says Claude now leads 26% of Anthropic's AI research and development work, up from under 1% in February 2026, assessed by a judge model against a task tree whose 542 nodes bottom out in 378 leaves with names like "serving incident postmortems". Printed beside it is a figure most organisations would have cut: on the automation-level scale, "model-versus-human exact agreement was 59%, human-versus-human was 35%". The people who defined the scale agreed on the exact level about a third of the time. The judge agreeing with humans more often than humans agreed with each other is not evidence that the judge is better than people. It is what you expect when a model sits near the middle of a noisy label distribution while human raters scatter around it. Publishing that ratio tells a reader precisely how much weight the 26% can bear. The oversight numbers come with no equivalent.

The strongest objection

It goes like this. You are demanding a recall figure over a population nobody can label, which is not a gap in the reporting but a fact about the world. And the design does not rest on monitors alone: agents carry persistent identities, and "Because agents can see each others' messages, they can correct each others' errors, and coordinate about shared work." That distributes oversight into the agent population itself, which scales in a way that fifty transcripts a week plainly does not. This is also a first publication, explicit that the monitoring has run for months rather than years, and released with its methodology so that others can copy it. Insisting on the hardest possible metric in version one is an efficient way to get no version two.

The first half of that is right about the wild population and wrong about the method. You do not label a billion decisions. You insert a known one. That is exactly what a control evaluation does. ControlArena builds settings containing a main task the model is supposed to do and a side task a misaligned model might pursue alongside it, runs an attack policy that pursues both, and scores whether the protocol's monitor caught it. The resulting safety number goes down when the monitor is bad, which is the property the three published measures lack. Inspect Petri, another open auditing agent, generates its own audit scenarios, runs them against a target model and scores the transcripts with a judge on a fixed rubric. The tooling is public and maintained. What a detection rate against planted behaviour costs is not feasibility. It is the willingness to publish a number that might come out low.

The second half, about agents correcting each other, is true and load-bearing and also correlated. Agents that can read each other's messages do catch each other's mistakes, but they are drawn from the same model and share its blind spots. Thirty thousand of them checking each other is not thirty thousand independent checks; it is closer to one check repeated many times, and the failure modes that matter are the ones the family does not see. The report's own identity design makes the series harder to read as it accumulates: "because the identity is not tied to a model, it persists through model upgrades". Continuity of the record is the right call. The consequence is that a shift in escalation rate between one quarter and the next could be a new model, a new threshold or new behaviour, and the published numbers will not separate them.

None of this is hypothetical pressure. A week before these measurements appeared, Anthropic's own threat intelligence report described external actors running parallel agent swarms unattended, with persistent campaign memory, across reconnaissance, exploitation and malware development. Whatever detects that is a monitor whose recall nobody has measured.

So: when you meet a monitoring statistic, ask what its denominator is. Blocks, flags and escalations are all numerator. Separate the four things a monitoring system can report, because they are routinely quoted as one: whether the pipe is connected, how fast it answers, where the threshold sits, and how much it catches. Only the last is about catching, and only the last requires an experiment. If you build agent monitoring yourself, plant faults deliberately from the first week, because manufacturing ground truth later is much harder than creating it as you go. And read the reliability disclosure before the headline figure. In this report, 35% is the more informative number than 26%.

The part of the announcement with real teeth is not one of the three metrics. It is the commitment to "embed independent third-party evaluators from multiple organizations at Anthropic, and give them access to internal processes, systems, and data comparable to what internal risk assessment teams have". Someone with that access can run the experiment the metrics do not contain, and can report a number the lab did not choose the threshold for.

A billion decisions passed a monitor last month, and about fifty conversations reached a human being. Fifty may well be the right number. At that scale no larger one would mean very much, and a lab that reviewed more would mostly be reviewing noise. The uncomfortable part is not the ratio. It is that nothing in the three published figures would read any differently if the fifty had been the wrong fifty.