Behind is not red

In an engineering post published on 14 September, Anthropic put down a number that most organisations would present as a triumph and that it presents as a problem. Its continuous integration ran twenty-five times as many jobs at the end of a six-month period as at the start. The test codebase grew tenfold. Engineering headcount stayed roughly flat. The cause is named without embarrassment: its engineers ship eight times as much code per quarter as they did between 2021 and 2025, and Claude writes eighty per cent of it.

What the post is actually about is not how they paid for that. It is about how they arranged not to.

The mechanism is test impact analysis, and it has two parts, both of which the post is careful to call deterministic. A listener records the result of every test in every CI run. A selector reads that accumulated history and decides, for a given pull request, which tests run — and, the operative half, which do not. The rest of the piece is the story of the listener falling behind, three times, each patch holding for less time than the last: doubling the CPU cores in October 2025 bought seventy days; sharding the listener by package in February 2026 bought twenty-nine; adding a daily restart in March 2026 lasted under a day. The redesign that followed took three weeks and is straightforwardly good engineering — stateless workers appending results to a journal, a consumer rolling that journal into per-test history every few seconds, the selector reading the rolled-up view. The author's own lessons are the right ones for a build team: keep state out of your processes, and assume twenty-five times the load inside two quarters.

They are not the only lessons. Buried in the account of the failure is a sentence worth reading twice, slowly, as an architect rather than as a build engineer. Twenty minutes of listener lag, the post says, could mean tens of thousands of test updates had not reached the selector, risking merged defective code, undetected flaky tests, and regressions from untested changes.

A twenty-minute lag in a service on a read path makes a page stale, and somebody complains. A twenty-minute lag in this service makes a verification result wrong. And a wrong verification result here does not arrive as a failure. It arrives as a pass.

That is the shift worth arguing about. Once selection replaces the full suite, a green check stops being a claim about the diff and becomes a claim about what a separate, stateful, lagging service believed at the moment it chose. Staleness in that service has no red state.

Running every test on every change was dumb and expensive, and it had one property nothing has replaced: it was a total function of the diff. The claim "the suite passed" had a fixed denominator, knowable in advance, reconstructible months later by anyone who could check out the commit. Nobody had to trust a second system to know what the first one had proved.

Selection makes that denominator a computed variable, and the computation belongs to somebody else. There are two families of it, and the difference between them is usually treated as an implementation detail. It is not.

The first family selects from structure. Jest's manual is admirably plain about what that costs: --onlyChanged "attempts to identify which tests to run based on which files have changed in the current repository", and it "requires a static dependency graph (ie. no dynamic requires)". That is a soundness precondition, stated in a CLI reference, in a parenthesis. pytest-testmon does the same job from the other end, building a dependency database from coverage and persisting it to a .testmondata file. Both are wrong when the graph is wrong — a dynamic import, a fixture resolved by name, a configuration file nothing declares a dependency on — and both are wrong in a way an engineer can reason about by reading the code.

Anthropic's selector belongs to the second family. It selects from what tests have done, not from what code calls. That is a more powerful tool and a weaker claim. "This test cannot be affected by this change" is a statement about a program. "This test has not been affected by changes like this one" is a statement about a dataset, and a dataset has a timestamp. Structure-based selection degrades when the code changes shape, which is a thing you can see in a diff. History-based selection degrades when the clock advances and the pipeline does not, which is a thing you can see nowhere in the change at all.

Everywhere else an engineering organisation puts a derived, lagging store, the degraded mode announces itself. A stale cache serves an old price. A stale index misses a document. A stale dashboard shows yesterday. Somebody notices, because being behind looks different from being current. Here, being behind means running fewer tests, and running fewer tests is faster and greener. This is the one derived store in the architecture whose staleness is rewarded by the interface that reports on it.

Nor is it recoverable afterwards, because the artefacts CI actually emits have no vocabulary for it. JUnit XML counts a suite's tests as the "total number of tests in this suite" and defines <skipped> as meaning "the test was not executed" — but a test the selector never chose produces no element at all, so it is not skipped, it is absent. CTRF, the newer open standard for test reporting, is tighter and no more help: a test's status "MUST be one of: passed, failed, skipped, pending, or other", and summary.tests is defined as "the total number of tests executed in the run". Neither format has a status, a count, or a field for a test that existed, was relevant, and was never asked. Neither can record how old the belief was that excluded it. The merge record cannot distinguish a build that ran the right tests from a build that ran what a service four minutes behind thought were the right tests, and it never could, because nobody wrote the field.

The rebuttal to all of this is strong and should be stated at full strength. The full suite was never the proof anybody treats it as. At the scale Anthropic is describing, a suite that is 99.9 per cent deterministic per test is red on most runs for reasons unrelated to the diff, which is why every large organisation retries, quarantines, and eventually stops reading. The denominator was already fictional; selection at least makes the sampling explicit, gives it an owner, and — as this very post demonstrates — gets it instrumented. Anthropic made the harder version of the case itself in February, measuring agentic coding evaluations rather than CI: the gap between the most- and least-resourced infrastructure setups on Terminal-Bench 2.0 was six percentage points, infrastructure error rates ranged from 5.8 per cent under strict enforcement to 0.5 per cent uncapped, and the conclusion was that leaderboard differences below three points deserve scepticism without documented infrastructure. If the environment moves a measured result by six points, the idea that running everything produces a clean verdict was gone before test selection arrived.

Concede it. Verification at scale has been sampling for years. But that concession is the argument rather than the reply to it. If CI is a sampling procedure, then its sampling parameters belong in the record and under a service level objective, the way the parameters of every other sampling procedure an engineering organisation trusts do. What the post describes is a sampling rate set by queue depth — an emergent property of how far behind a consumer happened to be that afternoon. Between sampling deliberately and sampling at whatever rate the journal is running at lies the entire difference between a measurement and an accident. The redesign restored throughput. It did not make the sampling rate a declared quantity, and nothing downstream would have somewhere to put it if it had.

The gate has also quietly changed owners. Selection turns the pull request check into a distributed production service with an availability requirement and, newly, a freshness requirement — and that service belongs to a build or developer-productivity team. Those teams are measured on how fast the gate opens. Nobody has told them they are now accountable for how much it proves. It is an ordinary architectural error, a correctness property relocated into a component whose owners are incentivised on latency, and it has an unusually large blast radius because it sits on the path of every merge.

And the volume forcing the whole arrangement is machine-authored. Evidence per change is being reduced at precisely the moment changes began to be produced by a process whose error distribution nobody has characterised, using a history of relevance accumulated when people wrote the diffs. Inference gets cheaper every quarter. CI compute does not. That pressure points one way.

Anthropic can tell you when its selector was twenty minutes behind because the problem bit them and they went and instrumented it. The teams that adopt this pattern next quarter will inherit the architecture — the journal, the stateless workers, the rolled-up history — without inheriting the reason it exists, and the tick on the pull request will look identical either way.

A red build has always told you that something is wrong. A green one now tells you only that nothing which ran went wrong. How much ran is a number no format records, no gate checks, and nobody has yet been asked to publish.