Long-Horizon Agent Reliability

A 99 percent reliable step sounds excellent until you run fifty of them. \(0.99^{50} = 0.61\). The agent that passed every unit test in your five-step evaluation fails four times out of ten on a realistic task, and every one of those failures is a step that looked fine in isolation.

This compounding is the central fact of agent engineering, and it explains a pattern that otherwise looks like bad luck: capability improvements that read as small on single-turn benchmarks produce large changes in what agents can finish.

The time-horizon reframing

METR proposed measuring capability in units of human time rather than benchmark accuracy: the 50 percent time horizon is the length of task, measured by how long a human expert takes, that a model completes with 50 percent success. Their fitted trend has this horizon doubling roughly every seven months since 2019, with a 95 percent bootstrapped confidence interval on the doubling time of 166 to 240 days (METR, 2025, arXiv:2503.14499).

Two things make this a better metric than raw pass rates. It is comparable across benchmark generations, because human time does not saturate. And it captures the thing product teams actually care about: not whether the agent can do the step, but how long it can go before something breaks.

Where the errors come from

Not all step failures are equal, and lumping them together hides the fix.

Independent slips. A malformed tool argument, a transient 503. These are the errors retries were built for, and they are the least interesting because a retry loop plus idempotent tools handles them.

Correlated errors. The agent misreads the schema once and every subsequent query is wrong the same way. Retrying reproduces the failure. Only re-grounding against the source fixes it.

Context-induced drift. As the transcript grows, earlier constraints lose salience, a degradation that has been measured directly on retrieval tasks well inside the advertised window (Chroma Research, 2025, Context Rot). The agent stops honouring an instruction it followed for the first twenty steps. See context rot.

Silent success. The worst class: the agent reports done, and the work is wrong. Nothing in the loop catches it because the loop's only success signal is the model's own claim.

What actually raises the exponent

Given \(P(\text{task}) \approx \prod_i p_i\), there are exactly three moves.

Raise \(p_i\). Better models help, and so does making each step easier: narrower tools, typed arguments, tools that fail loudly rather than returning an empty list.

Reduce \(k\). Every step removed is a multiplicative gain. Replacing a six-step "list, filter, sort, page, fetch, parse" sequence with one purpose-built tool is worth more than any prompt improvement to those six steps.

Break independence with verification. This is the one that changes the shape of the curve. If a checker catches a fraction \(c\) of step errors and the agent can recover, the effective per-step failure rate becomes \((1 - p_i)(1 - c)\). At \(p_i = 0.95\) and \(c = 0.8\), thirty steps goes from a 21 percent success rate to 74 percent. Verification is worth more than accuracy, because it is a different term in the product.

The corollary is that checkpointing matters as much as checking. An agent that detects a failure at step 28 and must restart from step 1 has gained nothing; one that can resume from step 25 has converted a task failure into a step retry.

When it breaks

Verifiers are not free and not perfect. A checker that costs a model call per step doubles the bill, and a checker with a 10 percent false-positive rate makes the agent thrash on work that was already correct.

τ-bench's pass^k is the honest metric. Running the same task \(k\) times and requiring all \(k\) to succeed exposes policy inconsistency that a single-run pass rate hides; reported pass^8 rates fall dramatically below pass^1 for frontier models (Yao et al., 2024, τ-bench, arXiv:2406.12045). If your agent is only right most of the time, it is not automatable.

Human time is a proxy, not a measure of difficulty. METR's own paper is careful about this: tasks that take humans a long time because they are tedious are not the same as tasks that take a long time because they are hard, and agents are much better at the first kind.