Suite Reliability Arithmetic

"One percent flaky" sounds acceptable, and for one test it is. For a suite it is arithmetic: a run is green only if every test is green, so the suite's reliability is the per-test rate raised to the number of tests.

At 200 tests and 1% each, P(green) = 0.99²⁰⁰ ≈ 0.134. About 87% of runs on correct code go red. The team's lived experience — "the build is always red, just re-run it" — is not a cultural failure. It is the only rational response to a suite that is wrong six times out of seven.

Running it backwards gives the budget. For a 95% green rate across 200 tests, each test must pass 0.95^(1/200) ≈ 0.99974 of the time: about 2.6 spurious failures per 10,000 runs.

Why it matters

It is the quantitative argument for the testing pyramid, and it is much stronger than the usual aesthetic one. You cannot buy 2.6-in-10,000 reliability from a test that crosses a network, a scheduler, a browser and a shared database, at any price. So either the count of such tests stays small or the build's verdict stops meaning anything — there is no third option, and no amount of discipline changes the exponent.

It also explains why suites degrade suddenly rather than gradually. Doubling the suite squares the failure probability, so a team that was at 60% green passes through 36% on its next doubling, which is the point at which people stop reading failures.

Implementation patterns

• Do the inverse calculation before sizing the suite. Target green rate to the power of 1/n gives the per-test rate required; compare it against the rate your existing tests actually achieve. This is the number that settles arguments about adding 50 more end-to-end tests.
• Recompute on every doubling. The exponent moves, so a suite that was tolerable at 100 tests is not at 200, and nobody redoes the sum unprompted.
• Cap the high-level suite by the arithmetic, not by ambition. If the calculation allows 12 network-crossing tests at the rate those tests can actually achieve, the suite has 12 and the remaining coverage moves down the pyramid.
• Separate the independent model from the correlated one. Plot failures over time before acting: evenly spread failures fit the exponent, while clustered ones mean a shared cause and the arithmetic gives the wrong remedy.
• Express the result as a rate per 10,000 runs, not a percentage, because percentages at this magnitude stop being intuitive and the conversation reverts to opinion.

Industry example

Google's published work on flaky tests reported that a substantial minority of its tests exhibited some flakiness and that a meaningful share of test executions were reruns, which is why large engineering organisations build automatic rerun-and-classify infrastructure rather than relying on engineers to report flakes. The infrastructure exists because the arithmetic makes manual triage hopeless at scale.

Failure scenarios

• Retry-until-green, where the pipeline reruns until it passes and a real regression is retried away.
• Silent coverage loss, where quarantine has no deadline and a quarter of the suite is non-blocking without anyone deciding that.
• Correlated failure mistaken for volume: the red afternoon is one shared fixture, and the team responds by deleting unrelated tests.
• The verdict losing meaning, after which a genuine break is indistinguishable from the background and ships.
• Budget set from the mean run, ignoring that flakes cluster, so the experienced rate is far worse than the calculated one.

Trade-offs

Meeting the budget means fewer high-level tests, and fewer high-level tests means some integration defects reach production. That is a real cost. It is paid against a suite whose signal is trustworthy, which is worth more, because a trustworthy red build stops a bad change and an untrustworthy one stops nothing. The exchange is coverage for signal, and signal is the thing the suite exists to provide.

When not to use it

When failures are correlated rather than independent, the arithmetic gives the wrong remedy. If the red tests cluster in time — one bad afternoon, then a week of green — the cause is a shared environment or fixture, and cutting the test count fixes nothing while removing coverage. Diagnose the correlation first: do failures cluster in time, or spread evenly across runs? Only the second is a per-test flake problem.

Interview question

Q: A team has 200 end-to-end tests each about 1% flaky and asks whether to add 50 more. Answer with arithmetic, then tell them what to do.

What a strong answer covers: 0.99²⁰⁰ ≈ 13% green, so 87% of runs are already spurious, and 250 tests takes it to about 8%; the inverse calculation giving a per-test budget near 2.6 in 10,000 for a 95% green rate; that this rate is unachievable for network-crossing tests, so the suite must shrink rather than grow; automatic rerun-and-classify to measure the real rate; and the check for correlation before acting on the independent model.

Quick check

Quiz: A 200-test suite needs a 95% green rate. What per-test flake rate does that allow? About 2.6 spurious failures per 10,000 runs — 0.95^(1/200) ≈ 0.99974 — which is why the suite must be small rather than merely well-written.

Flashcard: Why does "only 1% flaky" fail at suite scale? Because run reliability is the per-test rate to the power of the test count: 0.99²⁰⁰ ≈ 13% green, so 87% of runs go red on correct code and re-running becomes the rational response.