An end-to-end suite has 200 tests. Each fails spuriously about 1% of the time, independently of the others. Roughly what fraction of runs on correct code go red anyway?

The assumptions, stated

• 200 tests, each independently green on correct code with probability 0.99.
• A run is red if any test is red.
• Independence, which is generous and discussed below.

The arithmetic

P(all 200 pass) = 0.99^200. Take logs: 200 × ln(0.99) ≈ 200 × (−0.01005) ≈ −2.01, and e^(−2.01) ≈ 0.134.

So about 13% of runs are green and about 87% are red for no reason. The useful way to hold this is the rule of thumb that 0.99^n falls by a factor of e every 100 tests: 100 tests gives 37% green, 200 gives 13%, 300 gives 5%.

Which assumption dominates the error

Independence. Real flakes cluster, because they share a cause: one shared environment, one slow fixture, one clock. Clustering makes the distribution burstier — long green streaks punctuated by a red afternoon — and typically makes the average slightly better than the independent model. It does not rescue the order of magnitude, and it makes the experience worse, because a red afternoon is when people stop believing the suite.

Why the other options fail

• 1% treats the suite as one test. It is the answer people give instinctively, and it is what makes a 1% flake rate sound acceptable in a planning meeting.
• 13% is the green rate, not the red rate. It is the most common mistake among people who do the arithmetic correctly and then read the wrong end of it.
• 63% is the answer for 100 tests (1 − 0.37). Right method, wrong n, which is exactly what happens when a suite doubles and nobody redoes the sum.

What the number rules out

You cannot have 200 network-crossing tests at 1% flake and also have a red build mean something. Working backwards: for a 95% green rate at 200 tests, each test must be green 0.9997 of the time — roughly 2.6 spurious failures per 10,000 runs. That number is the real argument for the testing pyramid, and it is not an aesthetic one: you cannot buy that reliability from a test that crosses a network, a scheduler and a browser, at any price.

So either the suite shrinks to the dozen journeys that genuinely need end-to-end coverage, or the per-test rate improves by a factor of forty, and only one of those is achievable in a quarter.

When this is the wrong analysis

If the failures are not independent because they share an environment — as they usually are on a travel platform of Booking.com's shape, where one shared staging database backs every suite — the fix is not fewer tests, it is environment isolation per run. Counting tests when the cause is a shared fixture produces the wrong remedy, and the diagnostic is simple: do the red tests cluster in time, or in the suite?