End-to-End Test Economics

The reason to reason about this numerically is that the intuition "more coverage is better" hides a threshold past which the suite stops functioning.

Take a suite where each test passes reliably 99% of the time — which most teams would call good. Run 500 of them, and the probability that a clean build goes green is 0.99^500, under 1%. The suite now fails almost every run for reasons unrelated to the change, so the team's rational response is to re-run until green, at which point the suite has veto power without detection power.

Add the second cost: end-to-end failures do not localise. Someone spends an hour determining whether a failure is a real defect, a data problem, a timing issue or an environment fault, and that hour is paid per failure per day.

The conclusions that follow are uncomfortable but sound. Keep the end-to-end suite small and curated — the handful of revenue-critical journeys, not a regression net. Move everything else down to contract and component tests that fail deterministically. And treat suite reliability as a hard requirement: an end-to-end test that is not near-perfectly deterministic costs more than it returns and should be deleted rather than tolerated.