A test suite has a 4% flake rate. The team re-runs failed builds. What is the actual cost, and what policy fixes it?

The actual cost

The signal is gone. At a 4% flake rate, a red build is more likely to be noise than a real failure, so the rational response is to re-run — and the team has now trained itself to ignore failures, including the genuine ones. A real regression is re-run, passes on the second attempt for an unrelated reason, and ships.

Flaky tests are worse than no tests, because a missing test is a known gap while a flaky one is a false assurance.

Secondary costs: re-runs consume pipeline capacity and lengthen lead time, and engineers stop trusting the suite enough to act on it.

The policy that fixes it

• A hard flake budget, treated as a defect rather than as a fact of life. Above the threshold, fixing flakes takes priority over feature work.
• Automatic detection: run the suite against an unchanged commit periodically, and any test that fails is flaky by definition. This removes the argument about whether a failure was real.
• Automatic quarantine on detection: the test is removed from the blocking suite, an issue is created with an owner and a deadline, and it continues running in a non-blocking mode.
• A deadline on quarantine, after which the test is deleted. A quarantined test with no deadline is a test that has been deleted without anyone deciding to, and pretending otherwise makes the coverage figure a lie.
• No re-run button on the blocking suite, or the policy has no teeth. Re-runs are what make flakiness survivable and therefore permanent.

The usual causes

Timing — a sleep instead of a wait for a condition, or an assumption about ordering · shared state between tests, so results depend on execution order · real dependencies with their own variability · resource contention on the build agent · and non-determinism in the system under test, which is occasionally a real bug the flaky test is correctly detecting.

That last case is worth taking seriously before deleting anything: a test that fails one time in twenty may be finding a race condition that will fail in production one time in twenty thousand.

The structural fix

Push logic down to levels that can be tested deterministically. Most flakiness lives in end-to-end tests, and the number of those should be small and chosen deliberately rather than being whatever accumulated — because they are expensive to write, slow to run and the most likely to be flaky.