Flake Budget

At a few percent flakiness, a red build is more likely to be noise than a real failure. The rational response is to re-run, and the team has now trained itself to ignore failures — including the genuine ones. A real regression is re-run, passes for an unrelated reason, and ships.

Flaky tests are worse than no tests, because a missing test is a known gap while a flaky one is false assurance.

Why it matters

Test suites are trusted or ignored, and the transition is sharp. Once the team's default response to red is re-run, the suite's entire value has been lost regardless of how comprehensive it is — and the coverage number continues to look excellent.

Implementation patterns

• Automatic detection: run the suite against an unchanged commit periodically, and any test that fails is flaky by definition. This removes the argument about whether a given failure was real.
• Automatic quarantine on detection: removed from the blocking suite, an issue opened with an owner, still running non-blocking so the flakiness can be characterised.
• A deadline on quarantine, after which the test is deleted. A quarantined test with no deadline has been deleted without anyone deciding to, and pretending otherwise makes the coverage figure a lie.
• No re-run button on the blocking suite, or the policy has no teeth — re-runs are what make flakiness survivable and therefore permanent.
• A budget treated as a defect, so above the threshold fixing flakes takes priority over feature work.

The causes, and the one worth pausing on

Timing, from a sleep rather than a wait for a condition · shared state, so results depend on execution order · real dependencies with their own variability · resource contention on the agent.

And genuine non-determinism in the system under test, which is occasionally a real bug the flaky test is correctly detecting. A test failing one time in twenty may be finding a race condition that will fail in production one time in twenty thousand — which is worth investigating before deleting anything.

Industry example

Testing platforms such as BrowserStack see this from both sides: their own suites and their customers'. The consistent finding is that flakiness concentrates in end-to-end tests, which is why the number of those should be small and chosen deliberately rather than being whatever accumulated — they are the most expensive to write, the slowest, and the most likely to be flaky.

Failure scenarios

• A re-run button, which makes the problem permanent.
• Quarantine without a deadline, which is silent deletion.
• Flakiness accepted as normal, after which the suite is theatre.
• Deleting a flaky test that was detecting a real race.
• No detection mechanism, so every failure is argued about individually.

Trade-offs

Quarantining a test removes protection, and the deadline means it may be deleted rather than fixed — losing coverage that was real when the test was reliable.

The counter is that the coverage was already lost: a test whose failures are ignored provides nothing. The policy makes the loss explicit and time-bounded rather than silent and permanent, which is the entire argument.

Interview question

"Your suite flakes 4% of the time and the team re-runs. Tell me what that costs, and write me the policy you would propose — including what you would do about the test everyone insists is important and has never been reliable."