Flake Rate

A test suite's value is its ability to tell you that something is wrong. That value collapses sharply rather than gradually as flakiness rises, because at a few percent flake rate across a large suite, a red build is more likely to be noise than a real failure — so re-running becomes the rational response.

Once re-running is rational, the team has trained itself to ignore failures, including the genuine ones. A real regression is re-run, passes on the second attempt for an unrelated reason, and ships.

Flaky tests are worse than no tests, because a missing test is a known gap while a flaky one is false assurance — and the coverage figure continues to look excellent.

Why it matters

The transition is sharp and the loss is total. A suite whose default response to red is "re-run" provides no protection regardless of how comprehensive it is, and the organisation continues to pay for writing, maintaining and running it.

The secondary costs are also real: re-runs consume pipeline capacity, lengthen lead time, and consume engineer attention on failures that mean nothing.

Implementation patterns

• Automatic detection: run the suite repeatedly against an unchanged commit, and any test that fails is flaky by definition. This removes the argument about whether a given failure was real, which is otherwise litigated case by case and always resolved in favour of re-running.
• Automatic quarantine on detection — removed from the blocking suite, an issue opened with an owner, still running non-blocking so the flakiness can be characterised.
• A deadline on quarantine, after which the test is deleted. A quarantined test with no deadline has been deleted without anyone deciding to, and pretending otherwise makes the coverage figure a lie.
• No re-run button on the blocking suite, or the policy has no teeth — re-runs are what make flakiness survivable and therefore permanent.
• A flake budget treated as a defect, so above the threshold fixing flakes takes priority over feature work.
• Instrumentation of the suite: which tests fail, how often, and how often the failure was real. Most organisations lack this and it changes the conversation immediately.

The causes, and the one worth pausing on

Timing, from a sleep rather than a wait for a condition · shared state between tests, so results depend on execution order — one of the largest contributors · real dependencies with their own variability · resource contention on the build agent · and genuine non-determinism in the system under test.

That last category is worth taking seriously before deleting anything. A test failing one time in twenty may be detecting a race condition that will fail in production one time in twenty thousand — which is a real defect the test is correctly finding, and deleting the test removes the only evidence of it.

Industry example

Testing platforms and large engineering organisations report the same pattern from both sides: flakiness concentrates in end-to-end tests, which is the structural argument for keeping that layer small and deliberately chosen rather than comprehensive. They are the most expensive to write, the slowest, the hardest to diagnose, and the most likely to be flaky.

The consistent organisational finding is that flakiness is not fixed by exhortation but by removing the escape route — the re-run button — and by giving quarantine a deadline.

Failure scenarios

• A re-run button on the blocking suite, which makes the problem permanent.
• Quarantine without a deadline, which is silent deletion with extra steps.
• Flakiness accepted as normal, after which the suite is theatre.
• Deleting a flaky test that was detecting a real race, losing the only signal of a genuine defect.
• No detection mechanism, so every failure is argued about individually and re-running always wins.
• Suite instrumentation absent, so nobody knows which tests are the problem.
• Shared mutable state between tests, making results order-dependent — the most common structural cause.

Trade-offs

Quarantining a test removes protection, and the deadline means it may be deleted rather than fixed, losing coverage that was real when the test was reliable.

The counter is that the coverage was already lost: a test whose failures are ignored provides nothing. The policy makes the loss explicit and time-bounded rather than silent and permanent, which is the entire argument.

There is also a real cost in enforcing a no-re-run policy: a genuinely flaky infrastructure problem now blocks everyone until it is fixed, which is disruptive and is the intended pressure.

The trade is short-term disruption and some lost coverage in exchange for a suite whose red means something. For a team whose suite is already ignored, there is nothing to lose — which is the situation most teams adopting this are actually in.

Interview question

"Our suite flakes about 4% of the time and the team re-runs failed builds. Tell me what that actually costs us, write me the policy you would propose, and tell me what you would do about the test everyone insists is important and has never been reliable."