A suite has persistent flaky tests. Why is tolerating them worse than deleting them, and what policy works?
Show the full answer Hide the answer
Why tolerating is worse than deleting
A flaky test destroys the value of the reliable tests around it.
Once a suite fails intermittently for unrelated reasons, the rational response to any failure is to re-run. Everyone learns this quickly. From that point a genuine failure is also re-run, and re-run again, and eventually merged — because the failure looks like the flakiness everyone has learned to ignore.
A flaky test therefore has negative value: it provides no assurance and removes the assurance of every test next to it. Deleting it restores the signal.
The policy that works
1. Detect automatically. Track pass/fail history per test; a test that fails and then passes on the same commit is flaky by definition, and this can be measured rather than reported anecdotally.
2. Quarantine immediately. Remove it from the blocking suite the moment it is detected, so it stops degrading trust. Continue running it, non-blocking, to gather data.
3. Fix or delete within a defined window — commonly a week or two. Quarantine is a holding state, not a destination. A quarantine population that only grows is the same problem with an extra step.
4. Attribute to an owner, since an unowned flaky test is never fixed.
5. Track the flakiness rate as a headline metric, so the trend is visible and the worst offenders are known.
The common causes worth checking first
- Shared mutable test data, which makes tests order-dependent and defeats parallelism.
- Timing assumptions — fixed sleeps, races between an action and its observation.
- Real external dependencies, which fail for reasons outside the team's control.
- Non-deterministic inputs — random values, wall-clock time, ambient configuration.
- Resource contention in parallel execution.
Most of these are architectural rather than test defects, which is why flakiness frequently indicates that the code cannot be tested deterministically — and fixing the injection points removes whole categories at once.