Your test quarantine has grown to 140 tests over a year. What has gone wrong and how do you recover?
Show the full answer Hide the answer
What went wrong
Quarantine without a cap and without a deadline becomes a graveyard. Each individual decision was reasonable — move the flaky test aside, raise a ticket, keep shipping — and the aggregate is that coverage is now theoretical over whatever those 140 tests covered.
Worse, nobody knows what that is. The quarantine is a list of test names, not a statement of what risk is currently unverified.
Recover in three steps
1. Establish what is actually uncovered. Map the 140 tests to the behaviour they verified. Expect three groups: behaviour covered elsewhere at a lower level (genuinely redundant), behaviour covered nowhere (the real exposure), and tests for features that no longer exist.
2. Triage by that mapping, not by test. Delete the redundant and the obsolete — this is usually a large fraction and it is not a loss. For the genuinely uncovered behaviour, decide whether to fix the test or to cover it at a level where it will not be flaky, which is frequently the better answer: a flaky end-to-end test often becomes a reliable contract or integration test.
3. Investigate the causes rather than assuming. Categorise: timing and sleeps, shared state and ordering, external dependencies, and real non-determinism in the system. The last category matters most — a meaningful proportion of flaky tests are correctly detecting a race, a time dependency or an ordering assumption, and deleting them removes the only evidence of a genuine defect.
Prevent recurrence
A cap: no more than N quarantined at once, and exceeding it stops feature work. A deadline: after it, the test is fixed or deleted, deliberately. A visible metric: flakiness rate with a threshold. And treating a newly flaky test as a defect in the pull request that introduced it, while the author still has context.
The measure of success
Not zero quarantined tests. It is that red means broken and nobody reruns without reading — which is what the whole exercise is protecting.