A team's CI suite fails roughly one run in three for reasons unrelated to the change. Everyone reruns until green. How do you recover the situation?

Recognise what has actually been lost

The suite is no longer a gate. Once the team's reflex on red is "rerun", that reflex is applied to genuine failures too, and a real regression will be rerun into production. The damage is to the signal, and restoring the signal is the objective — not fixing every flaky test.

Immediate: quarantine, do not delete

1. Detect automatically. The same commit passing and failing is a sufficient definition; most CI systems can report it.
2. Move failing tests out of the blocking suite into a non-blocking job that still runs, so their behaviour stays visible.
3. Raise a ticket per quarantined test, with an owner and a deadline.

Within a day the blocking suite is trustworthy again, which is the point. Red means broken.

Two rules so quarantine does not become a graveyard

• A cap. No more than N tests quarantined at once; exceeding it stops feature work. Without a cap the quarantine grows until coverage is theoretical.
• A deadline. After it, the test is fixed or deleted. Deleting is a legitimate outcome — an unowned test nobody will fix is not providing coverage.

Then investigate causes rather than assuming

Categorise the quarantined tests. The distribution is informative:

• Timing and sleeps — waits for a fixed duration rather than a condition. Fixable in the test.
• Shared state and ordering — tests that pass alone and fail in parallel. Usually a shared database or fixture; fix by isolating.
• External dependencies — a real third party in an integration test. Virtualise it.
• Real non-determinism in the system. A race, a time-of-day dependency, an unhandled ordering assumption. This is the important category. A meaningful proportion of flaky tests are correctly detecting a genuine defect, and deleting them removes the only evidence.

The last category is why "just delete the flaky ones" is wrong advice, and why the investigation has to be real rather than a triage exercise.

Prevent recurrence

• Make the flakiness rate a visible metric with a threshold.
• Retries at the test-framework level are acceptable only if the retry is recorded and counted — a silent retry hides the problem permanently.
• Treat a new flaky test as a defect in the pull request that introduced it, while the author still has context.

What good looks like afterwards

Red means broken. Nobody reruns without reading. The quarantine list is short and shrinking, and two or three of the tickets closed with a fix to the system rather than to the test — which is the outcome that pays for the whole exercise.