Flaky Test

The damage from flakiness is not the wasted re-run. It is that a suite containing known-flaky tests teaches everyone the wrong reflex, and once "re-run it" is the default response to red, genuine failures are re-run too — sometimes several times, then merged.

Causes are a short and recurring list: dependence on wall-clock time or timezone, shared mutable state between tests, order dependence, race conditions in the test's own synchronisation, waiting on a fixed sleep instead of a condition, reliance on an external service, and data left behind by a previous run.

The management practice that works is unsentimental. Detect flakiness automatically by re-running failures and tracking per-test pass rates over time. Quarantine immediately — remove the test from the gating suite so it stops blocking while remaining visible. Then fix or delete on a deadline, because a quarantine with no expiry becomes a graveyard of tests nobody trusts or removes.

The cultural point that makes it stick: flakiness is a defect in the test, and often a signal of a genuine race condition in the system. Teams that investigate flakes rather than muting them find real concurrency bugs surprisingly often.