A CI pipeline runs 2,400 tests in 18 minutes and fails about one run in four for reasons unrelated to the change. To stop the interruptions, the team enables automatic retry: any failing test is re-run twice, and the build is green if it passes on any attempt. What happens over the following six months?

Week one: it works

Build failures from flakes drop from roughly 25% of runs to a few percent, because a test that fails independently 10% of the time passes at least once in three attempts about 99.9% of the time. Developers stop re-running builds by hand. Everyone is pleased, and this is the part that makes the change hard to reverse later.

Months one to three: the flake rate rises

Retries remove the pressure that was keeping flakiness down. Before, a flaky test caused visible pain and someone fixed it. Now it is invisible, so new flaky tests accumulate at whatever rate the codebase produces them and none are removed. A rate that was steady because it was being worked on becomes a rate that only increases.

The suite also gets slower in a way nobody attributes to the retries: a run with 30 retried tests spends minutes on re-execution, and the retries are serialised at the end where they extend the critical path.

Months three to six: the signal is gone

Two consequences arrive together.

• Real intermittent bugs are now reported as passes. A race condition in connection pooling that fails one run in ten is precisely a test that retry hides. The defect class that flaky tests are best at detecting is the class retries suppress — concurrency, ordering, timeouts, resource exhaustion — because those are the real bugs whose symptom is intermittency.
• Nobody can tell which is which. After six months of retries, the suite contains an unknown mixture of badly written tests and genuine intermittent defects, and separating them requires the historical per-attempt data that most teams were not recording.

The visible symptom is a production incident whose postmortem finds a test that had been failing and being retried for months.

What stops it

Not banning retries — a hermetic suite still meets the network occasionally. The mechanism is making retries expensive to ignore:

• Record every attempt, not the final result, and publish per-test flake rate as a first-class metric. A test is flaky the moment it passes on retry, and that event goes somewhere visible.
• Cap it: one retry, not two. One retry absorbs genuine infrastructure noise; two is a policy of not wanting to know.
• Quarantine on a threshold. A test above, say, 1% flake rate over 50 runs is removed from the gating suite automatically, assigned to its owning team, and deleted if not fixed within a fixed window. Quarantine without an expiry becomes a graveyard, which is the failure mode of the policy itself.
• Alert on the aggregate. Total retries per day trending up is the leading indicator; the build's green rate is a lagging one that retries were designed to flatter.

When this is the wrong diagnosis

If the flakiness genuinely came from the infrastructure and not from the tests — a shared CI cluster with occasional DNS failures, a container registry that times out — then retrying is correct and the fix is elsewhere. The distinguishing evidence is whether the failures cluster by test or by run. Failures spread across unrelated tests within the same run point at the environment. Failures concentrated in the same 30 tests point at the tests, and the team had a one-in-four failure rate, which is far too high for infrastructure alone.