Write the policy for handling flaky tests. What are the rules?

What the interviewer is testing

Whether you treat flakiness as an emergency with a mechanism, or as a backlog item.

The policy

Detection is automatic. Track per-test pass rate over time across all runs. A test that fails and then passes on re-run with no code change is flagged automatically. Do not rely on humans reporting it.

Quarantine on detection, same day. The test is removed from the blocking gate and continues running in a non-blocking suite, so it stays visible. This is not optional and does not require discussion — the harm of leaving it in the gate exceeds the harm of the coverage gap for a few days.

A fix deadline with an owner. Two weeks, assigned to the owning team. On expiry, the test is deleted. A quarantine with no expiry becomes a graveyard of tests nobody trusts or removes.

Quarantine has a cap. If more than a small percentage of the suite is quarantined, that is a delivery incident requiring a dedicated response, not business as usual.

Investigate before deleting. Flakiness is frequently a symptom of a genuine race condition in the system, not just in the test. Teams that investigate find real concurrency bugs surprisingly often, and the causes are a short list: wall-clock dependence, shared mutable state, order dependence, races in the test's own synchronisation, fixed sleeps instead of conditions, external service reliance, and data left by a previous run.

No blanket auto-retry in the gating suite. Retry as a detection mechanism is fine; retry as a way to go green is how the signal dies.

What a strong answer adds

The reason for the urgency: once "just re-run it" is the reflex, genuine failures are re-run too, sometimes several times, and then merged. The suite has stopped detecting anything while continuing to cost 90 minutes per run.

Common weak answers

Adding retries globally. Logging flaky tests in a backlog that is never prioritised.