A 90-engineer monorepo adds a merge queue that batches up to 8 pull requests, runs the full suite once per batch, and merges the batch if it passes. The suite has about a 4% chance of a spurious failure per run. Throughput is far worse than before. What happens, and in what order would you fix it?

The arithmetic first

A batch of 8 passes only if nothing fails spuriously. If the spurious-failure probability is 4% per run, a batch's chance of a clean run is 0.96 to the power of 1 for the suite as a whole — but the relevant failure surface scales with the number of tests exercised by 8 changes rather than 1. Take the conservative reading that the 4% applies per run: about 1 batch in 25 fails for no reason. Take the reading many teams actually experience, where each change's touched tests carry their own flake risk, and 8 changes give roughly 1 − 0.96^8 ≈ 28% of batches failing spuriously.

Then the mechanism that makes it much worse than those numbers suggest.

What happens, step by step

A failed batch tells you nothing about which change failed. The queue must bisect: split the batch, re-run, split again. A batch of 8 needs about 3 further rounds in the worst case, so one spurious failure costs 4 suite runs instead of 1.

Every one of those re-runs carries the same flake probability, so bisection can fail spuriously too, and the queue can spend several rounds concluding nothing.

Meanwhile the queue is blocked. Pull requests arrive at the same rate and are not merging, so the queue lengthens, which increases the number of changes per batch if the queue sizes batches dynamically, which raises the spurious-failure rate per batch. The congestion is self-reinforcing, and the usual experience is that the queue is fine for two weeks and then collapses on a Tuesday.

The second-order effect is social. Engineers learn that merging takes hours, so they batch more work per pull request, which makes each change larger, which makes each failure harder to attribute. The merge queue has made batch size worse, which is the opposite of what it was adopted for.

The fix, in order

1. Measure and publish the flake rate per test. Re-run failures automatically once and record which tests pass on retry. You cannot fix a flake rate you are not measuring, and the distribution is almost always concentrated: a handful of tests cause most of it.

2. Quarantine the worst offenders out of the gating suite, with an owner and a deadline, and run them non-gating. This is the one legitimate use of quarantine, and it needs a hard expiry or the quarantine list becomes the test suite.

3. Shrink the batch size to 2 or 3 until the flake rate is under about 0.5% per run. Batch size is the multiplier on both the spurious-failure probability and the bisection cost, and it is a configuration change available today.

4. Make bisection cheap with change-aware test selection, so a batch's failure re-runs only the tests reachable from each change rather than the whole suite.

5. Only then raise the batch size again, with the flake rate as the gate.

When a merge queue is the wrong tool

When the suite is slow rather than flaky, and the team is small. A queue adds latency to every merge in exchange for guaranteeing trunk is green. For 10 engineers with a 10-minute suite, merging directly and reverting the occasional break is cheaper, because the cost of a broken trunk is small and the cost of queue latency is paid on every change.

The queue earns its place when a broken trunk blocks many people at once — which is exactly the 90-engineer case — but only above a flake-rate threshold it cannot fix by itself. Adopting a merge queue onto a flaky suite converts a tolerable problem into a throughput collapse.

Common weak answers

• "Increase CI capacity." The queue is serialised by design; more runners shorten each run and do not reduce the number of rounds. It helps a little and misses the mechanism.
• "Re-run failed batches automatically." Without measurement this hides the flake rate and doubles the cost, and the queue still blocks. Retry is a data-collection tool here, not a fix.
• "Go back to merging directly." Defensible for a small team, and at 90 engineers it means trunk is broken often enough to block everyone. The flake rate is the actual problem and it will hurt either way.