Flake-Bound Merge Throughput
also called Merge Queue Collapse, Batch Flake Multiplier
The ceiling a merge queue's throughput hits as a function of batch size and per-run spurious failure rate - which collapses rather than degrades, because each false failure costs a bisection.
A 90-engineer monorepo adopts a merge queue batching 8 pull requests, running the suite once per batch, merging if it passes. Within three weeks merging takes hours. The suite has about a 4% spurious failure rate, and that one number, combined with the batch size, predicts the collapse.
Flake-bound merge throughput is the relationship: as batch size rises, the probability that a batch contains at least one spurious failure rises geometrically, and each such failure costs a bisection rather than a single re-run.
Why it matters
It explains a non-obvious observation: a merge queue added to a flaky suite makes delivery worse than no queue at all, and it does so suddenly. Teams describe it as working for two weeks and then collapsing, which is the signature of a threshold rather than a gradual degradation.
The arithmetic: with per-change flake risk f, a batch of n changes has a clean-run probability of
roughly (1 − f)^n. At f = 4% and n = 8, about 28% of batches fail for no reason. Each
failure then needs bisection — about log₂(n) further rounds, so 3 more suite runs for a batch of 8
— and every one of those rounds carries the same flake probability, so bisection can conclude
nothing and repeat.
Then congestion feeds back. Pull requests arrive at the same rate while nothing merges, so the queue
lengthens; if batches are sized dynamically they grow, which raises n, which raises the failure
rate. The loop is self-reinforcing, which is why the failure is a collapse.
Implementation patterns
- Measure the flake rate before adopting a queue. Auto-retry each failure once and record which tests pass on retry. The distribution is almost always concentrated in a handful of tests.
- Target under about 0.5% spurious failure per run before batching aggressively. At that rate a batch of 8 fails spuriously about 4% of the time rather than 28%.
- Size the batch from the measured rate, and treat batch size as a dial rather than a constant. Two or three is a reasonable holding position while the suite is being fixed.
- Quarantine the worst offenders out of the gating set, with a named owner and a hard expiry date. This is the one legitimate use of quarantine, and without the expiry the quarantine list becomes the test suite.
- Make bisection cheap with change-aware test selection, so a failed batch re-runs only the tests reachable from each change.
- Alert on queue wait time at the 90th percentile, not the mean, because the mean stays reasonable while the tail is what people experience.
Industry example
Merge queues exist in GitHub, GitLab and Bazel-based monorepo tooling precisely because a broken trunk blocks everyone at large team sizes, and the public engineering writing on them is consistent on the prerequisite: the gating suite must be reliable first. Google's published work on flaky tests through the 2010s reported that a substantial minority of its test failures were flaky rather than genuine, and that identifying and isolating them was treated as continuous infrastructure work rather than a project.
The pattern recurs in team accounts of adopting trunk-based development: moving 90 engineers onto a merge queue without first fixing the suite's reliability reliably produces a throughput incident, and the fix is always the flake rate rather than more CI capacity.
Failure scenarios
- Bisection storms, where a batch's failure cascades into many rounds that conclude nothing.
- Batch size growing with the queue, converting congestion into a higher failure rate.
- Engineers batching more work per pull request because merging is slow, so change size grows and attribution gets harder — the opposite of the queue's purpose.
- Automatic retry without measurement, which hides the flake rate and doubles the cost while the queue stays blocked.
- A quarantine list with no expiry, which drifts into being the real test suite.
- More runners bought as the fix, shortening each run without reducing the number of rounds.
Trade-offs
| Choose | Gains | Pays |
|---|---|---|
| Large batches | Fewer suite runs per merge when green | Geometric spurious-failure risk; expensive bisection |
| Small batches | Predictable merges; cheap attribution | More suite runs; more total CI minutes |
| No queue | No merge latency | A broken trunk blocks everyone, which at 90 engineers is the larger cost |
The deeper trade is that a merge queue converts a probabilistic cost — trunk occasionally broken — into a deterministic latency on every merge. That is a good trade above a certain team size and only if the suite is reliable.
When not to use it
For a small team with a fast suite. With 10 engineers and a 10-minute suite, merging directly and reverting the occasional break is cheaper, because the cost of a broken trunk is small and the queue latency is paid on every change.
And never as a response to flakiness. A queue does not reduce flake rate; it multiplies its consequences. If the suite is unreliable, the queue is the wrong next change and the flake rate is the right one.
Interview question
Q: Leadership has approved a merge queue for a 120-engineer monorepo to stop trunk breakages. You suspect the suite is flaky. What do you measure before agreeing, and what thresholds would make you say yes, not yet, or no?
What a strong answer covers: measuring per-run spurious failure rate by auto-retrying failures once and recording retry passes, plus the concentration of that rate across tests · computing the expected batch failure rate at the proposed batch size, and the bisection cost in suite runs · a "yes" threshold around 0.5% per run with small batches, a "not yet" with a named plan to quarantine and fix the concentrated offenders, and a "no" if the rate is high and nobody owns the suite · the alternative worth proposing in the meantime, which is change-aware test selection to make any failure cheap to attribute · why more CI capacity does not help, since the queue is serialised by design · and the metric to govern it afterwards: 90th-percentile queue wait, not the mean.
Quick check
Quiz: At a 4% per-change spurious failure rate, what share of 8-change batches fail for no
reason? — Roughly 28%, since 1 − 0.96^8 ≈ 0.28, and each failure then costs about 3 bisection
rounds that carry the same risk.
Flashcard: Why does a merge queue on a flaky suite collapse rather than degrade? — Because nothing merges while pull requests keep arriving, so the queue lengthens, batches grow, and larger batches fail spuriously more often. The congestion reinforces itself.