A merge queue tests each change against the current trunk plus the queue ahead of it, taking 20 minutes per run and merging serially. 60 engineers each want to merge twice a day. Estimate whether the queue can keep up, and say what you would change first.
Show the full answer Hide the answer
The assumptions, stated
20 minutes per verification run, serial merges, 60 engineers x 2 merges = 120 merges a day, and a working day in which merges arrive over roughly 8 hours rather than 24.
The arithmetic
Serial throughput is 60/20 = 3 merges an hour, so 24 in an 8-hour day, or 72 if the queue runs around the clock.
Demand is 120. The queue is short by a factor of 1.7 against a 24-hour day and a factor of 5 against the hours people actually work. Queue length therefore grows monotonically during the day: arrivals cluster in the afternoon, so by mid-afternoon the wait is hours and engineers begin batching changes to avoid it, which makes each change larger and each failure more expensive. This is a queueing system driven past its service rate, and the symptom is not slowness but a change in behaviour.
Which assumption dominates the error
The serial assumption. Everything follows from testing one candidate at a time, and it is the thing to attack, not the 20 minutes. Halving the pipeline to 10 minutes doubles throughput to 48 a day and still fails. Batching changes the shape of the curve rather than its height.
What I would change, in order
- Batch optimistically. Test the top N of the queue together as one candidate; if it passes, merge all N. With N = 5 the effective rate is 15 an hour, comfortably ahead of demand. On failure, bisect the batch — 3 extra runs for a batch of 5 — so the cost is paid only when something is actually broken. Batching is worth it while the failure rate is low; at a 30% failure rate the bisects dominate and you are better off serial. That is the flip condition, and it means the first prerequisite is a trunk whose failures are real.
- Shrink the verification set, not the tests. Run only what the change could affect, computed from the build graph. On a large repository this routinely takes a 20-minute run to 4 minutes for most changes, and it buys more throughput per hour of work than anything else here.
- Parallelise across machines. Throughput is a capacity purchase: 5 runners cut the wall-clock time of a run substantially if the suite shards cleanly.
- Move the slow, broad checks out of the merge gate and run them post-merge on a schedule, with a revert rule when they fail. Not every check has to be a gate; some can be a detector.
What the number rules in and out
It rules out a serial queue at this organisation size, permanently — no amount of pipeline tuning reaches 120 a day serially with a suite of any substance. It rules in either batching or change-aware test selection, and in practice both. It also tells you the honest headcount threshold: a serial merge queue with a 20-minute pipeline supports roughly 10 to 12 engineers, and the queue becomes the constraint above that.
When this is over-engineering
Below about 15 engineers, a serial queue is simpler, easier to reason about, and fast enough, and optimistic batching adds a bisect mechanism plus a failure mode where an innocent change is rejected. Build it when the measured wait, not the theoretical wait, starts changing how people work — the leading indicator is engineers saying they are holding a change until tomorrow.