intermediate 3 min answer

A merge queue tests each change against the current trunk plus the queue ahead of it, taking 20 minutes per run and merging serially. 60 engineers each want to merge twice a day. Estimate whether the queue can keep up, and say what you would change first.

merge-queuethroughputcibatchingcapacity
Show the full answer Hide the answer

The assumptions, stated

20 minutes per verification run, serial merges, 60 engineers x 2 merges = 120 merges a day, and a working day in which merges arrive over roughly 8 hours rather than 24.

The arithmetic

Serial throughput is 60/20 = 3 merges an hour, so 24 in an 8-hour day, or 72 if the queue runs around the clock.

Demand is 120. The queue is short by a factor of 1.7 against a 24-hour day and a factor of 5 against the hours people actually work. Queue length therefore grows monotonically during the day: arrivals cluster in the afternoon, so by mid-afternoon the wait is hours and engineers begin batching changes to avoid it, which makes each change larger and each failure more expensive. This is a queueing system driven past its service rate, and the symptom is not slowness but a change in behaviour.

Which assumption dominates the error

The serial assumption. Everything follows from testing one candidate at a time, and it is the thing to attack, not the 20 minutes. Halving the pipeline to 10 minutes doubles throughput to 48 a day and still fails. Batching changes the shape of the curve rather than its height.

What I would change, in order

  1. Batch optimistically. Test the top N of the queue together as one candidate; if it passes, merge all N. With N = 5 the effective rate is 15 an hour, comfortably ahead of demand. On failure, bisect the batch — 3 extra runs for a batch of 5 — so the cost is paid only when something is actually broken. Batching is worth it while the failure rate is low; at a 30% failure rate the bisects dominate and you are better off serial. That is the flip condition, and it means the first prerequisite is a trunk whose failures are real.
  2. Shrink the verification set, not the tests. Run only what the change could affect, computed from the build graph. On a large repository this routinely takes a 20-minute run to 4 minutes for most changes, and it buys more throughput per hour of work than anything else here.
  3. Parallelise across machines. Throughput is a capacity purchase: 5 runners cut the wall-clock time of a run substantially if the suite shards cleanly.
  4. Move the slow, broad checks out of the merge gate and run them post-merge on a schedule, with a revert rule when they fail. Not every check has to be a gate; some can be a detector.

What the number rules in and out

It rules out a serial queue at this organisation size, permanently — no amount of pipeline tuning reaches 120 a day serially with a suite of any substance. It rules in either batching or change-aware test selection, and in practice both. It also tells you the honest headcount threshold: a serial merge queue with a 20-minute pipeline supports roughly 10 to 12 engineers, and the queue becomes the constraint above that.

When this is over-engineering

Below about 15 engineers, a serial queue is simpler, easier to reason about, and fast enough, and optimistic batching adds a bisect mechanism plus a failure mode where an innocent change is rejected. Build it when the measured wait, not the theoretical wait, starts changing how people work — the leading indicator is engineers saying they are holding a change until tomorrow.