Pipeline Feedback Latency

A pipeline's duration is usually discussed as waiting: forty minutes is annoying, ten minutes is pleasant. That framing misses the effect that matters, which is on behaviour rather than on the clock.

Verification has a fixed cost per submission. When that cost is forty minutes, the cheapest way to ship five things is to submit them as one change — not because anyone decided to, but because that is what amortising a fixed setup cost looks like. Feedback latency therefore sets batch size, and batch size sets the size of every review, the number of candidate causes in every failed deploy, and the amount a rollback removes.

The second effect is attentional. A change stays in working memory for something on the order of 10 to 20 minutes. Past that, the result arrives as an interruption and is paid for twice.

Why it matters

Every delivery metric downstream is a function of it. Larger changes are reviewed less effectively, because review quality falls off past a few hundred lines in a sitting. Larger changes are harder to bisect, because the number of suspects is the number of changes in the deploy. Larger changes make rollback expensive, because reverting removes four things that were fine.

It also decides whether flaky tests get fixed. At 40 minutes per run, re-running is cheaper than investigating, so nobody investigates and the flake rate rises. At under 10 minutes the economics reverse.

Implementation patterns

• Split by contract, not by category. A commit stage under 10 minutes whose green means merge-safe, and later stages running the expensive suites against the built artefact, failing forward.
• Cut scope before buying hardware. Faster runners take a 40-minute suite to 25; running only the tests affected by the change takes it to 3.
• Measure the distribution, not the mean. The p90 is what engineers experience and remember; a mean of 12 minutes with a p90 of 38 feels like a 38-minute pipeline.
• Include the queue. Time from push to verdict includes waiting for a runner, which is often the largest term on a Monday morning and invisible in per-job timings.
• Quarantine flakes with an owner and a deadline, so removal is a decision rather than an erosion.

Industry example

The under-ten-minute commit stage has been the standard recommendation since Humble and Farley's Continuous Delivery (2010), and the DORA research programme has since repeatedly found lead time and change failure rate improving together rather than trading off — which is what you would expect if both are downstream of batch size rather than of care.

Failure scenarios

• Batch growth, the primary one: changes get larger until review and bisection stop working, and the cause is invisible because nobody attributes it to the pipeline.
• Flake tolerance, where re-running becomes the norm and the build's verdict stops carrying information.
• Merge queue collapse, where a long pipeline plus a busy repository means changes spend longer in the queue than in review.
• Hidden queueing, where per-job durations look fine and engineers still wait 40 minutes because runners are saturated.
• Optimising the mean by speeding up already-fast jobs while the slow tail, which is what everyone experiences, is untouched.

Trade-offs

Shortening feedback costs money and coverage. More parallel runners raise the bill; running only affected tests requires a dependency graph that is itself a maintained artefact and is wrong occasionally; moving suites after the merge means some defects reach main and are reverted rather than rejected. That last trade is only safe when changes are small — which is the property the short pipeline produces, so the two reinforce each other in both directions.

When not to use it

When the deployable unit is genuinely indivisible. Firmware, a database engine, a medical device, anything with a physical or certification step: verification is hours and cannot be split, and effort spent chasing a ten-minute pipeline is wasted. Keep a fast commit stage for the fraction that can be fast, and accept that batch size there is set by the release process. Chasing the metric in that context optimises a number that is not the constraint.

Interview question

Q: A team's pipeline takes 40 minutes and they say it is tolerable because they work on other things while waiting. Argue that it is not tolerable, using something other than the waiting time.

What a strong answer covers: the fixed-cost-amortisation mechanism that drives batch size up; the downstream effects on review quality, bisection and rollback; the attention-span argument for the ten-minute threshold; the flake economics that make re-running rational; and the remedies in order — split stages by contract, cut scope per stage, then buy hardware, measuring p90 including queue time.

Quick check

Quiz: Why does a slow pipeline raise change failure rate even when every test is correct? Because it pushes engineers towards larger changes, and larger changes are reviewed less effectively and harder to bisect when they break.

Flashcard: What does pipeline feedback latency actually control? Batch size. A fixed verification cost per submission gets amortised by submitting more at once, so review size, bisection difficulty and rollback blast radius all follow from the pipeline's duration.