A team's CI pipeline takes 40 minutes. Everyone agrees that is annoying but tolerable, because engineers do other work while they wait. Why does the 40 minutes change what engineers do rather than only how long they wait?
Show the full answer Hide the answer
The mechanism
A verification with a fixed cost gets amortised, and the way engineers amortise it is by making each change bigger. If every submission costs 40 minutes of waiting regardless of size, the cheapest way to ship five things is to submit them together. Nobody decides this; it is the rational response to a fixed setup cost, and it is the same arithmetic that sets batch size in a factory.
The consequences all follow from the batch size, not from the wait:
- A bigger change is a worse change to review. Review effectiveness is commonly reported to fall off sharply past a few hundred lines in one sitting, so the defects the second pair of eyes exists to catch start getting through.
- A bigger change is a worse change to debug. When a deploy breaks, the number of candidate causes is the number of changes in it. Bisecting five changes takes longer than bisecting one, and each bisect step costs another 40 minutes.
- A bigger change is a worse change to roll back, because reverting it removes four things that were fine.
The second effect: the loop does not close
Holding a change in working memory lasts on the order of ten to twenty minutes. At 40 minutes the engineer has started something else, so the result arrives as an interruption and is paid for twice: once in the switch away and once in the switch back. At under ten minutes the engineer waits, reads the result in context, and fixes it immediately. That threshold — a commit stage under ten minutes — has been the standard recommendation since Humble and Farley's Continuous Delivery in 2010, and it is about the attention span, not the CPU.
The third effect: flakes become unaffordable
At a 2% spurious failure rate, a re-run costs 40 minutes. The rational response to an expensive re-run is to re-run without investigating, because investigating costs more than the retry. The flake rate then rises, because nothing removes flakes except someone investigating them. A ten-minute pipeline makes investigation cheaper than superstition.
What to do, in order
- Split the pipeline into stages with different contracts. A commit stage under ten minutes that must be green to merge, and a slower stage that runs the expensive suites and can fail after merge.
- Cut scope per stage before buying hardware. Faster runners shorten a 40-minute suite to 25; running only the tests affected by the change shortens it to three.
- Quarantine flakes with an owner and a deadline, never silently.
When this is the wrong answer
When the deployable unit is genuinely indivisible — firmware, a database engine, anything with a long physical or certification step — the pipeline time is dominated by verification that cannot be split, and heroic attempts to make everything fast waste effort. The right move there is still a fast commit stage, with the honest acknowledgement that the full verification is hours and that batch size is set by the release process rather than by the pipeline.