intermediate 2 min answer

A team's pipeline takes 80 minutes from commit to a deployable artifact and engineers have stopped watching it. How do you approach fixing it?

cipipelinefeedbackdelivery
Show the full answer Hide the answer

Measure per stage before changing anything

Get the duration of every stage over the last few hundred runs, with the distribution rather than the mean. The shape is usually one or two stages dominating and a long tail of small ones, and the intuition about which is wrong more often than not.

Also measure queue time separately from execution time. A pipeline that executes in thirty minutes and waits fifty for a runner has a capacity problem, not a pipeline problem, and optimising the stages will change nothing.

The reductions, in order of return

Parallelise what is genuinely independent. Unit tests, static analysis, dependency scanning, licence checks and container linting all consume the same source and produce independent verdicts. Running them concurrently makes the pipeline as slow as the slowest rather than the sum. Watch for shared state — a common test database or a fixed port turns parallelism into flakiness.

Cache aggressively and correctly. Dependency resolution and compilation of unchanged modules should not repeat. Cache keys must include everything that affects the output, or the cache produces wrong results occasionally, which is worse than no cache.

Split the pipeline by feedback need. Fast checks — compile, unit tests, lint — must return in under ten minutes because that is roughly the attention window. Slow checks — full integration, performance, deep scanning — run after, or on a schedule, and block promotion rather than the merge.

Delete stages that prove nothing. Every mature pipeline has at least one. Feed it a deliberately broken artifact; if it passes, it is costing time and providing nothing.

The target and why

Under ten minutes to the first meaningful signal. Past that, engineers context-switch and return later, which converts a two-minute fix into a half-hour interruption — and that behavioural cost is larger than the machine time.

Full pipeline under thirty minutes. Past that, merge frequency falls, changes get batched, and every downstream problem gets worse.