Pipeline Change Coverage

A team reports deployment frequency of 12 a day, lead time of 40 minutes, change failure rate of 2% and recovery in 18 minutes. Elite on every published band. Incidents are up 40% over two quarters, and both facts are true at once.

All four metrics are computed on pipeline events. Anything that reaches production by another route is not in the denominator, so it cannot fail, and the measured change failure rate stays excellent while the system gets worse.

Why it matters

The four delivery metrics are well chosen and well researched, and DORA's published elite bands - deployment on demand, lead time under an hour, recovery under an hour - are a reasonable target. The failure is not the metrics. It is treating a measurement of the pipeline as a measurement of the system.

In most estates the unmeasured changes outnumber the measured ones: feature-flag flips, configuration services, infrastructure updates, base-image and package refreshes, vendor-pushed agent updates, hand-run database changes, cloud-provider maintenance, IAM edits, CDN and DNS changes. If the incidents originate there, the delivery dashboard is describing a different system from the one customers are using.

There is also a Goodhart effect. A process indicator with no outcome indicator beside it will be optimised - by routing risky changes around the pipeline, which improves every number and worsens every outcome.

Implementation patterns

• Compute the coverage ratio explicitly: changes that passed through the pipeline divided by all changes that altered production behaviour, per quarter. A figure below roughly 50% means the delivery metrics are not a description of the system.
• Classify one quarter of incidents by originating change type, not by affected service. This single exercise usually identifies the dominant unmeasured path immediately and is the evidence that carries an argument.
• Widen the denominator by emitting change events from every path - flag services, configuration stores, infrastructure automation - into the same stream the pipeline uses.
• Pair the four metrics with an outcome measure: customer-impacting minutes per month. Process indicators and outcome indicators are only trustworthy together.
• Report distributions with p50 and p95 and the sample size per service. A bimodal lead time - most changes fast, one class taking days - is invisible in a median.
• Define failure honestly. If change failure rate counts only rollbacks and hotfixes, a change that degrades something unmeasured or creates a latent condition is recorded as a success.

Industry example

The delivery-metrics literature is explicit that the four measures describe a delivery process, and the recurring industry experience since the metrics became widespread after 2018 is that organisations report elite figures from their best-instrumented services while incident rates move independently. The reliable tell is a coverage ratio nobody has calculated - when asked what fraction of production changes the numbers cover, teams usually have not considered the question.

Failure scenarios

• Risky changes routed around the pipeline because the pipeline is where the measurement is.
• Flag flips excluded from the change count, so the highest-frequency and highest-blast-radius change type is invisible.
• Scope creep in reporting, where figures from three instrumented services are presented as organisational performance.
• Median reporting hiding a slow class of changes that is where the frustration actually lives.
• Failure defined as rollback, so silent degradations and latent defects never count.
• Recovery time improving while incident count rises, presented as progress.

Trade-offs

Widening the denominator is politically expensive, because it usually halves the apparent performance of a team that has done genuinely good work. That cost is the reason it is not done, and naming it openly is what makes the conversation possible.

When not to use it

For a team whose only path to production is the pipeline - no manually applied infrastructure, no separate configuration service, no auto-updating agents - the denominator really is complete and the four metrics describe the system. That is rarer than teams assume, and the way to find out is to try to enumerate the other paths rather than to assume there are none.

It is also the wrong lens for a team whose problem is outcome rather than flow. If incidents are caused by design decisions rather than by delivery practice, improving the measurement of delivery will not help, and the effort belongs in architecture review.

Interview question

Q: You join an organisation reporting elite delivery metrics. What would you look at in your first two weeks to decide whether those numbers mean what they appear to mean?

What a strong answer covers: asking for the denominator first and computing the coverage ratio · listing the change paths that exist outside the pipeline and checking whether any emits events · classifying a quarter of incidents by originating change type · asking how change failure is defined and whether silent degradations could be counted · requesting distributions and per-service sample counts rather than headline medians · pairing the figures with customer-impacting minutes · and framing the finding as a scope question rather than as a criticism of the numbers, because the numbers are correct and it is the inference that is wrong.

Quick check

Quiz: How can change failure rate be 2% while incidents rise 40%? Because the rate is computed only over pipeline changes. Flag flips, configuration edits, infrastructure updates and vendor-pushed agent updates are outside the denominator, and in most estates they outnumber code deployments.

Flashcard: What single exercise identifies the unmeasured change path? — Classify one quarter of incidents by originating change type rather than by affected service. The dominant path usually becomes obvious immediately.