A platform team reports deployment frequency of 12 per day, lead time for changes of 40 minutes, change failure rate of 2% and recovery time of 18 minutes - elite on every band. Incidents have risen 40% over two quarters. Review the measurement.

What the numbers actually establish

That changes travelling through the pipeline are small, fast and mostly successful. That is a real and valuable property, and it is a statement about a population, not about production.

The four figures are all measured on pipeline events. Anything that reaches production by another route is outside the denominator entirely.

What is missing

1. The rest of the change surface. Feature-flag flips, configuration changes, infrastructure updates, base-image and package refreshes, vendor-pushed updates, database changes run by hand, cloud-provider maintenance. In most estates these outnumber code deployments, and if they cause incidents, change failure rate stays elite while incidents rise. This alone explains the pattern in the question.
2. The definition of "failure". Change failure rate usually counts changes that triggered a rollback or a hotfix. A change that degrades something nobody measures, or that creates a latent condition breaking three weeks later, is counted as a success.
3. Distribution, not mean. Lead time of 40 minutes is presumably a median. The p95 is what teams experience, and a bimodal distribution - most changes fast, a class of changes taking days - is invisible in the headline figure.
4. Scope. Which services are in the measurement? Elite figures from the three best-instrumented services say nothing about the forty others.
5. Incident cost, which is not a DORA metric at all. Recovery time of 18 minutes across a rising number of incidents may still be a worsening year for customers.

What I would change

• Widen the denominator to every change that reaches production. This is uncomfortable, usually halves the apparent performance, and is the only version of the metric that can be reconciled with the incident trend.
• Report distributions with p50 and p95 and the sample size per service.
• Pair the four metrics with an outcome measure - customer-impacting minutes per month - because the DORA metrics are process indicators and a process indicator without an outcome indicator will be optimised.
• Classify incidents by originating change type for one quarter. That single exercise usually identifies the unmeasured path immediately.

What I would leave alone

The four metrics themselves. They are well chosen, well researched and worth keeping - DORA's published elite bands (on-demand deployment, lead time under an hour, recovery under an hour) are a reasonable target. The failure here is not the metrics; it is treating a measurement of the pipeline as a measurement of the system.

How I would argue this in the review

Not as "your numbers are wrong" - they are correct. As a scope question: these describe 30% of the changes reaching production, and the incidents are coming from the other 70%. Ask for one quarter of incidents classified by change type, and let the data make the argument.

When this is the wrong critique

For a team whose only path to production is the pipeline - no manually applied infrastructure, no separate configuration service, no vendor agents auto-updating - the denominator really is complete and the four metrics describe the system. That is rarer than teams assume, and the way to find out is to try to enumerate the other paths.