A platform reports incident count down 40% year on year and presents it as evidence that reliability investment is working. Review that claim. What would you remove from the reporting, what would you change, and what would you keep?

What is actually required

A reliability metric has to be hard to improve without improving reliability. Incident count fails that test in three separate ways, and all three move it in the flattering direction.

Severity reclassification. The definition of what counts as an incident is set by the organisation reporting the number. A quiet tightening — "we only declare for customer-visible impact over 5 minutes now" — reduces the count with no change in reliability.

Under-reporting. If declaring an incident costs an engineer a postmortem, a review meeting and attention from leadership, the rational response at the margin is to fix it quietly. The incident count then measures willingness to declare, which is a cultural variable, and the number improves exactly as psychological safety degrades.

Silent failures are not counted at all. The incidents that never get declared are the ones with no error signal: a quality regression, a recommendation model trained on bad data, a stale cache serving wrong answers. Anthropic's published postmortem of September 2025 describes overlapping infrastructure bugs that degraded response quality on some platforms, which is the canonical shape: availability metrics stay green because nothing errors, and detection has to come from measuring output quality rather than output presence.

What I would remove

Incident count as a headline reliability metric. Keep it as an operational workload number — it is genuinely useful for staffing on-call and for noticing a trend in toil — and stop presenting it as evidence about the system.

Mean time to resolve. A mean over a handful of incidents with a long tail is dominated by one event, so it moves for reasons unrelated to capability. The distribution is informative; its mean is not.

What I would change

Report error-budget burn against user-journey SLOs. It is defined before the period, measured from the user's side, and cannot be improved by reclassifying anything. Nobody can quietly redefine whether a checkout succeeded.

Report the distribution of time-to-detect, not time-to-resolve. Detection is the part culture most affects and the part most predictive of future severity, and a rising p90 of detection time is the leading indicator that reporting quality is falling.

Add at least one output-quality SLI for any system whose failure mode is wrongness rather than unavailability: a model's evaluation score on a fixed held-out set, a sampled correctness check against a reference calculation, a reconciliation difference. Without one, the entire class of silent regression is outside every metric you have.

Count near misses and report them as a positive. A rising near-miss count is evidence the reporting culture is healthy. Leaving it out means the only safe way to look good is to say less.

What I would keep, even though it looks odd

The blameless postmortem requirement, including for small incidents — and specifically its cost. The obvious efficiency saving is to require postmortems only above a severity threshold, which immediately creates an incentive to classify below it. Keeping the obligation broad and keeping it cheap (a short template, a 20-minute review, no leadership attendance for low severities) is what keeps declaration costless.

When not to change any of this

If the organisation has no SLOs yet, do not start by removing its only reliability number. Incident count is a poor measure and it is a measure, and taking it away before a journey-level SLO exists leaves leadership with nothing, which reliably ends in a worse metric chosen by someone else. Sequence it: define 1 or 2 journey SLOs first, run them for a quarter, then demote incident count to an operational workload figure.

Also leave it alone for a team whose incident count is genuinely low and whose detection times are flat. The critique above is aimed at a 40% improvement presented as evidence; a stable small number alongside stable detection is not making a claim that needs dismantling.

How I would argue this in the review

Not by attacking the number. Ask for two things alongside it: the definition of "incident" as it stood 12 months ago compared with today, and the time-to-detect distribution over the same period. If the definition moved or detection slowed, the 40% is explained without any reliability improvement, and the conversation changes on its own.