Customers reported an outage 40 minutes before your monitoring did. How do you close that gap?

What the gap tells you

Every technical signal was green while users were failing. That is not a threshold-tuning problem; it means the system was monitoring components rather than outcomes.

The classic shape: a change breaks the checkout button in one browser. Error rates normal, latency normal, every instance healthy — and orders drop 30%.

Close it with three layers

Business metrics. Instrument the small number of events that constitute the product working — orders placed, payments succeeded, signups completed, messages delivered, searches returning results. Alert on anomalous drops compared with the same period last week, not a fixed threshold, because these metrics are strongly seasonal.

This is the layer that catches failures with no technical signature, and it is the one most often missing.

Synthetic monitoring. Scripted journeys — log in, search, add to basket, check out — running continuously from outside your network, on the browsers and regions your users actually use. It detects failures with no traffic to observe (overnight, in a small region) and it exercises the whole path including DNS, CDN and TLS, which server-side metrics never see.

Real user monitoring. Client-side telemetry captures what the user experienced — page load, rendering, JavaScript errors, and requests that never reached your servers. Server-side metrics are structurally blind to those, and they are a meaningful share of real failures.

Then check the other half of the gap

Was there a signal that nobody acted on? If the alert fired and was ignored, this is an alert fatigue problem, and the fix is the actionability work — deleting non-actionable rules, symptom-based alerting, burn-rate windows — not more monitoring.

Adding alerts to a system with a 40% actionability ratio makes detection worse, not better.

Make the customer channel a formal signal

Support ticket volume and social mentions are legitimate detection inputs, and in this incident they were the fastest one. Route a spike in support contacts into the alerting path, and give support a direct line to declare an incident rather than filing a ticket that queues.

Measure the thing you are fixing

Time to detect, tracked per incident and trended. Alongside time to mitigate, it is one of the two timeline gaps that point at concrete work — the first is a monitoring problem, the second a response problem.

Set a target and review it in every postmortem.

What a strong answer adds

Framing this as an SLO definition failure. If the SLI had been defined in terms of successful user journeys rather than successful HTTP requests, the burn-rate alert would have fired within minutes. The measurement layer was not merely incomplete — it was measuring the wrong events, and that is the finding worth carrying into the postmortem.