A marketplace's technical metrics are all healthy during an incident in which sellers cannot list items. Why did technical monitoring miss it, and what should be measured?
Show the full answer Hide the answer
Why technical monitoring missed it
The system was working; it was doing the wrong thing. Technical metrics measure whether requests are served: rate, errors, duration, saturation. A bug that returns a 200 response containing a validation failure, or a listing flow that silently drops the final step, produces perfect technical metrics.
The failure classes technical monitoring structurally cannot see:
- Successful responses with wrong content.
- A workflow step silently skipped, where each individual request succeeds.
- A third-party integration accepting requests and doing nothing — an image processor returning 200 and producing no output.
- A configuration or feature flag disabling a path, which is not an error, it is behaviour.
- A degraded upstream returning empty rather than failing — a search index returning zero results is a successful query.
That last one is worth emphasising: empty is not an error, and a system whose alerting cannot distinguish "no results" from "no results because the index is empty" will miss a total outage of its most important surface.
What should be measured
Business outcomes, at the same priority as technical ones:
- Listings created per minute
- Successful checkouts per minute
- Payment authorisation rate
- Search queries returning zero results, as a proportion
- New seller sign-ups completing
And the crucial detail: alert on deviation from expected pattern, not on an absolute threshold. These metrics are strongly seasonal — hourly, daily, weekly. A fixed threshold either fires every night or never fires. Comparing against the same period in previous weeks, with a tolerance band, is what makes them actionable.
Why these are the best detectors
They are closest to the thing that matters and therefore catch every failure mode, including ones nobody anticipated. A drop in listings created is a real problem regardless of whether the cause is a code bug, a misconfiguration, a third-party outage, or an unrelated change nobody connected.
They are also the metrics that make an incident's severity immediately legible to the business, which is what turns an engineering incident into an organisational response.
The implementation note
These belong in the same monitoring system as technical metrics, on the same dashboards, with the same alerting. Splitting business metrics into an analytics tool with hourly refresh means they detect nothing — the value is entirely in having them at operational latency, next to the technical signals that explain them.