An observability platform's incident process treats all incidents with the same severity and process. What problems does this create, and how should severity be defined?
Show the full answer Hide the answer
The problems with uniform severity
Serious incidents are under-resourced. If everything is an incident, the genuinely severe ones do not get the escalation, the incident commander and the executive attention they need — because the process provides the same response to everything.
Minor incidents are over-resourced. A single customer's degraded dashboard pulls in six engineers and a communications process. The cost is not just the wasted time; it is that people stop declaring incidents to avoid the ceremony, so real issues go unmanaged.
Alert fatigue becomes incident fatigue. When every issue triggers the full process, the process itself becomes the thing people route around.
Nobody can prioritise. With three concurrent incidents and no severity, there is no basis for deciding which gets the senior engineer.
How to define severity
By customer impact, not by technical cause — which is the distinction most organisations get wrong. A database failover is not a severity; the resulting customer experience is.
A workable ladder for a data platform:
| Severity | Definition | Response |
|---|---|---|
| SEV1 | Data loss, or complete unavailability for many customers | Immediate, all hands, incident commander, exec notified, customer comms |
| SEV2 | Significant degradation, or complete failure for a subset | On-call plus escalation, status page, hourly updates |
| SEV3 | Degraded but usable, or affecting few customers | On-call handles, tracked, no external comms |
| SEV4 | No customer impact, needs fixing | Ticket, next business day |
Two rules make the ladder work:
Anyone can declare, and declaring high is never punished. Under-declaring is far more costly than over-declaring, and any culture where declaring a SEV1 is career risk will systematically under-declare.
Severity can be revised in both directions, as understanding improves. The initial assessment is made with the least information available.
The observability-platform specific case
For a platform customers use to monitor their systems, two additional considerations apply.
Your outage blinds your customers during theirs. A monitoring platform's availability requirement is higher than a typical application's, because customers depend on it precisely when their own systems are failing.
Data loss is a distinct category from unavailability. Missing telemetry cannot be recovered — the moment has passed. A brief ingestion outage means permanent gaps in customer data, so it deserves higher severity than the same duration of query unavailability. Severity ladders that only measure "was it up" mis-rank this, and the ranking should encode which failures are recoverable and which are not.