A communication platform sets a 99.9% availability SLO. How should alerting on that SLO be structured so it catches both sudden outages and slow degradation?
Show the full answer Hide the answer
Why a single threshold fails
Alerting when availability drops below 99.9% over a fixed window has two failure modes depending on the window:
Short window — noisy. A brief blip breaches the threshold instantly and pages for something that resolved itself.
Long window — slow. A total outage takes a long time to move a 30-day average below the threshold, so the alert fires long after everyone knows.
Neither reflects what actually matters, which is how fast the error budget is being consumed relative to how much remains.
Burn-rate alerting
The error budget for 99.9% over 30 days is roughly 43 minutes. Burn rate is the multiple of the sustainable consumption rate:
- Burn rate 1 = the budget is consumed exactly over 30 days.
- Burn rate 14.4 = the entire budget is consumed in about 2 days.
- Burn rate 100 = consumed in about 7 hours.
Alert on burn rate rather than on the availability number.
Multi-window, multi-burn-rate
The standard configuration uses several alerts with different urgency:
| Condition | Meaning | Response |
|---|---|---|
| Very high burn over a short window | Severe outage in progress | Page immediately |
| High burn sustained over a medium window | Significant degradation | Page |
| Moderate burn over a long window | Slow leak consuming the budget | Ticket |
The short-window component prevents slow detection of a sudden outage. The long-window component catches persistent low-level degradation that no threshold alert would ever fire on. Together they cover both failure shapes with one framework.
A short confirmation window on each alert prevents a momentary spike from paging: require the burn rate to be sustained over both a long and a short window before firing, which filters transients without delaying real events.
What makes it work for this workload
Measure the SLI at the client, per user journey. For a conferencing platform, "the API returned 200" is not availability. The SLI is whether a participant joined a meeting within the expected time and stayed connected — measured where the participant is.
Segment. A regional network problem affecting one country is invisible in a global SLI and total for those users. Per-region burn rates catch it.
The connection to the error budget policy
Burn-rate alerts are the operational half of an error budget. The policy defines what happens when the budget is exhausted; the alerting tells you when exhaustion is approaching, with enough lead time to act. Without burn-rate alerting, the budget is a monthly report rather than a control.