intermediate 2 min answer

A team is launching a new service and asks what its SLO should be. How do you help them decide, and why is "99.99%" usually the wrong first answer?

slosreerror-budgetreliability
Show the full answer Hide the answer

What the interviewer is testing

Whether you treat reliability as a cost-benefit decision with a budget, or as a virtue to maximise.

Why not four nines

It costs an order of magnitude more per nine. 99.9% permits 43 minutes of failure a month and is achievable with a well-run multi-AZ deployment. 99.99% permits 4.3 minutes, which is less than the time it takes a human to read a page and open a laptop — so it requires automated failover, which requires the failure detection and the automation to be more reliable than the thing they protect.

It is often not achievable at all. The service's availability is capped by its dependencies. A service depending synchronously on three things at 99.9% cannot exceed about 99.6% no matter what it does. Promising four nines on top of that is promising something outside your control.

Users usually cannot perceive it. The user's own network, device and the client application contribute far more unavailability than the difference between three and four nines. Reliability nobody experiences is money spent on a number.

How to actually decide

1. Pick SLIs that reflect user experience. Availability as "the proportion of requests that returned a valid response within 500 ms", measured as close to the user as practical. Not CPU, not uptime of a host.

2. Establish what current or comparable performance is. An SLO should be a small, deliberate improvement on reality, not an aspiration disconnected from it.

3. Ask what the failure actually costs. Revenue per minute, contractual penalty, safety consequence, reputational effect. A payments API and an internal reporting dashboard should not have the same target, and pretending they should is how reliability budgets get wasted.

4. Check the dependency ceiling. Multiply through. If the arithmetic says the target is impossible, the conversation moves to removing a synchronous dependency — which is a much better conversation.

5. Start conservative and tighten. A target that is never missed teaches nothing. Set something the team will occasionally breach, learn from the breaches, then tighten.

What the SLO is for

The error budget. If the target is 99.9%, the budget is 43 minutes a month, and the agreement — made in advance, in writing — is that while budget remains the team ships, and when it is exhausted the team stabilises. That policy is the entire point; an SLO without an agreed consequence is a number on a dashboard.

What a strong answer adds

Alerting on burn rate rather than on threshold breaches: page when the budget is being consumed fast enough to exhaust it, which fires on real user impact and collapses dozens of cause-based alerts into a few symptom-based ones.

Also: keep the SLA looser than the SLO. Missing the internal target should be an internal signal, not a customer credit.