advanced 3 min answer

How does an error budget actually change behaviour, what makes it fail in practice, and what should happen when it is exhausted?

googlesresloerror-budgetreliability-culture
Show the full answer Hide the answer

What the mechanism is for

An SLO defines a target — say 99.9% of requests succeed over 28 days. The error budget is the complement: 0.1% of requests may fail without breaching the objective. Over 28 days that is roughly 40 minutes of full unavailability, or a much longer period of partial degradation.

The purpose is to convert an unwinnable argument into a measurement. "Should we ship features or improve reliability?" has no answer in the abstract and is resolved by whoever argues hardest. "Have we spent our error budget?" has an answer, and the answer determines the priority automatically.

Budget remaining → ship, and take risks; the budget is there to be spent. Budget exhausted → reliability work takes priority until it recovers. That symmetry is the whole design: it protects feature velocity as firmly as it protects reliability.

Why it fails in practice

  • The SLO is set at the wrong level. Too high, and the budget is always exhausted, so the policy is ignored within a month. Too low, and it never binds and changes nothing. The SLO should be set from what users actually need, and the most common error is choosing a number for its appearance rather than its meaning — 99.99% chosen because it sounds serious, then breached constantly.
  • No agreed consequence. A budget with no policy attached is a dashboard. The consequence must be agreed in advance, in writing, by the people who will resent it — because it will be invoked at the least convenient moment.
  • Measurement that does not reflect user experience. Server-side success rates that exclude client timeouts, or availability measured at the load balancer while users experience failures beyond it, produce a budget that stays healthy through a bad outage.
  • The budget is overridden every time it binds, which teaches everyone it is advisory and destroys it permanently. One override with a good reason is fine; a pattern of overrides is the end of the mechanism.
  • No ownership. If nobody owns the SLO, exhaustion is nobody's problem.
  • Budgets per service rather than per user journey, so every service is green while the journey that crosses five of them fails.

What should happen at exhaustion

The policy should be specific and pre-agreed:

  • Feature deployments to that service pause; reliability work and fixes continue. This is the standard and strongest version.
  • The team's priority shifts to the reliability backlog until the budget recovers on a rolling window.
  • An explicit, documented, time-boxed exception process for genuine emergencies, with a named approver — because a policy with no exception process is broken around rather than followed, and it is better to make the exception visible than to force it underground.
  • A review of why, distinguishing "one large incident" from "steady erosion", which have completely different remedies.

The subtlety that separates practitioners

A consistently unspent budget is also a finding. It means the system is more reliable than it needs to be, which is not free — it was paid for in velocity, in over-engineering, and in caution that had no justification. The correct response is to take more risk: deploy more often, run experiments, reduce review overhead.

Very few organisations act on that half of the mechanism, and the ones that do get most of the value — because the budget's real function is not to make systems more reliable but to make the reliability deliberate.