advanced 2 min answer

What would it cost to move a service from 99.9% to 99.99% availability, and how would you decide whether it is worth it?

availabilitycostslotradeoffsblast-radius
Show the full answer Hide the answer

What is being tested

Whether you can price reliability and whether you know the cheaper alternatives to buying another nine.

What the step costs

99.9% is 8.8 hours of downtime a year; 99.99% is 52 minutes. To get there:

  • Multi-zone with automatic, tested failover rather than a manual procedure. Roughly doubles infrastructure for the redundant tier.
  • Capacity headroom to survive losing a failure domain, which caps utilisation at (N−1)/N and therefore raises cost further.
  • Elimination of every single point of failure, including the ones in the control plane — identity, secrets, deployment, DNS.
  • A step change in operational maturity: fast detection (an hour of undetected downtime is more than the entire annual budget), tested runbooks, mature on-call, rigorous change management, canaries and automated rollback.

That last group is usually the real cost. Infrastructure doubling is a number; building the operational capability is a programme, and organisations consistently underestimate it.

The step to 99.999% — 5 minutes a year — requires multi-region active-active with all the data consistency complexity that entails, and it is rarely justified outside a few domains. It is also frequently claimed by systems that have not measured.

How to decide

By the cost of unavailability, not by aspiration:

  • Revenue lost per minute — and critically, whether it is lost or merely deferred. An e-commerce sale during a five-minute outage may simply happen five minutes later; treating deferred revenue as lost overstates the case substantially and is the most common error in these calculations.
  • Contractual penalties and service credits.
  • Regulatory consequences.
  • Reputational effect, which is real and hard to quantify — state it as a judgement rather than inventing a number.
  • Internal cost: engineers idle, orders processed manually.

Set the target where the marginal cost of the next nine exceeds its marginal value. That usually lands lower than instinct suggests.

Per-service, always

An estate-wide target is always wrong. Checkout may warrant 99.99%; a recommendation service that degrades gracefully may warrant 99.5%, because its failure is invisible when the fallback works. Paying for uniform reliability across services with wildly different failure consequences is one of the most common sources of over-spend.

The three cheaper alternatives

Usually better value than another nine:

1. Graceful degradation. If a component's failure no longer takes the service down, its availability stops mattering. Often the highest return available and far cheaper than redundancy.

2. Faster recovery. Availability is a function of both frequency and duration. Halving mean time to recovery improves availability arithmetically and usually costs less than preventing the failure — better detection, better runbooks, faster rollback.

3. Blast radius reduction. Cells or shards, so a failure affects 5% of users rather than 100%. This changes the impact of the same failure rate, which is frequently what actually matters to customers and to the business.

What a strong answer adds

Asking what the current availability actually is, measured. Many services claiming 99.9% have never measured it, and the first finding is usually that the real number is different from the assumed one — in either direction.