Operated Reliability

Availability calculations use component figures and a topology. Real systems achieve something different, usually worse, and the gap is explained by factors the calculation omits:

• Correlated failure. Replicas share code, configuration, deployment and dependencies, so they fail together. Redundancy multiplies availability only when failures are independent, and they rarely are.
• Change as the dominant cause. Most serious incidents come from a deploy, a configuration push or a dependency change — against which instance and region redundancy provide no protection at all.
• Operational capability. A design requiring expertise the team does not have will be operated badly, and badly-operated redundancy is worse than none because it creates confidence.
• Untested recovery paths. A failover that has never been exercised is a hypothesis. Configuration drift, expired certificates, missing quota and dependencies added since the design are discovered during the disaster.
• Complexity's own failure modes. Each additional mechanism is a new thing that can break, and the aggregate frequently exceeds what it prevents.

Why it matters for architectural comparison

A proposal that is more available in theory and much more complex to operate may be less available in practice. Comparing designed reliability makes the complex option look strictly better, which is how organisations acquire architectures they cannot run.

The comparison must include the failure modes the complexity introduces, not only the ones it removes.

Implementation patterns

• Enumerate new failure modes for each proposed mechanism, including the control plane that operates it.
• Ask whether the team can operate it, honestly, and treat a no as disqualifying rather than as a training need.
• Prefer mechanisms whose recovery path is the normal path. Active-active's genuine advantage is that failover is exercised constantly; a standby's disadvantage is that it is exercised annually at best.
• Invest in what addresses the dominant cause: staged rollout with blast-radius limits, rollback faster than rollout and independent of what broke, N-1 capacity headroom, cell isolation, and rehearsed degradation.
• Measure realised availability against the design figure, and investigate the gap rather than the calculation.

Industry example

The recurring pattern is an organisation with excellent redundancy and regular outages. The replicas are correct, the topology is sound, and the incidents are caused by configuration pushes, deploys and dependency changes that reach every replica simultaneously.

The interventions that move realised availability are consequently not more redundancy but a better change pipeline: validation before distribution, staged rollout by blast radius, health-gated promotion on real traffic signals rather than on "did it apply", and a rollback path that does not depend on the control plane a bad change may have broken.

That is why mature global platforms invest far more in the change pipeline than in the replica count.

Failure scenarios

• Availability quoted from a calculation that assumes independence.
• Redundancy sized at full utilisation, so losing one replica cascades into losing the rest.
• A complex design adopted by a team that cannot operate it, producing worse availability than the simple design it replaced.
• Recovery paths never exercised, so the first execution is during the disaster.
• Attributing the gap to bad luck rather than to correlated failure and change.

Trade-offs

Designing for operated reliability sometimes means choosing a less capable architecture because it is one the team can run — which feels like settling and is frequently correct. It also means investing in unglamorous pipeline and rehearsal work rather than in topology, which is harder to justify because the benefit is a non-event.

The honest framing is that an architecture is only as available as the organisation operating it, and that constraint belongs in the comparison rather than being discovered afterwards.

Interview question

"One design has a theoretical availability of five nines and requires a control plane your team has never run. The other has four nines and is operationally familiar. Which do you choose, and what would change your answer?"