A team can improve availability by adding several components, each of which is itself a failure point. How should that be evaluated?
Show the full answer Hide the answer
The evaluation
Compare the failure the addition prevents against the failures it introduces, using the organisation's own incident history rather than theoretical availability figures.
The specific question: what has actually caused outages here? For most organisations the answer is deployments, configuration changes and overload — not the infrastructure failures that redundancy addresses. Adding components to survive a class of failure you rarely have, at the cost of new components that fail in ways you have not seen, is frequently a net loss.
What makes a resilience addition genuinely worth it
- It addresses a failure that has occurred, or that is credibly likely, with an estimated frequency and consequence.
- It is simple enough to be understood by whoever is on call, since a component nobody understands is a component that extends every incident.
- Its own failure mode is benign — degradation rather than outage. A cache that fails to a slower path is worth adding; one whose failure takes the database down is not, unless the database is protected separately.
- It is exercised. A failover path that has never been tested is a more expensive way of having an outage, and untested resilience machinery routinely fails when first invoked.
The additions that usually pay
Staged rollout with automated halt conditions, because the most common cause of total outages in mature systems is a bad change reaching everything. Load shedding and admission control, because overload is the second. Both are simple, both address the actual failure history, and neither adds a component whose failure is total.
The additions that frequently do not
More replicas, when the failures are software rather than hardware. A message broker for a workload a database queue handles. A service mesh adopted for observability. Each is defensible in isolation and each adds operational surface that must be maintained by the same team that was already stretched.
The framing to use
"This prevents a failure of type X, which has happened N times, at a cost of Y — and it introduces a component that will need patching, monitoring, capacity planning and 3am understanding." Both halves stated together turn an architectural preference into an evaluable proposition.