Reliability versus Complexity

Definition

Redundancy, failover, replication, circuit breakers and clustering all improve reliability against the failures they address, and each adds components, states and interactions that can themselves fail.

The inflection point

There is a point beyond which added reliability machinery reduces reliability:

• A clustered configuration with automatic failover has more ways to go wrong than a single instance — split brain, flapping, a coordination bug, a failover triggered by a false positive.
• Automatic remediation that reacts to a signal an external dependency can move will eventually amplify an incident. A liveness probe checking a database is the canonical example.
• Every additional component is another thing to patch, monitor, understand and be woken by.
• Complexity that exceeds what the team can hold in their heads produces longer incidents, and duration is half of availability.

A meaningful share of production incidents are caused by the mechanisms intended to prevent incidents.

How to decide

For each proposed mechanism:

1. What specific failure does this address, and how likely is it? If it is protecting against something that has never happened and is not plausible, the complexity is unpaid.

2. What new failure modes does it introduce? Be explicit. A failover mechanism introduces false failovers, split brain and drift.

3. Can the team operate it? A sophisticated mechanism nobody understands is a liability. This is the question most often skipped, and it is frequently the decisive one.

4. Is there a simpler mechanism with most of the benefit? Graceful degradation is usually cheaper and less complex than redundancy, and it addresses more failures.

The alternatives that reduce complexity rather than adding it

• Graceful degradation — the component's failure no longer matters, so its availability stops mattering.
• Faster recovery — improves availability arithmetically without adding runtime machinery.
• Blast radius reduction — cells or shards, so the same failure rate affects fewer users.
• Removing components. The most reliable component is the one that does not exist.

Failure scenarios

• A cluster less reliable than a single instance, because of coordination failures.
• Automatic failover with no fencing, producing split brain.
• Retries at three layers, amplifying rather than absorbing.
• A mesh, a broker and distributed locks introduced for a workload that never needed distribution.

Interview question

"Give an example of a reliability mechanism that made a system less reliable, and explain the mechanism."