Give an example of a reliability mechanism that made a system less reliable, and explain the mechanism.

What is being tested

Whether you understand that reliability machinery has its own failure modes, and can explain one precisely.

The clearest example: a liveness probe that checks a dependency

The mechanism, step by step:

1. The database slows. Application instances are degraded but functioning.
2. The liveness probe calls the database and exceeds its timeout. The instance reports not alive.
3. The orchestrator does what liveness means: it kills and restarts the instance.
4. Every instance fails identically, because they all share the database. All are restarted.
5. Restarted instances have cold caches and empty connection pools, and immediately open new connections to an already-struggling database.
6. They fail again. Restart loop.

A degraded dependency became a total outage, executed by the mechanism installed to improve reliability.

Other examples worth having ready

Automatic failover without fencing. A network partition looks like a dead primary. The standby is promoted, the original primary is still serving clients, and both accept writes. Split brain — divergent histories that cannot be automatically merged, which is worse than the outage it prevented, because an outage ends and diverged data does not.

Retries at three layers. The mesh retries, the client library retries, the gateway retries. Three layers of three attempts is 27 requests for one user action, arriving precisely when the downstream is already struggling. Retries designed to absorb transient failures amplify a real one.

Aggressive autoscaling on a signal a downstream can move. A slow dependency raises latency, which raises concurrency, which triggers scale-out, which adds more connections to the struggling dependency.

A cluster more fragile than a single instance, because coordination, quorum and failover introduce more ways to fail than the single point of failure they replaced.

The general principle

Automated remediation must be more careful than the failure it responds to. Any mechanism that takes destructive action based on a signal an external dependency can influence will eventually amplify an incident.

How to evaluate a proposed mechanism

1. What specific failure does this address, and how likely is it?
2. What new failure modes does it introduce? Be explicit — a failover mechanism introduces false failovers, split brain and standby drift.
3. Can the team operate it? A sophisticated mechanism nobody understands lengthens incidents, and duration is half of availability. This question is skipped most often and is frequently decisive.
4. Is there a simpler mechanism with most of the benefit?

The alternatives that reduce complexity rather than adding it

Graceful degradation (the component's failure stops mattering), faster recovery (improves availability arithmetically with no runtime machinery), blast radius reduction (same failure rate, fewer users affected), and removing components — the most reliable component is the one that does not exist.