Health Check & Service Discovery  ·  View 21 of 21  ·  Assurance

Failure Classes

Twelve classes, what catches each one, and what it costs when the bound fails.

Editable source SVG draw.io All views
Caught by Bounded by Worst case if the bound fails Instance dead Probe + lease expiry 10 s p99 detection Black-holed requests Instance slow, not dead Passive outcomes Weight cut, local ejection Grey failure stays in rotation Observation path failed Second signal class Unknown state Healthy fleet drained Control plane unavailable Client cache age 60 min static serving Routing on hour-old truth Stale client view Connection failure 5 s withdrawal budget Requests to a dead address Correlated mass withdrawal Eligible fraction 50% guard disregards health Service at zero capacity Flapping Transition count Hysteresis + quarantine Every client's view churns Thundering herd on recovery Reconnect rate Slow start, 5% per 10 s Recovered target re-killed Registration leak Reconciliation diff 30 s lease expiry Zombie endpoints served Cross-region split-brain Replication lag + ARC Each region serves its own view Both regions declare the other dead Poisoned health signal Attribution at ingest Passive signal cannot be forged One workload denies another Policy error Staged rollout diff Budget + one-step reversal Wrong region evacuated Twelve Failure Classes — What Catches Each, And What It Costs Not one recovery in this table is a code change. Every bound is a number in the requirement, which is what makes the table checkable. v 1.0 · owner Reliability Architecture

What the table is for

  • Every bound in the middle column is a number in the requirement, so the table is checkable rather than reassuring.
  • Not one recovery in the table is a code change; each one is a limit the platform already enforces.
  • The right-hand column is written as the user-visible consequence, because that is what sizes the bound.

The two classes with the least cover

  • Cross-region split-brain: each region keeps serving its own view, which is correct and still means the regions disagree about reality.
  • Control plane unavailable: the 60-minute static window is a design claim that only a routine production drill turns into a fact.

Assumptions

  • Grey failure is assumed more common than process death; policy error is assumed as likely as component failure.
◀  Identity and Authorisation Flow All views
Solution Architecture v1.0 · Amazon Web Services with an open-source Envoy/xDS data plane · Reliability Architecture · 2026-10 · 21 views · 16 architecture decision records21-failure-classes