What questions characterise failure thinking, and which are most often skipped in design reviews?
Show the full answer Hide the answer
The questions
1. What happens if this dependency is unavailable? For each dependency: does the request fail, degrade, or proceed? An answer of "it fails" makes it a hard dependency that multiplies into the availability ceiling.
2. What happens if it is slow rather than down? The question most often skipped, and the one that causes more outages. Error-rate circuit breakers do nothing when a dependency succeeds slowly, while it holds threads and connections across the caller fleet and blows the latency budget.
3. What happens if this change is wrong and reaches everything? Configuration and deployment are the fastest global change mechanisms in most systems, so the rollback path — and whether it depends on what broke — matters more than the rollout path.
4. What happens if two things degrade at once? Reviews consider single failures; reality delivers a degraded dependency during a traffic peak. The realistic scenario is high load and something wrong.
5. What happens when it recovers? Second incidents happen during recovery — cold caches, empty pools, reconnect storms, and a backlog processed at many times the normal rate.
6. What must succeed for our recovery to work, and how likely is that during the failure? A recovery depending on an autoscaling API, a container registry or a control plane depends on something whose availability is negatively correlated with the need for it.
The ones most often skipped
Slow-not-down, two-at-once, and recovery. Each is where the expensive incidents actually live, and each is absent from most design reviews because the review considers the system rather than the system under stress.
The practice that makes it real
Ask them in the design review, and require an answer per dependency — recorded, so an untested fallback is visible as a hypothesis rather than assumed as a capability.
Then exercise the answers: fault injection continuously in production at low traffic share, because a fallback that is never run does not work, and that is the single most reliable finding in resilience engineering.