practice

Failure Thinking

Making "what happens when this fails?" a standing question applied to every component and every dependency in a design.

metaresiliencereview

The habit is to walk the design and, for each box and each arrow, ask three things: how does this fail, how will we know, and what does the system do next?

Most designs answer the first question adequately and the second and third not at all. A dependency whose failure mode is "returns 500" and whose detection is "somebody notices" and whose response is "the request fails" has inherited its availability into your system, and nobody wrote that down.

The specific failure modes worth walking each time, because they are the ones that get missed: slow rather than down (far more damaging), partially failed (some requests fine), failed and then recovered with stale state, and failed in a way that returns success. That last one is the reason health checks and validation matter more than uptime dashboards.