Search the practice set
126 questions, 454 terms and 400 topics in 20 areas.
57 results for “Failure Thinking”
Failure Thinking
Making "what happens when this fails?" a standing question applied to every component and every dependency in a design.
Constraint Thinking
Designing for the budget, timeline, skills, regulations and existing estate that actually exist, rather than for the ones a textbook assumes.
Failure Threshold
The condition that trips a circuit breaker — best expressed as a failure rate over a rolling window with a minimum request volume, not as a consecutive-failure count.
Grey Failure
A component that is degraded rather than down — slow, intermittently erroring, or failing for a subset of operations — which defeats health checks built for binary states.
Product Thinking
Treating what you build as something with users, a value proposition and a lifecycle, rather than as a project that completes.
Systems Thinking
Reasoning about a system in terms of the interactions and feedback loops between its parts, rather than the parts individually.
Active-Active vs Active-Passive
Whether all regions serve traffic simultaneously, or one serves while another waits to take over — a choice about which failure mode you would rather have.
Blameless Postmortem
An incident review that seeks the systemic conditions that made a failure possible, explicitly excluding individual fault.
Blast Radius
The set of things that break, or become reachable, when one component fails or is compromised.
Blast Radius Reduction
The set of deliberate partitions — accounts, regions, zones, cells, tenants, deployment stages — that bound how far any single failure or compromise can reach.
Byzantine Fault
A failure in which a component behaves arbitrarily or deceptively — returning wrong results rather than stopping — as distinct from simply crashing.
Capacity Planning
Deciding in advance how much capacity will be needed, given growth, seasonality and failure scenarios, and ensuring it can be there in time.
Cascading Timeout
The effect of independently-chosen per-hop timeouts summing to a total far longer than any caller is willing to wait.
Cell Isolation
Bulkheading at the level of a complete system copy, so that any failure — including ones nobody predicted — is contained to the customers assigned to one cell.
Cell-Based Architecture
Partitioning a service into complete, independent copies of itself, each serving a subset of customers, so a failure is bounded to one cell.
Chaos Engineering
Deliberately injecting failure into a system to discover, before an incident does, which of your resilience assumptions are false.
Circuit Breaker
A proxy that stops calling a failing dependency after a failure threshold, failing fast instead, and periodically tests whether it has recovered.
DORA Metrics
Four measures of software delivery performance — deployment frequency, lead time for change, change failure rate, and time to restore service.
Fail-Fast vs Fail-Safe
Whether a component should stop immediately on detecting a problem, or continue in a degraded but safe mode — a choice that depends entirely on which outcome is worse.
Fault Tolerance
Continuing to operate correctly despite the failure of some components, by design rather than by luck.
Game Day
A scheduled exercise in which a failure is deliberately introduced and the team responds as though it were real, to test the system and the response together.
Graceful Degradation
Continuing to deliver reduced but useful function when a dependency fails, instead of failing the whole request.
Lock Lease Expiry
The timeout on a distributed lock that prevents a crashed holder deadlocking the system — and the source of the pattern's hardest failure mode.
Partition Tolerance
The ability to keep operating when the network drops or delays messages between nodes — not a choice, but a property of any system spanning more than one machine.
Retry Budget
Capping retries as a proportion of overall traffic rather than as a count per request, so retries cannot multiply during the failure they are meant to survive.
Sequence Diagram
A diagram showing the ordered exchange of messages between participants over time, used to make an interaction's control flow and failure points explicit.
Static Stability
The property that a system keeps working on its existing state when its control plane or dependencies are unavailable, rather than needing them to keep running.
Synchronous vs Asynchronous Replication
Whether a write is acknowledged only after a replica has it, trading write latency against the amount of data a failure can lose.
Zonal vs Regional Services
Whether a cloud resource lives in one availability zone or is inherently spread across several — a property that determines what a zone failure takes with it.
A dashboard has been wrong for three weeks. Nobody knows which upstream produced the table it reads. What is the governance failure, and what fixes it?
The failures, and there are three 1. No lineage. The question "which upstream produced this?" should be answerable in seconds by following a derived dependency
Southwest cancelled about 16,700 flights in December 2022 when crew scheduling could not recover from a storm, after years of deferred modernisation. How do you make that argument before the failure rather than after?
The case, as publicly reported A severe winter storm caused widespread cancellations across US carriers. Most recovered within days; Southwest did not. Its crew
You are asked to make a multi-tenant SaaS resilient to "any single failure". You propose cells. What must you find before that claim is true?
What must be found: every shared dependency A cell is isolated only if nothing inside it depends on something shared with another cell. The work of adopting the
Your cluster fails over spuriously under load, but a real leader failure takes 45 seconds to detect. How do you resolve the tension?
The tension Detection time is set by the heartbeat interval times the failure threshold. Shorten it and you detect real failures faster and mistake slow but ali
A downstream service slows from 50 ms to 3 s. Within two minutes every service in the request path is down, including ones that do not call it. Explain the mechanism and how you would have prevented it.
What the interviewer is testing Whether you understand that most outages are amplification, not failure — and whether you can name the specific mechanism rather
A service becomes unresponsive during an incident in a dependency it barely uses. Investigation finds the client had no read timeout. Explain the full mechanism.
The mechanism, step by step 1. The dependency slows. It has not failed — it accepts connections and eventually responds, just very slowly. This is the important
One availability zone becomes unavailable. Walk through what happens to a typical three-tier application and what you would have changed.
What survives and what does not Load balancer — regional, survives, and stops routing to targets in the failed zone once health checks fail. Note the detection
One instance in a fleet of fifty is returning correct responses very slowly. Health checks pass and it stays in rotation. How do you detect and handle this?
Why it is not detected Differential observability. The instance believes it is healthy and reports so; its callers experience something else. A health check tha
Failure Thinking
Asking of every box and arrow how it fails and how you would know.
Constraint Thinking
Designing for the budget, skills and regulations that actually exist.
Failure Modes
Slow rather than down, partial, grey, and failing while reporting success.
Product Thinking
Treating platforms and services as products with users and a lifecycle.
Systems Thinking
Interactions and feedback loops rather than isolated components.
Availability Zones
The unit of correlated physical failure, and what zones do not protect against.
Backward Compatibility
Which changes are safe, and how to make breakage a build failure.
Capacity Modelling
Arithmetic before load tests, and headroom for failure as well as peak.
Chaos Engineering
Hypothesis-driven failure injection with a bounded blast radius.
Cost vs Reliability
Each nine costing an order of magnitude, and pricing the failure instead.
Distributed Systems
General material on partial failure, coordination and distributed reasoning.
Distributed Transactions
Two-phase commit, its blocking failure mode, and when it is still reasonable.
Fault Isolation
Cells, zones, tenants and the partitions that bound a failure.
Redundancy
N+1, N+2 and 2N, correlated failure, and headroom for the failure itself.
Reliability & Resilience
General material on designing for failure.
Reliability vs Complexity
Mechanisms that add availability and add failure modes.
Reserved & Committed Capacity
Committing the baseline, laddering terms, and expiry as a silent failure.
Routing & BGP
How traffic finds you, anycast, and route withdrawal as a failure mode.
Sequence Diagrams
Ordered message exchange, and walking the failure of each arrow.