Search the practice set
275 questions, 991 terms and 600 topics in 30 areas.
60 results for “Physical-World Failure Modes”
Clock Drift
Device clocks diverging from real time, which corrupts ordering, expires certificates early or late, and produces data that appears to arrive from the past.
Grey Failure
A component that is degraded rather than down — slow, intermittently erroring, or failing for a subset of operations — which defeats health checks built for binary states.
Byzantine Fault
A failure in which a component behaves arbitrarily or deceptively — returning wrong results rather than stopping — as distinct from simply crashing.
Fail-Fast vs Fail-Safe
Whether a component should stop immediately on detecting a problem, or continue in a degraded but safe mode — a choice that depends entirely on which outcome is worse.
File-Based Integration
Exchanging data as files on a schedule via SFTP or shared storage — dated, still ubiquitous, and workable when its failure modes are handled explicitly.
Change Failure Rate
The proportion of deployments that cause a production failure requiring remediation, and the DORA metric that keeps the others honest.
Correlated Failure
Failures that occur together because redundant components share a hidden dependency, invalidating the independence assumption behind redundancy calculations.
Failure Injection Testing
Deliberately introducing faults into a system under test to verify that timeouts, retries, fallbacks and circuit breakers behave as designed.
Failure Thinking
Making "what happens when this fails?" a standing question applied to every component and every dependency in a design.
Failure Threshold
The condition that trips a circuit breaker — best expressed as a failure rate over a rolling window with a minimum request volume, not as a consecutive-failure count.
Active-Active vs Active-Passive
Whether all regions serve traffic simultaneously, or one serves while another waits to take over — a choice about which failure mode you would rather have.
Blameless Postmortem
An incident review that seeks the systemic conditions that made a failure possible, explicitly excluding individual fault.
Blast Radius
The set of things that break, or become reachable, when one component fails or is compromised.
Blast Radius Reduction
The set of deliberate partitions — accounts, regions, zones, cells, tenants, deployment stages — that bound how far any single failure or compromise can reach.
Capacity Planning
Deciding in advance how much capacity will be needed, given growth, seasonality and failure scenarios, and ensuring it can be there in time.
Cascading Timeout
The effect of independently-chosen per-hop timeouts summing to a total far longer than any caller is willing to wait.
Cell Isolation
Bulkheading at the level of a complete system copy, so that any failure — including ones nobody predicted — is contained to the customers assigned to one cell.
Cell-Based Architecture
Partitioning a service into complete, independent copies of itself, each serving a subset of customers, so a failure is bounded to one cell.
Chaos Engineering
Deliberately injecting failure into a system to discover, before an incident does, which of your resilience assumptions are false.
Chaos Experiment
A controlled test that injects a specific failure to verify a hypothesis about the system's resilience, with a defined blast radius and abort condition.
Checkpoint Interval
How often a stateful processor persists its state and offsets, which trades steady-state overhead against how much work is redone after a failure.
Circuit Breaker
A proxy that stops calling a failing dependency after a failure threshold, failing fast instead, and periodically tests whether it has recovered.
Cross-Service Debugging
Investigating a failure that spans multiple services by moving between traces, logs, metrics and profiles along a single correlated request.
DORA Metrics
Four measures of software delivery performance — deployment frequency, lead time for change, change failure rate, and time to restore service.
Degradation Mode
A defined, intentional reduced state of service that the system enters under specified conditions, with known behaviour and known exit criteria.
Fan-Out Delivery
Delivering one published message to many independent subscribers, each with its own copy, position and failure handling.
Fault Domain
A boundary within which a single failure is contained, defined by the infrastructure and dependencies that components inside it share.
Fault Tolerance
Continuing to operate correctly despite the failure of some components, by design rather than by luck.
First-Principles Reasoning
Reducing a problem to the physical, mathematical or economic facts it rests on, then reasoning up, rather than reasoning from analogy or convention.
Game Day
A scheduled exercise in which a failure is deliberately introduced and the team responds as though it were real, to test the system and the response together.
Google Maps and Planetary-Scale Spatial Serving
Map serving is fast because almost nothing is computed on request — the world is precomputed into a pyramid of tiles, and space is indexed onto a one-dimensional curve.
Graceful Degradation
Continuing to deliver reduced but useful function when a dependency fails, instead of failing the whole request.
One instance in a fleet of fifty is returning correct responses very slowly. Health checks pass and it stays in rotation. How do you detect and handle this?
Why it is not detected Differential observability. The instance believes it is healthy and reports so; its callers experience something else. A health check tha
A dashboard has been wrong for three weeks. Nobody knows which upstream produced the table it reads. What is the governance failure, and what fixes it?
The failures, and there are three 1. No lineage. The question "which upstream produced this?" should be answerable in seconds by following a derived dependency
Southwest cancelled about 16,700 flights in December 2022 when crew scheduling could not recover from a storm, after years of deferred modernisation. How do you make that argument before the failure rather than after?
The case, as publicly reported A severe winter storm caused widespread cancellations across US carriers. Most recovered within days; Southwest did not. Its crew
Uber indexes the world with hexagons rather than squares for surge pricing and dispatch. Why does the shape matter, and what is the general lesson?
Why the shape matters A square has eight neighbours at two different distances — four edge adjacent, four corner adjacent and further away. So "the neighbouring
You are asked to make a multi-tenant SaaS resilient to "any single failure". You propose cells. What must you find before that claim is true?
What must be found: every shared dependency A cell is isolated only if nothing inside it depends on something shared with another cell. The work of adopting the
Your cluster fails over spuriously under load, but a real leader failure takes 45 seconds to detect. How do you resolve the tension?
The tension Detection time is set by the heartbeat interval times the failure threshold. Shorten it and you detect real failures faster and mistake slow but ali
A brief database slowdown caused a two-hour full outage. Explain the likely amplification chain and the fixes at each stage.
The chain Slow database → queries that took 10 ms take 2 s. Thread pool exhaustion. Requests hold threads for 200× longer. A pool of 50 that handled the load ea
A downstream service slows from 50 ms to 3 s. Within two minutes every service in the request path is down, including ones that do not call it. Explain the mechanism and how you would have prevented it.
What the interviewer is testing Whether you understand that most outages are amplification, not failure — and whether you can name the specific mechanism rather
A service becomes unresponsive during an incident in a dependency it barely uses. Investigation finds the client had no read timeout. Explain the full mechanism.
The mechanism, step by step 1. The dependency slows. It has not failed — it accepts connections and eventually responds, just very slowly. This is the important
Physical-World Failure Modes
Power loss, tampering, clock drift, thermal limits, and a truck through the fibre.
Failure Modes
Slow rather than down, partial, grey, and failing while reporting success.
Availability Zones
The unit of correlated physical failure, and what zones do not protect against.
Reliability vs Complexity
Mechanisms that add availability and add failure modes.
Service Mesh Operations
What a mesh genuinely solves, its failure modes, and the cost of running one.
Test Architecture Strategy
Choosing what to verify where, given the failure modes that actually occur.
Degradation Modes
Deciding in advance what is shed first and what is protected.
Failure Thinking
Asking of every box and arrow how it fails and how you would know.
Backward Compatibility
Which changes are safe, and how to make breakage a build failure.
Capacity Modelling
Arithmetic before load tests, and headroom for failure as well as peak.
Chaos Engineering
Hypothesis-driven failure injection with a bounded blast radius.
Cost vs Reliability
Each nine costing an order of magnitude, and pricing the failure instead.
Digital Twin
A server-side model of a physical thing's reported and desired state.
Distributed Systems
General material on partial failure, coordination and distributed reasoning.
Distributed Transactions
Two-phase commit, its blocking failure mode, and when it is still reasonable.
Fault Isolation
Cells, zones, tenants and the partitions that bound a failure.