Emergent Failure

In a system that has been running for a while, most incidents are not caused by a component failing. They are caused by components working exactly as designed, interacting in a way nobody designed.

A retry policy that is reasonable alone becomes a storm when three layers apply it. A timeout tuned against a healthy dependency is far too long once the pool is contended. A cache that improves latency leaves the origin unable to survive its own absence. Each of these passes every review of the component that contains it.

Why it matters

Component-level review, testing and metrics are structurally incapable of finding these, which means an organisation that reviews services and measures services will keep having incidents caused by their interaction — and each postmortem will correctly conclude that no single component was at fault.

Implementation patterns

• Map the actual dependency graph, including the undocumented shared ones: the shared cache, the shared database, the shared identity provider, the shared queue. The undocumented shared dependency is where the correlated failure lives.
• Inject faults under load. Injecting at 10% of peak proves the error path executes; injecting at peak reveals whether the timeout, pool size and breaker thresholds were correct — and they usually were not, because they were guesses. Inject latency, not only errors, since slow is worse than down.
• Review resilience policies together rather than individually. Retry, timeout, breaker, bulkhead and shedding interact, and reviewing each alone produces a set that is individually reasonable and collectively harmful. Deadline propagation and retry budgets only make sense at the system level.
• Trace the composed journey, so failure and latency are attributable across services rather than each reporting itself healthy.
• Classify incidents by contributing factor across the estate and review the aggregate, since one postmortem explains one failure while recurrence is a pattern visible only in aggregate.

Industry example

Logistics platforms such as Delhivery and Porter run deep dependency graphs across dispatch, tracking, pricing, notification and partner integrations. The characteristic incident is a small upstream slowdown that propagates into a total failure through retries and pool exhaustion — with every service's own dashboard green throughout.

The loops worth watching for specifically: retry amplification across layers · load shifting, where losing an instance moves its load to the others and accelerates the failure · autoscaling on a metric that scaling itself changes · feedback between subsystems, such as a dispatch radius widening during a shortage and pulling supply from adjacent regions, so the shortage propagates rather than resolving.

Failure scenarios

• Component-only review, which cannot see the class.
• Fault injection at low load, validating the code path rather than the system.
• Resilience policies chosen per service, summing to something harmful.
• No aggregate incident classification, so the pattern is never seen.
• Each team's metrics green during a user-visible outage, which is the signature.

Trade-offs

System-level analysis requires someone whose remit spans teams, which no team owns by default and which is frequently the first role cut. Fault injection at peak carries real customer risk, and the most informative experiment is the most dangerous one.

The mitigation is a small blast radius, a fast automated abort and a stated hypothesis — and a team that cannot do this safely at peak should conclude that its isolation is insufficient, which is itself the finding.

Interview question

"Your last three total outages each involved several services, none of which had failed. Describe what you would put in place to find the next one before it happens."