advanced 2 min answer

An organisation finds most incidents are caused not by individual service failures but by interactions between healthy services. What discipline addresses this, and what specifically should be done?

delhiverysystems-thinkingemergentdependenciesfault-injection
Show the full answer Hide the answer

What is happening

The failures are emergent: each component behaves as designed, and the composition does not. A retry policy that is reasonable alone becomes a storm when three layers apply it. A timeout that is fine at low load exhausts a pool at peak. A cache that improves latency makes the database unable to survive its own absence.

No component review finds these, because each component is correct.

What should be done

  • Map the actual dependency graph, including the ones nobody documented — the shared cache, the shared database, the shared identity provider, the shared queue. The undocumented shared dependency is where the correlated failure lives.
  • Fault injection under load, which is the essential combination. Adding latency at 10% load proves nothing; adding it at peak reveals whether the timeout, pool size and breaker thresholds were tuned correctly — and they usually were not, because they were guesses.
  • Review resilience policies together rather than individually. Retry, timeout, breaker, bulkhead and shedding interact, and reviewing each in isolation produces a set that is individually reasonable and collectively harmful. Deadline propagation and retry budgets are the two controls that only make sense at the system level.
  • Trace the composed path, so a journey's latency and failure can be attributed across services rather than each service reporting itself healthy.
  • Classify incidents by contributing factor across the estate, and review the aggregate. One postmortem explains one failure; recurrence is a pattern visible only in aggregate, and most organisations never do the aggregate review.

The specific loops worth looking for

  • Retry amplification across layers.
  • Load shifting — losing an instance moves its load to the others, accelerating the failure.
  • Autoscaling on a metric that scaling changes, such as queue depth.
  • Cache dependence, where the origin has never been sized for the traffic the cache hides.
  • Feedback between subsystems: a dispatch radius widening during a shortage pulls supply from adjacent regions, so the shortage propagates rather than resolving — a coupling neither subsystem's owner can see alone.

The framing

The system is the unit of analysis, not the service. An organisation that reviews services and measures services will keep having incidents caused by their interaction, and the only interventions that address it — dependency mapping, injection under load, aggregate incident review, and policies designed together — are all system-level activities that no team owns by default.