Failure Injection Testing
Deliberately introducing faults into a system under test to verify that timeouts, retries, fallbacks and circuit breakers behave as designed.
Resilience code is the least-exercised code in any system: written once, rarely triggered, and therefore routinely broken when it finally runs. A fallback that has never executed is a hypothesis.
The faults worth injecting, roughly in order of what they find:
Latency, not failure. A dependency returning an error is handled; a dependency taking 30 seconds is what exhausts thread pools and propagates. Timeout misconfiguration is far more common than missing error handling, and only latency injection finds it.
Errors — 500s, connection refused, and the awkward middle case of a malformed or partial response.
Resource exhaustion — connection pool saturation, memory pressure, disk full.
Partial failure — one instance failing while others succeed, which tests health checking and outlier ejection rather than the client's error handling.
Where to run it: in the service's own test suite using a mock or proxy, so it runs on every build and a regression fails the pipeline. This is the difference between resilience testing and chaos engineering — the latter validates the whole system in production, the former validates one service's behaviour continuously and cheaply.
The finding that recurs: retries configured at several layers multiplying into a retry storm that turns a brief dependency blip into an outage.