Resilience Testing
Verifying that failure-handling behaves as designed — a category of testing distinct from functional and load testing, and usually absent.
Definition
Resilience testing verifies the system's behaviour when things go wrong: dependencies fail, become slow, return garbage, or disappear mid-request. Functional tests verify the happy path; load tests verify capacity; neither exercises the code that runs during an incident.
What to test, in order of value
1. Dependency latency. The single highest-value test and the least performed. Most cascading failures begin with a slowdown, not an outage: a dependency going from 50 ms to 3 s exhausts thread and connection pools by Little's Law, and the caller then fails requests that never touched it. A system tested only against hard failures has not been tested against its most likely incident.
2. Dependency failure. Errors, connection refused, timeouts. Does the fallback run? Does it work? Does the circuit breaker trip at a threshold that is actually reachable?
3. Partial failure. One instance of three misbehaving; one shard slow; one region degraded. Harder to test and closer to reality than total failure.
4. Resource exhaustion. Disk full, memory limit, connection pool exhausted, file descriptors. These produce distinctive and usually unhandled failure modes.
5. Malformed and hostile responses. A dependency returning HTML instead of JSON, a truncated payload, an unexpectedly enormous response.
6. Recovery. What happens when the dependency comes back? Does the circuit breaker close? Is there a thundering herd of retries? Recovery behaviour is tested even less often than failure behaviour and causes just as many incidents.
How to test it
- Fault injection at the client library or proxy, so latency and errors can be introduced without modifying the dependency.
- Contract-level stubs that misbehave deliberately, not just stubs that return the happy path.
- Automated in CI for the critical paths, so resilience regressions are caught like any other.
- In production with a limited blast radius, for the behaviour that cannot be reproduced elsewhere.
The specific things it finds
Timeouts that are longer than the caller's own timeout, so they never fire. Retries at three layers multiplying into 27 requests. Fallbacks that call the failing dependency. Circuit breaker thresholds that cannot be reached before the pool is exhausted. Health checks that test dependencies and so amplify a downstream slowdown into a total outage.
Every one of those is invisible in code review and obvious in a test.
Interview question
"How would you test that your service degrades correctly when a dependency's latency rises tenfold?"