How would you test that your service degrades correctly when a dependency's latency rises tenfold?

What is being tested

Whether you can design a resilience test with a hypothesis and a measurable outcome, and whether you know what such a test typically finds.

The test

Inject latency at the client boundary, not in the dependency itself — via the HTTP client, a proxy, or a service mesh fault injection rule. This means you can test without a cooperating dependency and without affecting anyone else's traffic.

Structure:

1. State the hypothesis. "With the payment provider at 3 s instead of 300 ms, our p99 rises but stays under 2 s, the circuit breaker opens within 30 seconds, and requests not touching payments are unaffected."
2. Establish a baseline under representative load. Without load, the test measures nothing — the failure mode is about resource exhaustion, which only appears when resources are contended.
3. Ramp the injected latency, rather than stepping straight to 10x. The threshold at which behaviour changes is more informative than the endpoint.
4. Measure: p50/p99 of the affected endpoint, p99 of unaffected endpoints, thread and connection pool utilisation, error rates, circuit breaker state, retry counts at every layer, and queue depths.
5. Then test recovery. Remove the latency and watch what happens — recovery behaviour is tested even less often than failure behaviour and causes just as many incidents.

What it typically finds

• The pool exhausts before the circuit breaker opens. By Little's Law, in-flight requests equal arrival rate times latency, so a 10x latency increase requires 10x the concurrency at constant traffic. If the breaker's threshold is not reachable before the pool is empty, it never fires.
• Unaffected endpoints degrade too, which means there is no bulkhead — a shared connection or thread pool. This is the single highest-value finding and the most frequently missing mitigation.
• Timeouts longer than the caller's own timeout, so they can never fire. Extremely common.
• Retries multiplying. Three layers each retrying three times is 27 requests for one user action, arriving precisely when the dependency is already struggling.
• The fallback throws, because it has never run.
• Health checks failing, because they test the dependency — so the platform removes healthy instances and amplifies a partial problem into a total outage.

Making it routine

• Automate it in CI for the critical paths, so resilience regressions are caught like any other regression. A newly added dependency without a timeout should fail a build.
• Run it in production with a small blast radius for the behaviour that cannot be reproduced elsewhere.
• Test the neighbours, not just the service under test. Cascading failure is by definition about what happens to things that were not directly affected.

Why latency specifically

Because it is the most likely real incident and the least handled case. Code that handles connection-refused frequently does not handle a response that takes 30 seconds — and a system tested only against hard failures has not been tested against the failure it will actually experience.