A dependency fails only under high load, so normal testing never exposes it. How should load testing, fault injection, capacity experiments and traffic shadowing be combined to find it?

Why normal testing misses it

Load-dependent failures come from resource exhaustion and queueing effects that do not exist at low concurrency: connection pool saturation, lock contention, garbage collection pressure, thread starvation, buffer limits. At 10% of peak these are all invisible, and every functional test passes.

They are also emergent — the failure is in the interaction between components under contention, not in any one of them, which is why component-level testing cannot find them.

The combination that works

• Load testing to realistic peak with realistic data shape. Uniform synthetic traffic misses hot keys, skewed tenants and cache behaviour, which are frequently the actual cause. The data distribution matters as much as the volume.
• Fault injection during load, which is the essential combination. Adding latency to a dependency at 10% load proves nothing; adding it at peak reveals whether the timeout, the pool size and the circuit breaker were tuned correctly — and they usually were not, because the thresholds were guesses.
• Capacity experiments: deliberately reduce capacity under production load and observe where it breaks first. This finds the actual bottleneck rather than the assumed one, and it is cheap because it needs no new environment.
• Traffic shadowing to a new version at full production volume without serving the results, which exposes load-dependent behaviour before the change affects anyone.
• Game days for the human path, since knowing the system fails gracefully is only half the requirement — the other half is whether the on-call engineer can diagnose it at 3am.

The specific things to inject

Latency, not just errors — slow is worse than down and the tests that only inject failures miss the more dangerous case. Partial failures, where a dependency succeeds for some requests and not others. Dependency unavailability. Instance loss during peak. Clock skew. Cache flush, which is the single most instructive experiment most teams have never run.

The precondition

Do this in production, or accept that you have tested something else. A staging environment differs in data volume, cache state, traffic shape and dependency behaviour, and load-dependent failures live precisely in those differences. Production experiments need a small blast radius, a fast abort, and a clear hypothesis — but they are the only ones that test the system that exists.