Load-Coupled Fault Injection

Fault injection at low load proves that the error path executes. It proves almost nothing about whether the system survives, because the failures that matter come from resource exhaustion and queueing effects that do not exist at low concurrency: pool saturation, lock contention, garbage collection pressure, thread starvation.

Injecting the fault at peak is what reveals whether the timeout, the pool size and the circuit-breaker threshold were tuned correctly — and they usually were not, because they were guesses.

Why it matters

Load-dependent failures are emergent: the failure is in the interaction between components under contention, not in any one of them. Component testing cannot find them, functional testing passes, and the first real observation is an incident.

Implementation patterns

• Realistic data shape, not just realistic volume. Uniform synthetic traffic misses hot keys, skewed tenants and cache behaviour, which are frequently the actual cause.
• Inject latency, not only errors. Slow is worse than down — a failing dependency returns resources immediately while a slow one holds them — and tests that only inject failures miss the more dangerous case.
• Inject partial failure, where a dependency succeeds for some requests and fails for others, which is more common than total failure and behaves differently.
• Run capacity experiments: deliberately reduce capacity under production load to find the actual bottleneck rather than the assumed one. Cheap, because it needs no new environment.
• Flush the cache. The single most instructive experiment most teams have never run, and the one that most reliably reveals that the database cannot survive without it.
• Shadow traffic to a new version at full volume without serving results, exposing load-dependent behaviour before it affects anyone.
• Run game days for the human path, since graceful failure is only half the requirement — the other half is whether the on-call engineer can diagnose it at 3am.

Industry example

Delivery and dispatch platforms such as Swiggy have peaks that are both predictable and severe, with dependencies — mapping, payments, notification providers — that behave differently under contention than in isolation. The characteristic discovery from load-coupled injection is that a timeout tuned against a healthy dependency is far too long once the pool is contended, so a single slow dependency exhausts capacity long before the breaker notices.

Failure scenarios

• Injection at low load only, validating the code path and not the system.
• Errors injected but not latency, missing the more damaging failure.
• Testing in staging, which differs in data volume, cache state, traffic shape and dependency behaviour — precisely where these failures live.
• No hypothesis, so the experiment produces an incident rather than a finding.
• No abort criteria or blast-radius limit, which is what makes production experiments irresponsible rather than valuable.

Trade-offs

Production fault injection carries real customer risk, and doing it at peak carries the most. That is the tension: the most informative experiment is the most dangerous one.

The resolution is a small blast radius — one cell, one percentage of traffic, one availability zone — a fast automated abort, and a stated hypothesis so the experiment ends when it has answered the question. A team that cannot do this safely at peak should conclude that its isolation is insufficient, which is itself the finding.

Interview question

"You want to test what happens when your payments provider slows to five seconds. Design the experiment: when you run it, what you limit, what you measure, and what makes you stop."