Your load tests pass consistently and production still falls over at peak. What is wrong with the tests?

The workload model is almost certainly unrealistic

Data distribution is the usual culprit. Synthetic tests spread requests uniformly across keys, which produces an unrealistically high cache hit rate and no hot partitions. Production follows a power law: a small number of items take most of the traffic, and those items are where the contention is.

Traffic mix taken from assumption rather than telemetry. A test that is 95% reads against a 70/30 production reality proves nothing about the write path, which is where locks and replication live.

Constant arrival rate. Real traffic is bursty, and bursty arrivals produce longer queues than a constant rate at the same mean.

Data volume. Query plans and index behaviour on a small test dataset do not survive production scale.

Connection reuse. A test client that pools connections while real clients do not — or the reverse — changes handshake cost dramatically.

Cold state excluded. The minutes after a deployment, with empty caches and unwarmed pools, are the worst case and are usually not measured at all.

The test environment probably differs in ways that matter

Scaled-down infrastructure does not scale down proportionally — a database with one tenth the memory has a completely different cache profile. Mocked dependencies remove exactly the latency and failure behaviour that causes production problems. And a test environment with no other tenants has no noisy neighbours.

You are running the wrong kinds of test

Load testing validates the target. It does not tell you what happens past it.

Stress testing finds the breaking point and, more importantly, the behaviour there: fast rejection or collapse; recovery when load subsides or a retry backlog that prevents it. This is where retry storms and health-checks-failing-under-load are found, and both are common causes of "passed the test, fell over in production".

Soak testing finds what needs time — memory and descriptor leaks, cache degradation, log growth, token expiry. Note the pass criterion: compare the end state to the beginning, not merely that it stayed up. A run that passed while memory rose 40% found a defect and reported success.

Spike testing for the scheduled-event case, where the arrival gradient itself is the problem.

Fix the programme

Derive the workload model from production telemetry, not estimates. Test against production-like infrastructure and real dependencies. Include the cold-start window. Ramp progressively and record latency alongside throughput, so the saturation point is visible rather than a single maximum number.

Then close the loop: run it in CI against a budget, so a regression is caught by a build rather than by a peak.