A commerce platform load tests successfully at five times peak traffic, then fails during the real peak event. What is typically wrong with the load test?

What is typically wrong

The test is a scaled-up version of average traffic; the event is a different traffic shape.

1. Uniform data access. Load generators typically pick items randomly across the catalogue, which distributes load evenly and gives excellent cache behaviour. The real event concentrates on a handful of products, producing hot keys, lock contention and cache patterns the test never created.

2. Warm caches from the start. A test ramping up gradually warms caches during the ramp. A real event arrives as a step function against caches warmed for ordinary traffic, so the initial miss rate is far higher and the database sees a burst the test never produced.

3. Read-heavy generation. Load tests are easier to write for reads. The event is write-heavy — carts, orders, inventory decrements — and writes contend where reads do not.

4. No third-party constraints. Tests stub payment providers, tax services and shipping calculators. In the real event those are shared with everyone else's peak, and their rate limits and latency are part of the system.

5. Missing the long tail of user behaviour. Real users abandon carts, retry, open many tabs, and refresh repeatedly during slowness. The last of these is significant: user-initiated retries during degradation add load precisely when the system is struggling.

6. Perfect infrastructure. The test runs with everything healthy. The event coincides with a degraded dependency, a node failure, or a deployment.

What a meaningful test includes

• Realistic access-pattern skew, derived from actual traffic distributions rather than uniform random.
• Step-function arrival, not a gradual ramp, with caches in their ordinary state.
• Correct read/write mix, including the contention hot spots.
• Third-party dependencies at realistic latency, including their rate limits.
• An injected failure — a degraded dependency, a lost instance — because the realistic scenario is high traffic and something wrong.
• Test to failure, not to target. Knowing you clear the bar is far less useful than knowing where the cliff is and what breaks first.

The most valuable output

Not "we handled 5x". It is the ordered list of what breaks and at what load — which component saturates first, second and third. That list drives the capacity plan, the degradation ladder ordering, and the runbook, and it is the thing a pass/fail test never produces.