A load test passed at 3x expected traffic and production fell over at 1.2x. Name four ways the test could have been unrealistic.

What is being tested

Knowledge of the specific ways load tests mislead — all of which are common and all of which are avoidable.

The four most likely

1. Unrealistic data volume. The test ran against a small dataset. Indexes fit in memory, query plans were optimal, and scans were cheap. Production has 100 million rows: different plans, cache misses, and a completely different cost per query. This is the single most common cause.

2. Unrealistic request mix. The test hammered one or two endpoints that were easy to script. Production traffic is a distribution across many endpoints with wildly different costs, and the expensive one may be 3% of requests and 60% of load.

3. Warm cache. The test warmed up and then measured, so it tested the cache rather than the system. Real traffic has misses — new content, long-tail objects, cold starts after deployment — and the miss path is what falls over. A cache stampede on a hot key is invisible in a warm-cache test.

4. Uniform distribution of keys and tenants. The test used random IDs. Production has extreme skew: a hot tenant, a viral object, a celebrity user. Uniform load avoids exactly the hot-key and hot-shard problems that break production.

Others worth knowing

• No think time and persistent connections, producing a load shape that does not exist and understating connection churn.
• Test traffic from inside the network, excluding TLS handshakes, DNS, real network latency and slow-client effects.
• Scaled-down environment. A smaller environment has different bottlenecks; the result cannot be extrapolated.
• No concurrent background load. Production runs batch jobs, backups, analytics queries and deployments at the same time as serving traffic.
• Client retry behaviour absent, so the amplification effect that turns a slowdown into an outage never appears.
• The load generator saturated, so the measurement was of the test harness.

What to change

• Test against production-scale data, ideally a sanitised copy.
• Derive the request mix from production traffic logs, or replay them directly.
• Start cold, or explicitly test both cold and warm.
• Reproduce the skew — model the real distribution of tenant sizes and object popularity.
• Test in a production-like or production environment, with a small share of real traffic if possible.
• Test past the target, because how the system behaves when overloaded matters more than where the limit is.

What a strong answer adds

That the most valuable output of a load test is not "did it pass" but the shape of the curve: at what load does latency begin to rise non-linearly (the knee, which is the real capacity limit), and what happens beyond it. A test that only confirms the target was met has discarded most of the information it generated.