A load test shows p99 latency of 120 ms at 2,000 requests per second. Production at the same rate shows 900 ms. Why?
Show the full answer Hide the answer
What the interviewer is testing
Whether you know the specific design errors that make load tests confidently wrong.
The candidates, in order of likelihood
Data volume and shape. The test database has a fraction of production's rows. Different query plans, different index effectiveness, different cache hit rates. This is the most common cause by a wide margin.
Coordinated omission. A closed-model load generator waits for a response before sending the next request, so when the system slows, the generator slows too — systematically hiding the tail latency you are measuring. An open-model generator maintaining arrival rate independently is required.
Cache warmth and key distribution. The test requests the same 100 records repeatedly, achieving a cache hit rate production never sees. Production has a long tail of cold keys.
Workload mix. The test hammers one endpoint; production runs a mix, and expensive operations interleave with cheap ones, queueing behind each other.
Dependency realism. The test used mocks returning in 1 ms; production calls a partner API at 200 ms p99, holding connections and threads for far longer.
Concurrency and contention from background jobs, batch processes and scheduled work that the test environment does not run.
How to fix the test
Production-scale data with realistic distribution. Open-model generator. Realistic key distribution including cold keys. Production workload mix with think time. Dependencies virtualised with their latency and error behaviour, not stubbed instantly. Long enough duration for garbage collection patterns and leaks to appear.
What a strong answer adds
Always identify the bottleneck, not just the number. A test that reports latency without saying what limited it produces no actionable information, and the bottleneck frequently differs between test and production — which is itself the diagnosis.
And measure production directly: real user monitoring and server-side percentiles are ground truth, and the load test's job is predicting change, not establishing absolute values.
Common weak answers
Concluding that production has "more load". Adjusting the test until the numbers match production.