What makes a performance test predictive of production behaviour, and what is the most common design flaw?
Show the full answer Hide the answer
The most common design flaw
Uniform data access. Load generators pick items randomly across the dataset, which distributes load evenly, gives excellent cache behaviour and produces no contention.
Real traffic is steeply skewed — a small fraction of items takes most of the requests — producing hot keys, lock contention and cache patterns the test never created. This is the single most common reason a test passes at five times peak and the real peak fails.
What makes a test predictive
1. The real access distribution, derived from production statistics rather than assumed.
2. Step-function arrival, not a gradual ramp. A ramp warms caches during the ramp. A real event arrives as a step against caches warmed for ordinary traffic, so the initial miss rate is far higher.
3. The correct read/write mix, including the contention hot spots. Load tests are easier to write for reads; real events are frequently write-heavy, and writes contend where reads do not.
4. Third-party dependencies at realistic latency and rate limits, since in the real event those are shared with everyone else's peak.
5. User-initiated retries, which real users generate during slowness and which add load precisely when the system is struggling.
6. An injected failure, because the realistic scenario is high traffic and something degraded.
What to test for
Test to failure, not to target. Knowing you clear the bar is far less useful than knowing where the cliff is and what breaks first.
The most valuable output is the ordered list of what saturates first, second and third — which drives the capacity plan, the degradation ladder ordering and the runbook, and which a pass/fail test never produces.
The live-event addition
For a streaming platform, capacity is measured in connections, not requests per second — so CPU is the wrong signal and the test must generate realistic connection behaviour including reconnection, which is what a real disruption produces.