Load Testing in Practice

Definition

Load testing applies expected traffic to a system and measures whether it meets its objectives. Its value depends almost entirely on how closely the test resembles reality, and most load tests fail this test in predictable ways.

What makes a load test realistic

Representative request mix. Real traffic is a distribution across endpoints with wildly different costs. Hammering one cheap endpoint proves nothing about a system whose expensive endpoint is 3% of traffic and 60% of load.

Realistic data. A test against 1,000 rows tells you nothing about behaviour against 100 million — indexes behave differently, caches hit differently, query plans change. This is the single most common reason a load test passes and production fails.

Realistic cache state. A test that warms the cache and then measures is testing the cache. Real traffic has misses, and the miss path is what falls over.

Realistic think time and connection behaviour. Users pause. Clients reconnect. A test with zero think time and persistent connections produces a load pattern that does not exist.

Realistic concurrency distribution. Real traffic has a hot minority of tenants, users and objects. A uniform random distribution avoids exactly the hot-key problems that break production.

What to measure

Not just "did it pass". Latency percentiles across the range of load, the point at which latency begins to degrade non-linearly, error rate against load, saturation of every pool, and — critically — behaviour past the limit. What a system does when overloaded matters more than where the limit is.

Where to run it

Production-like or production. A scaled-down environment gives a number that cannot be extrapolated, because bottlenecks change with scale. Testing in production with a small share of traffic, or during low-traffic hours with careful limits, gives answers that are actually usable — and it is more common than people assume.

Failure scenarios

• Tested against an empty database.
• One endpoint tested, chosen because it was easy.
• Run once before launch, never repeated, so regressions are invisible.
• Passed at the target and never tested beyond it, so overload behaviour is unknown.
• Load generator saturated, so the measurement is of the test harness.

Interview question

"Your load test passes at 3x expected traffic and production falls over at 1.2x. Name four ways the test could have been unrealistic."