Stress Testing

Definition

Where load testing verifies behaviour at expected load, stress testing pushes past it. The question is not "can it handle the traffic" but "what does it do when it cannot?"

Why the failure mode matters more than the limit

Two systems with identical capacity can behave completely differently at 150% of it:

• System A sheds excess load, serves the rest normally, and recovers instantly when traffic drops.
• System B accepts everything, queues unboundedly, latency climbs until every request times out, memory exhausts, it crashes, restarts to a cold cache, and is overwhelmed again — a crash loop that outlasts the traffic spike by an hour.

System B is the default. Graceful overload behaviour must be built.

What to look for

• The knee. The load at which latency begins rising non-linearly. This is the real capacity limit, well below the point of errors, and it is what capacity planning should target.
• Behaviour past the knee. Does throughput plateau or collapse? Collapse means queueing and contention have taken over.
• The first thing to break, which is frequently not what anyone expected — a connection pool, an ephemeral port range, a file descriptor limit, a log volume, a third-party quota.
• Recovery. When load returns to normal, does the system recover immediately, slowly, or not at all without a restart? Slow recovery usually means an unbounded queue that must now drain.
• Blast radius. Does overload on one endpoint affect others? If so, there is no bulkhead.

What good overload behaviour looks like

• Admission control: reject early and cheaply rather than accepting work that cannot be completed. A request rejected after the database query has already cost you the query.
• Bounded queues with an explicit policy when full.
• Prioritisation: shed low-value traffic first, using a classification decided before the incident.
• Fast, clear rejection — a 429 or 503 with Retry-After, so well-behaved clients back off rather than retrying immediately and making it worse.
• Stable recovery without a restart.

Failure scenarios

• Never stress tested, so overload behaviour is discovered in production.
• Tested to the limit and no further, missing the collapse.
• Recovery not tested, which is where the long outages come from.
• The client's retry behaviour ignored in the test, so the amplification effect is absent from the results.

Interview question

"At 150% of capacity, what should your service do? Describe the mechanisms."