Breaking-Point Testing

A load test that stops at the target and passes has confirmed one thing: the system handles the target under the conditions tested. It has not revealed where the constraint is, how much margin exists, or what the failure looks like — which are the three things capacity planning actually needs.

Testing to failure gives all three, and it is the same test run for longer.

Why it matters

A team that has never seen its system fail in a test will see it fail for the first time in production, under the worst conditions, with no prior knowledge of the failure mode. Knowing that the system degrades gracefully at 130% and collapses at 160% is a different operational posture from knowing it passed at 100%.

It also reveals the order of constraints. Removing the first bottleneck reveals the second, and capacity planning must anticipate the sequence rather than treating each as a surprise.

Implementation patterns

• Ramp past the target until throughput stops increasing, and record both the level and the symptom.
• Realistic data shape, not just realistic volume. Uniform synthetic keys produce an unrealistically good cache hit rate and never create the hot-key contention that actually breaks things. Replaying or shaping production traffic finds more real problems than any other single change.
• Start cold in at least one scenario, since a warm-cache test is a different system from the event that matters.
• Production-scale data, or query plans are untested.
• Inject dependency latency during the test, because testing against healthy dependencies validates a scenario that will not occur.
• A separate soak test for hours to days, monitoring derivatives rather than levels, since leaks and accumulation produce a gradient rather than a threshold breach.
• Record the failure mode, not just the number: does it shed, queue, crash, or corrupt? A system that sheds at its limit is in a completely different position from one that queues unboundedly.

Industry example

Retail platforms such as Nykaa and Myntra face sale events where the traffic shape, cache state and third-party behaviour all differ from ordinary days simultaneously. The characteristic post-incident finding is that the load test used uniform keys and warm caches, and therefore tested a system that does not exist during a sale.

Failure scenarios

• Stopping at the target, learning nothing about margin or failure mode.
• Uniform key distribution, hiding hot-key contention.
• Mocked dependencies at one millisecond, hiding the real behaviour.
• A short test, missing leaks and accumulation.
• Passing the test and treating it as assurance, which is the outcome the test was designed to avoid.

Trade-offs

Testing to failure costs more time and, if run against production infrastructure, carries risk. It also produces findings the team must then act on, which is why it is sometimes quietly avoided.

The alternative is to discover the constraint during an event. The test costs a day; the discovery costs an incident — and unlike the incident, the test happens at a time of your choosing with everyone available.

Interview question

"Your load test passes at your projected peak. What do you do next, and what would you want to be able to say about your system that this test has not told you?"