A trading platform's functional tests all pass and production fails under load. What should the non-functional test strategy contain?

What it should contain

• A load test to and beyond the design point, with realistic data shape rather than uniform synthetic keys — which produce an unrealistically good cache hit rate and never create the hot-key contention that actually breaks things.
• Testing to failure, not to the target. A test that stops at the target confirms a number; one that finds the breaking point tells you the margin and the failure mode, which are the three things capacity planning needs.
• Fault injection during load, which is the essential combination — injecting latency at 10% load proves the error path executes, while injecting at peak reveals whether the timeouts, pool sizes and breaker thresholds were correct. They usually were not, because they were guesses.
• A soak test over hours to days, monitoring derivatives rather than levels — memory at 60% is meaningless, memory rising 2% per hour is the finding. This is the only test that finds accumulation.
• A cold-start test, since a fleet scaled up before the market open has empty caches and cold connection pools, and that is the state that matters.
• Latency budgets asserted as fitness functions in the pipeline, failing the build on regression — because architectural characteristics degrade through a long series of individually reasonable changes, which no reviewer can catch.

What the strategy must define

• The design point, as a number derived from the business rather than from imagination — for a brokerage, the market-open burst rather than the average.
• Which percentile matters, since p99 and p99.9 have very different costs and the choice should be argued from what a customer experiences when it is missed.
• What degradation is acceptable beyond the design point, which determines whether the system sheds, queues or fails.

The specific gap in most strategies

Testing with healthy dependencies, which validates a scenario that will not occur. The market-open burst coincides with every dependency's own peak — exchanges, market-data feeds, identity providers — so the correlated bad day is the design point, not the average one.

The precondition

Do it in production or accept that you tested something else. Staging differs in data volume, cache state, traffic shape and dependency behaviour, and load-dependent failures live precisely in those differences.