A service passes a 30-minute load test at target throughput with p99 well inside budget. Run for twelve hours at the same rate it degrades - p99 climbs steadily from hour three and the process is restarted nightly by a liveness probe. What classes of defect does duration expose that rate does not?
Show the full answer Hide the answer
Why duration is a separate axis from rate
A load test asks whether the system can serve a rate. A soak test asks whether it can serve that rate for as long as production will. The defects duration exposes are accumulations: something grows per request and is never released, so the symptom is proportional to total requests served, not to requests per second. Thirty minutes at 1,000 rps is 1.8M requests. Twelve hours is 43M. A leak of 100 bytes per request is 180 MB in the first case and 4.3 GB in the second, which is the difference between invisible and fatal.
The classes, with their signatures
- Heap growth that survives collection. A cache with no bound, listeners never unsubscribed, or a map keyed by request id. Signature: live-set size after a full collection rises monotonically, collection frequency rises with it, and p99 climbs because the process spends a growing share of its time collecting rather than serving.
- Handle and connection leaks. Sockets left in
CLOSE_WAIT, files not closed, a pool that creates on timeout and never reaps. Signature: open file descriptors climb towards the process limit, then every operation fails at once. The cliff, not the slope, is what you see in production. - Unbounded on-disk growth. Logs, temporary files, spool directories, and a write-ahead log with no checkpoint. Signature: a disk that fills on a schedule nobody wrote down.
- Fragmentation. Allocators that cannot return memory to the OS after long periods of mixed-size allocation. Resident memory climbs while live-set is flat, which is the pair of signals that distinguishes it from a leak.
- Things that expire. Tokens with an eight-hour lifetime, a certificate with a one-day rotation, a leader lease. A 30-minute test cannot observe a 60-minute expiry, and the renewal path is therefore the single least-tested code in many services.
- State the restart hides. A nightly restart is not a fix; it is the evidence. It also masks the defect from the test suite, because every test run starts from a clean process.
How long, and what to watch
Run at least one duration longer than the longest timer in the system (token, lease, certificate, cron, retention job), and at production-like concurrency rather than production-like rate alone: concurrency determines how many objects are live at once. Track live-set after collection, resident set, open file descriptors, thread count and disk free as first-class series. Alert on the slope, not the level — a 2 MB-per-minute rise is 2.8 GB a day and is visible in twenty minutes of regression if you are measuring gradients.
When this is the wrong answer
A batch job that runs for four minutes and exits does not need a soak test; process exit is the garbage collector. The same goes for a function-as-a-service handler whose runtime recycles containers every few minutes, where the leak is real but is paid for by the platform. Spend the test budget on duration only where processes are long-lived, and prefer one honest twelve-hour run before a launch to a nightly two-hour run that never reaches the interesting part of the curve.