Soak Testing

Definition

A soak test applies steady, moderate load over an extended period — typically many hours to several days — to find degradation that short tests structurally cannot detect.

What only a soak test finds

• Memory leaks. A slow leak is invisible in a 20-minute test and takes the service down after three days. This is the classic soak-test finding.
• Connection and file descriptor leaks. A path that fails to release a resource under an uncommon branch exhausts the limit eventually.
• Unbounded data growth. A cache, a queue, an in-memory index or a temporary table that grows without limit.
• Disk filling — logs, temporary files, unrotated data.
• Fragmentation and garbage collection degradation, where pause times grow over hours.
• Token and certificate expiry. A credential refreshed every 12 hours, with a bug in the refresh path, works perfectly for 12 hours.
• Scheduled interactions. A nightly batch job interacting badly with steady traffic, visible only if the test spans it.
• Cumulative correctness drift. Counters, reconciliation gaps and derived state that diverge slowly.

How to run one

• Realistic and steady load, not peak. The purpose is duration, not intensity.
• Long enough to cross the relevant cycles — token refresh, log rotation, batch windows, cache expiry, at minimum a full day.
• Monitor trends, not absolutes. The signal is a slowly rising line: memory, connections, latency, disk. A flat line at any level is fine; a gentle upward slope is the finding.
• Include deployments during the test, since production is never a static process.

Why it is skipped, and why that is a mistake

It takes days and finds nothing most of the time, which makes it easy to cut. But the failures it finds are exactly the ones that cause multi-hour production outages at inconvenient times, and they are nearly impossible to diagnose in production because the cause was hours or days before the symptom.

The cheap version, which is better than nothing: watch the trends in production on a long time axis. Memory, connections and latency plotted over 30 days will show a leak, if someone looks.

Interview question

"A service is restarted every night by a cron job and nobody remembers why. What do you suspect and how do you confirm it?"