A platform passes every load test yet degrades after several days of continuous operation. What class of problem is this, and what testing finds it?

The class of problem

Resource accumulation over time — problems whose effect is proportional to elapsed time rather than to load, so a test lasting an hour cannot find them regardless of intensity.

The usual causes:

• Memory leaks. Slow accumulation, invisible in an hour, fatal in a week.
• Connection or file-descriptor leaks, where a small percentage of operations fails to release.
• Unbounded in-memory caches without eviction, which grow until they exhaust memory.
• Fragmentation — heap or disk — degrading allocation performance gradually.
• Log and temporary-file growth filling disks.
• Database bloat: index fragmentation, dead tuple accumulation, statistics drifting until the planner chooses a worse plan.
• Thread or timer accumulation, where each request registers something never cleaned up.
• Certificate and token expiry, which is time-based by definition and never triggered by a short test.

What soak testing does

Runs a realistic sustained load for days, watching for trends rather than for thresholds. The signal is a metric with a slope, not a metric crossing a line:

• Memory usage rising without plateau across garbage collection cycles.
• Open connections or descriptors trending upward.
• Latency degrading slowly with no change in load.
• Disk consumption growing without bound.
• Query plan changes over time as statistics shift.

The essential discipline is to plot the trend and extrapolate. A memory graph rising 2% per day looks fine on any dashboard and exhausts the host in seven weeks.

Why it matters more for enterprise platforms

Consumer platforms deploy frequently, and deployment restarts processes, which masks accumulation problems — a leak that would take three weeks never has three weeks. Enterprise platforms with slower release cadences, long-lived processes and customer-managed deployments run for months without a restart, so accumulation problems reach their conclusion.

This means deployment frequency is a hidden variable in reliability: a system that only survives because it is restarted twice a week has an undiscovered problem, and that becomes visible the moment the release cadence slows.

The complementary practice

Trend-based alerting in production. Alerting on "memory above 90%" fires when it is nearly too late. Alerting on "memory trending to exhaustion within N days" gives a week of warning and turns an incident into scheduled work.

Most organisations alert on levels and not on slopes, which is why accumulation problems are usually discovered as outages.