A service performs well in load tests and degrades after several days in production. What class of problem is this, and how is it found before release?

The class of problem

Slow accumulation. Nothing is wrong at any instant; something grows without bound over time:

• Memory leaks, classically an unbounded cache, a growing collection, or listeners never removed.
• Connection or file descriptor leaks, where an error path fails to release.
• Heap fragmentation, where memory is available but not usable in the shapes required.
• Unbounded internal data structures — a deduplication set, an in-memory index, a metrics registry with a high-cardinality label.
• Data-driven degradation, where a table grows and a query that was fast at a million rows is slow at a hundred million. This one is not a leak and is frequently misdiagnosed as one.
• Log and metric accumulation filling disk or degrading the collection agent.

How it is found

A soak test: sustained realistic load for hours to days, with resource trends monitored rather than throughput. The signal is a line with a slope — memory, descriptors, connections, latency — rather than a threshold breach. The test is looking for a gradient, which is why a short test cannot find it regardless of intensity.

Specifically:

• Run for longer than a deployment interval. A service deployed daily may leak and never show it, which is a real mitigation and also means the leak is discovered the first time deployments pause.
• Monitor derivatives, not levels. Memory at 60% is meaningless; memory rising 2% per hour is the finding.
• Include the error paths. Leaks concentrate in error handling, which a happy-path soak test never exercises. Inject failures during the soak.
• Use realistic data growth, since the data-driven case requires the dataset to grow during the test.

The financial-platform specific case

For a product with financial state, the accumulation is often in unresolved intermediate states: pending transactions never reconciled, reservations never released, workflow instances with no terminal state. These grow monotonically and eventually dominate a table or a queue.

They are a correctness problem that manifests as a performance problem, which is why the resolution is a terminal state and a sweeper rather than more capacity — and why the soak test that finds them is more valuable than the load test that does not.