A service is restarted nightly by a cron job and nobody remembers why. What do you suspect and how do you confirm it?

What is being tested

Recognition of the "restart to fix it" pattern as a symptom of gradual resource exhaustion, and whether you know how to confirm it rather than removing the cron job and finding out the hard way.

What to suspect

Something that grows with time or with cumulative work and is never released:

• A memory leak. The classic. Retained references, an unbounded in-memory cache, accumulating listeners or subscriptions.
• A connection or file descriptor leak. A code path that fails to release a resource under an uncommon branch, exhausting the limit over days.
• An unbounded in-process collection — a cache with no eviction, a deduplication set, a metrics map with unbounded cardinality.
• Fragmentation or garbage collection degradation, where pause times grow over hours.
• A thread leak, where threads are created and never terminated.
• Disk filling — logs, temporary files, unrotated data.
• A token or connection that expires and is not refreshed correctly, so the service works for exactly N hours.

The last one is worth checking early because it produces the cleanest signature: failure at a precise interval rather than gradual degradation.

How to confirm it

Look at the trends over a long time axis — 7 to 30 days — for the metrics that would show it: process memory, heap usage, open file descriptors, connection counts, thread counts, garbage collection pause duration and frequency, and latency percentiles.

The signature is a sawtooth: a gradual rise, then a vertical drop at the restart time. If any of those graphs shows that shape, you have your answer immediately.

If the graphs do not exist, that is the first finding — and adding them is the prerequisite for everything else.

Then: disable the restart in a non-production environment under sustained load and watch. That is a soak test, and it is the only tool that finds this class reliably: the failure takes hours or days to appear, so it is structurally invisible to any short test.

Then: profile. A heap profile comparing a freshly started instance with one that has been running for 20 hours shows exactly what is being retained.

Why it persisted

Because the restart works. It is a rational local response that removes the symptom, the pressure to investigate, and the evidence — and it accumulates as institutional folklore until nobody remembers why it exists.

The risk is that the underlying growth rate changes. Traffic doubles, the leak doubles, and the service now dies at 14 hours instead of 30 — during the day, unexpectedly, with nobody expecting it.

What a strong answer adds

That the cheap version of soak testing is watching production trends on a long axis. A dedicated multi-day soak test is expensive and finds nothing most of the time, which is why it gets cut; plotting memory, connections and latency over 30 days costs nothing and shows a leak to anyone who looks.