A stream processing job has run fine for four months. It now fails with out-of-memory errors every few hours. No code has changed. Diagnose.

What the interviewer is testing

Whether you recognise unbounded state as the characteristic failure of long-running stateful jobs.

The diagnosis

State has been accumulating since deployment and has now crossed available memory. No code changed because none needed to — this was always going to happen, on a schedule set by the growth rate.

The usual sources:

Deduplication across all history. A job keeping every seen key to detect duplicates grows forever. This is the most common cause and it looks entirely reasonable in code review.

A stream-stream join with a long or unset window, retaining both sides indefinitely.

Keyed state with no time-to-live — per-entity aggregates for entities long since inactive, or a session store keeping every session ever created.

Growing key cardinality: the business added tenants, products or devices, so the number of keys grew rather than the state per key.

The fix

Bound the state explicitly. Every piece of keyed state needs a retention decision tied to what correctness actually requires:

Deduplication needs a window matched to the longest plausible retry or replay delay — hours or days, not forever. State beyond that window buys nothing.

Joins need a window sized from the measured arrival-time distribution between the two sides.

Keyed aggregates need a TTL so inactive keys expire.

Then configure the state backend to spill to disk rather than holding everything in memory, which converts a crash into a performance degradation — a far better failure mode.

What a strong answer adds

Monitor state size per operator with a trend alert. This failure is entirely predictable months in advance and is only ever discovered as an outage because nobody graphs it.

And the recovery implication that must be stated: large state means restore time is minutes, so that is the real time to recover, not process restart time. It belongs in the service's recovery objective, and it is routinely understated by an order of magnitude.

Common weak answers

Increasing memory, which buys weeks. Restarting on a schedule, which loses state and masks the cause.