A stream processing job has run for four months and now fails with out-of-memory errors every few hours. No code has changed. Diagnose.

What the interviewer is testing

Whether you know that unbounded state is the characteristic failure of long-running stateful jobs.

The diagnosis

Unbounded state growth. Nothing changed in the code because nothing needed to — state has been accumulating since deployment and has now crossed the available memory.

The usual sources:

Deduplication across all history. A job keeping every seen key to detect duplicates grows forever. This is the most common cause and looks entirely reasonable in code review.

A stream-stream join with a long or unset window, retaining both sides indefinitely.

Keyed state with no time-to-live — per-entity aggregates for entities that have long since become inactive. A session store keeping every session ever created.

Growing key cardinality: the business added tenants, products or devices, so the number of keys grew rather than the state per key.

The fix

Bound the state explicitly. Every piece of keyed state needs a retention decision tied to what correctness actually requires:

• Deduplication needs a window matched to the longest plausible retry or replay delay — usually hours or days, not forever. State beyond that window is buying nothing.
• Joins need an explicit window sized from the real arrival-time distribution between the two sides.
• Keyed aggregates need a TTL so inactive keys expire.

Then check the state backend configuration: spilling to disk rather than holding everything in memory changes the failure from a crash to a performance degradation, which is a far better failure mode.

What a strong answer adds

Monitoring state size per operator as a first-class metric with a trend alert. This failure is completely predictable months in advance and is only ever discovered as an outage because nobody graphs it.

And the recovery implication: large state means restore time is minutes, so that is the real time to recover, not process restart time. It should be stated in the service's recovery objective.

Common weak answers

Increasing memory, which buys weeks. Restarting on a schedule, which loses state and masks the cause.