A stateful stream processor's state grows continuously and recovery takes hours. What design changes address this?

Why state grows

Keys are never retired. A per-user or per-channel aggregation accumulates an entry for every key ever seen, and without expiry the state is the cumulative history of the system rather than its active set.

Contributing: windows held open by a conservative watermark; unbounded collections per key; and state schema that stores more than the computation needs.

The changes

1. State time-to-live per key. Entries not touched within a period are evicted. This is the single most important change — it makes state proportional to the active set rather than to cumulative history.

2. Bound what is stored per key. A rolling window of recent interactions rather than an unbounded list. Storing only what the computation reads.

3. Tighter watermarks with a side output for late data, so windows close and their state is released.

4. Incremental checkpointing, so a checkpoint writes only changes rather than the whole state — which is what makes checkpoint duration independent of total state size.

5. Local state with a remote backup, so recovery reads from local disk where possible and only falls back to remote restore when the instance is genuinely new.

6. Increased parallelism, so each instance holds a smaller share — bounded by the partition count, which is why partition count must be planned for growth.

Why recovery is slow

Restoring large state from remote storage across all instances simultaneously. The restore is bandwidth- bound and every instance competes.

Mitigations: incremental checkpoints, local state retention across restarts, staggered restarts rather than simultaneous, and — most effectively — less state.

The operational property to design for

Restarting a stateful job with a changed state schema. Existing state cannot simply be reused, so it must be migrated or discarded — and discarding means recomputing from history, which for long-window state is expensive.

This is discovered the first time a feature definition changes, and designing for it — versioned state, a migration path, or bounded windows that recompute cheaply — is far easier before that moment than after.