State Time-to-Live

A stateful streaming job keeps per-key state — a running count, a session, a join buffer. Absent an expiry policy, that state accumulates for every key the job has ever observed, so state size is a function of the job's entire history rather than of its current throughput.

State time-to-live evicts entries not touched within a defined period. It is the difference between state proportional to active users and state proportional to all users who ever existed.

Why it is the dominant lever

State drives almost everything expensive about a stateful pipeline at once:

• Memory, which sets instance size.
• Checkpoint size and duration, which sets how long a checkpoint blocks and how much storage it consumes.
• Recovery time, because restoring means reading that state back.
• Rebalance cost, because state moves when partitions move.

Which is why a single TTL setting frequently outperforms weeks of tuning elsewhere. It is also why lag and cost problems in stateful jobs tend to arrive together and slowly: nothing changes in the traffic, and the job degrades because its state has been growing since deployment.

Implementation patterns

• A TTL per state type, derived from the business meaning. A session times out after inactivity; a seven-day counter needs seven days; a join buffer needs the join window and nothing more.
• Bound what is stored per key. A rolling window or a sketch rather than an unbounded list — the per-key size matters as much as the key count.
• Incremental checkpointing, so checkpoint duration tracks the change rather than the total. Without it, checkpoint duration grows with state and eventually exceeds the interval.
• Local state with remote backup, so a restart on the same node reads locally.
• Tighter watermarks with a late-data side output, so windows close and release their state instead of waiting indefinitely for stragglers.
• State size and checkpoint duration as monitored SLIs, trended over weeks. Both grow slowly enough that point-in-time alerts miss them.

Industry example

The recurring pattern in real-time feature pipelines is a job that ran comfortably for months and now takes hours to recover from a restart. Nothing about the traffic changed; the state grew, checkpoint duration crept past the checkpoint interval, and a routine deploy became a multi-hour freshness incident.

The compounding problem is that recovery itself is a thundering herd — every instance restores large state from remote storage simultaneously, saturating exactly the resource recovery depends on. Staggered restarts and local state retention help; less state helps more.

Failure scenarios

• No TTL, so state is the job's full history and every operational property degrades with age.
• TTL longer than the business need, which is the same problem more slowly.
• Unbounded per-key collections, where the key count is fine and the payload is not.
• Full checkpointing at large state, where checkpoints overlap and the job never stabilises.
• State schema changes with no migration path, so the only option is discarding state and recomputing — expensive for long windows, and usually discovered the first time a definition changes.

Trade-offs

Expiry is data loss by design. A key that reappears after its TTL is treated as new, which is correct for sessions and wrong for a lifetime counter — those genuinely need durable storage outside the processor rather than a longer TTL.

Shorter TTLs also mean more eviction work and more recomputation for returning keys. The useful framing is that state in a stream processor is a working set, not a database: anything that must survive indefinitely belongs in a store designed for that, with the processor reading it rather than holding it.

Interview question

"Your streaming job's recovery time has grown from four minutes to two hours over a year, with no change in traffic. What happened, and what is the first thing you would change?"