Keyed State Size
The total state a job holds per key across all keys, which governs memory, checkpoint duration and recovery time.
Stateful operators accumulate per key: a join buffer, a session's events, a running aggregate, a deduplication set. Total state is the number of active keys multiplied by the state each holds, and it grows with cardinality rather than with throughput.
That distinction is what surprises teams. A pipeline handling a steady ten thousand events per second can have state that grows without bound if the key space does — a deduplication set keyed on a unique event identifier retains every identifier it has seen, forever, unless something expires them.
State time-to-live is therefore not an optimisation but a correctness requirement for any keyed state whose key space is unbounded. Without it the job runs well for weeks and then degrades: checkpoints take longer, memory pressure rises, and recovery time grows past the point where the job can catch up after a restart.
The related decision is the state backend. In-memory state is fast and bounded by heap; an embedded key-value store spills to disk and supports far larger state at some latency cost. Choosing the first and then growing past it is a rewrite rather than a configuration change.