A stream processor maintains per-vehicle state across events. What does that require operationally, and where does it break?

What it requires

• Durable checkpointed state, so a worker restart resumes rather than losing everything. The checkpoint interval is a trade between recovery time and steady-state overhead.
• State partitioned by the same key as the input, so a worker owns both the partition and its state and no cross-worker coordination is needed.
• A recovery path whose duration is understood. Restoring a large state on restart takes time proportional to its size, and if that exceeds the tolerance, the design is wrong regardless of how correct the processing is.
• Bounded state. Per-entity state that accumulates without expiry grows monotonically — the streaming equivalent of a memory leak, and it is discovered when a rebalance takes hours.

Where it breaks

• Rebalancing. Adding or losing a consumer reassigns partitions, and the new owner must restore that partition's state before processing resumes. A large state makes every rebalance an outage, and rebalances happen on every deployment.
• A feedback loop: processing occasionally exceeds the session timeout, the consumer is evicted, a rebalance is triggered, state must be restored, which makes the next timeout more likely. This looks like slow processing and is a stability problem.
• Skew. One partition's state far larger than the others makes its worker slow and its rebalance long, and the job's characteristics are set by the worst partition.
• State that outlives its entity, such as a vehicle that left the fleet — which is where the unbounded growth comes from and which needs an explicit expiry.

The design decisions that reduce the pain

• Keep the state small. A running aggregate rather than a buffer of events; the latest value rather than the history. What is kept in the processor should be only what the processing needs, with anything else in a store.
• Expire aggressively, with a defined time-to-live per key.
• Use an external state store where the state is large, trading latency for a much shorter recovery — which is the right trade above a threshold.
• Tune the session timeout against the observed processing time distribution, not against the mean, since the eviction is triggered by the tail.

The operational metric that matters

Rebalance frequency and duration. A pipeline that rebalances often, or whose rebalance takes minutes, is one deployment away from an incident — and neither number is visible in throughput or lag metrics, which is why the problem is usually diagnosed as something else.