Stream Processing Frameworks
Engines that process unbounded data with managed state, windowing and fault tolerance — chosen by state and time semantics rather than by throughput.
Definition
A stream processing framework runs continuous computations over unbounded input, managing partitioning, state, time semantics, checkpointing and recovery.
What the framework provides that you would otherwise build
- Managed state, checkpointed, so an operator resumes rather than recomputes after failure.
- Event-time processing with watermarks, so windows can be evaluated on when things happened rather than when they were seen.
- Exactly-once processing semantics, achieved by coordinating checkpoints with offsets — not exactly-once delivery, which is not available.
- Automatic scaling and rebalancing of partitions across workers.
If you need none of those, a simple consumer loop is dramatically less to operate, and that is a legitimate and under-chosen answer.
The decisions that matter
How much state? Stateless transformation is easy anywhere. Large keyed state — sessionisation, joins across streams, long windows — is what justifies a framework, and it is also what makes operation hard: state must be checkpointed, restored and eventually expired.
State TTL is mandatory. A keyed aggregation with no expiry grows until the job dies. This is the most common operational failure.
Windowing and lateness. Every windowed aggregation is a bet about how long to wait for late data, and what to do with what arrives afterwards — drop, side-output, or emit a correction. That is a business decision: a dashboard can drop, a payout cannot.
Watermark behaviour. The watermark is the minimum across partitions, so one idle or lagging partition stalls output entirely with no error. Idle-partition detection is usually off by default and is the fix.
The batch question
Frequently the honest answer is that hourly batch would be indistinguishable to users and an order of magnitude cheaper and simpler. Streaming is justified when freshness has a measurable business value — a dispatch decision on 5-second-old data is measurably better than on 60-second-old data — not when "real-time" was requested without a reason.
Failure scenarios
- Unbounded state, killing the job.
- A stalled watermark, so a healthy-looking job emits nothing.
- Late data silently dropped, so totals are quietly wrong.
- Two implementations of the same logic in a batch and a streaming path, which will diverge.
- A framework adopted for a stateless transformation that a consumer loop would have handled.
Interview question
"What justifies a stream processing framework over a simple consumer loop?"