What justifies a stream processing framework over a simple consumer loop?

What is being tested

Whether you can name the specific capabilities that justify substantial operational cost.

What the framework provides

1. Managed keyed state, checkpointed. Sessionisation, joins across streams, long windows, running aggregates. State must be partitioned, checkpointed, restored after failure and expired. Doing that correctly by hand is a project.

2. Event-time processing with watermarks, so windows are evaluated on when things happened rather than when they were seen. Necessary whenever devices buffer offline or partitions lag.

3. Exactly-once processing semantics, achieved by coordinating checkpoints with offsets — not exactly-once delivery, which is not available.

4. Automatic scaling and rebalancing of partitions across workers.

When a consumer loop is the better answer

Stateless transformation. Read, transform, write. A consumer loop with idempotent writes is dramatically less to operate, easier to debug, and adequate at surprisingly high volume.

Small state that fits in a database. If the aggregation can be an upsert into a table, the database does the durability and you avoid a distributed state system.

No event-time requirement, where processing-time windows are adequate.

This is a legitimate and under-chosen answer, and it should be the default until one of the four capabilities is genuinely needed.

The operational costs to weigh

State TTL is mandatory — a keyed aggregation with no expiry grows until the job dies, which is the most common streaming failure.

Watermark behaviour. The watermark is the minimum across partitions, so one idle or lagging partition stalls output entirely with no error. Idle-partition detection is usually off by default.

Lateness policy must be stated: how long to wait, and what to do with what arrives afterwards — drop, side-output, or emit a correction. A business decision, not a technical one.

The question underneath

Would hourly batch be indistinguishable to users? Frequently yes, and an order of magnitude cheaper and simpler.

Streaming is justified when freshness has measurable business value — a dispatch decision on 5-second-old data is measurably better than on 60-second-old data — not when "real-time" was requested without a reason.