A streaming aggregation stops emitting results. The job is healthy, consuming messages and using CPU. What is the most likely cause?

What is being tested

Understanding of event-time processing and watermarks — specifically that a healthy-looking job can stop producing output with no error at all.

The mechanism

A windowed aggregation emits a result when it believes the window is complete. Completeness is estimated by a watermark: the system's assertion that no more events with an event time earlier than T will arrive.

The watermark is computed as the minimum across all input partitions, because a window is only complete when every source has passed it. So a single partition that is idle — receiving no events — or lagging badly holds the global watermark back. Every other partition may be flowing normally; the job consumes, uses CPU, maintains state, and emits nothing.

There is no error, because nothing has failed. The system is correctly refusing to emit a result it cannot yet consider complete.

Diagnosis

• Per-partition watermark or event-time lag. The stalled one stands out immediately. This metric is the single most valuable thing to have on a streaming dashboard and is frequently absent.
• Idle partitions. Common with keys that are geographically or temporally sparse — a partition serving a region where it is 4am.
• A consumer lagging on one partition due to skew or a slow downstream.

Fixes

• Idle-partition detection: after a configured period with no data, allow that partition's watermark to advance on processing time so it stops blocking the group. Every mature streaming framework supports this and it is usually off by default.
• Fix the underlying lag if a partition is genuinely behind.
• Reconsider partitioning if idleness is structural rather than incidental.

The design lesson

Every windowed aggregation is making a bet about lateness, and the bet must be explicit: how long do we wait for late data, and what do we do with data that arrives after that? The options — drop, side-output for separate handling, or emit a correction — are business decisions rather than technical ones. A dashboard can drop. A courier payout cannot.

Related failure modes worth naming

• Unbounded state. A keyed aggregation with no TTL grows until the job dies. State expiry is mandatory, not optional.
• Late data silently dropped, so totals are quietly wrong and no metric reveals it. Always count what you drop.
• The system stopping rather than degrading. In a real-time marketplace — dispatch matching couriers to orders — an impaired streaming layer must fall back to a simpler heuristic on staler data, not stop assigning. A marketplace that stops matching is a marketplace that is down.

Why the other options are less likely

A failing sink usually produces errors and backpressure, showing as rising lag rather than silence. Memory thrashing shows as GC pressure and eventual crashes. A deleted topic produces immediate, loud errors.