Idle Source Stall

Event-time processing advances only when the watermark advances, and the watermark is the minimum across all input partitions. A partition producing no events contributes no timestamps, so its watermark does not move, so the overall watermark does not move — and every window in the pipeline stops closing.

The result is that output stops completely while every individual component is functioning correctly. No error is raised, no process crashes, throughput on the active partitions is normal, and the dashboards showing consumer lag look fine because the active partitions are being consumed.

Why it matters

It is a total output failure that presents as healthy. The pipeline is running, consuming and processing; the aggregations simply never emit. Detection typically comes from a downstream consumer noticing that a number has not changed, which is hours later and frequently reported by a business user.

The confusion during diagnosis is the expensive part: every obvious signal is green, so the investigation proceeds through the processor, the sinks and the consumers before anyone examines watermark progress per partition — which is the one metric that shows it immediately and the one most pipelines do not have.

Implementation patterns

• Configure idle source detection, so a partition producing nothing for a defined period is excluded from the watermark calculation rather than stalling it. This is a supported feature in mainstream processors and is off by default, which is why the failure is common.
• Monitor watermark lag per partition as a first-class metric, alerting when it diverges from processing time. This is the signal that identifies the problem in seconds rather than hours.
• Alert on output absence, not only on error rate. A window that has not emitted in an expected interval is an alert condition, and it is the one that catches this from the consumer's side.
• Understand the legitimate causes: a genuinely quiet partition — a low-traffic region, an inactive tenant, an overnight lull — an upstream producer that has stopped, a partition assigned to no producer, or a source that produces only during business hours.
• Set the idleness timeout deliberately, since too short discards genuine late events from a slow source and too long leaves the stall in place.
• Consider partition assignment, since a key space that leaves some partitions permanently empty guarantees the problem — and an empty partition is a partitioning defect as well as a watermark one.
• Distinguish stall from lag in alerting, because they have different causes and different responses.

Industry example

The failure is a standard operational lesson in Flink and Beam deployments, prominent enough that idleness handling is explicitly documented as a configuration every event-time pipeline should consider. It surfaces most often in two situations: a multi-region stream where one region is quiet overnight, and a partitioned source where the key distribution leaves some partitions empty.

The generalisable lesson is broader than streaming: the most expensive failures are the ones that present as healthy, because time-to-restore is dominated by time-to-diagnose and every green signal actively misdirects the investigation.

Failure scenarios

• Idle source handling not configured, which is the default.
• A quiet overnight region stalling a global pipeline every night.
• A producer stopped or misconfigured, leaving its partition permanently silent.
• Empty partitions from a key distribution that does not cover the partition count.
• Watermark progress unmonitored, so nothing surfaces the condition.
• Alerting only on errors and lag, both of which look normal.
• Idleness timeout set too aggressively, discarding real events from a genuinely slow source.
• The same failure in a downstream job, where a stalled upstream output stalls everything behind it.

Trade-offs

Idleness handling trades correctness for liveness: excluding a quiet partition from the watermark allows windows to close, and events that later arrive from that partition are late by construction and may fall outside the allowed lateness.

Setting the timeout is therefore a real decision. Too aggressive and a slow-but-active source has its events discarded; too generous and the stall persists for the duration of the timeout each time a source goes quiet.

The trade is a small amount of completeness in exchange for the pipeline continuing to produce output at all — which is almost always correct, because a stalled pipeline produces nothing and the alternative loses a fraction of events from one quiet source. The important part is that it is a configured decision rather than a default, and that the discarded events are visible in a late-data side output rather than silently dropped.

Interview question

"Our hourly metrics stopped updating at 2am and every dashboard is green — the job is running, lag is zero, no errors. Tell me your first hypothesis, the one metric you would look at, and what you would change so the next occurrence pages someone within five minutes."