A streaming aggregation reports lower totals than the batch job it replaced. Both read the same source. What is likely happening?

The likely cause: late events dropped past the watermark

The batch job reads a completed day and sees everything, including records that arrived hours after the event occurred. The streaming job closes each window when its watermark advances, and anything arriving later is discarded — silently, unless someone counted the drops.

Common sources of lateness that batch never noticed: mobile clients buffering offline, a slow or retried partition, a producer outage backfilling later, and clock differences at the edge.

How to confirm it

Instrument the drop count. Most stream processors expose late-record metrics; if not, route late records to a side output and count them. The number should reconcile with the shortfall — and if it does, you have the diagnosis in minutes rather than by inference.

Also check the watermark strategy and how much lateness it actually allows, and whether the shortfall correlates with periods of high lag.

The options, and their trade-offs

Increase allowed lateness. Hold windows open longer before final emission. Captures more, adds latency to every result, and increases state size. Choose the value from the measured distribution of event-time delay — the p99 of arrival lag — not from a guess.

Emit updates rather than dropping. The window fires on the watermark and re-fires when late data arrives, correcting the earlier result. Correct and complete, and it requires every downstream consumer to handle restatement — which is a contract change, not just a configuration one.

Route late records to a side output and reconcile them in a separate, slower path. Pragmatic: the main stream stays fast, and correctness is restored by a periodic job.

Accept the difference explicitly, with the drop rate monitored and a documented tolerance. This is legitimate for operational dashboards and not for anything billed or reported.

The wider point

Streaming and batch are answering slightly different questions, and expecting exact equality between them is usually a mistake. Batch has the luxury of a closed window; streaming trades completeness for latency. Where exact reconciliation is required — financial reporting especially — the usual arrangement is streaming for immediacy plus a batch reconciliation that produces the authoritative figure.

What a strong answer adds

Noting that watermarks are typically derived per partition and combined as the minimum, so a single idle or lagging partition stalls the whole computation — a distinct failure that presents as no output rather than low output, and one worth ruling out early.