A streaming aggregation must produce results promptly while some events arrive minutes late. How do watermarks, windowing and allowed lateness interact, and what is the business decision underneath?

The unavoidable tension

Event time is when it happened; processing time is when you saw it. They diverge because of network delays, mobile devices offline, retries, batching and partition skew.

A window over event time must decide when to stop waiting for events belonging to it. Waiting longer is more complete and less prompt; closing sooner is prompter and less correct. No mechanism removes this trade-off — the mechanisms make it explicit and configurable.

The mechanisms

Watermark: an assertion that no further events with a timestamp earlier than X are expected. It is a heuristic, typically derived as the maximum observed event time minus a fixed lateness allowance, and it can be wrong.

Window closure: when the watermark passes the window's end, the window is computed and emitted.

Allowed lateness: a further grace period after closure during which late events still update the result, producing a correction. State for the window is retained during this period, which is the cost.

Side output for very late data: events arriving beyond the allowed lateness, routed somewhere visible rather than dropped silently. This path must exist, because silent dropping is how a systematic data-loss problem goes unnoticed for months.

The design decisions

• Watermark delay set from the observed distribution of event lateness, not a guess. Measure the percentiles of the gap between event time and arrival time — the answer is usually surprising and it changes by source and by client type.
• Per-source watermarks, since a mobile client and a server-side producer have completely different lateness profiles, and a single global watermark is set by the worst.
• Idle source handling, because a partition producing no events stalls the watermark for the whole stream — a common and confusing failure where output stops entirely and nothing appears broken.
• State retention bounded, since allowed lateness holds window state and generous lateness across many keys is a memory problem.
• Emit updates rather than a single final answer where consumers can accept corrections, which allows a prompt provisional result plus refinement.

The business decision underneath

"How wrong may this be, and for how long?"

• A real-time operational dashboard wants promptness and tolerates a provisional figure that later corrects.
• A financial or regulatory figure tolerates no silent correction, and must either wait for completeness or publish explicitly as provisional with a defined finalisation point.
• A machine-learning feature usually prefers consistency between training and serving over either.

This is a product and business decision expressed as a watermark configuration, and engineers routinely make it alone by choosing a plausible-looking number. Surfacing it — "with this setting we will be wrong by roughly this much for roughly this long" — is the architect's contribution, and it frequently changes the answer.