A streaming aggregation with a 10-minute allowed lateness publishes hourly revenue. After a regional network outage, a mobile fleet batch uploads events that are 6 hours old. What happens to each affected window, and what should the pipeline do?

What happens, step by step

1. The watermark - the pipeline's estimate of "no more events older than this will arrive" - advanced past those six hours long ago, because it is derived from the event times the pipeline was actually seeing.
2. Each affected window therefore fired, emitted its result and released its state once the watermark passed window-end plus the 10-minute allowed lateness.
3. The uploaded events arrive for windows that no longer exist in the pipeline's memory. They are dropped, or routed to a late-data side output if one was configured. There is no in-between: state that has been released cannot be amended.
4. The six hourly figures are already downstream - in a dashboard, in a daily email, in a finance team's spreadsheet.

Where it amplifies

The technical loss is small. The organisational damage is the restatement: a number that has been used for a decision now changes. If the pipeline silently drops the events, the published figures are quietly wrong and nobody knows. If it emits corrections, every consumer must be able to handle a window being revised - and most dashboards and extracts are written as though yesterday's number is final.

What stops it

• A late-data side output, always. Events beyond the allowed lateness go to a durable location with their event time, not to the floor. This is one line of pipeline configuration and it is the difference between a correctable problem and a silent one.
• A defined restatement path. The log-as-source-of-truth design that The New York Times described in 2017 for its publishing pipeline is the architectural version of this: because every published asset is an event in a durable log, a view can be rebuilt by replaying the log rather than patched in place. A pipeline with that property treats a six-hour-late batch as a recompute, not an incident.
• Two speeds, stated explicitly. The streaming number is "live and provisional"; a daily batch over the complete data is "authoritative". Publishing both, labelled, removes the pressure to make the streaming path handle arbitrary lateness.

How to choose the allowed lateness

From the measured distribution of event lag, not from a round number. Instrument the gap between event time and processing time and set lateness to cover the 99^th percentile of normal operation - often seconds to minutes. Anything beyond that is not a late event, it is a backfill, and backfills belong to the batch path where state size is not a constraint.

The cost of a larger window is concrete: window state must be retained for the whole allowed lateness, so moving from 10 minutes to 6 hours multiplies the state the job keeps by roughly 36 and lengthens every recovery from a checkpoint.

When this is the wrong answer

If the aggregation is over a small enough key space to hold indefinitely - daily totals per country, say - a long or unbounded lateness with incremental corrections is perfectly affordable, and the two-speed architecture is unnecessary ceremony. The constraint that makes watermarks and lateness necessary is state size, so check the state size before designing around it.