Late-Data Policy

Event-time processing requires deciding when a window can be considered complete, because events arrive out of order and some arrive very late. A [[watermark]] is the mechanism for that decision; the late-data policy is what happens to everything the watermark excluded.

Every streaming system has one. The failure is having it by default rather than by choice, and discovering during an incident that the framework silently drops.

The three options and what each costs

• Drop. Cheapest, bounded state, and silently lossy. Acceptable only when the lost proportion is measured and known to be immaterial — which requires counting the drops, so even "drop" needs a counter.
• Side output. Late events routed to a separate stream for correction, reconciliation or analysis. The default worth defaulting to, because it converts silent loss into visible, actionable data.
• Window update. The window's result is recomputed and re-emitted. Correct, and it pushes the difficulty downstream: consumers must handle a value they already read changing, which is a contract, not a detail.

Why one watermark is rarely right

Mobile clients, offline-capable applications and unreliable networks produce a distribution with a tight common case and a very long tail. A watermark loose enough to capture the tail makes every result late; one tight enough for real-time use excludes the tail.

The resolution is to separate the paths by lateness rather than to pick a compromise:

• A tight watermark on the real-time path, serving decisions that need timeliness more than completeness.
• Late events to a side output rather than dropped.
• A batch correction path recomputing affected windows from the durable log.
• Both results published with a [[completeness-indicator]], so a consumer knows which answer they hold.

Industry example

Ride-hailing, delivery and field-operations platforms all hit this: a driver in a tunnel or on a poor connection buffers events and flushes them minutes or hours later. Two lessons recur.

First, client-side, send current state rather than the backlog — a location from four minutes ago has no dispatch value, and thousands of clients flushing simultaneously after a network event is an expensive burst carrying nothing useful. Discard stale events at the client, and again on arrival.

Second, treat client event time as untrusted. Device clocks are wrong in both directions, sometimes by years, and a single bad clock can advance a watermark and cause every other event to be marked late. Carry both client and server timestamps, and bound how far a client timestamp may deviate.

Failure scenarios

• Silent dropping, where nobody knows the loss rate because nothing counts it.
• A watermark driven by an unbounded clock skew, which lets one device close everyone's windows.
• Window updates with no downstream contract, so consumers double-count or read a value that quietly changed.
• A side output nobody consumes, which is dropping with extra storage cost.
• One number presented as both fast and complete, which is the underlying error the policy exists to make explicit.

Trade-offs

Allowing more lateness means holding state longer, which costs memory and checkpoint size, and delaying results. Allowing less means losing data. There is no setting that avoids the trade — which is exactly why it should be a stated product decision rather than a framework default.

Interview question

"Your mobile clients sometimes deliver events six hours late. Real-time dashboards need sub-minute freshness and finance needs the daily totals to be right. How do you serve both without maintaining two definitions of the metric?"