Mobile clients buffer events offline and deliver them hours later. How should watermarks and late data be handled?

The tension

A watermark asserts that events older than a threshold are no longer expected, allowing windows to close and state to be released. Set it conservatively and results are late and state grows; set it aggressively and late events are dropped.

With mobile clients buffering offline for hours, no watermark setting is correct for both the common case and the tail.

The resolution

Separate the paths by lateness rather than choosing one watermark.

1. A tight watermark for the real-time path, producing results at useful latency and accepting that a small proportion of events arrive after the window closed. This serves live dashboards and operational decisions, which need timeliness more than completeness.

2. Late events routed to a side output, not dropped silently — so they exist and can be accounted for.

3. A batch correction path that recomputes affected windows from the durable log, producing the complete result at a later time. Consumers needing completeness read this.

4. Both results published with an explicit completeness indicator, so a consumer knows whether they are reading the fast approximate answer or the settled one.

This is the practical resolution because it stops pretending one number can be both fast and complete.

The client-side half

Clients emerging from a coverage gap should send their most recent state, not flush the entire backlog. A location from four minutes ago has no value in a dispatch decision, and thousands of clients flushing simultaneously after a network event produces a burst that is both useless and expensive.

Server-side, discard stale events on arrival rather than processing them — which is the same selective reliability principle that applies to any time-sensitive data.

The correctness note

Event time must come from the client and be treated as untrusted. Client clocks are wrong in ways that are systematic rather than random, so events carry both client time and server receipt time, and the pipeline must handle timestamps that are impossible — far in the future, or before the device existed.