Watermark

A streaming aggregation over event time faces an unanswerable question: have all the events for this window arrived? In an unbounded stream there is no way to know — an event from an hour ago may arrive at any moment from a device that was offline.

A watermark is the system's answer: an assertion, derived heuristically, that events with a timestamp earlier than X will not be seen again. When the watermark passes a window's end, the window is computed and emitted.

The watermark is a guess, and it can be wrong. That is not a defect; it is the mechanism by which an unbounded problem becomes tractable, and the size of the guess is the configuration knob for the whole trade-off.

Why it matters

Waiting longer is more complete and less prompt; closing sooner is prompter and less correct, and no mechanism removes this. The watermark makes the choice explicit, configurable and measurable rather than accidental.

It is also where a business decision is routinely made by an engineer choosing a plausible number. A watermark delay of thirty seconds versus five minutes is a statement about how wrong the output may be and for how long — and the people who care about the answer are usually not in the room when it is set.

Implementation patterns

• Derive the delay from the observed distribution of lateness rather than from intuition: measure the percentiles of arrival time minus event time. The answer is usually surprising, and it differs sharply by source and client type.
• Per-source watermarks, since a mobile client and a server-side producer have completely different lateness profiles and a single global watermark is set by the worst contributor.
• Handle idle sources explicitly. A partition producing no events stalls the watermark for the entire stream — output stops completely and nothing appears broken, which is a common and deeply confusing failure.
• Allowed lateness as a grace period after closure, during which late events correct the result — with window state retained for that period, which is the cost and which becomes a memory problem across many keys.
• A side output for data beyond the allowed lateness, routed somewhere visible. This path must exist, because silent dropping is how a systematic data-loss problem goes unnoticed for months.
• Emit updates rather than a single final value where consumers can accept corrections, giving a prompt provisional answer plus refinement.
• Monitor watermark lag as a first-class metric, since a stalled or drifting watermark is invisible in throughput and error metrics.

Industry example

Watermarks are central to Flink, Beam and every event-time stream processor, and their design traces to the Dataflow model's treatment of completeness in unbounded data. In production the recurring lessons are operational rather than theoretical: idle partitions stalling watermarks, per-source lateness profiles diverging enough that a global watermark is useless, and late-data side outputs left unimplemented so that the loss is never observed.

The design question that reaches architects most often is the business one — whether an output may be provisional and later corrected — which determines the configuration entirely and is usually settled by default rather than by decision.

Failure scenarios

• Delay chosen by guess, either dropping a meaningful share of events or delaying output far beyond need.
• A global watermark set by the slowest source, penalising every prompt one.
• Idle partitions stalling the watermark, halting output with no error.
• No side output for very late data, making systematic loss invisible.
• Generous allowed lateness across many keys, producing unbounded state growth.
• Consumers assuming outputs are final when the pipeline emits corrections, so a downstream system double-counts.
• Processing time used where event time was needed, which produces plausible-looking output that is wrong during every delay and every backfill.
• Watermark lag unmonitored, so drift is discovered when someone questions a number.

Trade-offs

The core trade is completeness against latency, and it cannot be optimised away — only placed. A generous watermark produces correct results too late to act on; a tight one produces prompt results that are wrong for a measurable fraction of events.

Allowed lateness partially escapes the trade at the cost of state and of emitting corrections, which pushes complexity onto consumers: every downstream system must now handle a revised value, and many are not built to.

The trade is promptness against correctness, with state size as the price of having both partially. The right setting is determined by what the consumer does with the number — an operational dashboard should prefer promptness with corrections; a regulatory figure should wait or publish explicitly as provisional — and stating that consequence in the consumer's terms is what turns a configuration value into a decision.

Interview question

"Our hourly aggregates are missing about 2% of events and finance has noticed. Tell me what you would measure before changing anything, what you would change, and how you would decide between waiting longer and publishing a figure that gets corrected."