Watermarks and Allowed Lateness
The heuristic that lets a streaming system decide a window is complete, why it is always wrong in one of two directions, and how triggers and lateness policies turn a single answer into a sequence of refinements.
A watermark is an assertion the system makes to itself: "I believe I have seen all events with event time before \(T\)." It is the mechanism that converts an unbounded stream into finite, closeable windows, and it is a heuristic. Every watermark is either too early, dropping data that had not arrived yet, or too late, delaying every result unnecessarily. The design work is choosing which error to make and how to compensate.
How a watermark is produced
The common implementation is a bounded-out-of-orderness estimate: track the maximum event time seen, subtract a fixed delay, and emit that as the watermark. A delay of five minutes asserts that events arrive at most five minutes out of order, which is a guess informed by observed lateness distributions.
More sophisticated sources produce watermarks from the data infrastructure itself. A partitioned log where each partition is ordered by event time can emit a per-partition watermark and take the minimum across partitions, which is exact rather than heuristic as long as every partition is progressing.
That last condition is the classic operational failure. The watermark is the minimum across all input partitions, so one idle partition, or one that has stalled, holds the global watermark back and no window ever closes. Systems provide idleness detection to exclude non-advancing sources after a timeout, and forgetting to configure it is a common cause of a pipeline that produces nothing while looking healthy.
Triggers and refinement
Once you accept that the watermark is a guess, the natural response is to stop treating window emission as a single event. The Dataflow model formalises this with three orthogonal questions (Akidau et al., 2015, The Dataflow Model, VLDB): what results are computed, where in event time they are grouped, and when in processing time they are emitted.
Triggers answer the third. Early triggers emit speculative results before the watermark passes, giving low-latency partial answers. The on-time trigger fires at the watermark. Late triggers fire again when data arrives after the watermark, updating the result.
Accumulation mode determines what a later firing means: discarding mode emits only the new contribution, so a consumer must sum the firings; accumulating mode emits the full corrected value, so a consumer replaces the previous one. Getting this wrong between producer and consumer produces double counting that is very hard to diagnose because each component is behaving as documented.
Allowed lateness and state
Allowed lateness is how long after the watermark passes a window's state is retained so late events can update it. It is directly a memory decision: state for every open window in the lateness horizon must be held, so a day of allowed lateness on minute windows means 1,440 windows of state per key.
Beyond the horizon, state is dropped and later events go to a dead-letter path or are discarded. Counting what falls there is one of the more valuable pipeline metrics, since a rise in it is the earliest signal that upstream lateness has changed.
When it breaks
A watermark that is too aggressive silently loses data. Events arriving after the window closed are dropped, and unless dropped-record counts are monitored, the loss appears only as numbers that are quietly low. This is the failure mode most likely to survive to production undetected.
A watermark that is too conservative delays everything. A one-hour delay to accommodate a rare mobile straggler adds an hour of latency to every result, including the 99.9 percent of events that arrived promptly. Speculative triggers exist precisely so that this choice does not have to be made once for all consumers.
Late data forces downstream idempotency. A window that fires three times sends three results. Every consumer must handle updates to a value it already stored, which means keyed upserts rather than appends, and a sink without that property makes correct late handling impossible regardless of what the pipeline does.
Watermarks do not survive arbitrary transformations. Operations that change event times, such as joining with a delayed stream or assigning derived timestamps, invalidate the upstream watermark's meaning. Frameworks require an explicit new watermark assignment at those points, and doing it wrong produces windows that close before their inputs can possibly have arrived.
14 flashcards for this concept
Click a card to reveal the answer.