Watermark Lag
How far behind the newest event a watermark is held, which trades result latency against how much late data is included.
A watermark is the pipeline's assertion that no event older than time T will arrive. It is a heuristic, usually derived as the maximum event time seen minus a fixed allowance, and that allowance is the lag.
Set it small and windows close quickly, so results are fresh and more records arrive too late to be counted. Set it large and results are late but more complete. There is no setting that gives both, and there is no setting that is correct — a watermark is a bet about the tail of the arrival delay distribution.
The way to choose is to measure that distribution rather than guess. Record the difference between event time and processing time across a representative period and pick a percentile: a lag at p99 admits nearly everything at the cost of that much latency.
What must accompany it is a policy for what arrives after the watermark passed. Three options: drop it and count the drops, so the loss is visible; emit an update that revises the earlier result, which requires downstream consumers to handle revisions; or route it to a side output for separate handling. Choosing none of these means silently dropping late data, which is the default in several frameworks and the reason streaming totals quietly fail to match batch ones.