Provisional Result

Aggregating by event time means a window is never definitively complete: a device that was offline uploads twenty minutes of history, a client retries after a network failure, a lagging partition delivers late.

A pipeline that emits a result before the watermark passes is emitting a provisional value, and correcting it later is the right behaviour.

Undocumented, it is indistinguishable from a bug. A dashboard value that changes without explanation destroys trust in the whole platform, and the loss extends to the numbers that were never provisional.

Why it matters

The alternative — waiting for certainty — makes every result slow, which is unacceptable for an operational view. So the correct design emits provisionally and corrects, and the entire question is whether consumers know.

Implementation patterns

• Mark windows as provisional until the watermark passes, and expose that state in the API and the interface.
• Show when a value was restated, and by how much, so a user who noticed the change has an explanation.
• Separate the fast provisional path from the settled path. The live view reads provisional; the reporting layer reads aggregates computed after the watermark. One computation serving both means either the live view is slow or the reports are wrong.
• Decide explicitly what happens to events arriving after the watermark: dropped, counted separately, or triggering a recomputation. All three are defensible; leaving it undefined is not, and the default is silent dropping.
• Verify that downstream consumers tolerate restatement, since most are built assuming a value is final — an alert that fires on a provisional value and would not have fired on the settled one is a false positive the consumer cannot explain.
• Store both event time and processing time, since conflating them causes most time-related bugs in stream processing.

Industry example

Dispatch and delivery platforms such as Swiggy and Porter compute per-minute operational aggregates from devices that are intermittently connected, which makes late arrival routine rather than exceptional. The operational dashboard needs the provisional number and the business report needs the settled one, and the separation is what allows both to be correct for their purpose.

Failure scenarios

• Provisional values presented as final, so corrections look like bugs.
• One computation serving both live and reporting, forcing a bad trade.
• Late-arrival policy undefined, defaulting to silent dropping.
• Downstream alerts on provisional values, producing unexplainable false positives.
• Event time and processing time conflated, producing results that are wrong in a way nobody can localise.

Trade-offs

Exposing provisionality complicates the API and the interface, and it requires consumers to handle a state they would rather not think about. Some consumers genuinely only want the settled value and find the distinction noise.

The resolution is to serve them the settled path and reserve the provisional one for the operational consumers who need speed — which is the same separation, expressed as two endpoints rather than as a flag on one.

Interview question

"Your operations dashboard shows 412 deliveries in the last minute, and an hour later the same minute shows 447. Explain what happened, and tell me what you would change so nobody has to ask."