A daily reconciliation shows the streaming aggregation is 2% below the batch equivalent. Both read the same source. Why?
Show the full answer Hide the answer
What the interviewer is testing
Whether you understand that late data is dropped by default and that nobody usually measures how much.
The cause
Late-arriving events beyond the allowed lateness. The streaming job closes each window when the watermark passes its end, plus an allowance. Events arriving after that are dropped — silently, in most default configurations.
The batch job reads the same source hours later, when all events have arrived, and includes them. So the 2% is the tail of the lateness distribution the streaming job cut off.
Sources of lateness in practice: mobile devices that were offline, retries from an upstream system after an outage, a partition whose consumer lagged, and clock skew on event producers.
What to do
First, measure it. Route late events to a side output and count them, bucketed by how late they were. This turns an unknown loss into a distribution, and the distribution tells you what the allowance should be. Most teams have never looked, which is why the 2% was discovered by reconciliation rather than by monitoring.
Then choose deliberately, because there is no setting that avoids the trade:
| Option | Effect |
|---|---|
| Increase allowed lateness | Higher completeness, higher latency, more state retained |
| Keep the allowance, emit corrections | Full completeness, downstream must handle retractions |
| Keep the allowance, reconcile in batch | Streaming for speed, batch as the corrected system of record |
| Accept the loss | Legitimate if 2% does not affect the decision — but say so explicitly |
What a strong answer adds
Noting that the right answer depends on the consumer. An operational dashboard driving an immediate decision can accept 2%; a financial or regulatory figure cannot, and for those the streaming output should be explicitly labelled provisional with a batch-corrected figure as the authoritative one.
That hybrid — fast provisional, slow authoritative — is a common and defensible pattern, and it should be a stated design decision rather than an accident.
Common weak answers
Increasing the watermark allowance without measuring the distribution. Concluding the batch job is wrong.