An event pipeline partitions by event date. Measured over a quarter, 96% of a day's events have landed one hour after midnight, 99.3% after six hours and 99.9% after three days - mobile clients retry for days on bad networks. The team adds a completeness check and now publishes a partition only once it passes 99%. What has the business bought, what has it paid, and when does the bill arrive?

What is gained

Published numbers stop moving. Before the check, yesterday's figure read at 08:00 was about 4% low and the same figure read at lunchtime was higher, which teaches people that the warehouse is unreliable - the most expensive thing a data platform can teach. Publishing at a measured completeness threshold converts an invisible error into a stated delay. That is a real gain and it is why the check is right.

What is paid

Six hours of freshness on every daily number, every day, to protect against a 0.7% gap. The cost chains: downstream marts wait, the daily email moves to afternoon, an hourly operational view either stays on the incomplete partition - two sources of truth again - or goes away.

The subtler cost is in the composition of the missing data. The late 0.7% is not a random sample. It is concentrated in one client version, one network, one geography - the users whose events retry. So the early read was not merely 4% low; it was low in a way that moved the country mix and any per-platform metric. A completeness percentage tells you the size of the gap and nothing about its shape, which means the threshold protects the total and still misleads anyone slicing by the dimension that correlates with lateness.

When the bill arrives

Three moments, reliably:

1. The first month-end close where finance needs the figure before the hold-back expires, and somebody publishes manually. Now there is a documented override and the control is advisory.
2. An upstream incident that delays arrivals. Completeness never reaches 99%, the partition is never published, and the dashboard shows nothing - a blank report is read as an outage, and the team gets paged for working correctly.
3. A client release that changes retry behaviour and moves the distribution. The threshold was calibrated on last quarter's curve, and nothing re-derives it.

How to keep the option to reverse

Publish early and carry the fact. Two objects, one definition: a fast view with a completeness column on every row group and a documented restatement rule, and a stable view that only exposes partitions past the threshold. Consumers choose, and the choice is recorded. Alert on the completeness curve shifting rather than on the threshold being missed, because the curve moving is the thing that invalidates every number downstream.

The decision rule

Hold back when the consumer cannot restate. Payouts, regulatory filings, anything emailed to a customer or committed externally - publish once, late. Publish early with a completeness figure when the consumer is a dashboard a human reads, because a human can be told the number is 96% complete and will still make a better decision than with no number at all. The rule flips the moment a dashboard becomes the input to an automated action.

When this is the wrong answer

If 99.9% of events land within one hour - common for server-side events with no mobile retry path - the whole mechanism is unnecessary and a simple freshness check is enough. Measure your own arrival curve before buying any of this; the shape of that curve, not the existence of late data, is what justifies a hold-back window at all.