Completeness Indicator

A streaming output with no metadata is read as current and complete. That assumption is wrong most of the time and catastrophically wrong occasionally, and the consumer has no way to detect the difference.

A completeness indicator attaches two things to every result: an as-of timestamp (the event time up to which this result reflects reality) and a completeness measure (what proportion of expected input has been processed for that period).

Why availability is the wrong headline SLO

A pipeline that is running and forty minutes behind is worse than one that was briefly down and has caught up. Availability answers "is the job running", which passes every liveness check while the output grows steadily more wrong. What consumers actually depend on is freshness and completeness.

So the published SLIs should be:

• End-to-end lag at a percentile, as the headline.
• Completeness — the proportion of expected events processed — which catches silent loss that lag alone never reveals. A pipeline dropping 3% of events has excellent lag.
• Correctness, where measurable, via sampled reconciliation against a batch recomputation.
• Serving-layer availability separately, because the serving layer and the pipeline fail independently and a stale-but-available store is usually the better failure mode.
• Recovery time including catch-up. A ten-minute recovery with a two-hour catch-up is a two-hour freshness incident, and reporting the ten minutes is misreporting.

Implementation patterns

• The indicator carried on the data, not published on a separate dashboard. Consumers act on values, so the metadata must travel with the value.
• A defined behaviour when a value is stale or missing, chosen per consumer — fall back to a default, use a batch value, or refuse. The decision belongs to the consumer, which is only possible if they can see the staleness.
• Alert on the rate of change of lag, not only on its level. Lag rising steadily on a healthy job predicts a breach hours ahead, which is the difference between a planned intervention and an incident.
• Catch-up rate-limited, so recovery does not overwhelm downstream systems and cause a second incident.
• Both fast and settled results published, where a corrected batch path exists, each labelled — and their divergence treated as expected rather than as a defect.

Industry example

Real-time metrics platforms and live-event dashboards converge on the same design: approximate low-latency aggregates for anything perceptual, exact higher-latency processing for anything with consequences, and an explicit marker distinguishing them. Nobody perceives the difference between 812,000 and 814,000 concurrent viewers; the difference between an approximate and a settled revenue figure matters a great deal, and the only thing making that distinction safe is the label.

The pattern generalises: the two paths will diverge, and the architecture should make the divergence legible rather than treat it as a bug to be eliminated.

Failure scenarios

• Availability as the only SLO, so a lagging pipeline reports healthy.
• No completeness measure, so silent event loss is invisible.
• Metadata on a dashboard rather than on the data, so automated consumers cannot use it.
• Recovery time excluding catch-up, which understates every incident.
• Unlabelled fast and settled numbers, which produces a recurring argument about which system is wrong.

Trade-offs

Carrying metadata adds payload size and forces consumers to handle a case they would rather ignore — and many will ignore it anyway unless the client library makes staleness handling the default path.

The cost is small and the alternative is that every consumer silently assumes currency. The design question worth asking early is not whether to publish the indicator but what a consumer should do when it is bad, because an indicator nobody acts on is decoration.

Interview question

"Your streaming pipeline reports 99.95% availability and the business says the numbers are wrong. Both are true. What are you not measuring, and what would you publish instead?"