Completeness SLO

Streaming pipelines are routinely measured on latency and on consumer lag, because both are easy to obtain. Whether every event arrived is measured rarely, and it is the failure that costs most.

A silently dropped batch produces a chart that looks fine and is wrong. Latency is normal, lag is zero, no error was raised, and the aggregate is short by an amount nobody can see.

Why it matters

Every other streaming metric measures the health of the events that made it. Completeness is the only one that measures the events that did not, and loss is precisely the failure mode that produces no signal.

It is also the metric that catches a class of bugs nothing else does: a filter that is too aggressive, a partition that stopped being consumed, a serialisation failure quietly discarding records, a quarantine path nobody monitors.

Implementation patterns

• A source-side count to compare against, per interval and per partition. Without an independent count there is nothing to measure completeness against, which is why it is skipped.
• Compare per partition rather than in aggregate, since skew means one partition can be losing everything while the total looks approximately right.
• Reconcile against the authoritative source periodically, which catches loss that in-flight counting misses.
• Alert on the quarantine and dead-letter rate, since those are the deliberate loss paths and a steady non-zero rate is an unfixed bug being absorbed.
• Distinguish deliberate filtering from loss in the accounting, or a filter looks like a defect and a defect looks like a filter.
• Set the target per data class. A price feed and a settlement stream have completely different requirements, and one target across both is either unaffordable or meaningless.

Industry example

Financial and trading platforms such as CoinDCX and Zerodha have data classes where completeness is not negotiable — a missed trade or settlement record is a reconciliation break — alongside classes where sampling is acceptable. Applying one standard to both is what makes completeness measurement look unaffordable, and the per-class target is what makes it practical.

Failure scenarios

• Completeness never measured, so silent loss is discovered by a customer or a reconciliation.
• Aggregate comparison only, hiding a single failing partition.
• No source-side count, leaving nothing to compare against.
• Quarantine and dead-letter rates unalerted, so deliberate loss paths absorb an unfixed bug indefinitely.
• One target across all data classes, making the measurement either unaffordable or useless.

Trade-offs

Producing an independent source-side count is real work and, for some sources, is not possible at all — a third-party feed that does not report what it sent leaves you with no reference.

Where the count is unavailable, reconciliation against the authoritative state is the substitute: compare the derived result against the source of truth periodically. It is slower and less precise, and it is the only mechanism that detects the loss at all — which makes it the difference between finding the problem yourself and having a customer find it.

Interview question

"Your pipeline's latency and lag are both healthy and a downstream report is missing 3% of yesterday's records. What would have detected that, and why did you probably not have it?"