What SLOs should a streaming pipeline publish, and why is availability the wrong headline metric?
Show the full answer Hide the answer
Why availability is the wrong headline
A streaming pipeline that is up and forty minutes behind is worse than one that is briefly down and caught up. Availability answers "is the job running", which passes every liveness check while the output is increasingly wrong.
The properties consumers actually depend on are freshness and completeness, and neither is captured by uptime.
The SLOs to publish
1. End-to-end lag — from event occurrence to availability in the serving layer, at a percentile. This is the headline. It is what consumers plan around and what degrades first.
2. Completeness — the proportion of expected events processed for a period, which detects silent loss that lag does not.
3. Correctness where measurable — reconciliation against a batch recomputation, sampled. This catches the processor producing plausible wrong results, which no operational metric shows.
4. Availability of the serving layer, separately from the processing pipeline — because the decoupling means they fail independently and consumers care about them differently.
5. Recovery time after a failure, including catch-up, since a pipeline that recovers in ten minutes and takes two hours to catch up has a two-hour freshness incident.
What consumers need alongside
An as-of timestamp and a completeness indicator on every output. A consumer must be able to tell whether they are reading current, lagging or incomplete data — otherwise they will assume current, and a stale value presented as fresh is worse than a gap.
The alerting that works
Alert on lag and on completeness, not on job health. And alert on the rate of change of lag as well as its level, because rising lag with a healthy job is the signature of a pipeline that will fall over hours from now — which is far more actionable than the alert that fires when it already has.
The recovery property that must be designed
Catch-up must be rate-limited. Processing at many times the normal rate to catch up can overwhelm the downstream store and cause a second incident at the moment of recovery — which is a failure mode specific to streaming and frequently discovered the hard way.