Streaming SLOs
End-to-end latency, consumer lag and completeness as commitments rather than dashboards.
5 to work through
-
intermediate
Consumer lag has grown from 30 seconds to 4 minutes over a week. Throughput and error rates are normal. What is happening and how urgent is it?
2 min answer -
intermediate
What SLOs should a streaming pipeline publish, and why is availability the wrong headline metric?
2 min answer -
intermediate
You are handing a streaming pipeline to an operations team who have never run one. What must exist before they accept it?
2 min answer -
advanced
Tencent's DAGOR (SoCC 2018) detects overload in WeChat's microservice fleet from the average queuing time in a server's pending queue rather than from CPU, treats 20 ms as the overload threshold, and piggybacks the server's current admission level onto its responses so callers shed doomed work themselves. A streaming ingest tier has the same shape. What transfers, and where would copying it be a mistake?
3 min answer -
advanced
What SLOs should a market-data streaming pipeline have, and what does each protect?
2 min answer
4 terms in this topic
Completeness Indicator
Metadata published alongside a streaming result stating how current it is and what proportion of expected input it reflects - so consumers can reason…
metricCompleteness SLO
The proportion of produced events that actually reached the destination - the streaming SLO that is measured least often and whose absence conceals t…
metricConsumer Lag
The gap between the latest offset written to a partition and the offset a consumer group has processed, expressed in records or in time.
conceptIdle Source Stall
A stream processor whose event-time progress halts because one partition or source has stopped producing - so output ceases entirely while every comp…
Neighbouring topics
Streaming & Real-Time Data
General material on continuous processing of unbounded data.
Streaming vs Batch
The freshness requirement that actually justifies streaming, and the cost of assuming one.
Exactly-Once Semantics
What the phrase really means, where it holds, and the idempotent sink underneath it.
Stream Processing Frameworks
Flink, Kafka Streams, Spark Structured Streaming — state, checkpointing and recovery.
Windowing
Tumbling, sliding and session windows, and the aggregation each one answers.
Watermarks & Late Data
Deciding a window is complete when events can still arrive, and what to do when they do.
Stateful Stream Processing
Keyed state, state backends, checkpoint size, and the restore time that follows.
Stream-Table Duality
A changelog and a table as two views of the same thing, and materialising between them.
Kappa vs Lambda
One pipeline replayed versus two pipelines reconciled, and the maintenance each carries.
Streaming Schema Evolution
Changing an event's shape while a retained log still holds every older version of it.
Streaming Joins
Joining two unbounded streams, the buffering it needs, and the enrichment alternative.
Backfill & Reprocessing
Replaying history through changed logic without double-counting the live output.
CDC to Stream
Turning database changes into an event log, and how that differs from a domain event.
Real-Time Serving Layer
Where a low-latency read of a streaming aggregate actually lands.
Feature Freshness
How stale a feature can be before the model degrades, and the pipeline that follows.
Partition Keys & Ordering
Ordering guaranteed only within a partition, and choosing the key that makes that enough.
Dead Letter Handling
The poison message that blocks a partition, and the queue nobody reads.
Streaming Cost
Always-on compute, retention and cross-zone traffic as the three bills that surprise.
Real-Time Analytical Stores
Druid, Pinot and ClickHouse — ingest-and-query engines for sub-second aggregation.