Search the practice set
275 questions, 991 terms and 600 topics in 30 areas.
57 results for “Streaming SLOs”
Consumer Lag
How far behind the newest record a consumer is, expressed in time rather than in message count, and its rate of change.
gRPC Streaming
Four call patterns — unary, server streaming, client streaming and bidirectional — built on HTTP/2 streams.
CDC Initial Snapshot
The consistent full copy taken when a CDC pipeline starts, before streaming begins — and the step that determines whether the target is correct.
Change Data Capture
Publishing a stream of a database's row-level changes by reading its replication log, without modifying the application that owns it.
Consumer Group
A set of consumers that cooperatively read one stream, with each partition assigned to exactly one member, so the group collectively processes every message once.
Event Stream
An append-only, retained log of events that many independent consumers read at their own position, and can re-read.
Freshness Requirement
How stale data may be before the decision it supports degrades — the only question that justifies streaming over batch.
Internal Consumer SLO
A reliability commitment made to teams who cannot switch supplier, which is why it must be measured from their side rather than from the platform's.
LinkedIn and the Origin of Kafka
Kafka was built to replace point-to-point data integration between many systems with a single durable log that any system could publish to and any number could read.
Log Compaction
A retention policy that keeps only the most recent value for each key rather than deleting by age, so the log becomes a durable snapshot of current state.
Log-Based CDC
Capturing changes by reading the database's own write-ahead log, which sees every change with no load on the source and no application involvement.
Offset Management
How a consumer records its position in a stream, and the decision that determines whether processing is at-least-once or at-most-once.
Proxy Buffering
Whether a reverse proxy accumulates a response before forwarding it, which protects the backend from slow clients but breaks streaming.
Reactive Streams
A specification for asynchronous stream processing in which the consumer requests a specific number of items, making backpressure part of the protocol rather than an afterthought.
Retention Cost
The storage bill for keeping a log replayable, which is set by retention multiplied by throughput multiplied by the replication factor.
Schema Compatibility Mode
The rule stating which schema changes a registry will accept, which encodes whether producers or consumers are expected to upgrade first.
Server-Sent Events
A one-way streaming protocol over plain HTTP in which the server pushes text events to the client on a long-lived response.
Stream Enrichment
Attaching reference data to a stream by lookup against a materialised table rather than by joining two unbounded streams.
Stream-Table Duality
The equivalence between a stream of changes and a table of current state — each can be derived from the other.
Unbounded Dataset
Input with no known end, which removes the option of waiting for completeness and forces every aggregate to be provisional.
Watermark
A moving assertion that no events older than a given event-time will arrive, which is what allows an event-time window to be closed and emitted.
Windowing
Grouping an unbounded stream into finite chunks so aggregation can produce results, defined over event time rather than arrival time.
gRPC
A contract-first RPC framework using Protocol Buffers over HTTP/2, with generated clients and servers and first-class streaming.
You are handing a streaming pipeline to an operations team who have never run one. What must exist before they accept it?
The signals they will be paged on Consumer lag in time , not in messages — ten thousand messages means nothing without the rate. And alert on the derivative : g
A streaming aggregation reports lower totals than the batch job it replaced. Both read the same source. What is likely happening?
The likely cause: late events dropped past the watermark The batch job reads a completed day and sees everything, including records that arrived hours after the
A vendor claims their streaming platform provides exactly-once processing. How do you evaluate the claim?
Ask where the guarantee ends Nearly always at the platform's boundary. Within it, state and offsets commit together, so internal state reflects each input once.
Your streaming aggregate reports 2% lower daily revenue than the batch reconciliation. Both are "correct". Explain what is happening and how you resolve it.
The likely cause: silently dropped late data The streaming job windows by event time and closes each window when the watermark passes. Records arriving after th
A CDC pipeline feeding your warehouse falls three hours behind during a source system's batch job, and the source's transaction log retention is 24 hours. What is the risk and what do you change?
The immediate risk Lag consumes the retention window. At three hours behind against a 24 hour retention, you have 21 hours of margin. If the consumer stops enti
A business sponsor asks for a real-time data platform because "the competition has one". Reporting is currently a nightly batch that lands at 06:00 and nobody has complained. How do you handle this?
Do not answer the technology question "Real time platform" is a solution, and it has arrived without a problem attached. Answering it directly leads either to a
Application teams say the platform is unreliable. The platform team's dashboard shows 99.95% on every component. How do you resolve this?
Both are right, and that is the finding The platform measured its components. Consumers experience journeys. The gap between those two views is where trust erod
Consumer lag on a Kafka topic grows during peak and does not recover overnight. You add consumers and nothing improves. Why?
The mechanism Each partition is assigned to exactly one consumer within a group. With ten partitions and ten consumers, an eleventh consumer joins the group, is
Streaming SLOs
End-to-end latency, consumer lag and completeness as commitments rather than dashboards.
Event Streaming
Retained ordered logs, consumer offsets, partitions and replay.
Platform SLOs
Committing to reliability for internal consumers who cannot choose another provider.
Streaming & Real-Time Data
General material on continuous processing of unbounded data.
Streaming Cost
Always-on compute, retention and cross-zone traffic as the three bills that surprise.
Streaming Data
Windowing, watermarks, late arrivals and exactly-once semantics.
Streaming Joins
Joining two unbounded streams, the buffering it needs, and the enrichment alternative.
Streaming Schema Evolution
Changing an event's shape while a retained log still holds every older version of it.
Streaming vs Batch
The freshness requirement that actually justifies streaming, and the cost of assuming one.
Backfill & Reprocessing
Replaying history through changed logic without double-counting the live output.
CDC to Stream
Turning database changes into an event log, and how that differs from a domain event.
Dead Letter Handling
The poison message that blocks a partition, and the queue nobody reads.
Exactly-Once Semantics
What the phrase really means, where it holds, and the idempotent sink underneath it.
Feature Freshness
How stale a feature can be before the model degrades, and the pipeline that follows.
Kappa vs Lambda
One pipeline replayed versus two pipelines reconciled, and the maintenance each carries.
Partition Keys & Ordering
Ordering guaranteed only within a partition, and choosing the key that makes that enough.
Real-Time Analytical Stores
Druid, Pinot and ClickHouse — ingest-and-query engines for sub-second aggregation.
Real-Time Serving Layer
Where a low-latency read of a streaming aggregate actually lands.
Stateful Stream Processing
Keyed state, state backends, checkpoint size, and the restore time that follows.
Stream Processing Frameworks
Flink, Kafka Streams, Spark Structured Streaming — state, checkpointing and recovery.
Stream-Table Duality
A changelog and a table as two views of the same thing, and materialising between them.
Watermarks & Late Data
Deciding a window is complete when events can still arrive, and what to do when they do.
Windowing
Tumbling, sliding and session windows, and the aggregation each one answers.
gRPC Transport
HTTP/2 multiplexing, binary encoding and streaming semantics.