Streaming & Real-Time Data
General material on continuous processing of unbounded data.
4 to work through
-
advanced
A delivery platform assigns each order to the nearest available driver as soon as the order is ready. Critique this.
2 min answer -
advanced
A live platform computes real-time metrics during events where concurrency changes by an order of magnitude within minutes. What must the streaming architecture provide?
2 min answer -
advanced
Kafka replaced ZooKeeper with KRaft and added tiered storage. What problems did each address, and how do they change how a platform is operated and sized?
2 min answer -
advanced
You are the architect for a platform where 30 teams both publish and consume events, and leadership asks you to set the company standard for exposing real-time data between teams. What do you standardise, what do you deliberately leave to teams, and how would you know the standard is working?
3 min answer
4 terms in this topic
Backpressure in Streaming
A slow downstream stage signalling upstream to slow down, so that queues stay bounded instead of consuming memory until the job dies.
practiceCatch-Up Throttling
Deliberately limiting how fast a recovered consumer drains its backlog, so that recovery does not become a second and larger incident.
case-studyDoorDash Dispatch: Matching as an Optimisation Problem
Assigning deliveries greedily to the nearest driver is locally sensible and globally poor, so the assignment is batched and solved as an optimisation.
conceptUnbounded Dataset
Input with no known end, which removes the option of waiting for completeness and forces every aggregate to be provisional.
Neighbouring topics
Streaming vs Batch
The freshness requirement that actually justifies streaming, and the cost of assuming one.
Exactly-Once Semantics
What the phrase really means, where it holds, and the idempotent sink underneath it.
Stream Processing Frameworks
Flink, Kafka Streams, Spark Structured Streaming — state, checkpointing and recovery.
Windowing
Tumbling, sliding and session windows, and the aggregation each one answers.
Watermarks & Late Data
Deciding a window is complete when events can still arrive, and what to do when they do.
Stateful Stream Processing
Keyed state, state backends, checkpoint size, and the restore time that follows.
Stream-Table Duality
A changelog and a table as two views of the same thing, and materialising between them.
Kappa vs Lambda
One pipeline replayed versus two pipelines reconciled, and the maintenance each carries.
Streaming Schema Evolution
Changing an event's shape while a retained log still holds every older version of it.
Streaming Joins
Joining two unbounded streams, the buffering it needs, and the enrichment alternative.
Backfill & Reprocessing
Replaying history through changed logic without double-counting the live output.
CDC to Stream
Turning database changes into an event log, and how that differs from a domain event.
Real-Time Serving Layer
Where a low-latency read of a streaming aggregate actually lands.
Feature Freshness
How stale a feature can be before the model degrades, and the pipeline that follows.
Streaming SLOs
End-to-end latency, consumer lag and completeness as commitments rather than dashboards.
Partition Keys & Ordering
Ordering guaranteed only within a partition, and choosing the key that makes that enough.
Dead Letter Handling
The poison message that blocks a partition, and the queue nobody reads.
Streaming Cost
Always-on compute, retention and cross-zone traffic as the three bills that surprise.
Real-Time Analytical Stores
Druid, Pinot and ClickHouse — ingest-and-query engines for sub-second aggregation.