Event Streaming
Retained ordered logs, consumer offsets, partitions and replay.
6 to work through
-
intermediate
A downstream consumer has been failing for four days. Your event log retains three days. What happens?
2 min answer -
intermediate Multiple choice
A platform processes billions of events per day and requires ordering per customer but not globally. How should partition keys and consumer groups deliver that without creating a serialisation bottleneck?
2 min answer -
intermediate Multiple choice
You have 14 systems exchanging data through bespoke point-to-point pipelines. Adding a system means building integrations to six others. Propose an alternative.
2 min answer -
advanced
A ride-hailing platform must process billions of location and trip events per day, with per-trip ordering. Design the partitioning, consumer scaling and failure handling.
3 min answer -
advanced Multiple choice
Consumer lag on a Kafka topic grows during peak and does not recover overnight. You add consumers and nothing improves. Why?
2 min answer -
advanced
Design the event ingestion path for a recommendation platform receiving billions of interaction events daily, where the same events must feed real-time personalisation and offline model training. How should the pipeline be structured?
3 min answer
4 terms in this topic
Consumer Group
A set of consumers that cooperatively read one stream, with each partition assigned to exactly one member, so the group collectively processes every …
patternEvent Streaming
A durable ordered log of facts that many independent consumers read at their own pace, retaining events after consumption rather than deleting them.
case-studyLinkedIn: Kafka and the Unified Log
LinkedIn replaced a tangle of point-to-point data pipelines with a single durable log, turning an O(n²) integration problem into an O(n) one.
conceptOffset Management
How a consumer records its position in a stream, and the decision that determines whether processing is at-least-once or at-most-once.
Neighbouring topics
Distributed Systems
General material on partial failure, coordination and distributed reasoning.
CAP & PACELC
What you must give up during a partition, and the latency choice the rest of the time.
Consistency Models
Linearizable, sequential, causal, eventual, and the session guarantees between them.
Idempotency
Making an operation safe to repeat, because a client that times out cannot know.
Retries & Backoff
Exponential backoff, jitter, retry budgets, and how retries become the outage.
Timeouts & Deadlines
Per-hop timeouts that do not compose, and the deadline budget that replaces them.
Circuit Breakers
Failing fast on a broken dependency, and what you fail fast to.
Backpressure & Flow Control
Telling callers to slow down instead of buffering into congestion collapse.
Load Shedding
Rejecting some work deliberately so the rest can be served correctly.
Bulkheads & Isolation
Partitioning resources so one dependency cannot starve the others.
Leader Election
Agreeing who is in charge, and fencing the one who no longer is.
Consensus Protocols
Raft, Paxos and quorums — what they guarantee and what they cost.
Distributed Locking
Mutual exclusion across machines, and why it is harder than it looks.
Distributed Transactions
Two-phase commit, its blocking failure mode, and when it is still reasonable.
Sagas & Compensation
Replacing atomicity with semantic undo, and ordering the irreversible steps last.
Service Discovery
Finding a healthy address for something whose instances are ephemeral.
Messaging & Queues
Decoupling producer from consumer, and the semantics that come with it.
Clocks & Ordering
Why wall clocks lie, and how logical clocks and versions restore order.
Failure Modes
Slow rather than down, partial, grey, and failing while reporting success.