Dead Letter Handling
The poison message that blocks a partition, and the queue nobody reads.
4 to work through
-
beginner Multiple choice
A platform team sets one default for every stream consumer: retry a failing record three times then route it to a dead-letter topic and carry on. The ledger team's consumer derives account balances from ordered per-account events. What trade-off does that default make on their behalf and what should they run instead?
3 min answer -
intermediate
A streaming pipeline sends failed messages to a dead-letter queue. What makes that a real control rather than a data-loss mechanism?
2 min answer -
intermediate
How should unprocessable messages be handled, and what makes dead-letter handling useful rather than a graveyard?
2 min answer -
intermediate
You find 40,000 messages in a dead letter queue, the oldest from seven months ago. Nobody knew. What do you fix?
2 min answer
3 terms in this topic
Dead Letter Queue
A separate destination for messages that cannot be processed successfully, so one bad message does not halt the stream behind it.
metricDead-Letter Budget
A declared ceiling on the share and the age of records a pipeline may divert before it counts as failing rather than coping - with an owner and a def…
conceptPoison Message
A record that a consumer cannot process and cannot skip, which halts its partition entirely until someone intervenes.
Neighbouring topics
Streaming & Real-Time Data
General material on continuous processing of unbounded data.
Streaming vs Batch
The freshness requirement that actually justifies streaming, and the cost of assuming one.
Exactly-Once Semantics
What the phrase really means, where it holds, and the idempotent sink underneath it.
Stream Processing Frameworks
Flink, Kafka Streams, Spark Structured Streaming — state, checkpointing and recovery.
Windowing
Tumbling, sliding and session windows, and the aggregation each one answers.
Watermarks & Late Data
Deciding a window is complete when events can still arrive, and what to do when they do.
Stateful Stream Processing
Keyed state, state backends, checkpoint size, and the restore time that follows.
Stream-Table Duality
A changelog and a table as two views of the same thing, and materialising between them.
Kappa vs Lambda
One pipeline replayed versus two pipelines reconciled, and the maintenance each carries.
Streaming Schema Evolution
Changing an event's shape while a retained log still holds every older version of it.
Streaming Joins
Joining two unbounded streams, the buffering it needs, and the enrichment alternative.
Backfill & Reprocessing
Replaying history through changed logic without double-counting the live output.
CDC to Stream
Turning database changes into an event log, and how that differs from a domain event.
Real-Time Serving Layer
Where a low-latency read of a streaming aggregate actually lands.
Feature Freshness
How stale a feature can be before the model degrades, and the pipeline that follows.
Streaming SLOs
End-to-end latency, consumer lag and completeness as commitments rather than dashboards.
Partition Keys & Ordering
Ordering guaranteed only within a partition, and choosing the key that makes that enough.
Streaming Cost
Always-on compute, retention and cross-zone traffic as the three bills that surprise.
Real-Time Analytical Stores
Druid, Pinot and ClickHouse — ingest-and-query engines for sub-second aggregation.