A streaming pipeline sends failed messages to a dead-letter queue. What makes that a real control rather than a data-loss mechanism?

What makes it real

• A named owner and an alert on depth. A dead-letter queue nobody monitors is a data-loss mechanism with extra steps, and that is its default state.
• Enough context to diagnose: the original message, the error, the consumer version, the attempt count and the timestamp. A DLQ containing only the payload requires reproducing the failure to understand it, which nobody does.
• A replay path that reprocesses a corrected message or a batch of them, since the usual resolution is a code fix followed by a replay — and without the path, the fix does not recover the data.
• Categorisation. A malformed message, a transient downstream failure and a genuine bug need different responses, and a single queue mixing them is triaged by nobody.
• A retention policy and a decision about what happens at the end of it, since a DLQ that silently expires messages has lost them.

Distinguishing the failure types before the DLQ

• Transient downstream failure: retry with backoff. Should not reach the DLQ unless the retry budget is exhausted.
• A poison message that fails deterministically: retrying is pointless and it blocks the partition if processed in order. This is the case the DLQ exists for, and the signature is repeated redelivery of one offset while other partitions progress.
• A consumer bug: everything fails. Filling the DLQ with the entire stream is the wrong response — the consumer should stop, be fixed, and resume from its offset, and a circuit breaker on the DLQ rate is what prevents the flood.

That last distinction is the one most implementations miss, and it turns a bug into a large-scale reprocessing exercise.

The multi-destination complication

When one event fans out to many destinations, each needs its own dead-letter path and its own owner, because a failure at one destination says nothing about the others — and a shared DLQ makes the per-destination resolution impossible.

The measure

Dead-letter rate by category, per destination, with a target of approximately zero for the poison category. A steady non-zero rate is an unfixed bug that the DLQ is quietly absorbing — which is the state it exists to make visible and frequently ends up concealing.