How should unprocessable messages be handled, and what makes dead-letter handling useful rather than a graveyard?

Why it is necessary

A message that consistently fails will otherwise block or consume workers forever. In an ordered partition it blocks everything behind it — head-of-line blocking with an unbounded duration. In a competing-consumer queue it cycles through workers, consuming capacity that looks like healthy utilisation.

Without a retry ceiling and a dead-letter path, one poisonous message silently reduces effective capacity or halts a partition.

What makes it useful rather than a graveyard

1. A retry ceiling with backoff, distinguishing transient failures (retry) from permanent ones (dead-letter) where possible. Sending a transient failure straight to the dead letter loses data unnecessarily.

2. The failure reason recorded with the message, plus the attempt history. A dead letter with no context is unactionable, and unactionable messages accumulate.

3. Alerting on the rate, not just the existence. A steady trickle is a data-quality problem; a sudden spike is usually a deployment or a schema change, and the two need different responses.

4. A replay path, so a corrected consumer can reprocess the accumulated messages. This is what makes the dead letter a recovery mechanism rather than a bin — without it, the messages are lost with extra steps.

5. An owner and a review cadence. An unmonitored dead-letter queue is a place where data goes to be forgotten, and its depth is a leading indicator nobody watches.

6. Retention long enough to cover discovery and correction, since a schema problem found a week later needs the week's messages still present.

The categories worth separating

• Malformed messages — a producer problem, requiring the producer to be told.
• Schema incompatibility — usually a deployment problem, and usually correctable by replay.
• Business rule rejections — arguably not failures at all, and better routed to a normal outcome path than to a dead letter.
• Downstream unavailability — a transient failure that should be retried far longer before dead-lettering, since dead-lettering it converts an outage into data loss.

Conflating these means the queue contains problems with different owners and different remedies, which is why nobody processes it.