You find 40,000 messages in a dead letter queue, the oldest from seven months ago. Nobody knew. What do you fix?

What the interviewer is testing

Whether you recognise that a DLQ without operational treatment is a data loss mechanism.

What this actually is

Seven months of business events that were never processed: orders not fulfilled, updates not applied, notifications not sent. The system has been silently incomplete and the downstream data is wrong in ways nobody has correlated.

The DLQ did its job — it prevented a poison message from blocking the partition. Everything after that was missing.

The immediate work

Triage the 40,000 by failure reason. They will cluster into a handful of causes — typically one schema change, one unhandled null, one downstream outage — rather than 40,000 distinct problems.

Assess the business impact per cluster with the owning team. Some are harmless duplicates; some are orders that never shipped, and those need a remediation plan independent of the technical fix.

Fix the defects, then replay in controlled batches, with idempotency verified first — replaying 40,000 messages into a system that is not idempotent creates a second incident.

The permanent fixes

Alert on any arrival. A healthy stream produces zero dead letters, so any message is an alert, not a threshold. This is the single fix that would have prevented seven months.

A named owner for the queue, with the alert routed to them.

Retention that does not silently delete unresolved records, and a size alert well before it.

Replay tooling that exists and has been used, rather than being written under pressure.

Sufficient context on each message: original topic and offset, failure reason, stack trace, consumer version, timestamp. Without it, triage of 40,000 unexplainable payloads is impossible.

What a strong answer adds

Distinguishing transient failures — which belong in retry with backoff and should never reach the DLQ — from permanent ones. A DLQ receiving transient failures fills with messages that would have succeeded on retry, which both hides the real poison messages and inflates the remediation.

Common weak answers

Replaying everything immediately. Deleting the queue to clear the alert.