A consumer diverts records it cannot process to a dead-letter topic created with the cluster defaults. Monitoring counted about 60000 diverted records after a bad release in March. When the team replays in May the topic holds 11 records. Which setting explains the loss?

The mechanism

A dead-letter topic is an ordinary topic. It is created with whatever the cluster default retention is - commonly seven days - and nobody chooses that number, because nobody chooses anything when a topic is auto-created. Records older than the retention are deleted by the broker on its own schedule, with no error anywhere and no change to any dashboard that counts diversions rather than stored records.

The eleven survivors are simply the records diverted in the last few days. The sixty thousand from March were deleted in March, roughly a week after they arrived, while the incident ticket was still open.

The diversion counter and the topic contents measure different things, and the team has been watching the counter.

What to set, and to what

• Retention on a dead-letter topic is a triage service-level objective expressed in days. If the team's commitment is "a human looks at every diverted record within 30 days", retention is at least 90 days: the commitment plus the time to fix the bug plus the time to replay.
• Alert on the age of the oldest record in the topic, not on the count. A count alert fires during the incident, when everyone already knows. An age alert fires when the backlog is being ignored, which is the failure that actually loses data.
• Create dead-letter topics explicitly with the rest of the infrastructure definition, so the retention is a reviewed line rather than a cluster default.
• Size it: 60000 records at a few kilobytes each is well under a gigabyte, so there is no cost argument for a short retention here. For a topic diverting millions of records a day the answer is different and the real fix is upstream.

Why the other options fail

• Log compaction. Compaction is off by default and would only be on if someone configured it. If it were on it would also keep one record per key, so a topic with thousands of distinct keys would still hold thousands of records rather than eleven.
• Acks of 1 and a failover. This loses records at the moment of a leader change, in bursts, not steadily over two months, and it would leave an uneven pattern rather than only the newest few days.
• Offset retention expiry. Real, and a genuine hazard for a long-idle consumer group, but it changes where a reader starts rather than what the topic contains. The team is looking at the topic's contents here.

When this is the wrong answer

If the pipeline diverts tens of thousands of records every normal day, long retention on the dead-letter topic is treating a design fault as a storage problem. A dead-letter rate above roughly one record in ten thousand is a routing or validation defect, and the fix is a schema check at the producer or a separate topic for the known-bad shape, not a bigger graveyard.