A service rebuilds its cache from a compacted topic after being offline for two weeks. It now holds records that were deleted. Why?

What the interviewer is testing

Whether you know how deletes work in a compacted log and the specific window in which they are lost.

The mechanism

A delete in a compacted topic is a tombstone — a record with a null value for the key. Compaction retains the latest value per key indefinitely, but tombstones are retained only for a configurable period (commonly the delete retention setting) before being removed entirely.

The purpose of that window is to give consumers time to observe the delete. Once it passes, the tombstone is compacted away and the key simply does not appear in the topic at all.

A consumer offline longer than that window and then replaying from the beginning sees no record for the key — and therefore no instruction to delete it. Its cache retains the stale entry permanently.

The fixes

Set the tombstone retention longer than the maximum plausible consumer outage, and document that as the supported offline window. This is a deliberate decision, not a default.

Rebuild by replacement, not by merge. A consumer bootstrapping from offset zero should build a fresh cache and swap it in atomically, rather than applying records to its existing state. Anything absent from the topic is then correctly absent from the cache. This is the more robust fix and it removes the dependency on tombstone retention entirely.

Alert on consumers exceeding the supported offline window, so a two-week outage is a known event requiring a full rebuild rather than a silent resume.

What a strong answer adds

Periodic reconciliation between the source of truth and the derived cache, comparing key sets. Derived state diverges for many reasons — a missed event, a bug, an out-of-order apply — and a comparison job is the only thing that detects silent divergence. Building it at the same time as the cache is what separates a robust implementation from one that erodes trust.

Common weak answers

Increasing compaction frequency, which is unrelated. Assuming the delete was never published.