Log Compaction
Retaining only the most recent value for each key in a topic, so the log becomes a durable snapshot of current state rather than a bounded window of history.
Ordinary retention deletes by age, which means a consumer starting fresh sees only the recent window and has no way to learn the current state of keys that have not changed lately. Compaction retains the latest record per key indefinitely, so replaying the topic from the beginning reconstructs the complete current state.
That property is what makes several common patterns work. A CDC topic can be replayed to rebuild a downstream table in full. A new service can bootstrap its local cache of reference data by consuming from offset zero. Stream processors restore their state from a compacted changelog.
The mechanics have implications worth knowing. Deletion is expressed as a tombstone — a record with a null value — and tombstones are themselves retained for a configurable period before removal, long enough for consumers to observe them; a consumer offline longer than that window can miss a delete permanently and keep a record that no longer exists. Compaction runs in the background, so intermediate values remain visible for some time, and it is not a guarantee about what any given consumer will see.
The design rule: compacted topics require a meaningful key on every record, and a topic whose records are events rather than state changes is usually not a candidate.