Log Compaction
A retention policy that keeps only the most recent value for each key rather than deleting by age, so the log becomes a durable snapshot of current state.
Normal retention deletes by time or size, which means history is eventually lost and a new consumer cannot rebuild state from the beginning. Compaction instead guarantees that the latest value for every key is retained indefinitely, while superseded values are eventually removed.
This turns a stream into something with the properties of a table: replaying it from the start yields the current state of every key. It is what makes change-data-capture topics, configuration streams and materialised-view rebuilds practical, and it is how Kafka stores its own consumer offsets.
Two mechanics worth knowing. Deletion is expressed as a tombstone — a record with a key and a null value — which is retained long enough for consumers to observe the delete and then removed. And compaction is asynchronous, so duplicates for a key can be present for a while; consumers must handle seeing an older value followed by a newer one.
The design consequence: choosing a key for a compacted topic is choosing the granularity of the snapshot, and it cannot be changed without republishing the topic.