Offset Management
How a consumer records its position in a stream, and the decision that determines whether processing is at-least-once or at-most-once.
The offset is simply the position of the next message to read. Where and when it is committed decides the delivery semantics, and the choice is more consequential than it looks.
Commit after processing gives at-least-once: a crash between processing and committing causes redelivery, so a message may be handled twice.
Commit before processing gives at-most-once: a crash after committing loses the message entirely.
Auto-commit on an interval — the default in several clients — gives neither reliably, because the commit fires on a timer unrelated to what has actually been processed. It is the source of a recurring class of bug in which messages are silently skipped after a crash, and disabling it is usually the first thing to do in a serious consumer.
The stronger arrangement is to store the offset atomically with the processing result, in the same transaction as the downstream write. Then position and effect cannot diverge, and the consumer is effectively-once with respect to that store.
Manual offset control also enables replay, which is the operational superpower of streams: reset a group's offsets and reprocess history after fixing a bug.