A team turns on transactional producers and read_committed consumers to get exactly-once stream processing. What have they gained, what have they given up, and when does the bill arrive?

What is gained, precisely

Atomicity across a write and its offset commit: the records a processor produces and the position it advances to land together or not at all. That removes the duplicate-side-effect window that at-least-once leaves behind on every crash between produce and commit. For a pipeline whose output is itself a topic, it means a downstream reader never sees the output of work that was later reprocessed. Introduced in Kafka through KIP-98 (2017).

What it does not give: exactly-once anywhere outside Kafka. A transaction cannot cover a charge to a payment provider or a row in a warehouse. The guarantee ends at the log boundary, and teams routinely assume otherwise.

What is paid

• Coordinator and marker overhead. Each transaction adds coordinator round trips and writes a commit or abort marker into every partition it touched. The cost is per transaction, so throughput loss is roughly inversely proportional to transaction size - batching 10 ms of work per transaction is cheap, one record per transaction is not.
• A new failure domain, the transaction coordinator and its state topic, with its own recovery behaviour.
• The last stable offset, which is where the real bill is. A read_committed consumer may not read past the LSO - the offset before the earliest still-open transaction. One producer that dies between its first write and its commit holds the LSO back, and every consumer of that partition stops, including consumers that do not care about the transaction, and including records written non-transactionally after it. Consumer lag climbs while the brokers are healthy, the disks are fine and nothing errors.

When the bill arrives

At the first hung transaction. The coordinator aborts an abandoned transaction only when its timeout expires, so the stall is bounded by configuration rather than by seconds - and a genuinely stuck one may need the detection-and-abort tooling added in KIP-664 (2020), or for retention to age the records out. The condition is invisible on ordinary dashboards: graph the per-partition LastStableOffsetLag, not only consumer lag, or the first indication is a downstream team asking why their data stopped.

When not to turn it on

When the sink can be made idempotent - which is most sinks. A natural key with an upsert, a deduplication table keyed by event id, or a target that is naturally idempotent (setting a value rather than incrementing it) gives the same end state with none of the above. Reach for transactions when the side effect is inside Kafka and genuinely cannot be made idempotent; otherwise spend the effort on the idempotent write and keep the simpler failure model.

How to decide in a review

Ask what the duplicate actually costs. If a duplicate means a double-counted metric in a dashboard, pay nothing for it. If it means a second notification to a customer, fix it at the notification service with a deduplication key, which also covers the retries that Kafka cannot see. Exactly-once in the log is a narrow guarantee bought at the price of a shared stall.