A team wants exactly-once processing in a streaming pipeline. Where do transactional producers, idempotent consumers and output-side deduplication each fit, and which failure modes still require reconciliation?

What "exactly-once" actually means

Exactly-once delivery across a network boundary does not exist. A sender that cannot confirm receipt must either retry (risking a duplicate) or not retry (risking a loss), and no protocol removes that choice.

Exactly-once processing does exist, and it is achieved by at-least-once delivery plus deduplication or transactional state updates. Systems advertising exactly-once are providing this, which is genuinely useful and is not what the phrase suggests.

Where each mechanism fits

Idempotent producer. The broker deduplicates retries from a single producer session using a producer identifier and a sequence number per partition. This eliminates duplicates caused by producer retries, which is the most common source, and it costs almost nothing.

Transactional producer. Writes to multiple partitions, plus the consumer's offset commit, are committed as one atomic unit. This is what makes read-process-write pipelines exactly-once within the streaming system: the offset advances if and only if the output was written, so a crash cannot leave the two inconsistent.

Consumers must read committed only, or they see aborted transactions' records.

Idempotent consumers, for anything outside the streaming system. Transactions cover the broker and its offsets. They do not cover a write to an external database, an email, a payment, or a third-party API — the moment the pipeline touches anything else, at-least-once returns and the consumer must handle it.

Output-side deduplication or idempotent effects for those external writes: an upsert keyed on the event identifier, a conditional write on a version, or a processed-identifier table written in the same transaction as the effect — because a crash between the effect and the record of the effect reintroduces the duplicate.

What still requires reconciliation

• Any external system without a transactional boundary shared with the consumer. A payment gateway, an email provider, a partner API — the best available guarantee is idempotency at their end or deduplication at yours, and neither is perfect.
• Ordering across partitions, which is never guaranteed. Consumers must not assume global order; use version checks and reject stale updates.
• Producer failure before the record was written, which is a lost event rather than a duplicated one, and is invisible to every downstream deduplication mechanism.
• Reprocessing after a bug, which by design produces duplicates that deduplication windows may have expired.
• Clock skew and late-arriving data, where an event processed as on-time is later found to belong to a closed window.

A reconciliation process comparing source and destination totals is therefore not an admission of failure; it is a required component in any pipeline whose correctness matters.

The cost, stated honestly

Transactional semantics add latency — a commit boundary per batch — reduce throughput, and introduce transaction coordinator state that can itself become an operational concern. Long-running transactions block consumers reading committed records.

Many pipelines do not need it. A pipeline feeding a metrics dashboard, a search index, or an analytical aggregate tolerates duplicates trivially if the downstream operation is idempotent — and making the effect idempotent is usually cheaper and more robust than making the delivery transactional.

The engineering question is never "how do we avoid duplicates" but "where does deduplication live, what is its window, and what reconciles the residue."