A team requires exactly-once processing across a message platform and its external providers. What is achievable, and what should be built instead?
Show the full answer Hide the answer
What is achievable
Exactly-once effect within a system whose state you control — via at-least-once delivery plus idempotent processing, or via transactional coupling of consumption and output within one system.
Exactly-once delivery across a network to a system you do not control is not achievable. The sender cannot distinguish "not delivered" from "delivered, acknowledgement lost", and no protocol removes that ambiguity.
What that means practically
Design for at-least-once with idempotent consumers, and be precise about where the guarantee begins and ends.
For a communications platform, the honest contract is: a retry with the same idempotency key will not create a second send. It is not: the recipient receives exactly one message — because once the message is handed to an external carrier, the platform cannot know what happens, and the carrier may itself duplicate.
Publishing the weaker true guarantee is better engineering than implying the stronger false one, because customers design against what they are told.
What to build
1. An idempotency key from the caller, since only the caller can identify the logical operation, and it must be recorded in the same transaction as the effect it protects.
2. Idempotent consumers keyed on a business identifier or an event id, because at-least-once means every step reruns occasionally.
3. A transactional outbox where a state change must produce an event, so "state changed" and "event will be published" commit atomically.
4. A stable event id in every outbound message, so customers can deduplicate — which is what makes at-least-once harmless on their side.
5. Reconciliation, because in any sufficiently asynchronous system drift is a certainty and the only question is whether you detect it or a customer does.
The framing that resolves the argument
"Exactly-once" in stream processing means exactly-once state update within the processing system's own state, not exactly-once side effects on the outside world. Teams that hear the phrase and assume the second build systems whose correctness depends on a guarantee that was never offered.