A vendor claims their streaming platform provides exactly-once processing. How do you evaluate the claim?
Show the full answer Hide the answer
Ask where the guarantee ends
Nearly always at the platform's boundary. Within it, state and offsets commit together, so internal state reflects each input once. That is real and it is half the problem.
The other half is the write to the outside world — your database, your API, your file. No processor can make that atomic with its own offset commit. On failure between the two, the record is written and the offset is not, so it is written again.
So the correct question is not whether the platform is exactly-once, but what your sink does with a duplicate.
Test it rather than accepting it
Kill the consumer between the external write and the offset commit, deliberately, and observe. This is a ten-minute experiment and it settles the question definitively.
Do the same for the sink: write the same record twice and check the result.
What makes a sink safe
A keyed upsert. A write conditioned on a sequence number. A deduplication table keyed on an event identifier.
What is not safe, and is the common case: appending to a table, incrementing a counter, sending an email, calling an API without an idempotency key.
The honest architecture
At-least-once delivery with idempotent sinks, which is what most systems described as exactly-once actually implement. Saying so plainly is better than claiming a guarantee the deployment does not have, because the claim is what stops people designing the deduplication.
The vendor question to ask
"Show me the failure mode where a record is delivered twice, and tell me what my sink must do." A vendor who can answer that clearly is being straight with you; one who repeats the marketing term is not.