Exactly-Once Semantics, Precisely

A sender whose acknowledgement is lost cannot distinguish "not received" from "received, ack lost", so it must retransmit (risking duplicates) or not (risking loss). No third option exists. What is achievable is exactly-once processing: deliver at least once and ensure duplicate deliveries produce no additional effect on state or output.

When the computation has no deterministic key, or when the sink lacks keyed upserts, or when the operation accumulates. An increment is not idempotent, so aggregations cannot be made safe by upsert alone; they need transactional commits with checkpointed state instead. Set assignment works, addition does not.

The pipeline periodically snapshots its state, and output writes are staged in a transaction committed atomically with that snapshot. On failure it restores the last snapshot, reprocesses from the corresponding input offsets, and aborts the uncommitted transaction, so the reprocessed records produce the only committed output. Flink checkpoint barriers with Kafka transactional producers is the reference implementation.

At-least-once. Guarantees are properties of a path, and the weakest link sets the strength. A sink that cannot participate in a two-phase commit downgrades the whole pipeline regardless of what the engine provides internally, which is why the sink's capabilities belong in the design discussion before the engine's.

Wall-clock time, unseeded randomness, and iteration over unordered collections. Replay must reproduce identical results; if it does not, the transaction still commits successfully and the output is simply wrong. A visible failure would be preferable, because this one leaves no trace and no alert.

Because with a transactional sink, output only becomes visible at commit boundaries, which coincide with checkpoints. Tuning the interval purely for recovery time therefore changes how stale every consumer's view is. Short intervals mean constant snapshotting overhead; long intervals mean both expensive recovery and high output latency.

A sender whose acknowledgement is lost cannot distinguish "not received" from "received, ack lost", so it must retransmit (risking duplicates) or not (risking loss). No third option exists. What is achievable is exactly-once processing: deliver at least once and ensure duplicate deliveries produce no additional effect on state or output.

When the computation has no deterministic key, or when the sink lacks keyed upserts, or when the operation accumulates. An increment is not idempotent, so aggregations cannot be made safe by upsert alone; they need transactional commits with checkpointed state instead. Set assignment works, addition does not.

The pipeline periodically snapshots its state, and output writes are staged in a transaction committed atomically with that snapshot. On failure it restores the last snapshot, reprocesses from the corresponding input offsets, and aborts the uncommitted transaction, so the reprocessed records produce the only committed output. Flink checkpoint barriers with Kafka transactional producers is the reference implementation.

At-least-once. Guarantees are properties of a path, and the weakest link sets the strength. A sink that cannot participate in a two-phase commit downgrades the whole pipeline regardless of what the engine provides internally, which is why the sink's capabilities belong in the design discussion before the engine's.

Wall-clock time, unseeded randomness, and iteration over unordered collections. Replay must reproduce identical results; if it does not, the transaction still commits successfully and the output is simply wrong. A visible failure would be preferable, because this one leaves no trace and no alert.

Because with a transactional sink, output only becomes visible at commit boundaries, which coincide with checkpoints. Tuning the interval purely for recovery time therefore changes how stale every consumer's view is. Short intervals mean constant snapshotting overhead; long intervals mean both expensive recovery and high output latency.