A billing platform sends a webhook, the customer's endpoint times out, and the customer's system retries the operation several times. How should idempotency keys, durable event state, delivery attempts, ordering and reconciliation prevent duplicate billing?

Why exactly-once is the wrong goal

The provider cannot distinguish "the consumer did not receive it" from "the consumer processed it and the acknowledgement was lost." Any delivery system facing that ambiguity must choose to risk duplicates or risk loss. For billing events, loss is worse, so at-least-once is the only defensible choice — and duplicates become the consumer's problem to absorb.

Saying this explicitly in the documentation is part of the architecture, because a consumer who believes delivery is exactly-once will not write the deduplication and will find out during your first retry storm.

What the provider must supply

• A stable, immutable event ID that is identical across every delivery attempt of the same event. If the ID is regenerated on retry, deduplication is impossible no matter what the consumer does.
• A durable event store, so an event survives a broker failure and can be redelivered days later.
• Explicit delivery-attempt metadata — attempt number, first-sent timestamp — so a consumer can tell a retry from a genuine new event and a support engineer can reason about what happened.
• Exponential backoff with jitter and a cap, so a customer's brief outage does not become a self-inflicted denial of service against them when they recover.
• A replay API, because eventually every consumer needs to reprocess a window, and without it they will ask you to re-send events manually.
• Ordering scoped to the subscription, not globally. Global ordering across all customers is a single serialisation point that cannot scale. Per-subscription ordering is what actually matters — you cannot deliver a cancellation before the creation — and it partitions cleanly.

What the consumer must do

Record the event ID before acting, in the same transaction as the side effect. A consumer that processes and then records is not idempotent; it just has a smaller window.

Ordering is not enough on its own

Even ordered delivery breaks if a consumer processes events concurrently for speed. Ordering guarantees at the transport must be preserved by the consumer's concurrency model, which means per-key serialisation on the consumer side — usually the most commonly missed half of this design.

The backstop is reconciliation: a daily comparison of the provider's authoritative state against what the consumer believes. It is the only mechanism that catches an event that was never delivered at all.