A billing platform must deliver events to thousands of customer endpoints with wildly varying reliability. What guarantees should it offer, and what must it require of consumers?

The guarantee to offer

At-least-once delivery with a stable event identifier, stated explicitly in the documentation. Exactly-once is not achievable across a network you do not control — the provider cannot distinguish "the consumer did not receive it" from "the consumer processed it and the acknowledgement was lost" — and for billing events, loss is worse than duplication.

Saying this in the documentation is part of the architecture. A consumer who believes delivery is exactly-once will not write deduplication, and will discover the truth during your first retry storm.

What the provider must supply

• A stable, immutable event ID, identical across every delivery attempt. If it is regenerated on retry, deduplication is impossible however good the consumer is.
• A durable event store, so an event survives broker failure and can be redelivered days later.
• Delivery attempt metadata — attempt number, first-sent timestamp — so a consumer can distinguish a retry from a new event and a support engineer can reconstruct what happened.
• Exponential backoff with jitter and a cap, so a customer's brief outage does not become a self-inflicted denial of service when they recover.
• A replay API, because every consumer eventually needs to reprocess a window, and without one they will ask you to resend manually.
• Ordering scoped to the subscription, not globally. Global ordering across all customers is a single serialisation point that cannot scale; per-subscription ordering is what actually matters — a cancellation must not precede its creation — and it partitions cleanly.
• Signed payloads, so the consumer can verify origin without a callback.
• An automatic disable with notification for endpoints failing persistently, or a dead customer's endpoint consumes delivery capacity indefinitely.

What must be required of consumers

Record the event ID before acting, in the same transaction as the side effect. A consumer that processes and then records is not idempotent; it merely has a smaller window.

And preserve ordering in their own concurrency model: a consumer that reads an ordered stream and hands events to a thread pool has discarded the guarantee it was given. This is the most commonly missed half of the design.

The backstop

Reconciliation. A periodic comparison of the provider's authoritative state against what the consumer believes is the only mechanism that catches an event never delivered at all — and it is what turns a delivery system into a correctness system.