A communications platform delivers webhooks to customer endpoints that are frequently slow, occasionally down, and sometimes process the same event twice. Design the delivery system and state the guarantee you can honestly publish.

The guarantee you can honestly publish

At-least-once delivery, no ordering guarantee across events, with an event id for deduplication.

Not exactly-once — that is not achievable across a network to a system you do not control. Not ordered — guaranteeing order requires serialising delivery per destination, which means one slow event blocks everything behind it for that customer.

Publishing the weaker true guarantee is better engineering than implying a stronger false one, because customers build on what you tell them. A platform that implies ordering will have customers whose correctness depends on it, and they will be broken by an ordinary retry.

The design

1. Durable queue per destination, with independent progress. One customer's dead endpoint must not consume the delivery fleet or delay anyone else. Per-destination isolation is the core structural requirement.

2. Exponential backoff with jitter, over a long window. Retries over hours or days, not minutes. Customer endpoints go down for deployments, and a system that gives up in five minutes loses events during every customer's release.

3. Circuit breaking per destination. After sustained failure, stop attempting immediately and probe periodically. Without this, a customer who deletes their endpoint consumes delivery capacity indefinitely.

4. Signed payloads with a timestamp. So recipients can verify authenticity and reject replays. Signing is not optional for a platform whose webhooks trigger financial or operational actions on the customer side.

5. An event id, stable across retries. This is what makes at-least-once workable: the customer deduplicates on it. Documenting this prominently is more valuable than any amount of delivery engineering, because the customer's idempotency is the only thing that makes duplicate delivery harmless.

6. A dead-letter path with customer visibility. Undeliverable events are retained and listable, so a customer recovering from an outage can see and replay what they missed. Without this, a customer's four-hour outage is permanent data loss they discover later.

7. Delivery attempt logs exposed to the customer. Response codes, timing, payloads. This eliminates the largest category of support burden — "we never received it" — by letting customers answer it themselves.

The failure modes to design against

Retry storms against a recovering endpoint. When a customer comes back up, every queued event for them arrives at once, knocking them down again. Rate-limit delivery per destination and ramp up gradually — the same courtesy you would want from an upstream.

Head-of-line blocking from a single poisonous event. One event that consistently causes a 500 must not block the queue forever. A retry ceiling moves it to the dead-letter path and lets the rest proceed.

Unbounded queue growth for a permanently dead endpoint, which needs a retention limit and eventual disabling with notification.

Payload growth over time. Adding a field to a webhook payload is a breaking change for customers who validate strictly. Payload evolution needs the same versioning discipline as an API — which is why webhooks belong high on the list of surfaces that constrain a platform permanently.