Design outbound webhook delivery for a platform with tens of thousands of subscribers whose endpoints have wildly varying reliability. What guarantees can you offer and how do you prevent one bad subscriber from harming the rest?

The guarantee you can actually offer

At-least-once delivery, with ordering not guaranteed. Anything stronger is a promise you cannot keep across an internet boundary to endpoints you do not control.

State this explicitly in the documentation, because the consumer must build for it: idempotent handling via a delivery/event identifier, and tolerance of out-of-order arrival. Every webhook consumer that does not deduplicate has a latent double-processing bug, and the provider's documentation is where that is prevented.

Preventing one subscriber from harming others

This is the core architectural requirement, and the failure mode is direct: a subscriber whose endpoint takes 30 seconds to respond will consume the delivery workers' capacity and delay everyone else's events.

• Per-subscriber concurrency limits and queues. Each subscriber has bounded in-flight deliveries, so a slow endpoint backs up its own queue and nobody else's.
• Aggressive per-request timeouts — a few seconds. The webhook contract must state that the endpoint should acknowledge quickly and process asynchronously, and the timeout is what enforces it.
• Circuit breaking per subscriber: after sustained failures, stop attempting, back off substantially, and probe occasionally. Continuing to hammer a dead endpoint is work with a known-zero success rate.
• Separate worker pools by subscriber health, so known-slow subscribers cannot occupy the pool serving healthy ones.
• A queue depth cap per subscriber, beyond which events are dropped or diverted to storage with a notification. Unbounded per-subscriber queues turn one dead endpoint into a storage incident.

Retry policy

Exponential backoff with jitter over a long window — minutes to hours to a day — because most endpoint failures are transient deployments and outages, and a subscriber down for an hour should not lose events.

Retry only on retryable conditions: 5xx, timeouts, connection failures. A 4xx means the request is malformed or unauthorised and will never succeed — retrying it is pure waste, and specifically 410 Gone should disable the subscription.

A dead-letter destination after exhaustion, with the events retained and retrievable, plus a notification to the subscriber. And a replay API, so a subscriber who was down can fetch what they missed rather than opening a support ticket — this single feature removes most webhook-related support load.

Security, which is not optional

• Signature over the payload and a timestamp, with a shared secret per subscriber, so the receiver can verify authenticity — and the timestamp is what prevents replay.
• Secret rotation with an overlap period, allowing two valid secrets at once.
• Stable source IP ranges published, for subscribers who firewall.
• No sensitive data in the payload beyond an identifier where possible, so the webhook says "this changed" and the subscriber fetches the detail over an authenticated API. This limits the damage from a misconfigured endpoint and keeps the payload small.

The observability the subscriber needs

A delivery log the subscriber can see: what was sent, when, the response, and the retry state. Most webhook support burden is "did you send it?" and a self-service answer eliminates the entire category.