You operate webhook delivery for 50000 integrators. One endpoint is down for two days. Design the delivery system.

What is being tested

Whether you isolate a failing consumer from everyone else, and whether you provide a recovery path rather than only a retry policy.

The immediate requirement: isolation

A slow or failing endpoint must not consume delivery capacity. Without isolation, workers block on timeouts to a dead endpoint and delivery degrades for all 50,000 integrators — one partner's outage becomes your outage.

Mechanisms:

• Per-endpoint circuit breaking. After consecutive failures, stop attempting for a period. Do not keep hammering a dead endpoint.
• Bounded concurrency per endpoint, so one integrator cannot occupy the pool.
• Separate queues by endpoint health, so healthy deliveries are never behind a backlog of retries to a broken one.
• Aggressive timeouts. Two seconds, not thirty. The consumer's job is to acknowledge, not to process.

The retry schedule

Exponential backoff with jitter over a defined window — commonly a few days — then a terminal state. The schedule must be published, so integrators can reason about it, and it must end, because infinite retries accumulate forever.

The recovery path, which is the part usually missing

After two days there is a large backlog. Three decisions:

1. Do you replay the backlog? Delivering two days of events at full speed to a just-recovered endpoint will knock it over again. Replay must be rate-limited, and the integrator should be able to trigger it themselves rather than having it happen automatically.

2. Are old events still useful? For state-change notifications, often not — the current state is what matters. Offering "here are the object IDs that changed; fetch their current state" is frequently better than replaying two days of individual events, and much cheaper for both sides.

3. Reconciliation via the API. This is the essential mechanism. Webhooks are an optimisation over polling, not a guarantee. Some events will be missed — an endpoint down beyond the retry window, a deployment that lost them, a bug that dropped them silently. A robust integration reconciles periodically against a list API, and the platform must provide one that supports "what changed since timestamp X".

A design that treats webhook delivery as authoritative will eventually diverge with no way to detect it.

Notification and visibility

• Notify the integrator that their endpoint is failing, through a channel other than the endpoint.
• A dashboard showing delivery state, failures with response codes, and a manual replay control.
• Disable persistently dead endpoints and require re-enabling, so you are not retrying to an address that was decommissioned a year ago.

The consumer-side guidance to publish

Receivers should acknowledge immediately and process asynchronously. A consumer that processes synchronously inside the handler times out under load, you retry, and they get duplicate work plus eventual endpoint disabling — a failure caused entirely by an unstated expectation.

Plus: verify signatures (with a timestamp to prevent replay), deduplicate on event ID, and do not assume ordering.