Design a webhook delivery system for a platform with 10,000 customers. What are the hard parts?

Delivery, and its failure modes

Persist the event first, deliver asynchronously. Delivery in the request path couples your latency and availability to every customer's endpoint.

Retry with exponential backoff and jitter over a long window — hours to days — because customer endpoints have outages and deployments. Jitter matters: without it, recovery produces a synchronised burst against an endpoint that has just come back.

Per-endpoint circuit breaking. A customer whose endpoint has been failing for a day should not consume delivery capacity indefinitely. Slow their rate, and stop after a threshold with a notification.

Per-customer isolation. Without it, one slow endpoint consumes the shared worker pool and delays every other customer's deliveries. Separate queues or concurrency limits per destination — this is bulkheading, and it is the difference between one customer's outage and yours.

A dead letter destination plus a replay API, so a customer who was down can recover the events they missed without asking support.

Security, where implementations are commonly wrong

HMAC signature over the raw request body, with a timestamp inside the signed payload and a tolerance window to prevent replay.

Document the four ways consumers break verification: verifying after parsing (re-serialised JSON has different bytes), non-constant-time comparison, no timestamp, and no support for two active secrets during rotation.

Support two valid secrets simultaneously so customers can rotate without dropping deliveries.

Consider SSRF. Customer-supplied URLs are attacker-controllable input. Block private address ranges and link-local metadata endpoints, and re-validate after redirects.

The contract you publish

At-least-once with possible duplicates — consumers must be idempotent on event ID. Say it plainly.

No ordering guarantee, because retries reorder by construction. Include a sequence number or timestamp so consumers can ignore regressions.

Tell consumers to return 2xx immediately and process asynchronously, and to reserve 4xx for genuinely malformed requests — a 4xx for a business-logic problem usually stops retries and drops the event permanently.

The observability customers need

Per-endpoint delivery success rate, latency and failure reasons, exposed in the customer's dashboard. Otherwise every failure is a support ticket, and support cannot answer it either.

What a strong answer adds

Offering an alternative for high-volume consumers: a polling API or a managed event stream. Webhooks suit low-volume, low-latency notification; at high volume, pull-based delivery is more reliable and shifts flow control to the consumer, where it belongs.