The retry you are counting on was never promised
How platforms deliver webhooks to endpoints their customers run: retry policy, endpoint circuit breaking, sender placement, and the published failure record.
Reconstructs the sender's side of webhook delivery from GitLab's incident reviews and shipped source, GitHub's staff answers, three open-source delivery services and the Standard Webhooks specification. An architect leaves knowing which retry camp their senders and their own product belong in, the endpoint-breaker design every implementation converged on, and the three failure classes the public record actually contains.
GitHub and GitLab, whose webhooks trigger most of the world's CI, do not retry failed deliveries at all; the at-least-once guarantee everyone assumes is implemented by the receiver or not at all, while the webhook-service ecosystem standardised multi-day retries.
What you get out of it
- Senders split into two coherent camps: no retries plus a redelivery API where events point into a pullable system of record, multi-day exponential retries where events are the record.
- Four independent implementations converged on the same endpoint circuit breaker; the only contested part was 4xx semantics, and GitLab reversed itself in public after transient 404s broke integrations.
- Every published webhook delay incident was a shared-substrate incident: Sidekiq or an eventing service degraded, and webhooks were the symptom.
- GitLab's delivery bookkeeping became the bottleneck itself: fan-in bursts turned one webhook's status row into database-wide lock contention.
- WooCommerce's tracker documents both silent event loss (shutdown-hook queueing) and a self-amplifying delivery loop (payload reads that write), the two costs of embedding the sender in the application.
Scope
Why this, now. The Standard Webhooks spec has cross-vendor adoption and GitLab shipped its reversal of webhook auto-disabling in 17.11, so for the first time both retry camps and a full public design argument are citable side by side.
What it does not cover. Receiver-side idempotent processing (covered by the 2026-09-01 guide), and the engineering-blog, paper and talk layers, which the research session's egress policy blocked; the guide says so explicitly and ships the queries for that missing layer.
Other field guides
Where the languages meet: ten years of Dropbox, read from its own repositories
Reconstructs a decade of Dropbox's language and interface decisions from its repositories, CppCon slide decks, SDK issue threads and package registri…
31 sources · 8 organisations · 5 postmortemsWhat the vendor cannot host, you will maintain: ten years of Alibaba's published middleware
A decade of Alibaba's published middleware, for architects choosing which parts of their own system to delegate to a large company's components. The …
36 sources · 9 organisations · 6 postmortemsSwapping the system behind the name: ten years of LinkedIn
LinkedIn wrote Kafka, published it, and in 2025 announced its replacement. Over the same decade it also left its own RPC framework, its own service d…
28 sources · 15 organisations · 3 postmortems