The retry you are counting on was never promised
How platforms deliver webhooks to endpoints their customers run: retry policy, endpoint circuit breaking, sender placement, and the published failure record.
Reconstructs the sender's side of webhook delivery from GitLab's incident reviews and shipped source, GitHub's staff answers, three open-source delivery services and the Standard Webhooks specification. An architect leaves knowing which retry camp their senders and their own product belong in, the endpoint-breaker design every implementation converged on, and the three failure classes the public record actually contains.
GitHub and GitLab, whose webhooks trigger most of the world's CI, do not retry failed deliveries at all; the at-least-once guarantee everyone assumes is implemented by the receiver or not at all, while the webhook-service ecosystem standardised multi-day retries.
What you get out of it
- Senders split into two coherent camps: no retries plus a redelivery API where events point into a pullable system of record, multi-day exponential retries where events are the record.
- Four independent implementations converged on the same endpoint circuit breaker; the only contested part was 4xx semantics, and GitLab reversed itself in public after transient 404s broke integrations.
- Every published webhook delay incident was a shared-substrate incident: Sidekiq or an eventing service degraded, and webhooks were the symptom.
- GitLab's delivery bookkeeping became the bottleneck itself: fan-in bursts turned one webhook's status row into database-wide lock contention.
- WooCommerce's tracker documents both silent event loss (shutdown-hook queueing) and a self-amplifying delivery loop (payload reads that write), the two costs of embedding the sender in the application.
Scope
Why this, now. The Standard Webhooks spec has cross-vendor adoption and GitLab shipped its reversal of webhook auto-disabling in 17.11, so for the first time both retry camps and a full public design argument are citable side by side.
What it does not cover. Receiver-side idempotent processing (covered by the 2026-09-01 guide), and the engineering-blog, paper and talk layers, which the research session's egress policy blocked; the guide says so explicitly and ships the queries for that missing layer.
Other field guides
A timeout is not an answer: making retried requests safe
A client that times out on a mutating call holds no evidence about whether it happened, so it must resend, and something has to make the resend harml…
28 sources · 25 organisations · 4 postmortemsYou cannot delete the dual write, only move it
A service commits a change and must announce it, and no mechanism makes both happen or neither. This guide reconstructs where eighteen organisations …
28 sources · 24 organisations · 3 postmortemsA million clients, four hundred connections
Reconstructs the two-line connection-rationing architecture that GitLab, Notion, Figma, OpenAI, Cloudflare and Supabase all converge on, with the mea…
33 sources · 26 organisations · 5 postmortems