Integration & APIs 07 Sep 2026 28 min read

The retry you are counting on was never promised

How platforms deliver webhooks to endpoints their customers run: retry policy, endpoint circuit breaking, sender placement, and the published failure record.

Reconstructs the sender's side of webhook delivery from GitLab's incident reviews and shipped source, GitHub's staff answers, three open-source delivery services and the Standard Webhooks specification. An architect leaves knowing which retry camp their senders and their own product belong in, the endpoint-breaker design every implementation converged on, and the three failure classes the public record actually contains.

The finding that surprised me

GitHub and GitLab, whose webhooks trigger most of the world's CI, do not retry failed deliveries at all; the at-least-once guarantee everyone assumes is implemented by the receiver or not at all, while the webhook-service ecosystem standardised multi-day retries.

What you get out of it

  • Senders split into two coherent camps: no retries plus a redelivery API where events point into a pullable system of record, multi-day exponential retries where events are the record.
  • Four independent implementations converged on the same endpoint circuit breaker; the only contested part was 4xx semantics, and GitLab reversed itself in public after transient 404s broke integrations.
  • Every published webhook delay incident was a shared-substrate incident: Sidekiq or an eventing service degraded, and webhooks were the symptom.
  • GitLab's delivery bookkeeping became the bottleneck itself: fan-in bursts turned one webhook's status row into database-wide lock contention.
  • WooCommerce's tracker documents both silent event loss (shutdown-hook queueing) and a self-amplifying delivery loop (payload reads that write), the two costs of embedding the sender in the application.

Scope

Why this, now. The Standard Webhooks spec has cross-vendor adoption and GitLab shipped its reversal of webhook auto-disabling in 17.11, so for the first time both retry camps and a full public design argument are citable side by side.

What it does not cover. Receiver-side idempotent processing (covered by the 2026-09-01 guide), and the engineering-blog, paper and talk layers, which the research session's egress policy blocked; the guide says so explicitly and ships the queries for that missing layer.

Open the field guide → Self-contained: it loads nothing at read time, follows your system theme, and prints cleanly.