How platforms actually push events to servers other people run, reconstructed from the layer of the record that marketing never touches: GitLab's incident reviews and shipped source, GitHub's staff answers, three open-source delivery services, and the one specification their builders wrote together. Read it before you assume "at-least-once" means what you think it means, on either side of the wire.
The problem, who has solved it in production, what this guide covers, and what it deliberately leaves out.
Strip the technology name away and the problem is this: when something happens inside your system, you must tell thousands of other organisations' servers about it, promptly and at least once, over HTTP endpoints they wrote, hosted on infrastructure you cannot see, with capacity you cannot know. At any moment some of those endpoints are slow, some are dead, some return errors that mean "stop" and some return errors that mean "try later", and none of them will tell you which. Every delivery you attempt consumes your compute; every delivery you skip breaks someone's integration. The sender carries all of the operational burden and receives none of the operational signal.
The systems studied here have all solved it in production, in public, with their working shown: GitLab (whose webhook reliability programme is fully visible in its issue tracker, incident reviews and shipped Ruby source), GitHub (through its staff answers, its published receiver contract and a 2026 incident thread), Svix, Convoy and Hookdeck's Outpost (three open-source delivery services whose configuration files are their design documents), WooCommerce (whose tracker records what happens when the sender lives inside the application), and the Standard Webhooks consortium (Svix, Zapier, Twilio, Mux, ngrok, Supabase, Lob and Kong), whose specification is the closest thing the field has to a shared decision record.
The surprise sits in the first two tiles. The two platforms whose webhooks trigger most of the world's CI do not retry failed deliveries. GitHub staff said it flatly in a community thread: "There are no retries of failed webhook deliveries and no API for listing or retrieving failed webhook events" (an inspection and redelivery API arrived later, but redelivery remains something the receiver requests, not something GitHub does). GitLab's own feature proposal for retry policies, open since March 2022, describes the current behaviour as "drop: as current - allow messages to be dropped". Meanwhile the webhook-as-a-service ecosystem standardised the opposite: the Standard Webhooks specification recommends "a retry schedule spanning multiple days, with an exponential backoff", and Svix, Convoy and Outpost all ship it. At-least-once delivery is real in this field, but which side of the wire implements it depends entirely on who your sender is, and most receiving teams have never checked.
This guide covers the sender's side: retry policy, endpoint health, isolation of the delivery path, and the security of calling URLs your customers typed. The receiver's half, idempotent processing of duplicates, is the subject of the earlier guide Making retried requests safe in this collection and is not repeated here.
The evidence base is deliberately and unusually narrow in kind: source code, decision records, issue threads and incident reviews, all on github.com and gitlab.com. The research session that produced this guide ran behind an egress policy that blocked every other host, including the engineering blogs of Stripe and Segment and the archives of arXiv and USENIX, so accounts known to exist there are not cited here; nothing was cited that was not fetched and read. The hunt section includes the queries to run for that missing layer. Treat the absence of blog, paper and talk evidence in this guide as an artefact of the constraint, not of the public record.
The common shape across GitLab, Svix, Convoy and Outpost, with the divergence points marked. Every box below is attributable to at least two systems.
Across four independent implementations the delivery pipeline has the same skeleton. An
event is matched against subscriptions and fanned out into one delivery task per endpoint.
Tasks land on a durable queue: Sidekiq on Redis at GitLab, a Redis task queue at Svix
("PostgreSQL for the storage of events, Redis for the task queue and cache", per the README),
Redis with an experimental Postgres option at Convoy, and "Redis, PostgreSQL, a supported
message queue" at Outpost. Dispatch workers with bounded concurrency (Svix ships
worker_max_tasks = 500 as the default cap) pull tasks, make the HTTP call with a
hard timeout, and write two records: an attempt log the customer can see, and a per-endpoint
failure counter that drives the health machinery. Nobody delivers synchronously from the
request path, and every one of the four keeps per-attempt history rather than only a current
status.
Every system tracks consecutive failures per endpoint and stops calling endpoints that keep failing. GitLab's shipped constants: a grace threshold of 3 (disabled on the 4th consecutive failure), backoff starting at 1 minute and doubling to a 1-day cap, permanent disablement past 39. Svix disables after 120 hours of unbroken failure; Convoy "opens the circuit when an endpoint fails consecutively, then probes it".
Sources: auto_disabling.rb, Svix config, Convoy
Svix persists every attempt "not only to retry failed attempts but to give users visibility"; Convoy markets itself on the debugging surface; GitLab exposes recent deliveries per hook and grew a customer request for resending "when they have many failures (hundred or more)" programmatically. The 2026 consumer complaint thread on GitHub's forum names the absence of this surface as the industry's biggest gap: "Providers give almost no visibility into what actually happened."
Sources: Svix, GitLab #372826, GitHub community
The sender makes HTTP requests to URLs its customers typed, which is a server-side request forgery machine unless filtered. Svix blocks "SSRF attacks and Internal IP addresses" by default and warns that whitelisting private subnets "is a security risk"; Convoy's tracker carried a live SSRF fix for endpoint probing as recently as August 2026, and the fix died unmerged. Static egress IPs double as the customer's firewall story.
Sources: Svix config, Convoy PR #2827
Where the four diverge is instructive. GitLab's sender is not a service at all; it is a worker class inside the same Sidekiq fleet that runs everything else on GitLab.com, which is why, in every incident this guide catalogues, webhook latency moved when something unrelated saturated Sidekiq. The three dedicated services made the opposite choice, and Outpost pushes the divergence one step further: its destination list is "Webhooks, Hookdeck Event Gateway, Amazon EventBridge, AWS SQS, AWS S3, GCP Pub/Sub, RabbitMQ, and Kafka", which quietly concedes that for high-volume consumers the right delivery mechanism is not HTTP at all but a handoff to a bus the receiver already operates. That option changes who owns the retry problem, which is the subject of the next section.
Five forks in the road, each with the choice a real system made, the stated reason, and the condition that flips the answer.
Notice what the split correlates with. The platforms that refuse to retry are the ones
whose events are advisory pointers into a pullable system of record; a missed push event is
recoverable by listing commits. The systems that retry for days are the ones sold to senders
whose events are the system of record, where a dropped invoice.paid is money.
Neither camp is wrong; the error is inheriting a camp without checking which kind of event
you emit. GitLab's tracker states the consequence of not deciding explicitly: users with
essential messages and users with chatty CI noise get the same drop-on-failure treatment,
because the sender cannot tell them apart.
Decision 2 is the strongest convergence in the whole corpus: four independent implementations, one mechanism. The disagreement that remains is only about thresholds and semantics, and GitLab ran the experiment in public. Its 2022 design treated a 4xx as proof of receiver misconfiguration and disabled the hook on the spot; the issue tracker then filled with self-hosted administrators whose integrations broke every deploy window, one calling it "rather harsh to break self-hosted environments for the sake of protecting SaaS". The 2025 redesign (issue #396577, shipped in 17.11) lets every failure mode self-heal and reserves permanence for the 40th consecutive failure, which the code comment translates into operator time: a hook fails with maximum backoff for "at least 1 month" before the platform gives up on it. A public feedback issue is still tuning it in 2026. That is a complete, documented argument cycle most teams re-run privately and expensively.
recent_failures,
disabled_until).| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Retry failed deliveries? | No (GitHub, GitLab); multi-day backoff (Svix, Convoy, Outpost, spec) | The other camp's answer | Storage cost and event criticality vs. pullable systems of record | GitHub staff, GitLab #355721 |
| Persistently failing endpoint | Circuit breaker: backoff, then disable + notify | Trying forever; instant permanent disable on 4xx | Platform protection; then the 4xx theory broke real integrations | Epic 8083, #396577 |
| Sender placement | Dedicated service (Svix, Convoy, Outpost) | Shared job fleet (GitLab), request shutdown (WooCommerce) | Shared substrate couples delivery latency to every neighbour | Incident 20791, WC #44199 |
| Delivery state placement | Per-attempt rows; derived counters | Hot counters on the subscription row | Fan-in bursts turn one row into a database-wide lock queue | GitLab #352245 |
| Transport | HTTP push by default; bus handoff at volume | HTTP-only forever | Inbound HTTP is the receiver's bottleneck and security objection | Outpost |
Five published failures in three classes. The recurring lesson: webhook incidents are almost never caused by webhooks.
Group the public record by failure class and a pattern appears that should reorganise how you monitor this subsystem. Class one: the shared substrate degrades and webhook delivery degrades with it, while the webhook code is blameless. Class two: the delivery machinery's own bookkeeping becomes the bottleneck under fan-in. Class three: the sender is embedded in an application whose lifecycle silently eats or multiplies events. No incident in this corpus was caused by the part teams design most carefully, the HTTP call itself.
What is missing from the record matters too. No public account in this corpus describes the classic theorised catastrophe, a recovering endpoint flattened by its own retry backlog arriving at once. The spec and every implementation guard against it with backoff and jitter, so either the guard works or the failure goes unreported; both readings justify keeping the guard. Equally, no sender in this corpus publishes an account of losing events it had accepted, and GitLab's incident review is explicit that delay, not loss, was the impact. The delivery pipelines hold; it is the edges (the substrate below, the receiver beyond, the bookkeeping within) that fail.
Every figure with its context, date and source. Measured, shipped-default and derived are marked apart.
| Metric | Value | At | Context | Kind | As of | Source |
|---|---|---|---|---|---|---|
| Receiver response budget | 10 s | GitHub | 2xx required within it, or the delivery counts as failed | Documented contract | 2026-09 | GitHub docs |
| Automatic retries after failure | 0 | GitHub, GitLab | Receiver recovers via redelivery API (GitHub) or manual resend (GitLab) | Documented contract | 2026-09 | GitHub staff, GitLab #355721 |
| Backoff on failing endpoint | 1 min → 24 h, ×2 | GitLab | Starts on 4th consecutive failure; counter resets on any 2xx | Shipped constant | 2026-09 | auto_disabling.rb |
| Permanent disable threshold | 40 consecutive failures | GitLab | Code comment: at least 1 month of failing at max backoff first | Shipped constant | 2026-09 | auto_disabling.rb |
| Default retry schedule | 5 s, 5 m, 30 m, 2 h, 5 h, 10 h, 10 h | Svix OSS | retry_schedule in the shipped config; ~27.6 h cumulative (derived sum) | Shipped default | 2026-09 | config.default.toml |
| Endpoint disable window | 120 h | Svix OSS | Endpoint disabled after failing continuously this long; any success resets | Shipped default | 2026-09 | config.default.toml |
| Dispatch concurrency cap | 500 tasks | Svix OSS | worker_max_tasks default per worker process | Shipped default | 2026-09 | config.default.toml |
| Delay under substrate incident | up to 15 min | GitLab.com | 2025-10-29, Sidekiq slowdown; no deliveries lost | Measured | 2025-10 | Incident 20791 |
| Delay under upstream incident | 40 min max, 10 min avg | GitHub | 2026-02-03, push webhooks, eventing-service CPU saturation | Measured | 2026-02 | Incident thread |
| Tenant burst that hurt | ~23,000 jobs | GitLab.com | One project, minutes, 2022-02-07; drove the webhook rate-limit work | Measured | 2022-02 | Incident 6297 |
The shipped defaults are what operators get, not what any operator measured as optimal;
Svix's schedule and GitLab's thresholds encode judgement, and both projects have changed
them (GitLab's backoff redesign removed a backoff_count column as recently as
the 18.x series). The 27.6-hour figure is this guide's arithmetic on Svix's schedule, not a
published claim. Unknown, because nobody publishes it: sender-side cost per delivery,
fleet sizes, and p99 delivery latency in steady state. If those numbers matter to your
design, you will have to measure a pilot; no public account supplies them.
Every source behind this page, graded. All from open trackers and repositories, fetched and read 2026-09-07; the ledger with per-claim quotes ships beside this page as sources.md.
Severity-3 incident review: Sidekiq worker slowdowns delayed webhooks up to 15 minutes for two and a half hours. Notable for what did not happen: no deliveries were lost.
One project generated ~23K jobs in minutes and delayed CI platform-wide for an hour. The corrective action list is where GitLab's webhook rate limiting was born.
An abusive project's async jobs took page loads and API requests down for 15 minutes; mitigation was manually deleting the offender's jobs from the shared queue.
Staff updates in the official community thread: eventing-service connection churn and CPU saturation delayed push webhooks up to 40 minutes and Actions starts with them.
The original containment design: 5xx backoff to 24 h, 4xx disabled immediately, and the stated motive, protecting the platform "from the potential abuse or misuse of a small few".
The public reversal of the 4xx-is-permanent rule after it broke integrations that 404 during deploys; shipped in 17.11 with uniform backoff and permanence only at 40 straight failures. A feedback issue (#503733) still tunes it.
Documents drop-on-failure as current behaviour and proposes drop / retry-N / enqueue tiers, priced as a paid feature because of storage cost. Contains the corpus's most honest sentence about event criticality.
The cross-vendor consensus document: multi-day exponential retry, endpoint disablement with out-of-band notification, webhook-id as idempotency key, HMAC with timestamp tolerance. Adopters listed include OpenAI, Anthropic and PagerDuty.
The post-ship feedback channel for the 17.11 redesign, kept open by the product team; evidence that endpoint-health thresholds are tuned continuously, not set once.
The shipped breaker: thresholds 3 and 39, backoff 1 minute doubling to 1 day, project hooks only, behind an ops feature flag, with the month-to-permanent budget explained in a comment.
The lock-contention finding: simultaneous job completions firing one hook stall the database on that hook's status row, degrading throughput for unrelated jobs.
A customer with "many failures (hundred or more)" asks for programmatic resend; the shape of recovery work when the sender does not retry.
A production sender's whole posture in one config file: the retry schedule, the 120-hour endpoint disable window, worker concurrency, SSRF blocking on by default, Redis queue durability guidance.
Independent Go implementation converging on the same controls: constant or exponential-with-jitter retries, per-endpoint rate limiting, circuit breaking with probing, endpoint disable with notification, Redis queue.
A community fix for server-side request forgery in endpoint probing that died on CLA and CI friction; the vulnerability class is alive in 2026 even in dedicated webhook infrastructure.
At-least-once delivery, automatic and manual retries, and a destination list that extends past HTTP to EventBridge, SQS, S3, Pub/Sub, RabbitMQ and Kafka.
Maintainer-adjacent account of webhooks queued at request shutdown never being queued at all when another plugin exits first; silent loss in the world's most-deployed commerce platform.
An operator's incident report in issue form: payload reads trigger sync writes that fire the same event again, growing the queue and CPU while every individual delivery succeeds.
The staff answer that anchors this guide's surprise: no retries, use polling; later comments point to the Deliveries API as the receiver-driven recovery path.
A consumer-side systematisation of what is still broken: late or missing events, empty payloads, opaque errors, and WAFs silently eating deliveries. Zero replies from providers at fetch time.
The receiver contract: respond fast with 2xx, process async, prepare for duplicates, return 4xx only for true misconfiguration; plus the documented disable thresholds and the Idempotency-Key header held constant across retries.
The 10-second response budget, async processing guidance, redelivery as the receiver's responsibility, and X-GitHub-Delivery as the stable dedup key.
The vendor's own adoption post-mortem: cost anxiety, integration inertia and unawareness, not technical disagreement, keep teams on homegrown senders.
Governance signal: the spec's committee spans competing vendors (Svix, Zapier, Twilio, Mux, ngrok, Supabase, Lob, Kong), and its open issues show live argument about signature schemes and header conventions, not about retry or breaker mechanics.
Six rungs from an evening's dispatcher to a system whose failure modes you have met on purpose. The crossing from toy to real is rung 4.
One process: accept an event, POST it to a registered URL with a 10-second timeout, a signature header (HMAC over timestamp plus body, per the Standard Webhooks spec), and an attempt record in SQLite. No retries yet.
Done when: a delivery to a slow endpoint gives up at exactly 10 s and the attempt row says so. Teaches: the timeout is the sender's only unilateral power.
Re-enqueue failures on Svix's shipped schedule (5 s, 5 m, 30 m, 2 h, 5 h, 10 h, 10 h) read from a config file, with jitter. Keep the delivery ID constant across attempts.
Done when: an endpoint that dies for an hour receives exactly one copy of each event afterwards, and your log shows which attempt succeeded. Teaches: at-least-once is a schedule plus a stable ID, nothing more mystical.
Implement Figure 3: consecutive-failure counter, disable at 4, backoff 1 minute doubling to a day, permanent at 40, counter reset on success, manual re-enable. Watch GitLab's constants stop making sense for your event rate and change them.
Done when: a dead endpoint costs the fleet one probe per backoff window instead of one timeout per event. Teaches: thresholds are functions of event frequency; there is no universal constant.
Build the receiver zoo: one endpoint that tarpits at 9.5 s, one that returns 200 instantly then 500 for an hour, one that 404s during a simulated deploy window, one that is your own metadata service IP. Run a steady event stream against all four.
Done when: the tarpit does not delay the healthy endpoint's deliveries (this forces per-endpoint concurrency), and the metadata-service URL is rejected at registration. Teaches: head-of-line blocking and SSRF are the two failures you must design out, not patch out.
Fire 20,000 events at one endpoint in a minute (GitLab's 23K-job incident, scaled to your laptop) while the health bookkeeping writes to a single row. Measure throughput, then batch or sample the bookkeeping and measure again.
Done when: you can show the contention on a graph and its removal on another. Teaches: GitLab issue #352245, personally.
Add the two endpoints every real system grew: list recent deliveries with status and latency, and bulk-replay failures since a timestamp. Then delete a day of deliveries and recover a consumer using only these APIs.
Done when: recovery from your own outage needs no database access. Teaches: the attempt log is the product; delivery is just its write path.
The queries that produced this guide, plus the ones to run on the layer this session could not reach. The tracker queries are the transferable technique: every large open-source platform's webhook argument is public.
site:gitlab.com gl-infra production "incident review" webhook sidekiqsite:gitlab.com "Delayed Webhooks" production incidentsite:github.com orgs/community/discussions webhook delayed incidentgithub availability report webhook delivery delaysgitlab auto-disable failing webhooks issue designsite:gitlab.com issues "retry" webhooks "problem to solve"repo:frain-dev/convoy is:pr is:closed is:unmergedstandard-webhooks spec "operational considerations"svix-webhooks config.default.toml retry_schedulepath:app/models/concerns/web_hooks repo:gitlab-org/gitlabwoocommerce webhook shutdown "action scheduler" delivery issuesegment centrifuge "billions of events" deliverybrandur webhooks stripe operationalstripe webhook retry "72 hours" documentationshopify webhooks eventbridge "at scale" engineering