Portfolio ↗
Evidence ledger 24 sources Checked 07 Sep 2026

Evidence ledger

One row per claim in The retry you are counting on was never promised: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.

Working ledger, one row per claim. All URLs fetched in-session on 2026-09-07.

Note on evidence base: this session's network egress allowed github.com, raw.githubusercontent.com and gitlab.com only. Engineering-blog, paper and talk layers (segment.com, github.blog, brandur.org, docs.stripe.com, shopify.dev, arxiv.org and others were attempted and blocked at the proxy) are therefore absent by constraint, not because the material does not exist. The guide's scope section says the same. Everything cited below was fetched and quoted this session.

# Org Title Tier Published Checked URL Claim I take from it Supporting quote or figure
1 GitLab Epic 8083: Auto-disable failing webhooks adr 2022 (15.6 target) 2026-09-07 https://gitlab.com/groups/gitlab-org/-/epics/8083 GitLab's stated motive for auto-disabling webhooks is protecting the platform from a few bad endpoints "protect the GitLab and users across the system from the potential abuse or misuse of a small few"
2 GitLab Epic 8083 adr 2022 2026-09-07 https://gitlab.com/groups/gitlab-org/-/epics/8083 Original design: 5xx backoff from 10 min up to 24 h; 4xx disabled immediately "5xx errors ... Initial disable period: 10 minutes ... Maximum disabled period: 24 hours; 4xx errors ... disabled immediately" (design summary)
3 GitLab Issue 396577: allow all autodisabled webhooks to self-heal adr 2023-03 2026-09-07 https://gitlab.com/gitlab-org/gitlab/-/issues/396577 The 4xx-permanent rule broke recoverable integrations (404 during deploys); users called it harsh; GitLab reversed to uniform self-healing + permanent disable at 40 consecutive failures "a 404 may happen in operation - while we deploy/migrate/initialize"; "rather harsh to break self-hosted environments for the sake of protecting SaaS"; "All webhooks can self-heal ... permanently disabled after 40 consecutive failures"
4 GitLab webhooks.md (gitlab-org/gitlab master docs source) vendor current (checked 2026-09-07) 2026-09-07 https://gitlab.com/gitlab-org/gitlab/-/raw/master/doc/user/project/integrations/webhooks.md Shipped auto-disable behaviour: 4 consecutive failures → disabled 1 min, backoff to 24 h; 40 consecutive failures → permanent; failures include 4xx/5xx, timeouts, HTTP errors "After 4 consecutive failures, webhooks are disabled for one minute, extending up to 24 hours"; "After 40 consecutive failures, webhooks ... are not automatically re-enabled"
5 GitLab webhooks.md docs source vendor current 2026-09-07 https://gitlab.com/gitlab-org/gitlab/-/raw/master/doc/user/project/integrations/webhooks.md Receiver contract: respond fast, process async, misuse of status codes is a receiver bug "Respond quickly with a 200 or 201 status. Avoid processing webhooks in the same request. Use a queue to handle webhooks after receiving them."
6 GitLab webhooks.md docs source vendor current 2026-09-07 https://gitlab.com/gitlab-org/gitlab/-/raw/master/doc/user/project/integrations/webhooks.md GitLab sends an Idempotency-Key header held constant across retries Docs note an Idempotency-Key header "consistent across webhook retries"
7 GitLab infra Incident 20791: 2025-10-29 Delayed Webhooks postmortem 2025-10-29 2026-09-07 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20791 Webhook delay incident: up to 15 min delay, 14:10–16:45, Sidekiq worker slowdown; no deliveries lost "Webhooks were delayed by as much as 15 minutes"; "Slowdowns in sidekiq workers, including WebHookWorker and related jobs, caused elevated webhook latency"; "No webhooks failed to deliver, but the delays disrupted some customer workflows"
8 GitLab infra Incident 6297: 2022-02-07 high number of queued Sidekiq jobs postmortem 2022-02-07 2026-09-07 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6297 One project's job storm (~23K jobs) delayed CI/MR processing for ~1 h; corrective action was webhook rate limiting "one project created ~23K of jobs"; corrective action: "Rate limit webhook execution and backoff"
9 GitLab infra Incident 6586: 2022-03-14 issues with async jobs postmortem 2022-03-14 2026-09-07 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6586 An abusive project's async jobs overwhelmed Sidekiq; mitigation was deleting the project's jobs "removing the jobs created by the abusive project, so we can alleviate the Sidekiq queue"; impact "Unable to load pages, failure of various API requests" (17:11–17:26 UTC)
10 GitLab Issue 352245: rate limit webhook execution and backoff source 2022-02 2026-09-07 https://gitlab.com/gitlab-org/gitlab/-/issues/352245 The webhook bookkeeping row itself became the bottleneck: thousands of simultaneous job completions fire hooks whose log workers contend on one WebHook row "several thousands of jobs finish at roughly the same time, causing web hooks to fire"; "The concurrent updates to a single row could create a lock contention in the database"; contention "causes all these jobs to wait, decreasing throughput and increasing the backlog of other jobs"
11 GitLab Issue 355721: specify web-hook retry policies adr 2022-03 2026-09-07 https://gitlab.com/gitlab-org/gitlab/-/work_items/355721 GitLab's default is drop-on-failure; retry/enqueue policies remain an open proposal, priced as an EE feature because of storage cost Proposed policies: "drop (as current), retry up to N times ..., and enqueue"; "a good candidate for an EE feature ... the cost of storing these records"
12 GitHub Community discussion 186279 (2026-02-03 incident thread) postmortem 2026-02-03/05 2026-09-07 https://github.com/orgs/community/discussions/186279 Push webhooks delayed up to 40 min (avg 10 min) by eventing-service connection churn and CPU saturation "connection churn in our eventing service, which caused CPU saturation and delays for reads and writes, with subsequent downstream delivery delays"; push webhooks "delayed up to 40 minutes (average 10 minutes)", 14:00–17:40 UTC
13 GitHub Webhook best practices (github/docs source, main) vendor current 2026-09-07 https://raw.githubusercontent.com/github/docs/main/content/webhooks/using-webhooks/best-practices-for-using-webhooks.md Sender timeout contract is 10 s; recovery after receiver downtime is the receiver's job (redelivery API), not automatic "Your server should respond with a 2XX response within 10 seconds of receiving a webhook delivery."; "If your server goes down, you should redeliver missed webhooks once your server is back up."
14 GitHub Webhook best practices (github/docs) vendor current 2026-09-07 https://raw.githubusercontent.com/github/docs/main/content/webhooks/using-webhooks/best-practices-for-using-webhooks.md Dedup key: X-GitHub-Delivery stays the same on redelivery "Use the X-GitHub-Delivery header to ensure that each delivery is unique per event." ... "If you request a redelivery, the X-GitHub-Delivery header will be the same as in the original delivery."
15 Standard Webhooks Spec 1.0.0 (repo main) adr 2023-11 (v1.0.0) 2026-09-07 https://raw.githubusercontent.com/standard-webhooks/standard-webhooks/main/spec/standard-webhooks.md The ecosystem's stated problem is fragmentation of bespoke implementations "the ecosystem is fragmented, with each webhook provider using different implementations and varying quality. Even high quality implementations vary, making them inherently incompatible"
16 Standard Webhooks Spec 1.0.0 adr 2023-11 2026-09-07 https://raw.githubusercontent.com/standard-webhooks/standard-webhooks/main/spec/standard-webhooks.md Spec's operational recommendations: multi-day exponential retry; disable persistently failing endpoints and tell the owner out-of-band; webhook-id as idempotency key; timestamp tolerance against replay "It's recommended to retry delivery following a retry schedule spanning multiple days, with an exponential backoff."; "notify the consumers using other channels (e.g. email), and is recommended to disable future delivery to the endpoint"; "Use the webhook-id header as an idempotency key"; "verify the webhook-timestamp header has a timestamp that is within some allowable tolerance"
17 Standard Webhooks Repo landing page adr checked 2026-09-07 2026-09-07 https://github.com/standard-webhooks/standard-webhooks Steering committee spans Svix, Zapier, Twilio, Mux, ngrok, Supabase, Lob, Kong; adopters incl. OpenAI, Anthropic, PagerDuty, Twilio "adopted by a variety of companies including: OpenAI, Anthropic, Google Gemini, Kong, Svix, Supabase, Vanta, Drata, Etsy, PagerDuty, Twilio, TaskRabbit"; 1.7k stars
18 Svix svix-webhooks README (main) source checked 2026-09-07 2026-09-07 https://raw.githubusercontent.com/svix/svix-webhooks/main/README.md Open-source delivery service shape: Postgres for events, Redis task queue, DLQ with redrive endpoint, SSRF blocking by default PostgreSQL "for the storage of events"; Redis "for the task queue and cache"; monitor "svix.queue.depth_dlq"; "SSRF attacks and Internal IP addresses" blocked by default
19 Svix config.default.toml (svix-server, main) source checked 2026-09-07 2026-09-07 https://raw.githubusercontent.com/svix/svix-webhooks/main/server/svix-server/config.default.toml Concrete production defaults: retry schedule [5s, 5m, 30m, 2h, 5h, 10h, 10h]; endpoint disabled after 120 h of consistent failure; 500 concurrent workers "retry_schedule = [5,300,1800,7200,18000,36000,36000]"; "endpoint_failure_disable_after = 120" (hours); "worker_max_tasks = 500"
20 Convoy (Frain) Convoy repo/README source checked 2026-09-07 2026-09-07 https://github.com/frain-dev/convoy Independent OSS gateway converges on the same controls: retries (constant or exponential+jitter), rate limiting, circuit breaking, endpoint disable with notification "retries, rate limiting, static ips, circuit breaking, rolling secrets" (feature list); disables endpoints on consecutive failure and notifies
21 Convoy (Frain) PR #2827 Fix SSRF through unchecked GET (closed unmerged) source 2026-08 2026-09-07 https://github.com/frain-dev/convoy/pull/2827 SSRF in endpoint probing was a live vulnerability class in 2026; the community fix died unmerged (CLA + CI issues, closed by author) "Fix SSRF vuln on endpoint probing"; closed by submitter, CLA signed by 3 of 6 committers
22 WooCommerce Discussion 44199 (Web API feedback) source 2024 2026-09-07 https://github.com/woocommerce/woocommerce/discussions/44199 Embedded-sender loss mechanism: webhooks queued on the shutdown hook are silently lost when any plugin exits first "If any of those plugins does an exit or causes an error, the remaining shutdown tasks are never executed, resulting in WooCommerce's pending webhooks to never be added to the Action Scheduler queue."
23 WooCommerce Issue 62492: HPOS sync-on-read repeated order.updated webhooks source 2026 2026-09-07 https://github.com/woocommerce/woocommerce/issues/62492 Feedback loop: delivery reads the order, the read triggers a sync write, the write fires another webhook; operator saw backlog + CPU + stalled analytics "Webhook delivery reads the order to build a REST payload ... HPOS performs a sync/write during this read. That write triggers order.updated. The webhook is queued again, repeating the cycle."; "Large numbers of pending Action Scheduler jobs, primarily woocommerce_deliver_webhook_async"; "Elevated CPU usage caused by self-originating HTTP requests"
24 Convoy (Frain) Wiki: why developers do not use webhook gateways vendor 2023 2026-09-07 https://github.com/frain-dev/convoy/wiki/Why-Developers-do-not-use-Webhook-Gateways-today Adoption friction for dedicated senders is cost/awareness/inertia, not disagreement about the mechanics "Many engineers are used to managing webhook integrations manually, and are hesitant to switch to a new platform."
25 GitLab auto_disabling.rb (gitlab-org/gitlab master) source current 2026-09-07 https://gitlab.com/gitlab-org/gitlab/-/raw/master/app/models/concerns/web_hooks/auto_disabling.rb Shipped constants: grace threshold 3, permanent at 39 recent failures, backoff 1 min doubling to 1 day; auto-disabling applies to project hooks only and sits behind an ops feature flag "TEMPORARILY_DISABLED_FAILURE_THRESHOLD = 3"; "PERMANENTLY_DISABLED_FAILURE_THRESHOLD = 39"; "INITIAL_BACKOFF = 1.minute"; "MAX_BACKOFF = 1.day"; "BACKOFF_GROWTH_FACTOR = 2.0"; "ENABLED_HOOK_TYPES = %w[ProjectHook]"; comment: "A webhook will be failing and being temporarily disabled for the max backoff of 1 day (MAX_BACKOFF) for at least 1 month before it becomes permanently disabled on its 40th failure."
26 GitLab Issue 355721: specify web-hook retry policies (full text) adr 2022-03, still open 2026-09-07 https://gitlab.com/gitlab-org/gitlab/-/issues/355721 GitLab cannot distinguish must-deliver from nice-to-have events; drop-on-failure is the documented current behaviour; storage cost prices retries as a paid feature "Users have no way to tell us how important their processes are, and we have no way to distinguish nice-to-have messages that will tolerate failure to essential messages that must be delivered eventually."; "drop: as current - allow messages to be dropped."; "the cost of storing these records (and potential abuse), this would be a good candidate for an EE feature"
27 GitLab Issue 372826: allow resending failed webhook requests with the API source 2022-09 2026-09-07 https://gitlab.com/gitlab-org/gitlab/-/issues/372826 Manual recovery does not scale: a customer with hundreds of failures asked for programmatic resend "A customer would like the ability to be able to resend any failed requests using the API so they can resend them programmatically when they have many failures (hundred or more)."
28 GitLab Issue 503733: feedback on self-heal feature adr 2024-11 2026-09-07 https://gitlab.com/gitlab-org/gitlab/-/issues/503733 The self-heal redesign shipped and is still being tuned via a public feedback issue "Thanks for using Allow all autodisabled webhooks to self-heal feature introduced with ... #396577. This issue serves the purpose of collecting feedback"
29 Hookdeck Outpost README (main) source checked 2026-09-07 2026-09-07 https://raw.githubusercontent.com/hookdeck/outpost/main/README.md Third independent OSS sender; at-least-once; retries; and a bus-handoff option: destinations include EventBridge, SQS, S3, Pub/Sub, RabbitMQ, Kafka, not just HTTP "Production-ready infrastructure for sending webhooks and delivering events from your platform to your customers' systems"; destinations: "Webhooks, Hookdeck Event Gateway, Amazon EventBridge, AWS SQS, AWS S3, GCP Pub/Sub, RabbitMQ, and Kafka"; "At least once delivery"
30 GitHub Community discussion 24721: handling GitHub webhook retry source 2022+ 2026-09-07 https://github.com/orgs/community/discussions/24721 GitHub staff confirm no automatic retries; receivers are told to poll; a Deliveries API later enabled receiver-driven redelivery Staff (lee-dohm): "There are no retries of failed webhook deliveries and no API for listing or retrieving failed webhook events."; suggested "polling or some other system to find the events that happened while your system was unresponsive"
31 GitHub community Discussion 185003: why webhooks still fail us in 2026 source 2026 2026-09-07 https://github.com/orgs/community/discussions/185003 Consumer-side view in 2026: late/missing events and opaque failures are still the norm "Events arrive late or not at all. Providers give almost no visibility into what actually happened."
32 Convoy (Frain) README (main, raw) source checked 2026-09-07 2026-09-07 https://raw.githubusercontent.com/frain-dev/convoy/main/README.md Convoy self-description and queue architecture: Redis queue (Postgres experimental), server+agent split "an open source high-performance webhooks gateway used to securely ingest, persist, debug, deliver and manage millions of events reliably"; "retries, rate limiting, static ips, circuit breaking, rolling secrets and more"