Evidence ledger
26 sources
Checked 25 Sep 2026
Evidence ledger
One row per claim in Deciding who gets told no: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
One row per claim. Every URL was fetched during the research session on 2026-09-25. The research environment's outbound network policy allowed github.com, raw.githubusercontent.com, gitlab.com, pkg.go.dev and the public package registries; engineering blogs (Stripe, Cloudflare, Figma, GitHub's blog), paper archives (arXiv, USENIX, ACM) and video hosts were unreachable. The guide is therefore deliberately built from the repository record: incident reviews on GitLab's public production tracker, source code, merge requests, design blueprints, KEPs, protobuf contracts and in-repo documentation. Where a well-known account is named in the prose (Stripe's rate-limiters post, Cloudflare's counting post, GitHub's sharded Redis limiter post, the SIGCOMM 2007 distributed rate limiting paper), it is explicitly marked as not verifiable in this session and no evidence card or link is built on it. GitHub issue and pull request threads were also unavailable (the session's GitHub API access is repository-scoped), so the argument-in-review layer comes from GitLab merge requests and from git history fetched over the git protocol.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | GitLab | Incident Review: Uptick in 429 errors, unexpected authentication errors | postmortem | 2024-06 | 2026-09-25 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18174 | A load-balancer configuration change broke the rate-limit bypass path, so the traffic that normally skips the limiter got limited; the biggest flow through a mature limiter is the exempt flow. | "approximately 2,000,000 requests were bypassing rate-limiting in a 5-minute window before the incident. During the incident, this number dropped to 15,000, indicating that approximately 1,985,000 requests were affected every 5 minutes." Root cause: "A configuration change to our HAProxy which changed how out application level rate-limiting is handled for users." Impact: "Customers in around 50 root namespace were impacted by this, mainly customers who are in our allowlist for rate-limiting." |
| 2 | GitLab | Incident Review #18174 (detection) | postmortem | 2024-06 | 2026-09-25 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18174 | The failure of the exception path was detected by a customer, not a monitor. | "The customer reached to our support team through PagerDuty"; improvement listed: "Having an alert on increase in 429 could have been a good indicator." |
| 3 | GitLab | 2022-11-30: GitLab login fails with 429 error | postmortem | 2022-11-30 | 2026-09-25 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8101 | A rate limiter converts someone else's fast failure loop into user-facing 429s: an OAuth secrets migration broke logins, the resulting redirect loop hit the login throttle, and users saw "too many requests" for six and a half hours. | "For approximately 7 hours, starting at 2022-11-30 03:50 UTC customers who attempted to login through OAuth providers GiHub, Salesforce and Bitbucket on GitLab.com were unable to login and presented with 429 errors." "Impact Duration: 3:50 utc - 10:20 utc (390 minutes)"; Root cause label: "RootCause::Config-Change". |
| 4 | GitLab | Incident Review: INC-12449: gitlab.com HTTP timeout over 30s | postmortem | 2026-07-27 | 2026-09-25 | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22584 | A single user's unthrottled download burst produced a database lock convoy and a Severity 1 outage; the throttle that would have covered the path existed but was disabled in production. | "A burst of Nix generic-package download traffic from a single external user repeatedly updated packages_packages.last_downloaded_at on every GET request without throttling, creating a PostgreSQL lock convoy"; "A contributing factor was that the application-level RackAttack rate limit for authenticated package API requests (throttle_authenticated_packages_api_enabled) was disabled in production". |
| 5 | GitLab | Incident Review INC-12449 (blast radius and mitigation) | postmortem | 2026-07-27 | 2026-09-25 | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22584 | Measured impact of the missing limit, and the emergency mitigation being an edge rate limit deployed mid-incident then codified. | "~9.49M HTTP 503 responses during the peak 19:43–19:58 UTC window"; "PgBouncer waiters peaked at 3,680"; Severity 1; "Total Duration: 1 hour, 17 minutes"; "A temporary Cloudflare rate-limiting rule was designed and deployed quickly as a preventive measure, then codified into Terraform (config-mgmt MR 14886)". |
| 6 | GitLab | Incident Review: 2025-03-06: Some succeeded gitlab-org jobs are stuck in running | postmortem | 2025-03-07 | 2026-09-25 | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/19447 | The limiter's identity classifier disagreed with the API's authenticator, silently, until traffic growth crossed the unauthenticated threshold; CI runners were throttled as anonymous users for 39 hours. | "requests being incorrectly classified as unauthenticated, resulting in lower IP-based rate limits"; "If the job token was passed in the header under Private-Token, then Rack Attack attempted to look up the user associated with the PAT, and finding none, it assumed this was an unauthenticated request."; "Total Duration: 39.15 hours". |
| 7 | GitLab | Incident Review #19447 (latency of the defect, corrective direction) | postmortem | 2025-03-07 | 2026-09-25 | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/19447 | Classification defects in a limiter are latent by design: a wrong verdict changes a threshold, not an outcome, so nothing fails until load arrives. The stated fix is one identity code path for limiter and API. | "This Auth could have been failing silently for a long time because it won't actually block the request it just changes the rate limit"; "We should address this by using a single source of truth for the job token auth code in both RackAttack and in the api. Additionally, we should consider ... alert if they fail the authentication at one layer and pass it at another."; trigger chain: "GitLab Runner v16.10 added the PRIVATE-TOKEN header to support Cells routing." |
| 8 | GitLab | MR 183764: Fix Rack Attack incorrectly rate limiting runner API | source | merged 2025-03-07 | 2026-09-25 | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/183764 | The fix for the misclassification: make the limiter's token lookup fall through to the job-token parameter instead of raising on the header. | "This commit makes access_token return blank if it has the CI build token prefix so that the exception is not raised. That way Rack Attack can then ensure the request is authenticated with the job token." |
| 9 | GitLab | Next Rate Limiting Architecture (blueprint, in-repo) | adr | 2022-09-08 | 2026-09-25 | https://gitlab.com/gitlab-org/gitlab/-/blob/v16.0.0-ee/doc/architecture/blueprints/rate_limiting/index.md | GitLab's own architecture document says binary bypass for big customers is the wrong exception mechanism, and that enforcement is scattered across at least five opaque systems. | "We allow disabling rate-limiting for some marquee SaaS customers, but this increases a risk for those same customers. We should instead be able to set higher limits."; "We enforce a number of different policies via opaque external systems (Pipeline Validation Service, Bouncer, Watchtower, Cloudflare, HAProxy)."; limit/quota/policy definitions and the warn: 2B / day / soft: 100k / s / hard: 500k / s YAML sketch. |
| 10 | GitLab | Epic 2021: Rate Limits, Phase 1: Rails unification | adr | 2026-04-17 | 2026-09-25 | https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2021 | Four years after the blueprint, application rate limiting is still fragmented across seven or more implementations with inconsistent dry run and bypass, and new endpoints ship unlimited by default; the unification plan centres on identity, not counting. | Fragmentation table: RackAttack ("Mix of hardcoded values and ApplicationSetting DB columns"), ApplicationRateLimiter ("Some via Application Settings UI, some via API, some hardcoded"), "5+ other limiter types"; "New endpoints don't get rate limits by default."; "Each rate limit check has two pieces of identity: a call site name ... and a request identifier". |
| 11 | GitLab | Issue 656: Enable Rack::Attack rate limiting for authenticated and unauthenticated requests | source | 2020-11-05 | 2026-09-25 | https://gitlab.com/gitlab-com/gl-infra/observability/team/-/work_items/656 | The initial thresholds were tuned in dry-run mode for two months before enforcement, and dry-run telemetry caught a transposition of the two authenticated limits before any user was affected. | Settings table: 2020-11-20 "1500/m (dry) / 1000/m (dry) / 2000/m (dry)" with "Note the numbers were inadvertently transposed for the two auth options"; corrected 2020-11-27; unauth dropped to "500/m (dry)" 2020-12-22; enforced 2021-01-18 at "500/m / 2000/m / 1000/m". |
| 12 | GitLab | Production change #3034: Enable RackAttack rate-limiting in dry-run mode | source | 2020-11-16 | 2026-09-25 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3034 | Enforcement was explicitly staged behind a dry run to measure accuracy before affecting users. | "First step is to enable it in Dry Run mode for the new throttles, so we can evaluate the accuracy and potential effectiveness before we affect users for real." |
| 13 | GitLab | Production change #22496: Add new nodes for the 6th shard to redis-cluster-ratelimiting | source | 2026-07-10 | 2026-09-25 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/22496 | Counting is a workload with its own capacity planning: GitLab runs a dedicated Redis Cluster for rate limiting and grew it to 18 nodes to afford running two limiters in parallel during a shadow migration. | "adds a 6th shard (master + 2 replicas) to the existing gprd redis-cluster-ratelimiting cluster, growing it from 5 → 6 shards (15 → 18 nodes)"; "ahead of the Rack::Attack → Labkit::RateLimit shadow rollout (both rate limiters run in parallel during comparison, roughly doubling counter operations)"; nodes are reserved c2-standard-4 across three zones. |
| 14 | GitLab | User and IP rate limits (administration documentation, in-repo) | vendor | checked at master | 2026-09-25 | https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/settings/user_and_ip_rate_limits.md | The operational surface of the exception plane: a bypass header the edge must erase on untrusted traffic, a user-ID allowlist, per-throttle dry run, and a documented procedure for un-locking yourself out. | "It is important that your load balancer erases or overwrites the bypass header on all incoming traffic. Otherwise, you must trust your users to not set that header and bypass the GitLab rate limiter."; GITLAB_THROTTLE_DRY_RUN "logs a message ... while letting the request continue"; troubleshooting heading: "Disable throttling after accidentally locking administrators out". |
| 15 | GitLab | MR 233066: Unify RackAttack configuration, add per-throttle dry_run and allowlist DB columns | source | opened 2026-04-23, closed unmerged | 2026-09-25 | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/233066 | Even the exception plane's configuration is under live rework: a change adding per-throttle dry-run and allowlist columns for all 21 throttles was closed without merging while the unification plan was re-cut around labkit. | "Adds two JSONB columns to application_settings ... rack_attack_throttle_dry_run: dry-run flag per throttle name; rack_attack_throttle_allowlists: user ID allowlists per throttle name"; "Dynamic accessors for all 21 throttle names are generated via RACK_ATTACK_THROTTLE_NAMES"; state closed, merged_at null (GitLab API); the superseding epic (row 10) describes the replanned phases. |
| 16 | GitLab | Gitlab::ApplicationRateLimiter (source, master) | source | checked at master | 2026-09-25 | https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/application_rate_limiter.rb | The second limiter implementation inside the same application: per-action throttled? checks with threshold, interval and a users allowlist argument, separate from the middleware limiter. |
def throttled?(key, scope:, resource: nil, threshold: nil, interval: nil, users_allowlist: nil, peek: false, ...); per-call threshold override "Optional threshold value to override default". |
| 17 | Envoy (Lyft origin) | envoyproxy/ratelimit README | source | checked at master | 2026-09-25 | https://github.com/envoyproxy/ratelimit | The reference global limiter is a separate gRPC service over Redis, keyed by domain plus descriptors, with a local negative cache for keys already over limit and a global shadow mode for introducing limits. | "Applications request a rate limit decision based on a domain and a set of descriptors. The service reads the configuration from disk ... composes a cache key, and talks to the Redis cache."; local cache "stores the over-the-limit cache keys, and thus avoids reading the redis cache again for the already over-the-limit keys"; "There is a global shadow-mode which can make it easier to introduce rate limiting into an existing service landscape." |
| 18 | Envoy | HTTP rate limit filter contract (rate_limit.proto) | source | checked at main | 2026-09-25 | https://github.com/envoyproxy/envoy/blob/main/api/envoy/extensions/filters/http/ratelimit/v3/rate_limit.proto | The failure posture is a config field defaulting to fail open, the decision budget defaults to 20 ms, and a runtime fraction lets operators fail partially closed. | "The filter's behaviour in case the rate limiting service does not respond back. When it is set to true, Envoy will not allow traffic in case of communication failure between rate limiting service and the proxy. bool failure_mode_deny = 5;"; timeout comment: "If not set, this defaults to 20ms"; failure_mode_deny_percent example: "when the rate limit service is unavailable, 50% of requests will be denied (fail closed) and 50% will be allowed (fail open)". |
| 19 | Envoy | Commit cfdbd87: redis: optionally close pooled connections on READONLY error replies (#1191) | source | 2026-07-27 | 2026-09-25 | https://github.com/envoyproxy/ratelimit/commit/cfdbd87a1e1b8c7de0509fc92fcad812c516311a | The limiter's own store failover is a live operational concern in 2026: after a Redis failover the service holds connections to a demoted primary and needs explicit READONLY handling. | Commit title: "redis: optionally close pooled connections on READONLY error replies (#1191)", committed 2026-07-27 (git history fetched over git protocol). |
| 20 | Envoy | Global rate limiting (architecture overview, in-repo) | vendor | checked at main | 2026-09-25 | https://github.com/envoyproxy/envoy/blob/main/docs/root/intro/arch_overview/other_features/global_rate_limiting.rst | When global counting is worth it (many hosts converging on few, low latency), and the two-stage pattern: a local token bucket absorbs bursts so the global service survives them. | "The most common case is when a large number of hosts are forwarding to a small number of hosts and the average request latency is low"; "a local token bucket rate limit can absorb very large bursts in load that might otherwise overwhelm a global rate limit service. Thus, the rate limit is applied in two stages."; of the quota-based mode: "Open source reference implementation of the rate limiting service is currently unavailable." |
| 21 | Google / YouTube | Doorman design doc (in-repo) | adr | repo pushed 2016 | 2026-09-25 | https://github.com/youtube/doorman/blob/master/doc/design.md | The problem statement for cooperative client-side limiting, the cost argument against pure server-side refusal, and the three named failure postures: pessimistic, optimistic, and a configured safe capacity. | "There is a cost in sending and then rejecting requests. We have already burnt the network capacity and done some processing on the request."; "1. In pessimistic mode it can behave as if a capacity of zero has been granted ... 2. In optimistic mode it can behave as if it got all of the capacity it requested ... 3. In safe mode the client is allowed to use the resource up to the per-client safe capacity limit"; "Experience has shown us that exponential back-off is a lossed art (much like speling)". |
| 22 | Google / YouTube | Doorman repository status | source | last commit 2016-05-02 | 2026-09-25 | https://github.com/youtube/doorman | The open-sourced global limiter stopped moving ten weeks after its first commit: 70 commits total, last on 2016-05-02, with the README's short-term plans (C++ and Python clients, Docker image) never shipped. | README: "Doorman should be currently considered Alpha quality software."; git log: last commit 2016-05-02, 70 commits (fetched over git protocol). |
| 23 | Kubernetes | KEP-1040: Priority and Fairness for API Server Requests | adr | created 2019-02-28, status implemented | 2026-09-25 | https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/1040-priority-and-fairness/README.md | The failure record that motivated replacing a request-count limit with concurrency shares: undifferentiated max-inflight lets one misbehaving client crowd out heartbeats and controllers. | "Apart from the distinction between mutating and readonly, no other distinctions are made among requests; consequently there can be undesirable scenarios where one subset of the request load crowds out other parts of the request load."; "Deployment of Doom. We had a situation where a bug in the Deployment controller caused it to run amuck under certain circumstances, issuing requests in a tight loop."; "Kubelet Amuck ... we would like only the guilty individual to suffer, not all its peers." |
| 24 | Kubernetes | KEP-1040 (seats and borrowing) | adr | checked at master | 2026-09-25 | https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/1040-priority-and-fairness/README.md | The unit of admission is an execution seat, allocated by shares and lendable between priority levels by percentage. | "This is the number of execution seats available at this priority level."; "NominalCL(i) = ceil( ServerCL * ACS(i) / sum_acs )"; "LendableCL(i) = round( NominalCL(i) * lendablePercent(i)/100.0 )"; "BorrowingCL(i) = round( NominalCL(i) * borrowingLimitPercent(i)/100.0 )". |
| 25 | Netflix | concurrency-limits README | source | checked at master | 2026-09-25 | https://github.com/Netflix/concurrency-limits | The argument against RPS as the unit: stress-tested RPS limits go stale under autoscaling; measure concurrency with Little's law and adapt it with TCP congestion algorithms; partition the limit between traffic classes. | "in large distributed systems that auto-scale this value quickly goes out of date and the service falls over by becoming non-responsive"; "Limit = Average RPS * Average Latency"; "we borrow from common TCP congestion control algorithms by equating a system's concurrency limit to a TCP congestion window"; example partitions live traffic 0.9 and batch 0.1. |
| 26 | gRPC | gRFC A6: client retries (retry throttling) | adr | checked at master | 2026-09-25 | https://github.com/grpc/proposal/blob/master/A6-client-retries.md | The client is enlisted in refusal: retries are throttled by a per-server token count and stop below half the maximum; servers can push back with an explicit do-not-retry. | "Every failed RPC will decrement the token_count by 1. Every successful RPC will increment the token_count by tokenRatio."; "If token_count is less than or equal to the threshold, defined to be (maxTokens / 2), then RPCs will not be retried until token_count rises over the threshold."; example config maxTokens: 10, tokenRatio: 0.1. |
| 27 | AWS | botocore adaptive retry mode (source) | source | checked at develop | 2026-09-25 | https://github.com/boto/botocore/blob/develop/botocore/retries/adaptive.py | The AWS SDK ships a client-side rate limiter that runs a CUBIC congestion controller against throttling responses, wired into every request send and retry decision. | rate_adjustor = throttling.CubicCalculator(starting_max_rate=0, ...); token_bucket = bucket.TokenBucket(max_rate=1, clock=clock); throttling_detector = standard.ThrottlingErrorDetector(...); registered on before-send and needs-retry events. |
| 28 | nginx | ngx_http_limit_req_module.c | source | checked at master | 2026-09-25 | https://github.com/nginx/nginx/blob/master/src/http/modules/ngx_http_limit_req_module.c | The most widely deployed limiter is an in-process leaky bucket: per-key excess state in a shared memory zone, no external store, no cross-node view. |
Struct fields ngx_uint_t excess; (node state) and the request path computing excess per key (lines 27 and 200 of the module at master; fetched raw). |
| 29 | Brandur Leach | redis-cell README (GCRA Redis module) | source | checked at master | 2026-09-25 | https://github.com/brandur/redis-cell | GCRA gives a rolling window without a drip process, returns the refusal metadata (limit, remaining, retry-after) in one atomic call, and takes limits per invocation so they reconfigure on the fly. | "Implements the fairly sophisticated generic cell rate algorithm (GCRA) which provides a rolling time window and doesn't depend on a background drip process."; CL.THROTTLE <key> <max_burst> <count per period> <period>; response includes "The number of seconds until the user should retry ... Equivalent to Retry-After"; "Rate limiting parameters are provided with every invocation so that limits can easily be reconfigured on the fly." |
| 30 | Rack::Attack | rack-attack README | source | checked at master | 2026-09-25 | https://github.com/rack/rack-attack | The Rails-ecosystem middleware's shape: throttles are named, keyed by a caller-supplied discriminator, with state in a pluggable cache defaulting to Rails.cache. | "Throttle state is stored in a configurable cache (which defaults to Rails.cache if present)"; "make your ruby-block argument return the discriminator. This discriminator is how you tell rack-attack whether you're limiting per IP address, per user email or any other." |
| 31 | Go team | golang.org/x/time/rate (pkg.go.dev) | vendor | checked | 2026-09-25 | https://pkg.go.dev/golang.org/x/time/rate | The standard Go limiter exposes three distinct refusal semantics in its API: drop now (Allow), schedule for later (Reserve), or block the caller (Wait). | "Package rate provides a rate limiter."; API surface: func (lim *Limiter) Allow() bool, func (lim *Limiter) Reserve() *Reservation, Wait, SetBurst, SetLimit (package index). |
| 32 | npm ecosystem | express-rate-limit (registry record) | source | first 2014-12-11, latest 2026-08-29 | 2026-09-25 | https://registry.npmjs.org/express-rate-limit | Per-endpoint refusal has been commodity middleware for over a decade; the registry's first publish is December 2014 and the package is still releasing. | Registry JSON: description "Basic IP rate-limiting middleware for Express. Use to limit repeated requests to public APIs and/or endpoints such as password reset."; time.created 2014-12-11; latest 8.7.0 published 2026-08-29. |
| 33 | Python ecosystem | limits (PyPI record) | source | v5.8.0 current | 2026-09-25 | https://pypi.org/pypi/limits/json | The Python ecosystem's shared limiter library, current at 5.8.0. | PyPI JSON: name limits, version 5.8.0, summary "Rate limiting utilities". |
| 34 | Rust ecosystem | governor (crates.io index record) | source | v0.10.4 current | 2026-09-25 | https://index.crates.io/go/ve/governor | The Rust ecosystem's shared limiter crate, current at 0.10.4. | Index record: latest version 0.10.4 (sparse index line, fetched 2026-09-25). |
Named but not verifiable in this session
These are deliberately cited without links in the prose, marked as unreachable, and carry no evidence cards: Stripe, "Scaling your API with rate limiters" (stripe.com blog, 2017); GitHub, "How we scaled the GitHub API with a sharded, replicated rate limiter in Redis" (github.blog, 2021); Cloudflare, "How we built rate limiting capable of scaling to millions of domains" (blog.cloudflare.com, 2017); Figma, "An alternative approach to rate limiting" (figma.com blog); Raghavan et al., "Cloud Control with Distributed Rate Limiting" (SIGCOMM 2007). Their hosts were blocked by the session's egress policy; search results confirmed the titles exist but the pages were not fetched, so nothing in the guide depends on their contents.