How production systems refuse requests: the classifier, the counter, the exception path and the failure posture behind every 429, reconstructed from GitLab's public incident tracker and six years of its rate-limiting record, the Envoy and Kubernetes admission machinery, Google's abandoned Doorman, and the self-throttling clients AWS and gRPC ship. After reading it you can design the refusal path of an API, and you will know that the counting algorithm is the part least likely to hurt you.
A shared service with finite capacity, many independent clients, and a verdict that must be produced per request, in microseconds, on servers that share no memory.
Strip the technology names away and the problem is this: some of your callers can each, alone, generate more work than the whole system can absorb. You must decide, for every request, whether to do the work or refuse it; the decision has to be roughly consistent across hundreds of processes; and it has to cost almost nothing, because it runs before every piece of work you actually get paid for. Every system in this guide answers with the same five parts: a classifier that decides who a request belongs to, an exception plane that decides who is exempt, a counter against a threshold, a verdict contract that tells the client what to do about the refusal, and a rollout mode that lets you watch a limit before you enforce it.
The surprise in this corpus, and the reason to read on: in four production incidents at GitLab between 2022 and 2026, spanning a Severity 1 outage with 9.49 million failed requests, the counting was never the thing that failed. The classifier lied, the exception plane broke, a limit sat disabled, or the limiter faithfully amplified a failure that started somewhere else. The algorithm section of every rate-limiting tutorial, the token buckets and sliding windows, is the one part of this machinery with a clean production record. Design reviews spend their time on the wrong box.
Who the record comes from. GitLab.com is the spine of this guide because it is the only large SaaS operator whose incident reviews, threshold-tuning worksheets, architecture blueprints and merge request arguments about rate limiting are all public and cross-linked. The supporting cast: the Envoy ratelimit service (Lyft's design, now the de facto open-source global limiter), Kubernetes' API Priority and Fairness (KEP-1040), YouTube's Doorman, Netflix's concurrency-limits, the gRPC retry throttle, the AWS SDK's adaptive client limiter, and nginx's leaky bucket, each read from its own repository.
This guide covers request-level admission control: deciding per request, per client, whether to serve. It does not cover volumetric DDoS defence, WAF rules, billing quotas, or utilisation-triggered load shedding (for the overload feedback loops that shedding breaks, see this series' retry-storms guide). One honesty note: this session's research environment could reach repository hosts and package registries but not engineering blogs, paper archives or video hosts. The well-known Stripe, Cloudflare, Figma and GitHub blog accounts of their limiters, and the SIGCOMM 2007 distributed rate limiting paper, are named where relevant but were not fetched, are not linked, and carry no claims here. The blog and paper layers of this topic exist; this page is built from the layer below them, and says so rather than citing from memory.
One shape recurs across every system in the corpus; the divergences are where the counter lives and what it counts.
The reference shape has admission control at three altitudes, and mature operators run all
three at once. At the edge, a proxy holds coarse per-IP rules and the emergency knobs:
when GitLab took a Severity 1 outage from an unthrottled package endpoint in July 2026, the
mitigation deployed mid-incident was a Cloudflare rule, "then codified into Terraform" per the
incident
review. In the application, middleware sees authenticated identity and applies named
throttles per user, per IP, per endpoint class; GitLab runs 21 named throttles in its Rack
middleware (MR
233066) plus a second per-action limiter,
ApplicationRateLimiter,
inside the same codebase. Behind everything sits the counter store: GitLab gives counting
its own failure domain, a Redis Cluster named redis-cluster-ratelimiting that grew
to six shards and 18 nodes in July 2026
(change #22496).
Turns a request into a limit key: which throttle applies, and what identity to count it under. It re-implements authentication, which is exactly where it goes wrong: GitLab's middleware "attempted to look up the user associated with the PAT, and finding none, assumed this was an unauthenticated request" while the API layer authenticated the same request fine, for months.
Record: GitLab #19447, rack-attack discriminators, Envoy descriptors
Fixed windows in Redis (GitLab), leaky bucket in shared memory
(nginx's per-key excess), token bucket in-process (Go's
x/time/rate), GCRA when you want rolling windows and a Retry-After from one
atomic call (redis-cell). Every over-limit key is also cached locally in Envoy's service so
the store isn't re-asked about a client it already refused.
Record: nginx, x/time/rate, redis-cell
Allowlists, bypass headers, dry-run flags, per-customer disables. At GitLab this plane carried roughly two million requests per five minutes, two orders of magnitude more traffic than enforcement ever touched. It is configured in load balancers, environment variables and database columns at once, which is why it breaks.
Record: GitLab admin docs, #18174
What the refused client is told: HTTP 429 with Retry-After, rate-limit headers, or gRPC's
explicit server pushback ("retry after a given delay or ... not retry at all"). The Go
limiter API is a compact statement of the three possible semantics: Allow
(refuse now), Reserve (admit later), Wait (block the caller).
Record: gRFC A6, x/time/rate
Serious platforms do not leave backoff to the caller's goodwill; they ship it. The AWS SDK runs a CUBIC congestion controller against throttling responses inside botocore. gRPC stops its own retries when a per-server token count falls below half its maximum. Doorman's 2016 argument for this still stands: a refused request has already "burnt the network capacity and done some processing".
Record: botocore adaptive.py, gRFC A6, Doorman design
Dry run and shadow are not conveniences, they are part of the architecture. GitLab ran its first throttles in dry-run mode for two months before enforcing; Envoy's ratelimit service has a global shadow mode "to introduce rate limiting into an existing service landscape"; and GitLab's 2026 limiter replacement runs old and new limiters in parallel, recording decision divergence.
Record: change #3034, ratelimit README, #22496
The deepest divergence across the corpus is not where the counter lives but what it counts. The Rack, nginx and Envoy families count requests per unit time. Kubernetes and Netflix both concluded that requests are the wrong unit. KEP-1040 replaced the API server's flat max-in-flight with "execution seats" allocated by shares and lendable between priority levels, because a LIST costs a different amount than a GET and "one subset of the request load crowds out other parts". Netflix's concurrency-limits library makes the same move from the other end: an RPS limit derived from a stress test "quickly goes out of date" under autoscaling, so measure concurrency (Little's law: limit equals average RPS times average latency) and adapt it with TCP congestion algorithms. Two teams, two directions, one conclusion: when request cost varies or capacity moves, count occupancy, not arrivals.
Five forks, each with the condition that flips it. The first two get all the interview attention; the last three cause the incidents.
failure_mode_deny defaults to false: on limiter failure, traffic flowsfailure_mode_deny_percent: deny a configured fraction while the service is down| Decision | Chosen in the record | Rejected | Because | Evidence |
|---|---|---|---|---|
| Unit | Requests per window at the API edge; seats or concurrency inside the platform | One unit everywhere | RPS is a sellable contract; occupancy tracks true cost | KEP-1040, Netflix |
| Counter placement | Local, or dedicated shared store, or 20 ms decision service | Exact global counting | Bursts overwhelm a global service; stage local buckets in front | Envoy arch docs |
| Failure posture | Fail open by default, fractional deny available | Fail closed by default | A limiter blip must not become a platform outage | rate_limit.proto, Doorman |
| Client role | Ship backoff in the SDK | Cooperative capacity leases | Leases need owned clients; Doorman stalled in 2016 | gRFC A6, doorman repo |
| Exceptions | Bypass and allowlists (widespread) | Higher limits that keep counting | Operator's own blueprint calls bypass a risk to the bypassed | GitLab blueprint |
| Rollout | Dry run, then enforce; shadow for replacements | Enforce on deploy | Dry run caught transposed thresholds before users did | issue 656 |
Four incidents, four failure classes, one pattern: the counter is innocent. Every failure lives in identity, exceptions, configuration, or amplification.
These four are all from one operator, which is a limit of the public record, not of the lesson: GitLab is the only organisation publishing rate-limiter incidents at this depth. Treat the classes, not the company, as the finding. Each class is a different answer to the question "what did the limiter believe that was false?"
What is missing from the public record matters as much: no postmortem in this corpus attributes an outage to the counting algorithm being wrong, and none attributes one to a counter store failing while configured fail-open. Either that failure mode is rare, or, like fail-open failures generally, it is invisible: when the limiter silently admits everything and the backend survives, nobody writes it up. Envoy's July 2026 commit adding READONLY handling after Redis failovers shows the store-failure path is being hit; the absence of incident reports about it is a detection gap, not evidence of safety.
Everything here is measured or read from a primary artifact; nothing is a vendor benchmark. Dates matter: thresholds move.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| First enforced thresholds | 500/min unauth IP; 2,000/min auth API; 1,000/min auth web | GitLab.com | After two months of dry-run tuning | 2021-01 | issue 656 |
| Dry-run period before enforcement | ~2 months | GitLab.com | 2020-11-20 dry run to 2021-01-18 enforcement; caught transposed limits | 2021 | issue 656 |
| Named throttles in middleware | 21 | GitLab.com | Rack::Attack alone, excluding ApplicationRateLimiter and edge rules | 2026-04 | MR 233066 |
| Dedicated counting cluster | 6 shards, 18 nodes (c2-standard-4) | GitLab.com | Redis Cluster for rate limiting only; grown for shadow rollout doubling counter ops | 2026-07 | change #22496 |
| Traffic on the bypass path | ~2.0M requests / 5 min | GitLab.com | Allowlisted traffic normally skipping the limiter; measured when it broke | 2024-06 | incident #18174 |
| Decision budget for a remote verdict | 20 ms default | Envoy | Timeout on the gRPC rate limit call; expired timeout admits unless deny is configured | 2026-09 (checked) | rate_limit.proto |
| Client retry throttle | maxTokens 10, tokenRatio 0.1, stop below maxTokens/2 | gRPC | Example config; failures cost 1 token, successes restore 0.1 | 2026-09 (checked) | gRFC A6 |
| Seat allocation formula | NominalCL(i) = ceil(ServerCL × ACS(i) / sum_acs) | Kubernetes | Concurrency shares per priority level, with lendable and borrowing percentages | 2026-09 (checked) | KEP-1040 |
| Cost of the missing limit | 9.49M 503s / 15 min; 77 min S1 | GitLab.com | One user, one unthrottled endpoint, lock convoy | 2026-07 | INC-12449 |
| Latency of a classifier defect | months latent, 39.15 h to resolve | GitLab.com | Misclassification changed a threshold, not an outcome, until load crossed it | 2025-03 | incident #19447 |
| Life of the cooperative limiter | 70 commits, last 2016-05-02 | YouTube / Doorman | Open-sourced global client-side limiter; planned clients never shipped | 2026-09 (checked) | doorman repo |
| Age of commodity middleware | first publish 2014-12-11; v8.7.0 2026-08-29 | npm / express-rate-limit | Per-endpoint refusal has been a package install for over a decade | 2026-09 | npm registry |
The GitLab thresholds are that operator's numbers for that traffic in 2021; they are a worked example of how to derive limits (from observed percentiles, in dry run), not values to copy. The 20 ms Envoy budget and the gRPC token numbers are shipped defaults, which is a statement of what their authors consider sane, not a measurement. Nobody in this corpus publishes the steady-state cost of counting; the closest artifact is GitLab's 18 reserved c2-standard-4 nodes, which is a lower bound on what fleet-wide fairness costs at their scale as of mid-2026.
Every source behind this page, graded. The full ledger, with one quoted claim per row, ships beside this file as sources.md. Blog, paper and talk tiers are absent because their hosts were unreachable in this research session; that absence is stated, not papered over.
An HAProxy change broke the rate-limit bypass; ~1.985M requests per 5 minutes that normally skip the limiter were refused. Impact concentrated on allowlisted customers. Detected by a support ticket.
Rack::Attack classified runner job-updates as unauthenticated because it tried the wrong token first, while the API authenticated them fine. Latent for months; 39 hours to resolve once load crossed the lower threshold.
One user's package-download burst, an UPDATE on every GET, a PostgreSQL lock convoy, and a Severity 1: ~9.49M 503s in the 15-minute peak. The covering throttle existed and was disabled in production.
A secrets migration broke OAuth sign-in; the redirect loop tripped the login throttle, and for 390 minutes users saw "too many requests" instead of the real failure.
The operator's own assessment: limits defined in many places, enforced by five opaque systems, and marquee customers exempted by full bypass, which the blueprint itself names a risk to those customers.
Four years after the blueprint: seven-plus limiter implementations, inconsistent dry run and bypass, new endpoints unlimited by default. The unification design centres on a call-site name plus request identifier, i.e. on identity, not on counting.
The dated table of every threshold change from first dry run (2020-11-20) to enforcement (2021-01-18), including the week the two authenticated limits were accidentally transposed, caught by dry-run telemetry.
The dedicated redis-cluster-ratelimiting grows to 18 reserved nodes, sized ahead of a shadow rollout that runs old and new limiters in parallel, roughly doubling counter operations.
A change giving each of the 21 throttles its own dry-run flag and allowlist columns, closed without merging as the plan was re-cut around the labkit unification. The exception plane's own configuration is still in motion in 2026.
The merged repair for incident #19447: the limiter's token lookup returns blank for CI build tokens instead of raising, so authentication falls through to the job token.
The operational contract of the exception plane: a bypass header the edge must erase on untrusted traffic, user-ID allowlists, per-throttle dry run via environment variable, and a documented recovery for locking your own administrators out.
A gRPC service over Redis keyed by domain and descriptors, with a local cache of over-limit keys so refused clients stop costing Redis reads, and a global shadow mode for introducing limits into an existing landscape.
failure_mode_deny defaults to false (fail open), the remote verdict gets 20 ms, and failure_mode_deny_percent lets an operator deny a fraction while the limiter is down: a shipped dial between open and closed.
The limiter's own store failover is live operational work in 2026: pooled connections can stick to a demoted primary and answer READONLY until explicitly closed.
When global counting earns its keep: many hosts converging on few, latency too low for circuit breaking to bite. And the two-stage pattern: local token buckets absorb bursts so the global service survives them.
The cooperative alternative: clients lease capacity from a server hierarchy ahead of use, because refusal burns the network and compute you already spent. Names three degraded postures: pessimistic, optimistic, and a configured safe capacity.
70 commits, the last on 2016-05-02; README status "Alpha quality"; the planned C++ and Python clients never appeared. The most architecturally ambitious limiter in the corpus is also the shortest-lived.
The motivation section is a failure catalogue: Deployment of Doom, Kubelet Amuck, self-maintenance crowded out. The design replaces undifferentiated max-inflight with seats, shares, and borrowing between priority levels.
Stress-tested RPS limits go stale as systems autoscale; the library measures the concurrency limit instead, adapting it with TCP congestion algorithms (Vegas, Gradient2) and partitioning it between traffic classes, e.g. live 90% / batch 10%.
The client refuses on the server's behalf: a per-server token count, minus one per failure, plus 0.1 per success, and no retries below half the maximum. Servers can push back with an explicit do-not-retry.
The SDK carries a client-side rate limiter: a token bucket whose fill rate is set by a CUBIC congestion calculator reacting to throttling errors, hooked into every request send and retry decision.
The most-deployed limiter is a leaky bucket in C: per-key excess tracked in a shared memory zone, per node, no external store, no cross-fleet view. The baseline every fancier design is implicitly compared against.
The generic cell rate algorithm gives a rolling window with no drip process; CL.THROTTLE returns the verdict, the remaining budget and the retry-after in one atomic call, with limits passed per invocation so they change without redeploys.
The standard Go limiter's API is a taxonomy of refusal semantics: Allow (drop it now), Reserve (admit it later), Wait (make the caller absorb the delay). Most designs pick one without noticing the other two exist.
Every packaging ecosystem carries a standard limiter: npm's express-rate-limit (first published 2014-12-11, v8.7.0 in August 2026), Python's limits (5.8.0), Rust's governor (0.10.4). The algorithms are solved; everything above them in this guide is not.
The two limiter families inside one Rails codebase: middleware throttles keyed by a caller-supplied discriminator with state in Rails.cache, and a per-action limiter with its own thresholds and allowlist arguments. GitLab's epic counts "5+ other" beyond these.
Six rungs. The crossing from toy to real is rung four, where you break your own limiter and have to pick a posture.
Wrap one endpoint of a toy API with a token bucket (use your ecosystem's standard library: x/time/rate, limits, governor). Drive it with a load generator at 2x the limit and plot admitted vs refused per second.
Done when: a burst of exactly burst-size is admitted, steady state holds the configured rate, and refusals carry Retry-After. Teaches: burst vs rate are independent knobs, and refusal is an API response you design, not an error.
Run two instances behind a load balancer, move the counter to Redis with INCR plus EXPIRE, then make the read-check-write atomic (Lua or GCRA via redis-cell). Compare per-node counting (limit/2 each) against the shared counter under skewed load.
Done when: you can state the over-admission of each design under a 90/10 traffic skew, with numbers. Teaches: what fleet-wide fairness costs, and what per-node counting silently tolerates.
Add authentication, then three named throttles: per-IP unauthenticated, per-user authenticated, per-endpoint expensive-path. Log every verdict with throttle name and identity, GitLab-style.
Done when: a request with a bad token and a request with no token hit different throttles, and you can grep the log for who was refused by which rule. Teaches: the classifier is authentication re-implemented, with all its edge cases.
Stop Redis mid-load-test. Implement all three Doorman postures behind a flag: optimistic (admit all), pessimistic (refuse all), safe capacity (per-node local bucket at a fraction of the limit). Add Envoy's refinement: a percentage dial between open and closed.
Done when: you can produce a table of over-admission and false-refusal for each posture during a 60-second store outage. Teaches: the posture is a business decision you must be able to defend per endpoint; login and checkout want different answers.
Add a dry-run flag per throttle that logs would-refuse without refusing. Replay a day of access logs, and derive thresholds from the observed per-identity percentiles, the way GitLab's issue 656 worksheet does.
Done when: you can name how many real identities your proposed limit would have refused yesterday, before any user feels it. Teaches: thresholds are empirical; the two-month gap between GitLab's dry run and enforcement was the product.
Add a bypass header stripped at your edge, a per-throttle allowlist, and a counter for bypassed traffic. Alert when bypass volume shifts by more than N% hour over hour. Then break the edge stripping deliberately and watch the alert fire.
Done when: a synthetic canary detects both failure directions: exempt traffic getting counted, and untrusted traffic self-exempting. Teaches: the exception plane is the highest-traffic, least-tested path; GitLab's 2024 incident was exactly this, undetected by monitoring.
The queries that found this material, adapted for a reader with an open network. The GitLab tracker searches work today, verbatim.
site:gitlab.com gl-infra/production "incident review" "rate limit"RackAttack search in gitlab-com/gl-infra/production issues"429" "incident review" allowlist bypass"rate limiter" postmortem "fail open"repo:envoyproxy/ratelimit is:pr is:closed is:unmergedfailure_mode_deny site:github.com"shadow mode" OR "dry run" throttle merge requestpath:keps "priority and fairness" alternatives"safe capacity" rate limiting leaseGCRA "generic cell rate" redis"retry throttling" tokenRatio"concurrency limit" "Little's law" adaptivestripe "scaling your api with rate limiters"cloudflare "counting things" rate limitinggithub "sharded, replicated rate limiter" redis"distributed rate limiting" SIGCOMM 2007 RaghavanNamed but not fetched in this session, and therefore carrying no claims above: Stripe's "Scaling your API with rate limiters" (2017), GitHub's "How we scaled the GitHub API with a sharded, replicated rate limiter in Redis" (2021), Cloudflare's "How we built rate limiting capable of scaling to millions of domains" (2017), Figma's "An alternative approach to rate limiting", and Raghavan et al., "Cloud Control with Distributed Rate Limiting" (SIGCOMM 2007). Their hosts were outside this session's network policy; treat them as the next layer of reading, not as sources of this page.