Reliability & Operations 25 Sep 2026 29 min read

Deciding who gets told no

How production systems refuse requests: the classifier, counter, exception plane and failure posture behind rate limiting and admission control, read from the repository record.

Read the field guide → The evidence ledger

A field guide to request-level admission control, built from GitLab's public incident tracker and six years of its rate-limiting change record, the Envoy ratelimit service and filter contract, Kubernetes API Priority and Fairness, Google's abandoned Doorman, Netflix's adaptive concurrency limits, and the self-throttling clients gRPC and the AWS SDK ship. It reconstructs the five-part refusal machinery, the five decisions that shape it, and the four failure classes the incident record actually contains.

The finding that surprised me

In every incident in the corpus the counting algorithm was innocent: what failed was the classifier that decides who a request is, the exception plane that decides who is exempt, a limit that sat disabled, or the limiter faithfully amplifying someone else's failure, and at GitLab the bypass path carried two orders of magnitude more traffic than enforcement ever touched.

What you get out of it

The classifier is authentication re-implemented, and its defects are latent by design: a wrong verdict changes a threshold rather than an outcome, so GitLab's runner misclassification hid for months and then cost 39 hours.
The exception plane (bypass headers, allowlists, per-customer disables) is the highest-traffic, least-tested path: roughly 2M requests per 5 minutes bypassed GitLab's limiter, and its breakage was detected by a customer, not a monitor.
Failure posture is a config field most teams never consciously choose: Envoy defaults to fail open, ships a percentage dial between open and closed, and Doorman named all three postures (pessimistic, optimistic, safe capacity) in 2016.
Requests are the wrong unit when cost varies or capacity moves: Kubernetes replaced max-inflight with seats and shares, and Netflix derives concurrency limits from Little's law with TCP congestion algorithms; both rejected RPS independently.
Rollout mode is architecture, not process: GitLab tuned thresholds in dry run for two months (catching transposed limits before users did) and is migrating limiters in 2026 by running both in parallel and recording decision divergence.

Scope

Why this, now. GitLab is replacing its production rate limiter in 2026 by running two limiters in parallel on a dedicated 18-node Redis cluster, and its July 2026 Severity 1 shows what one disabled throttle still costs; the machinery of saying no is under live rework at the operators who publish.

What it does not cover. Volumetric DDoS defence and WAF rules, billing quotas, utilisation-triggered load shedding (see the retry-storms guide in this series), and the engineering-blog, paper and talk evidence layers, whose hosts were unreachable under this session's network policy; the well-known Stripe, Cloudflare, Figma and GitHub accounts are named but carry no claims.

Open the field guide → Self-contained: it loads nothing at read time, follows your system theme, and prints cleanly.

Keep digging

Other field guides

Reliability & Operations

Deciding who gets told no

What you get out of it

Scope

Other field guides

Adding capacity under fire

The router finds out last: taking a server out of service without dropping requests

A backup is a claim, a restore is the proof