Rate Limiting in Practice

Definition

Constraining the rate at which a caller may consume a resource, rejecting or delaying requests beyond the limit.

Why it matters

Three distinct purposes, and conflating them produces bad designs:

Protection. Preventing any one caller from consuming capacity everyone needs — including callers who are not malicious but have a retry loop with no backoff.

Fairness. Sharing finite capacity across tenants so one large customer's batch job does not degrade everyone.

Commercial. Enforcing plan entitlements, where the limit is the product.

These want different algorithms, different scopes and different responses. A protection limit should be generous and fail fast; a commercial quota should be exact and clearly reported.

Implementation patterns

Fixed window. Count per calendar interval. Simplest, and permits double the intended rate at a window boundary — a caller can spend a full quota at the end of one window and again at the start of the next.

Sliding window. Smooths the boundary effect at the cost of more state.

Token bucket. Tokens accrue at a fixed rate up to a cap; each request consumes one. Allows a controlled burst while limiting the sustained rate, which matches real client behaviour far better than a hard cap. The usual choice for public APIs.

Leaky bucket. Enforces a strictly smooth output rate — right when protecting a fragile downstream that cannot absorb bursts at all.

Concurrency limiting rather than rate limiting: bound in-flight requests instead of requests per second. Frequently the better control, because it maps directly to resource occupancy.

Failure scenarios

Independent enforcement across instances. Twenty gateway instances each enforcing "100 per minute" is a limit of 2,000. Correctness requires shared state, which puts a datastore in the request path — so the practical compromise is local counters with periodic reconciliation and a deliberately accepted margin of error. The failure is not the approximation; it is treating an approximate control as an exact one.

The same problem at the edge, worse. A limit enforced independently at fifty edge locations is fifty times the limit, usually discovered during an abuse incident.

IP-based scoping, which penalises everyone behind a shared address and is trivially evaded.

No feedback to clients, so well-behaved callers cannot pace themselves and retry immediately, making the overload worse.

Industry example

The two-tier arrangement is what most large platforms converge on: a generous approximate limit at the edge to absorb volumetric abuse cheaply, and an accurate limit at the origin for anything contractual or security-relevant. Cloudflare's position in front of origins is precisely this division of labour — the edge handles volume, the origin handles correctness.

Trade-offs

Exact enforcement costs a shared-state round trip on every request. Approximate enforcement is fast and lets some callers exceed the limit. Choose per limit, and document which is which.

Generous limits protect capacity and permit abuse; tight limits frustrate legitimate bursty clients. Token bucket is the compromise most APIs settle on.

Interview question

You enforce 100 requests per minute per user at the edge across 40 locations. A user makes 3,000. Explain and fix.

The candidate should identify independent local counters and propose the two-tier design — approximate at the edge for volumetric protection, accurate at the origin for the limit that matters. A strong answer generalises: anything with a global invariant does not belong at the edge without coordination.