Rate Limit Design

The algorithm choice has visible consequences. A fixed window counter is simplest and allows double the intended rate at a window boundary, since a client can spend a full quota at the end of one window and again at the start of the next. A sliding window smooths that. A token bucket allows a controlled burst while limiting the sustained rate, which matches real client behaviour better than a hard cap and is the usual choice for public APIs. A leaky bucket enforces a strictly smooth output rate, which suits protecting a fragile downstream.

The scope decision matters as much: per user, per tenant, per API key, per endpoint, or per IP — where IP-based limiting is the crude fallback that penalises everyone behind a shared address and is trivially evaded.

The distributed problem is the one that is easy to get wrong: limits enforced independently at each of twenty gateway instances give twenty times the intended limit. Correctness requires shared state, which puts a datastore in the request path, so the practical compromise is local counters with periodic reconciliation and a deliberately accepted margin of error.

The client-facing behaviour completes the design: return 429 with a Retry-After, publish the limit, remaining quota and reset time in headers so well-behaved clients can pace themselves, and distinguish throttling from failure so clients back off rather than retrying immediately and making it worse.