Design rate limiting for a multi-tenant API where a single customer's traffic spike currently degrades service for everyone.

Name the problem precisely

This is the noisy neighbour problem. Rate limiting is one control for it, and on its own it is incomplete — a customer within their limit can still saturate a shared resource if the limit was set without reference to actual capacity.

Algorithm

Token bucket. Tokens refill at a fixed rate up to a capacity; an idle client accumulates them and may burst, while sustained load is capped at the refill rate.

Real traffic is bursty. An algorithm that rejects a burst the client could legitimately afford produces avoidable failures.

Alternatives and why not: fixed window is trivial and permits a double burst across the boundary. Sliding window fixes that with more state. Leaky bucket smooths output to a constant rate, which suits protecting a downstream that cannot absorb bursts at all.

Dimensions, because one limit is never enough

Limit by tenant for fairness, by endpoint because an expensive endpoint warrants a lower limit than a cheap one, and by cost where request costs vary widely — charging tokens proportional to work rather than counting requests.

Add a global limit as the backstop that protects the service when the sum of per-tenant limits exceeds capacity, which it usually does.

Implementation

Shared counters across instances — otherwise the effective limit is the configured limit × instance count. A central store with atomic operations, or approximate local counters synchronised periodically, trading accuracy for latency.

Enforce at the gateway, so rejected requests never consume backend capacity.

The client contract

429 for this client exceeding its limit; 503 for general overload. They mean different things and clients should treat them differently.

Retry-After on every rejection — without it the client guesses, and a guessing client typically retries immediately, deepening the overload the rejection was meant to relieve.

Limit, remaining and reset headers on every response, so clients can stay within the limit rather than discovering it by failing.

What rate limiting does not solve

Fair queuing for requests within limits, load shedding when the service is genuinely overloaded (shedding by priority, not at random), concurrency limits as well as rate limits — one long-running request holds resources regardless of rate — and bulkheading, giving tenants or tiers separate resource pools.

A complete answer names rate limiting as one layer of overload protection and identifies which others this platform needs.