A public API is being overwhelmed by a handful of clients. Where should rate limiting live, which algorithm, and how should limits differ across users, tenants, endpoints and infrastructure?

What is being tested

Whether you think in layers rather than reaching for a single limiter, and whether you distinguish protecting the business from protecting the infrastructure.

Where it should live: all of these, doing different jobs

At the network edge. Volumetric attacks and obvious abuse should be dropped as close to their source as possible, before consuming bandwidth or connection capacity. This is the model Cloudflare-style edge networks exist to provide: mitigation distributed across many locations rather than concentrated at a single scrubbing point, because the attack traffic never has to converge on one place.

At the API gateway. Per-client business limits — plan quotas, per-tenant fairness, per-endpoint costs. This is where the policy lives, because it is business logic about who is entitled to what.

In the service. A concurrency limit as a last line of defence. Even if everything above fails, the service must not accept more concurrent work than it can process; beyond that it should shed fast. This protects it from internal callers too, which the gateway never sees.

At the database and downstream. Connection pool limits per caller, so one workload cannot exhaust the shared resource.

Each layer protects against a different failure. A single limiter at the gateway does nothing when the attack is volumetric and nothing when the pressure is internal.

Algorithm

Token bucket for most API limiting: it allows a burst up to the bucket size while bounding the sustained rate, which matches how legitimate clients actually behave — bursty but bounded. A fixed window is simpler and has an ugly edge, allowing double the limit across a window boundary. A sliding window log is precise and expensive. A leaky bucket smooths output and is right when the downstream needs an even rate rather than a bounded average.

Concurrency limiting rather than rate limiting is underused and often the better tool for protecting a service, because what actually saturates is in-flight work, not requests per second. A slow downstream raises concurrency without raising the request rate, and a rate limiter does not notice.

Dimensions

Weight limits by cost, not by count. One request that triggers a full table scan is not equivalent to a thousand key lookups. Assigning a cost to each endpoint and limiting on cost units consumed is more work and considerably more accurate.

Behaviour when limited

• Return 429 with a Retry-After header. This is not a courtesy; it is what stops a well-behaved client from retrying immediately and making it worse.
• Include limit, remaining and reset headers so clients can self-regulate.
• Never simply drop legitimate traffic silently — a client that gets no answer retries, which is the opposite of what you want.
• Prefer shedding cheap work to shedding expensive work when overloaded, and shed at the earliest possible point, because a request rejected after the database query has already cost you the query.

Distributed enforcement

Counters must be shared across gateway instances, usually in a fast central store. That store is now on the critical path, so it needs a defined behaviour when unavailable: fail open for availability, fail closed for protection. Choose deliberately per endpoint — failing open on a login endpoint during an outage is how credential stuffing succeeds.

A common compromise is local counters with a share of the global budget per instance, periodically reconciled. Slightly imprecise, no hot dependency on the shared store, and precision was never the requirement.

What a strong answer adds

Distinguishing abuse from a legitimate customer who has grown. The response to the first is blocking; the response to the second is a conversation and a higher tier. A limiter that cannot tell them apart will eventually throttle your best customer during their most important week.