An AI platform serves customers from individual developers to large enterprises, where a single request can be a thousand times more expensive than another. How should rate limiting be designed?

Why request counting is the wrong unit

A limit of "1000 requests per minute" is meaningless when one request generates ten tokens and another generates ten thousand. Two customers within the same request limit can impose vastly different loads, so the limit protects nothing and is unfair to whoever uses it modestly.

The unit must reflect the actual scarce resource: for inference, that is accelerator time, which correlates with tokens processed and generated.

The design

1. Limit on the resource, not on requests. Tokens per minute, or a cost-weighted unit combining input and output. Requests per minute remains useful as a secondary guard against pathological client behaviour, but it is not the primary control.

2. Token buckets rather than fixed windows. Fixed windows produce a stampede at the boundary — every client retries at the top of the minute — and refuse legitimate bursts that the system could easily absorb. A token bucket permits bursting up to a bucket size while enforcing a sustained rate, which is both fairer and smoother.

3. Multiple simultaneous limits. Concurrency (in-flight requests), sustained rate, and burst capacity are different constraints. Concurrency matters most for expensive long-running requests, because a customer with high concurrency occupies accelerators regardless of their token rate.

4. Tiers with different shapes, not just different numbers. An enterprise customer with predictable bursts needs a large bucket; a batch workload needs sustained throughput and tolerates queueing. Selling the same limit shape at different sizes forces every workload into one pattern.

5. Priority classes, so shedding is not uniform. When capacity is short, batch and low-priority traffic is shed before interactive traffic. A single limit per customer cannot express "this request matters more than that one".

6. Limits visible in every response. Remaining capacity, limit, reset time, and a Retry-After on rejection. A client that can see its budget paces itself; a client that cannot, retries blindly — which is amplification.

The enterprise burst problem

Legitimate customers exceed normal limits during predictable events — a quarterly batch, a product launch, a migration. Handling this badly is how platforms lose enterprise customers.

The mechanisms that work:

• Pre-arranged temporary increases, requested through an API or a console rather than through a support ticket.
• Burst credit accumulated during under-use, so a customer who is quiet for hours can spend that allowance.
• A queued tier where over-limit requests are accepted with a longer deadline instead of rejected — appropriate for batch workloads and unacceptable for interactive ones.

The capacity connection

Rate limits are only half the system. They are a per-customer fairness mechanism; they do not protect the platform when the aggregate of all customers' limits exceeds capacity — which it always does, because limits are sold optimistically.

So rate limiting must sit alongside global admission control that sheds based on actual system state. The customer-facing limit answers "what did you buy"; admission control answers "what can we deliver right now". A platform with only the first will honour every customer's limit right up until it collapses.