Cost-Weighted Rate Limiting

When one endpoint serves both a trivial lookup and a query that scans an enormous range, a limit expressed in requests per second treats them identically — which means it is set for the expensive case and starves the cheap one, or set for the cheap case and provides no protection at all.

Cost-weighted limiting assigns each request a cost and gives each caller a budget in those units.

Why it matters

It is the only limit that reflects the actual constraint. It also changes customer behaviour in a way no technical control achieves: when an expensive query pattern visibly consumes a budget, callers restructure their queries. Technical limits produce complaints; economic signals produce different code.

Implementation patterns

• Classify before execution, from method and parameters — range size, depth, expected cardinality. The estimate must be computable without doing the work.
• Charge the estimate, reconcile with the actual. A query that turned out far more expensive than estimated should consume the difference from the caller's budget, or the classifier becomes a target.
• Reject unbounded requests at admission rather than timing them out, because a timeout still pays for the work already performed.
• Separate pools per cost class, so an expensive query cannot hold a connection that cheap ones need. A budget limits how much a caller consumes; isolation limits what their consumption does to others, and both are needed.
• Publish the cost model. A budget the caller cannot predict is indistinguishable from random failure, and it produces support load rather than behaviour change.
• Cache the immutable. Anything about a finalised record never changes, which is an unusually good caching property and removes cost before it is charged.

Industry example

Blockchain RPC providers such as Polygon's infrastructure serve requests spanning several orders of magnitude in cost — a current-height query against a wide historical log scan — through one endpoint with one shape. Compute-unit pricing is now the norm in that domain precisely because request counting is meaningless there.

The same structure appears in GraphQL platforms where query complexity varies arbitrarily, in analytics APIs where a segmentation query can scan billions of rows, and in AI inference APIs where token count rather than request count is the resource.

Failure scenarios

• Request-count limits on a variable-cost API, protecting nothing.
• A cost model that cannot be computed before execution, making admission control impossible.
• Estimates never reconciled with actuals, so callers learn to construct requests that look cheap.
• Budgets without isolation, so a caller within their budget still monopolises a shared pool.
• An unpublished cost model, producing unpredictable failures and support load instead of adapted clients.

Trade-offs

Cost models add complexity for both the platform and the customer, and they are always approximate — a model that is too detailed is unpredictable, and one that is too coarse is unfair. There is also a real risk of the model becoming a specification that constrains future implementation changes.

The alternative is a flat limit, which is simple, predictable, and a business model that subsidises the expensive minority at the expense of everyone else. For an API with a narrow cost range, flat is correct; for one spanning orders of magnitude, it is not.

Interview question

"Your API's cheapest request costs a microsecond and your most expensive costs eight seconds of CPU. Design the rate limit, and tell me what a customer sees when they exceed it."