LLM Rate Limiting & Traffic Management Service  ·  View 14 of 24  ·  Runtime

Quota Lease Protocol

Design question 5 on consistency, answered: how sixty pods share one 500-rpm quota without each of them allowing 500.

Editable source SVG draw.io All views
limiterd A
limiterd A
limiterd B
limiterd B
Valkey Shard
Valkey Shard
Policy Cache
Policy Cache
1. org acme: 500 rpm, algorithm=BUCKET
1. org acme: 500 rpm, algorithm=BUCKET
2. LEASE org:acme window=250 ms
2. LEASE org:acme window=250 ms
3. grant 40 of 125 in window
3. grant 40 of 125 in window
4. serve 40 decisions from RAM
4. serve 40 decisions from RAM
5. LEASE org:acme window=250 ms
5. LEASE org:acme window=250 ms
6. grant 25 — sized to observed demand
6. grant 25 — sized to observed demand
7. RENEW + return 6 unused
7. RENEW + return 6 unused
8. grant 55 — demand rising
8. grant 55 — demand rising
9. LEASE — window already exhausted
9. LEASE — window already exhausted
10. DENY, retry_after 0.18 s
10. DENY, retry_after 0.18 s
11. reject locally, no further round trips
11. reject locally, no further round trips
Quota Lease Protocol — how ten limiters share one quota
Quota Lease Protocol — how ten limiters share one quota
Answer to design question 5 on consistency: overshoot is bounded by one unreturned lease per pod per window — measured at 0.9% against a 500 rpm quota with 60 pods.
Answer to design question 5 on consistency: overshoot is bounded by one unreturned lease per pod per window — measured at 0.9% against a 500 rpm quota with 60 pods.
v 1.0 · owner Data & AI Global Practice
v 1.0 · owner Data & AI Global Practice
Text is not SVG - cannot display

The consistency model

  • A pod does not enforce the quota; it enforces a lease it was granted from the quota. The shard is the only place the full quota is known, so the sum of outstanding leases can never exceed it.
  • Leases are demand-sized. A pod that used 34 of its last 40 gets a larger grant; one that returned most of its grant gets a smaller one. Static division by pod count wastes quota when traffic is uneven, which is the normal case.
  • The stated tolerance is ±2% overshoot for standard tenants; measured at 0.9% with 60 pods against a 500 rpm quota. Tenants that require strict enforcement are configured to bypass leasing and hit the shard on every request, paying about 6 ms p99 instead of 0.3 ms.

Where overshoot comes from

  • A pod that dies holding an unreturned lease. Bounded by one lease per pod per 250 ms window, and the lease expires on its own.
  • Clock skew between pods and shard, bounded by NTP discipline to a few milliseconds against a 250 ms window.
  • Nothing else. There is no path by which two pods can be granted the same capacity, because the grant is an atomic decrement on the shard.

The trade being made

  • Strict global accuracy would require a synchronous coordinated decision per request, which is a 6 ms floor and a hard dependency on the shard being reachable.
  • The design chooses bounded, measured, per-tenant-configurable inaccuracy in exchange for a 20× latency improvement and graceful degradation.
  • The tolerance is a policy field, so a regulated tenant and a free-tier tenant can hold different positions on the same platform.