LLM Rate Limiting & Traffic Management Service  ·  View 07 of 24  ·  Data

Where Rate-Limit State Lives

Design question 1, answered: three tiers, with authority in exactly one of them per kind of state.

Editable source SVG draw.io All views
L1 · In-process — per limiter pod (RAM)
L1 · In-process — per limiter pod (RAM)
Lease Buckets
granted RPM / TPM slice
Lease Buckets...
Policy Cache
last-known-good
Policy Cache...
Negative Cache
denied scope, 200 ms
Negative Cache...
≈ 0.3 ms p99
92% of decisions
≈ 0.3 ms p99...
L2 · Shared coordination
L2 · Shared coordination
RPM / TPM Counters
windowed, TTL-scoped
RPM / TPM Counters...
Concurrency Semaphores
sorted set + TTL
Concurrency Semaphores...
Reservation Ledger
estimate held 120 s
Reservation Ledger...
Budget Counters
spend to date
Budget Counters...
≈ 1.5 ms p99
8% of decisions
≈ 1.5 ms p99...
L3 · Durable — system of record, never on the hot path
L3 · Durable — system of record, never on the hot path
PostgreSQL
policy + hierarchy
PostgreSQL...
ClickHouse
usage ledger
ClickHouse...
MinIO Archive
Iceberg, 7 years
MinIO Archive...
10–200 ms
0% of decisions
10–200 ms...
Cold pod: no lease yet
first call goes to L2
Cold pod: no lease yet...
Kafka policy.v1 + usage.v1
the only L3 → L1 path
Kafka policy.v1 + usage.v1...
refill every 250 ms
refill every 250 ms
committed usage
committed usage
outbox
outbox
policy push
policy push
budget rollup, 60 s
budget rollup, 60 s
Where Rate-Limit State Lives — three tiers, one hot path
Where Rate-Limit State Lives — three tiers, one hot path
Application we own
Application we own
Decision point
Decision point
Data store
Data store
Risk / gap
Risk / gap
Queue / topic
Queue / topic
synchronous
synchronous
event / async
event / async
Answer to design question 1: state lives in all three tiers, but authority lives only in L2 for counters and L3 for policy. L1 holds a bounded, expiring claim on L2.
Answer to design question 1: state lives in all three tiers, but authority lives only in L2 for counters and L3 for policy. L1 holds a bounded, expiring claim on L2.
v 1.0 · owner Data & AI Global Practice
v 1.0 · owner Data & AI Global Practice
Text is not SVG - cannot display

The decision

  • State lives in all three tiers, but authority does not. Valkey is authoritative for counters; PostgreSQL is authoritative for policy; L1 holds only a bounded, expiring claim on L2 and can be discarded at any moment without loss.
  • Local memory alone was rejected: N pods each enforcing the full quota multiplies it by N. A database alone was rejected: the p99 budget is 10 ms and a durable write is 10 to 200 ms.
  • The claim is what makes both properties available at once — L1 gives the latency, L2 gives the correctness, and the lease is the contract between them.

Numbers

  • L1 ≈ 0.3 ms p99, serving 92% of decisions. L2 ≈ 1.5 ms p99, serving 8%. L3 serves 0%.
  • Counter keys carry a TTL of two windows, so a shard that loses its data rebuilds correct state within 500 ms rather than needing a repair job.
  • Budget counters are refreshed from the ledger every 60 s — cost enforcement is deliberately eventually consistent, and view 09 explains why that is acceptable.

Risks

  • A cold pod has no lease and sends its first decision for each tenant to L2. During a large scale-out this raises L2 traffic sharply; leases are pre-warmed for the top 200 tenants at pod start.
  • Budget enforcement lagging by up to 60 s means a tenant can overspend by one minute of peak traffic. That is accepted and stated in the tenant contract; hard cost stops would need a synchronous ledger read on the hot path.