LLM Rate Limiting & Traffic Management Service  ·  View 12 of 24  ·  Runtime

Atomic Multi-Limit Commit

Design question 2, answered: how five counters are checked and incremented all-or-nothing, and what happens to the one that cannot be.

Editable source SVG draw.io All views
Resolve keys
Resolve keys
Dry-run all limits
Dry-run all limits
Commit or abort
Commit or abort
Reserve
Reserve
Respond
Respond
limiterd (Go)
limiterd (Go)
Build key set
hash tag {org:acme}
Build key set...
Single EVALSHA
one round trip
Single EVALSHA...
Record request_id
Record request_id
ALLOW / REJECT
+ limiting scope
ALLOW / REJECT...
Tenant shard (Valkey)
Tenant shard (Valkey)
Org, team, user, model keys
same slot by design
Org, team, user, model keys...
Read every counter
no mutation yet
Read every counter...
INCR all or none
Lua is single-threaded
INCR all or none...
ZADD reservation
score = expiry
ZADD reservation...
Verdict + retry_after
Verdict + retry_after
Provider shard (Valkey)
Provider shard (Valkey)
Global provider key
different slot
Global provider key...
Read provider counter
Read provider counter
Compensating release
if tenant leg aborts
Compensating release...
Provider headroom
Provider headroom
Atomic Multi-Limit Commit — all limits, or none
Atomic Multi-Limit Commit — all limits, or none
Answer to design question 2: tenant scopes are made atomic by co-locating their keys on one slot; the provider scope cannot be, so it is a second leg with a compensating release.
Answer to design question 2: tenant scopes are made atomic by co-locating their keys on one slot; the provider scope cannot be, so it is a second leg with a compensating release.
v 1.0 · owner Data & AI Global Practice
v 1.0 · owner Data & AI Global Practice
Text is not SVG - cannot display

The mechanism

  • All tenant-side keys carry the hash tag {org:acme}, so Redis Cluster maps them to one slot and a single Lua script can read and mutate them together. Lua execution is single-threaded, which is what makes the check-then-commit atomic without any lock.
  • The script is genuinely two-phase: it reads every counter first and mutates nothing unless all pass. A naive incr-then-check leaks quota on the failing path and is the classic bug in this design.
  • The provider counter is global and cannot share the tenant's slot. It is a second leg with a compensating release — the one place the design accepts a compensating action instead of atomicity, and it is bounded to a single counter.

Numbers

  • One EVALSHA per lease refill, not per request. Measured 1.5 ms p99 at 90k script executions per shard per second.
  • Script SHAs are pinned to the release and loaded on every shard at deploy time (view 19), so a cold shard never falls back to sending the script body.
  • The compensating release window is under 2 ms, so provider-counter drift is bounded well below the ±2% overshoot tolerance.

Risks

  • Hash tagging concentrates one organisation's keys on one slot. That is the deliberate trade for atomicity, and it is also the scaling ceiling — view 24 shows the sub-sharding that relieves it.
  • If limiterd crashes between the tenant commit and the compensating release, the provider counter is over-counted until its window rolls. It fails in the safe direction: the upstream is under-used, never overwhelmed.