LLM Rate Limiting & Traffic Management Service  ·  View 10 of 24  ·  Runtime

Authorization Hot Path

One request from arrival to reconciliation, showing exactly which steps are skipped on the common case.

Editable source SVG draw.io All views
Application
Application
Envoy Gateway
Envoy Gateway
limiterd
limiterd
Valkey Shard
Valkey Shard
llm-gateway
llm-gateway
LLM Provider
LLM Provider
1. POST /v1/chat/completions
1. POST /v1/chat/completions
2. Check(tenant, model, 5k in, 2k out)
2. Check(tenant, model, 5k in, 2k out)
3. resolve scope chain from cached policy
3. resolve scope chain from cached policy
4. try local lease — exhausted
4. try local lease — exhausted
5. EVALSHA check_and_commit(keys)
5. EVALSHA check_and_commit(keys)
6. ALLOW + lease 40 req / 250 ms
6. ALLOW + lease 40 req / 250 ms
7. reserve 7 000 tokens + slot, TTL 120 s
7. reserve 7 000 tokens + slot, TTL 120 s
8. ALLOW request_id=req-123
8. ALLOW request_id=req-123
9. forward with request_id
9. forward with request_id
10. upstream completion
10. upstream completion
11. completion + usage block
11. completion + usage block
12. Commit(actual 5 800)
12. Commit(actual 5 800)
13. release 1 200 tokens + free slot
13. release 1 200 tokens + free slot
14. 200 OK
14. 200 OK
Authorization Hot Path — one request, p99 under 10 ms
Authorization Hot Path — one request, p99 under 10 ms
Steps 5 to 7 are skipped on roughly 92% of requests, where the local lease still has capacity. That is what buys the latency budget.
Steps 5 to 7 are skipped on roughly 92% of requests, where the local lease still has capacity. That is what buys the latency budget.
v 1.0 · owner Data & AI Global Practice
v 1.0 · owner Data & AI Global Practice
Text is not SVG - cannot display

Read this first

  • Steps 5 to 7 — the Valkey round trip — are skipped on roughly 92% of requests, where the pod-local lease still has capacity. That is where the latency budget comes from.
  • The reservation is taken before the response is returned, not after. A request that is allowed but never executed still holds quota until the reaper releases it, which is the correct conservative behaviour.
  • Commit is asynchronous and off the caller's critical path. The tenant's response is not delayed by accounting.

Latency budget

  • Envoy filter overhead 0.4 ms · scope resolution from cache 0.1 ms · local lease check 0.2 ms.
  • On a lease miss: Valkey EVALSHA 1.5 ms p99 · reservation write 0.6 ms · total 6 ms p99.
  • Budget: 10 ms p99 for the decision. Measured p99 across the mix, 6.2 ms; p50, 0.5 ms.

Failure behaviour

  • If step 5 times out at 3 ms, the pod falls back to its local bucket and the tenant's fail_mode decides the verdict — view 23 has the ladder.
  • If step 12 never arrives, the reservation expires after 120 s and the reaper releases it. The tenant loses that quota for the interval; it is never lost permanently.
  • A duplicate commit for the same request_id is a no-op, because reservation state moves HELD → COMMITTED once.