LLM Rate Limiting & Traffic Management Service
A distributed, multi-tenant admission control plane that sits between applications and LLM providers and answers one question in under 10 milliseconds: may this request proceed? The architecture is organised around four load-bearing decisions — quota is leased to pod-local buckets so most decisions never leave the process, tenant scopes are made atomic by co-locating their keys on one Redis slot, estimated tokens are reserved and reconciled against actuals with a reaper behind them, and the behaviour when coordination fails is a per-tenant policy field rather than a global constant.
24 views, each in three formats.
Open a view to read it in full. Every SVG carries its diagram source inside it, so it opens in diagrams.net fully editable with no import step; the draw.io files are the same diagrams as plain source.
-
01
System Context
Who sends traffic through the service, which providers it protects, and what it deliberately refuses to hold.
-
02
High-Level Architecture
The path one request takes, and which parts of it are on the latency budget.
-
03
Layered Architecture
What each layer is responsible for, and the one call direction that is allowed to break the rule.
-
04
Container Architecture (C4 Level 2)
The deployable units, which plane each belongs to, and the contracts between them.
-
05
Integration Architecture
Every interface in and out, with the protocol and cadence on each one.
-
06
Integration Modes
Three ways to adopt the platform, and which of them can guarantee that reconciliation actually happens.
-
07
Where Rate-Limit State Lives
Design question 1, answered: three tiers, with authority in exactly one of them per kind of state.
-
08
Policy and Usage Data Model
How a limit at any level in the hierarchy is expressed without a schema change, and what is recorded about a completed request.
-
09
Usage Accounting
How a completed request becomes a ledger row, a budget counter and eventually an invoice line, without ever being counted twice.
-
10
Authorization Hot Path
One request from arrival to reconciliation, showing exactly which steps are skipped on the common case.
-
11
Hierarchical Limit Evaluation
Five scopes that must all pass, evaluated in a fixed order so the rejection reason is deterministic.
-
12
Atomic Multi-Limit Commit
Design question 2, answered: how five counters are checked and incremented all-or-nothing, and what happens to the one that cannot be.
-
13
Token Reservation and Reconciliation
Design question 3, answered: the loop that turns an estimate into an accurate charge, and the sweep that makes over-reservation safe.
-
14
Quota Lease Protocol
Design question 5 on consistency, answered: how sixty pods share one 500-rpm quota without each of them allowing 500.
-
15
Priority, Tiers and Congestion Behaviour
What each tier actually experiences as load rises and as coordination degrades — the two axes read together.
-
16
Provider Limits, Routing and Failover
How the platform keeps applications from overwhelming an upstream, and what it does when one is already saturated or down.
-
17
Deployment Architecture
Two active regions, three availability zones each, and the deliberate choice not to replicate rate-limit state between them.
-
18
Policy Control Plane
Design question 5 on propagation, answered: from an administrator's edit to a live decision, and how the delay is measured rather than assumed.
-
19
Delivery Pipeline
Four things ship on independent cadences — service code, Lua scripts, policy and infrastructure — and each gets its own gate.
-
20
Observability and SLOs
Every signal the brief asks for, placed against the stage that emits it, with the SLO that makes it actionable.
-
21
Security Trust Zones
Six zones by decreasing exposure, and the one structural property that limits the blast radius of a decision-plane compromise.
-
22
Identity and Tenant Isolation
How a tenant is proved, and why one tenant cannot spend another's quota even if it asks to.
-
23
Failure Modes and the Degradation Ladder
Design question 4, answered: fail open, fail closed or fall back locally — and why that is a tenant policy field rather than a global constant.
-
24
Scaling to One Million Decisions per Second
Design question 6, answered: what scales linearly, what does not, and where the real ceiling actually sits.
Everything as it was delivered.
These files are served exactly as they were produced — the diagram pages keep their own house style because that is the artifact, not a rendering of it.