LLM Rate Limiting & Traffic Management Service

Architecture Views

24 views, in reading order. Every view ships three ways: an HTML page, an SVG that re-opens in diagrams.net fully editable, and draw.io source.

A distributed, multi-tenant admission control plane that sits between applications and LLM providers and answers one question in under 10 milliseconds: may this request proceed? The set reads in six acts — what sits inside the boundary, how the parts fit, where state lives and who owns it, what happens at runtime, how it is operated, and why it is safe. Four decisions carry the whole design: quota is leased to pod-local buckets so most decisions never leave the process (views 07 and 14); tenant scopes are made atomic by co-locating their keys on one Redis slot (view 12); estimated tokens are reserved and reconciled against actuals with a reaper behind them (view 13); and the behaviour when coordination fails is a per-tenant policy field rather than a global constant (view 23).

Context and scope

What the service is accountable for, who depends on it, and the deliberate limit on what it is allowed to see.

01 System Context Who sends traffic through the service, which providers it protects, and what it deliberately refuses to hold. HTML page SVG draw.io

02 High-Level Architecture The path one request takes, and which parts of it are on the latency budget. HTML page SVG draw.io

Structure

The layering rule, the deployable containers, the integration surface, and the three ways a workload can adopt the platform.

03 Layered Architecture What each layer is responsible for, and the one call direction that is allowed to break the rule. HTML page SVG draw.io

04 Container Architecture (C4 Level 2) The deployable units, which plane each belongs to, and the contracts between them. HTML page SVG draw.io

05 Integration Architecture Every interface in and out, with the protocol and cadence on each one. HTML page SVG draw.io

06 Integration Modes Three ways to adopt the platform, and which of them can guarantee that reconciliation actually happens. HTML page SVG draw.io

Data

Where rate-limit state lives across three tiers, what the policy and usage model looks like, and how consumption reaches billing.

07 Where Rate-Limit State Lives Design question 1, answered: three tiers, with authority in exactly one of them per kind of state. HTML page SVG draw.io

08 Policy and Usage Data Model How a limit at any level in the hierarchy is expressed without a schema change, and what is recorded about a completed request. HTML page SVG draw.io

09 Usage Accounting How a completed request becomes a ledger row, a budget counter and eventually an invoice line, without ever being counted twice. HTML page SVG draw.io

Runtime

The hot path itself: how one decision is made, how five scopes are checked atomically, how tokens are reserved and reconciled, and how quota is shared across instances.

10 Authorization Hot Path One request from arrival to reconciliation, showing exactly which steps are skipped on the common case. HTML page SVG draw.io

11 Hierarchical Limit Evaluation Five scopes that must all pass, evaluated in a fixed order so the rejection reason is deterministic. HTML page SVG draw.io

12 Atomic Multi-Limit Commit Design question 2, answered: how five counters are checked and incremented all-or-nothing, and what happens to the one that cannot be. HTML page SVG draw.io

13 Token Reservation and Reconciliation Design question 3, answered: the loop that turns an estimate into an accurate charge, and the sweep that makes over-reservation safe. HTML page SVG draw.io

14 Quota Lease Protocol Design question 5 on consistency, answered: how sixty pods share one 500-rpm quota without each of them allowing 500. HTML page SVG draw.io

15 Priority, Tiers and Congestion Behaviour What each tier actually experiences as load rises and as coordination degrades — the two axes read together. HTML page SVG draw.io

16 Provider Limits, Routing and Failover How the platform keeps applications from overwhelming an upstream, and what it does when one is already saturated or down. HTML page SVG draw.io

Operations

How the service is deployed across regions, how policy reaches a running pod, how change is released safely, and what is watched.

17 Deployment Architecture Two active regions, three availability zones each, and the deliberate choice not to replicate rate-limit state between them. HTML page SVG draw.io

18 Policy Control Plane Design question 5 on propagation, answered: from an administrator's edit to a live decision, and how the delay is measured rather than assumed. HTML page SVG draw.io

19 Delivery Pipeline Four things ship on independent cadences — service code, Lua scripts, policy and infrastructure — and each gets its own gate. HTML page SVG draw.io

20 Observability and SLOs Every signal the brief asks for, placed against the stage that emits it, with the SLO that makes it actionable. HTML page SVG draw.io

Assurance

The trust boundary, how a tenant is proved, what happens when each dependency fails, and where the real scaling ceiling is.

21 Security Trust Zones Six zones by decreasing exposure, and the one structural property that limits the blast radius of a decision-plane compromise. HTML page SVG draw.io

22 Identity and Tenant Isolation How a tenant is proved, and why one tenant cannot spend another's quota even if it asks to. HTML page SVG draw.io

23 Failure Modes and the Degradation Ladder Design question 4, answered: fail open, fail closed or fall back locally — and why that is a tenant policy field rather than a global constant. HTML page SVG draw.io

24 Scaling to One Million Decisions per Second Design question 6, answered: what scales linearly, what does not, and where the real ceiling actually sits. HTML page SVG draw.io

Open svg/<view>.svg or drawio/<view>.drawio in draw.io Desktop or at app.diagrams.net to edit. The SVG carries the diagram inside it, so it is both the picture and the source. This folder is self-contained — copy it whole and every link still resolves.