# LLM Rate Limiting & Traffic Management Service

**Solution Architecture v1.0 · Data & AI Global Practice · 2026-08 · 24 views · open-source stack**

A distributed, multi-tenant admission control plane that sits between applications and LLM providers and answers one question in under 10 milliseconds: may this request proceed? The architecture is organised around four load-bearing decisions — quota is leased to pod-local buckets so most decisions never leave the process, tenant scopes are made atomic by co-locating their keys on one Redis slot, estimated tokens are reserved and reconciled against actuals with a reaper behind them, and the behaviour when coordination fails is a per-tenant policy field rather than a global constant.

---

## What is here

| Path | Contents |
|---|---|
| `diagrams/index.html` | The view index — 24 views grouped into six acts, every format linked |
| `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning it deliberately omits, with copy / PNG / PDF export |
| `diagrams/svg/*.svg` | The same 24 views as SVG with the diagram XML embedded — re-opens in diagrams.net fully editable |
| `diagrams/drawio/*.drawio` | draw.io native source |
| `specs/views.json` | Diagram specifications — the source of truth for every view (assembled from `part-a..d.json`) |
| `specs/manifest.json` | Acts, page titles, subtitles and the decision / assumption / risk cards (assembled from `manifest-a..b.json`) |
| `scripts/build.sh` | Rebuilds every deliverable from the specs |
| `ask.md` | The original requirement |

Every component carries its own mark — 341 icons embedded in the files, so the deliverable renders identically anywhere with no external requests.

To rebuild after editing a spec:

```bash
bash scripts/build.sh          # requires Node 20+ and draw.io Desktop
```

Edit `specs/part-a.json` … `part-d.json` and `specs/manifest-a.json` / `manifest-b.json`; the build script assembles `views.json` and `manifest.json` from them.

---

## The six acts

| Act | What it lands | Views |
|---|---|---|
| 1 · Context and scope | The boundary, the traffic sources, the providers, and the data the service refuses to hold | 01–02 |
| 2 · Structure | Layering rule, deployable containers, integration surface, three adoption modes | 03–06 |
| 3 · Data | Where rate-limit state lives, the policy and usage model, how consumption reaches billing | 07–09 |
| 4 · Runtime | The hot path, hierarchical evaluation, atomic commit, reservation, leasing, priority, provider failover | 10–16 |
| 5 · Operations | Multi-region deployment, policy propagation, delivery pipeline, observability | 17–20 |
| 6 · Assurance | Trust zones, tenant isolation, failure modes, the real scaling ceiling | 21–24 |

---

## The six core architecture questions, answered

The brief poses six design questions. Each is settled here and traceable to the view that carries it.

| Question | Position taken | View |
|---|---|---|
| **1 · Where does rate-limit state live?** | All three tiers, but authority in exactly one per kind of state. Valkey is authoritative for counters, PostgreSQL for policy. Pod-local memory holds only a bounded, expiring *lease* on Valkey capacity — discardable at any moment without loss. Local-only multiplies the quota by pod count; database-only blows the latency budget by 10–20×. | 07 |
| **2 · How do you atomically check multiple limits?** | All tenant-side keys carry the hash tag `{org:acme}` so Redis Cluster maps them to one slot, and a single Lua script reads every counter before mutating any of them. Lua is single-threaded, so check-then-commit is atomic without a lock. The provider counter is global and cannot share the slot — it is a second leg with a compensating release. | 12 |
| **3 · How do you handle token reservation?** | Reserve the pessimistic figure (estimated input + max output), then release the delta on commit. A reaper sweeps holds whose 120-second TTL expired, which is what makes pessimism affordable: a client that vanishes after ALLOW costs its tenant at most 120 s of quota. Concurrency slots follow the identical lifecycle. | 13 |
| **4 · What happens when Redis goes down?** | `fail_mode` is a column on the organisation, not a global constant. Default is **LOCAL** — each pod falls back to `quota ÷ pod count × 0.8`, keeping enforcement approximately correct. HIGH-tier tenants fail **OPEN** (availability wins), free tier fails **CLOSED** (cost wins). Counters are never repaired; they carry a two-window TTL and rebuild correct state in 500 ms. | 23 |
| **5 · How do policies propagate?** | Transactional outbox written in the same commit as the policy row, relayed to a log-compacted Kafka topic, consumed by every pod. Compaction means a restarting pod replays only the current policy set and reaches steady state without PostgreSQL. Every pod exports its policy version as a gauge, so propagation delay is measured rather than claimed. Target p99 < 2 s. | 18 |
| **6 · How do you scale to 1M decisions/sec?** | Linearly in pods and shards, because a decision touches one tenant's key space. Maglev hashing on `org_id` at the edge gives routing affinity, which lifts the lease hit rate from 78% to 92% and keeps Valkey at 8% of traffic. The binding constraint is not aggregate rate — it is one organisation's keys landing on one Redis slot, relieved by deterministic sub-sharding above 5k rps. | 24 |

---

## Requirements traceability

| Requirement | Where it is satisfied | View |
|---|---|---|
| FR1 · Request authorization | `Check` RPC and `/v1/authorize`; ALLOW/REJECT with `request_id`, reason and `retry_after` | 10, 11 |
| FR2 · Request-based limiting | Algorithm Engine as a strategy; token bucket in V1, sliding window selected per policy row | 03, 11 |
| FR3 · Token-based limiting | Estimated input + max output reserved, actuals reconciled | 13 |
| FR4 · Concurrency limits | Semaphore slots held from ALLOW, released on commit, failure or sweep | 07, 13 |
| FR5 · Hierarchical limits | Five scopes evaluated in fixed order, all must pass; guardrails enforce child ≤ parent | 11, 12 |
| FR6 · Model-specific limits | `policy.model_id` nullable — a model override is an insert, not a migration | 08, 11 |
| FR7 · Provider-specific limits | Global provider counters above tenant quotas, plus adaptive throttle from upstream 429s | 16 |
| FR8 · Dynamic policy management | Policy-as-code → policy-api → outbox → compacted Kafka → every pod, no redeploy | 18 |
| FR9 · Usage accounting | Kafka → Flink → ClickHouse ledger and cost marts; idempotent upsert on `request_id` | 09 |
| FR10 · Reservation & reconciliation | The reserve → execute → observe → reconcile → sweep loop | 13 |
| FR11 · Retry information | Six distinct reason codes; `retry_after` derived from the limiting scope's own window | 11 |
| FR12 · Priority / tiers | Reserved (not borrowed) headroom for HIGH; deterministic shedding order by tier | 15 |
| NFR1 · Low latency | p99 < 10 ms; measured 6.2 ms p99, 0.5 ms p50, 92% served from process memory | 02, 10 |
| NFR2 · High throughput | 100k/s initial, 1M/s design ceiling; unit economics per pod and per shard | 24 |
| NFR3 · High availability | 99.99%; two active regions, 3 AZ, replica promotion under 15 s | 17 |
| NFR4 · Horizontal scalability | Stateless decision pods, sharded coordination, no central bottleneck | 17, 24 |
| NFR5 · Distributed consistency | Lease protocol; ±2% tolerance stated, 0.9% measured; strict tenants bypass leasing | 14 |
| NFR6 · Fault tolerance | Per-tenant fail mode, degradation ladder, quantified blast radius, no repair jobs | 23 |
| NFR7 · Multi-tenancy | Scope keys derived from signed claims only; per-tenant quota, budget and concurrency | 22 |
| NFR8 · Security | Limiter structurally outside the prompt data path; mTLS/SPIFFE, Vault-leased provider keys | 21, 22 |
| NFR9 · Observability | Signal type × pipeline stage matrix, all correlated by `request_id`, with SLOs | 20 |

---

## The technology stack — open source throughout

Only the managed LLM providers are commercial, as the brief allows.

| Capability | Choice | Why |
|---|---|---|
| Edge gateway | **Envoy Proxy** | `ext_authz` v3 is a stable gRPC contract, so the default integration mode needs no application code; Maglev hashing gives tenant affinity |
| Decision service | **Go** (`limiterd`) | Predictable GC matters more than raw throughput when the SLO is a p99 |
| Coordination state | **Valkey Cluster** (Redis protocol) | Server-side Lua gives multi-key atomicity in one round trip — the property view 12 depends on |
| Policy store | **PostgreSQL 16 + Patroni** | Versioned relational policy with row-level security; never on the hot path |
| Event bus | **Apache Kafka (KRaft)** | One bus, two topic families: log-compacted `policy.v1` for propagation, partitioned `usage.v1` for accounting |
| Stream processing | **Apache Flink** | Windowed aggregation with checkpointed exactly-once effect at the sink |
| Usage ledger | **ClickHouse** + **dbt** | Column store sized for high-cardinality per-request rows; dbt for cost marts |
| Cold archive | **Apache Iceberg on MinIO** | 7-year audit retention off the hot store |
| Identity | **Keycloak** (OIDC, realm per tenant) + **SPIFFE/SPIRE** | Tenant identity at the edge, workload identity between services |
| Secrets | **Vault / OpenBao** | Dynamic, short-TTL provider credentials — rotation without redeploy |
| Observability | **OpenTelemetry · Prometheus · Grafana · Loki · Tempo** | One correlation id across metrics, logs and traces |
| Orchestration | **Kubernetes · Argo CD · Argo Rollouts · Helm · OpenTofu** | GitOps delivery with SLO-gated canaries |
| Self-hosted inference | **vLLM** serving **Mixtral 8x7B** | The last-resort failover route, and the residency-safe option |
| Managed providers | Azure OpenAI · Anthropic Claude · Google Gemini (Vertex AI) | Behind adapters; adding a fifth is an adapter, not an architecture change |

---

## The numbers this architecture claims

| Metric | Target | Basis |
|---|---|---|
| Decision latency | p99 < 10 ms | 0.3 ms on a lease hit, 6 ms on a lease refill |
| Lease hit rate | ≥ 90% | 92% measured with routing affinity, 78% without |
| Throughput | 100k/s initial, 1M/s ceiling | 18k decisions/s per 4-vCPU pod; 90k Lua/s per shard |
| Availability | 99.99% (52 min/year) | Two active regions, 3 AZ each |
| Quota overshoot | ≤ 2% standard, 0.9% measured | One unreturned lease per pod per 250 ms window |
| Policy propagation | p99 < 2 s | Compacted topic, ~1.2 s cold start at 5,000 orgs |
| Usage ledger lag | < 5 min | Kafka → Flink → ClickHouse |
| Budget refresh | 60 s | Cost enforcement is deliberately eventually consistent |
| Reservation TTL | 120 s | ~4× p99 completion latency of the slowest supported model |
| Provider failover | < 5 s | Breaker trips on 3 failures in 2 s |

---

## Assumptions on the record

These are exercise assumptions and would be confirmed during capacity planning.

- 100k authorization decisions/sec at launch, 1M/sec design ceiling; peak-to-average ratio 4:1.
- 5,000 organisations, four providers, roughly 20 models at launch.
- Every calling workload can obtain an OIDC token — there is no anonymous traffic path.
- ±2% quota overshoot is acceptable for standard tenants; tenants needing strict enforcement are configured to bypass leasing and pay 6 ms p99.
- Budget enforcement lagging by up to 60 s is acceptable and is stated in the tenant contract. Hard cost stops would require a synchronous ledger read on the hot path.
- The self-hosted vLLM pool is sized for 15% of peak, which caps how much provider failover it can absorb.

## Deliberately out of scope

- Prompt safety, PII redaction and content moderation — a separate guardrail service this platform can call.
- Semantic caching and response reuse — it changes token economics substantially and deserves its own design.
- Fine-tuning, model hosting and GPU scheduling beyond routing to the vLLM pool.
- Weighted fair queuing across tenants within a tier, deferred to V2; V1 is first-come within a tier.

---

## Known risks carried forward

| Risk | View | Position |
|---|---|---|
| Hash tagging concentrates one org's keys on one Redis slot | 12, 24 | Accepted as the price of atomicity; relieved by sub-sharding above 5k rps, at the cost of overshoot rising to ~3% |
| `llm-gateway` sees prompt content in memory | 21 | Highest-value target; restricted syscall profile, no shell, no debug endpoints in production |
| Kafka and ClickHouse are primary-region only | 17 | Primary loss stops usage aggregation; billing catches up from 7-day retention, live budget enforcement degrades to last-known values |
| Provider usage block formats differ and change | 05, 09 | Contract tests per adapter; where counts are missing on a stream, rows are marked estimated and excluded from invoices (currently < 2% of traffic) |
| Failover changes model behaviour | 16 | Opt-in per model family; the ledger records which provider actually served each request |
| Mode B clients own their own commit | 06 | Reaper bounds the damage; per-tenant commit-ratio alert below 0.98 over 15 minutes |
| Region failover causes spurious 429s until quota rebalances | 17 | Reconciler drops to a 10 s cycle when a region is marked down |

---

## Open decisions for the sponsor

1. **Strict-tenant tier.** Leasing is bypassed for tenants requiring exact enforcement, at 6 ms p99 instead of 0.3 ms and a hard dependency on Valkey. Confirm whether this tier is offered commercially and at what price.
2. **Default fail mode.** LOCAL is proposed as the default. Confirm that a free-tier tenant failing CLOSED during a Valkey outage is commercially acceptable.
3. **Budget enforcement lag.** 60 s means a tenant can overspend by one minute of peak traffic. Confirm this is acceptable, or fund the synchronous-ledger path.
4. **vLLM pool sizing.** Currently 15% of peak. A correlated two-provider outage exceeds it; confirm whether to fund more GPU capacity or accept tier-based shedding.
5. **Region count.** Two active regions meet 99.99%. A third would be driven by data residency, not availability.