LLM Rate Limiting & Traffic Management Service  ·  View 03 of 24  ·  Structure

Layered Architecture

What each layer is responsible for, and the one call direction that is allowed to break the rule.

Editable source SVG draw.io All views
Client & Edge
Client & Edge
Application SDK
authorize + commit
Application SDK...
Envoy Gateway
ext_authz, mTLS
Envoy Gateway...
Tenant Router
Maglev hash
Tenant Router...
Admission Control
tier shedding
Admission Control...
Decision
Decision
Authorize API
gRPC + HTTP
Authorize API...
Policy Evaluator
scope cascade
Policy Evaluator...
Algorithm Engine
bucket · sliding window
Algorithm Engine...
Reason & Retry Builder
429 semantics
Reason & Retry Builder...
Coordination
Coordination
Lease Manager
250 ms grants
Lease Manager...
Atomic Limit Scripts
Lua · single slot
Atomic Limit Scripts...
Reservation Ledger
estimate vs actual
Reservation Ledger...
Concurrency Semaphores
slot lease + reaper
Concurrency Semaphores...
Egress & Providers
Egress & Providers
LLM Gateway
OpenAI-compatible
LLM Gateway...
Provider Adapters
four upstreams
Provider Adapters...
Failover Router
weighted + health
Failover Router...
Usage Extractor
reads token counts
Usage Extractor...
Control
Control
Policy API
Go · REST
Policy API...
Policy Store
PostgreSQL 16
Policy Store...
Policy Bus
Kafka · compacted
Policy Bus...
Admin Console
React
Admin Console...
Accounting & Insight
Accounting & Insight
Usage Collector
at-least-once
Usage Collector...
Stream Aggregator
Apache Flink
Stream Aggregator...
Usage Ledger
ClickHouse
Usage Ledger...
Cost & Budget Marts
dbt models
Cost & Budget Marts...
Platform
Platform
Kubernetes
3 AZ per region
Kubernetes...
Identity & Secrets
Keycloak · Vault
Identity & Secrets...
Observability
OTel · Prom · Grafana
Observability...
GitOps Delivery
Argo CD · Rollouts
GitOps Delivery...
Layered Architecture
Layered Architecture
A layer calls only the layer below it. The one deliberate exception is Accounting, fed asynchronously from Decision and Egress and never called back by them.
A layer calls only the layer below it. The one deliberate exception is Accounting, fed asynchronously from Decision and Egress and never called back by them.
v 1.0 · owner Data & AI Global Practice
v 1.0 · owner Data & AI Global Practice
Text is not SVG - cannot display

The layering rule

  • A layer calls only the layer below it. The single exception is Accounting, which is fed asynchronously from Decision and Egress and never called back by them — that keeps billing off the latency budget.
  • Control is a peer of Decision, not above it. Policy arrives by push; the decision path never calls the control plane synchronously, which is why a PostgreSQL outage cannot stop traffic.
  • The Algorithm Engine is a strategy, not a fixed implementation. Token bucket ships in V1; sliding window is selected per policy row without a code change (FR2).

Numbers

  • Seven layers, 28 capabilities. Every capability maps to at least one functional requirement in the brief.
  • Lease refill interval 250 ms — short enough that a policy change takes effect within one window, long enough to keep Valkey traffic at 8% of decisions.
  • Policy cache holds the full tenant set in roughly 40 MB per pod at 5,000 organisations.

Risks

  • Reason & Retry Builder is load-bearing for client behaviour. A wrong retry_after turns a rate limit into a retry storm; it is derived from the limiting scope's window, never from a constant.
  • Provider Adapters are the layer most likely to churn, since each vendor changes its usage block format independently. Contract tests per adapter are mandatory.