The service is an admission control plane, not an LLM proxy that happens to count. Every capability on the diagram exists to make or account for an allow/reject decision.
Prompts and completions traverse the egress gateway but are never persisted, indexed or logged. The limiter receives token counts and signed scope claims only — that is what makes NFR8 an architectural property rather than a policy promise.
Providers are treated as untrusted, rate-limited dependencies with their own quotas, not as an unlimited backend. Protecting the upstream is a first-class requirement (FR7), not a side effect.
Assumptions
Every calling workload can obtain an OIDC token from the enterprise IdP; there is no anonymous traffic path.
Four providers at launch — Azure OpenAI, Anthropic Claude, Google Gemini on Vertex AI, and Mixtral self-hosted on vLLM. Adding a fifth is an adapter, not an architecture change.
Initial scale is 100k authorization decisions per second, designed to reach 1M. Both figures are exercise assumptions and are revisited in view 24.
Out of scope
Prompt safety, PII redaction and content moderation. Those belong in a separate guardrail service that this platform can call but does not implement.
Semantic caching and response reuse. It would change token economics substantially and deserves its own design.
Fine-tuning, model hosting and GPU scheduling beyond the vLLM pool this service routes to.
