AI Gateway Pattern
also called LLM Proxy, Model Router
A single control point through which all model calls pass, providing routing, cost control, caching, logging, guardrails and provider abstraction.
Without one, every application integrates with a provider SDK directly, and the organisation ends up with no aggregate view of spend, no consistent logging, no shared rate limiting, credentials scattered across services, and no way to switch providers without touching every codebase.
The gateway centralises what should only be built once. Cost attribution and budgets per team and per application, with hard limits, which is the capability finance asks for first. Routing by task, sending simple classifications to a small cheap model and hard reasoning to a large one — often the largest available cost reduction in the whole system. Caching of identical or semantically similar requests. Failover between providers when one is degraded or rate-limiting. Logging of prompts, retrieved context and responses, which is the prerequisite for debugging anything. Guardrails applied uniformly on input and output. And credential isolation, so provider keys live in one place.
The costs to weigh honestly: another hop in a latency-sensitive path, another component that must be highly available since it is now in front of everything, and an abstraction that will lag provider-specific features — teams will want the newest capability before the gateway supports it, and a gateway that blocks adoption gets bypassed.
The pragmatic position: keep the gateway thin, pass through provider-specific parameters rather than normalising everything, and make it a transparent proxy rather than a lowest-common-denominator abstraction.