An AI gateway now terminates streaming responses for six products and holds a semantic cache shared across tenants. Two properties of that design have caused real outages and real data exposure at large AI providers. What has the organisation taken on and when does the bill arrive?

What is gained

The gateway is worth having. Centralised keys, per-tenant token budgets, one place to swap providers, one audit log, one prompt-injection filter. Six teams stop each building the same rate limiter badly. That is not in dispute.

What is paid: a shared failure domain

Every product's availability is now the product of the gateway's and the provider's. Two nines-and-a-half each is 99.8% combined, which is roughly 90 minutes a month, and that is before correlated failures.

The sharper cost is that a change to the shared path is a change to every product at once. OpenAI's December 2024 incident is the general lesson: a new telemetry service, deployed on 11 December, caused every node in every cluster to issue expensive Kubernetes API calls whose volume scaled with cluster size. The control planes buckled, DNS-based service discovery broke, and all services were degraded or unavailable from 15:16 to 19:38 Pacific, about four and a half hours. DNS caching delayed the symptom by roughly 20 minutes, so the change reached every cluster before the first failure appeared, and engineers were locked out of the control plane they needed to fix it.

The pattern transfers exactly. A gateway is a control plane for model access, a change to it lands everywhere, and its own staged rollout is the only thing standing between a config edit and a total outage. Budget for: per-cluster canaries, a break-glass path that bypasses the gateway with direct provider keys, and rate-limited control-plane access reserved for operators.

What is paid: a shared data path

A semantic cache keyed across tenants is a cross-tenant read path. OpenAI's March 2023 incident showed how thin the margin is: a bug in the redis-py asyncio client left cancelled requests able to receive another user's data, which exposed some users' chat titles and, for a small number, partial payment details. Connection-pool and cache code is exactly where this class of bug lives, because the correctness condition is an invariant nobody tests.

The containment is structural, not careful coding. Include the tenant id in the cache key and enforce it at read time as well as write time. Never cache a response whose inputs included retrieved documents, because the cache then inherits the document's access-control rules and no cache implements those. Semantic caches make this worse than exact-match caches: a near-miss key can legitimately serve a different question's answer, so the blast radius of a key-scoping bug is wider.

What is paid: streaming

A proxy that buffers a response destroys time-to-first-token. A product whose first token appeared in 300 ms now waits for the full generation, which for a long answer is 8 to 15 seconds of apparent hang. Any guardrail that must see the whole output forces buffering, so the design choice is chunk-level scanning with the ability to retract, or accepting the latency on the small share of traffic that needs a full-output check.

When the bill arrives

Not at launch. It arrives at the first gateway change that is not a code deploy (a routing table, a model alias, a quota file) and at the first security review that asks who else can read the cache. Both happen within a year.

When not to introduce a gateway

Two teams and one provider do not need a gateway. A shared client library with per-team keys gives most of the control with no new failure domain and no new data path, and it can be replaced by a gateway later without changing call sites. Introduce the gateway when you have three or more teams, more than one provider, or a compliance requirement for a single audit log.