A platform routes all model calls through an internal gateway. What belongs there, and what must not?

What belongs at the gateway

1. Provider and model routing, so model choice is configuration and can change without deployments, and so traffic can be split for comparison.

2. Quota and rate limiting per team, per feature and per tenant, denominated in tokens or a cost-weighted unit rather than request count — because requests differ by orders of magnitude in cost.

3. Cost attribution and metering. Every call tagged with team, feature, tenant and model. Without this, cost cannot be attributed, unprofitable usage is invisible, and the product cannot be priced.

4. Caching, where inputs repeat. Frequently a large fraction of cost, and centralising it means every feature benefits without each implementing it.

5. Retries, timeouts and failover between providers, with the retry policy owned in one place rather than implemented differently by each team.

6. Observability — latency, token counts, error rates, truncation and refusal rates — in a consistent shape across every caller.

7. Redaction and logging policy, so sensitive content is handled consistently rather than per feature.

What must not be there

Prompt content and business logic. A gateway that owns prompts becomes a bottleneck every team must change and nobody owns, and it separates the prompt from the code and evaluation that give it meaning.

Output validation and guardrails specific to a use case. What constitutes an acceptable output is domain knowledge. A generic filter at the gateway either blocks legitimate outputs or misses domain-specific problems.

Evaluation. It belongs with the feature, against that feature's held-out set.

The property most often missed

Graceful behaviour when a provider is degraded. The gateway is a hard dependency for every AI feature, so its failure policy is the platform's failure policy. It needs per-provider circuit breaking, a defined behaviour when all providers are degraded, and — importantly — the ability for a caller to say whether it wants a slow correct answer, a fast degraded one, or a fast failure.

A gateway that only routes is a single point of failure with a configuration file.