Six teams are calling model providers directly from their applications. Justify an AI gateway, and say what it should not do.
Show the full answer Hide the answer
What the interviewer is testing
Whether you can justify a shared component on capabilities that must exist once, and whether you know the failure mode of over-abstraction.
What breaks without one
No aggregate view of spend. No consistent logging, so no way to debug a bad answer. Provider credentials scattered across six codebases. No shared rate limiting, so teams compete for the same provider quota. No way to switch providers without touching every application. And no uniform guardrails.
What the gateway should provide
Cost attribution and budgets per team and per application, with hard limits. Usually the capability finance asks for first.
Routing by task — simple classifications to a small cheap model, hard reasoning to a large one. Frequently the largest cost reduction available across the estate.
Caching of identical and semantically similar requests.
Failover between providers when one is degraded or rate-limiting.
Logging of prompts, retrieved context and responses — the prerequisite for debugging anything.
Guardrails applied uniformly on input and output.
Credential isolation, so provider keys live in one place.
What it should not do
It should not become a lowest-common-denominator abstraction. Teams will want the newest provider capability before the gateway supports it, and a gateway that blocks adoption gets bypassed — at which point you have the complexity and none of the control.
Keep it thin: pass through provider-specific parameters rather than normalising everything, and behave as a transparent proxy rather than a unifying API.
The costs to acknowledge
Another hop in a latency-sensitive path, and another component that must be highly available since it now sits in front of everything. Both are real and both argue for keeping it simple.
What a strong answer adds
Logging as the highest-value capability, above cost control. Without the prompt, the retrieved context and the response recorded together, quality problems are undiagnosable — and quality is what determines whether the AI features survive.
Common weak answers
Justifying it on governance principle without naming capabilities. Designing a full abstraction layer over provider APIs.