LLM Gateways and Routing

You start with one provider key in an environment variable. Six months later you have four providers, fifteen models, three internal teams sharing the bill, two compliance reviews open, and an outage every time Anthropic's us-east region degrades. The LLM gateway is the piece of infrastructure that turns "we have an API key" into "we have a managed LLM platform." Either you build one or you run one of the open options.

What a gateway does that direct SDK calls do not

The single biggest argument for a gateway: when you have N apps and M models, you have N*M wiring problems. A gateway makes it N + M.

The open options

LiteLLM (BerriAI). Python SDK + standalone proxy server. Unifies 100+ providers behind the OpenAI-compatible Chat Completions API. The proxy supports virtual keys per team/user, spend tracking, budgets, fallbacks, retries, and a usable admin UI. MIT-licensed, the de-facto default for self-hosted gateways. Easiest path: run the proxy in a container, point all your apps at it, give each team a virtual key.

Portkey. Hosted (with a self-host option) gateway with strong observability and prompt-management UX. Routes through 250+ models, ships caching, guardrails, and a prompt library. Picks itself when the team wants a dashboard rather than a config file.

OpenRouter. A unified billing layer in front of every model. Not really self-hosted - it is a marketplace. One API key, you can call Claude, GPT, Llama, Mistral, anything. The pitch is operational simplicity at the cost of putting your traffic through a third party. Strong for prototyping, agent development, and teams that want one invoice instead of twelve.

Cloudflare AI Gateway. Edge-deployed gateway with built-in caching and analytics. Picks itself if you are already on Cloudflare and want zero infra.

Roll your own. Sometimes the right answer at scale. A 500-line FastAPI service can do 80% of what LiteLLM does, with exactly the policies you want and no compatibility tax. Defensible if you have one team writing this and three teams using it; indefensible if every team builds their own.

Policy-based routing

The interesting policies are not "always use GPT-4." They are:

# Cheap-first with confidence escalation
def route(prompt, user_tier):
    response = call("groq/llama-3.1-70b", prompt)
    if response.logprob_confidence < 0.7 or user_tier == "pro":
        response = call("anthropic/claude-sonnet-4-5", prompt)
    return response

# Fallback on rate limit / provider error
def call_with_fallback(prompt):
    chain = [
        "anthropic/claude-sonnet-4-5",      # primary
        "bedrock/anthropic.claude-sonnet",  # same model, different cloud
        "openai/gpt-4.1",                   # different vendor, similar quality
    ]
    for model in chain:
        try:
            return call(model, prompt, timeout=30)
        except (RateLimitError, ProviderError, TimeoutError):
            continue
    raise AllProvidersFailed()

# Tenant-aware routing
def route_for_tenant(prompt, tenant):
    if tenant.requires_data_residency == "EU":
        return call("bedrock-eu-west/claude-sonnet", prompt)
    if tenant.compliance_tier == "HIPAA":
        return call("vertex-private/gemini-pro", prompt)
    return call("anthropic/claude-sonnet-4-5", prompt)

LiteLLM expresses these as router config; Portkey expresses them as policies in the UI. Either way, the policy is declarative, per-request observable, and changeable without a deploy.

Per-tenant quotas

Once multiple tenants share an upstream key, you need quotas at the gateway, not in the app. Otherwise one tenant's runaway loop consumes the rate limit and every other tenant gets 429s. Standard pattern:

LiteLLM ships all of these out of the box. Roll-your-own usually starts with Redis counters and a Lua script for atomic increment-and-check.

The observability story

The minimum useful telemetry per request:

• request_id, tenant_id, user_id, feature (your app dimension)
• model_requested, model_actually_used (after fallback)
• prompt_tokens, cached_tokens, completion_tokens
• cost_usd (computed from your price table, not the provider's bill)
• ttft_ms, total_latency_ms
• finish_reason, error_class (if any)
• prompt_hash and response_hash (for dedup, never the raw text in metrics)

Then ship this to whichever observability stack you use - LiteLLM and Portkey both export to Langfuse, Datadog, Prometheus, OpenTelemetry. Raw prompts and responses go to a separate audit log (S3 with object-lock, typically), not into your metrics pipeline.

When it falls down

• Gateway becomes a SPOF. You centralised all LLM traffic through one service. Run it in HA, keep the config out of the request path, and have a documented "bypass the gateway" runbook for outages.
• Latency tax. A gateway adds one network hop and some bookkeeping. Budget 5-15 ms for in-region, 50+ ms cross-region. Acceptable for chat; sometimes not for tight agent loops.
• Fallback hides regressions. If GPT-4 silently falls back to Claude on every error, your eval set is no longer evaluating GPT-4. Alert on fallback rate, do not just hide it.
• OpenRouter's data policy. Routing through a third-party gateway means a third party has your prompts. Read the data-retention terms and decide whether that fits your compliance posture.

Further reading

• LiteLLM on GitHub - the proxy and SDK, with the router and fallback patterns in the docs.
• LiteLLM cost tracking documentation - virtual keys, per-team spend, daily breakdowns.
• Portkey documentation - the hosted-gateway take with prompt management and guardrails.
• OpenRouter quickstart - the marketplace API pattern, useful even if you self-host for the unified model catalogue.