When the model provider goes down: a field guide to the LLM gateway

The territory

A capability you sell is rented, per request, from a company you cannot page. The question is what stands between your code and theirs.

99.58%

Anthropic status-page uptime over the months before June 2025; OpenAI's was 99.80%. Three-plus hours of provider downtime a month is the baseline, not the exception

Assembled, 2025

25T

Tokens per week routed through OpenRouter alone by May 2026, five times the volume of six months earlier. The indirection layer is now infrastructure

Pulse2, 2026

16M

Queries per month through Uber's internal GenAI Gateway from about 30 teams, one year after it shipped

InfoQ, 2024

0.14%

Share of gateway-fronted requests that still failed during a roughly 12-hour Anthropic API incident, and every listed cause was fallback misconfiguration, not the outage. Vendor-reported

Portkey, 2025

State the problem without naming the technology: the capability at the centre of your product is produced by someone else's computers, reached over HTTP, billed by usage, rate-limited at their discretion, and revised under your feet. Five or six companies sell it. Their status pages record multi-hour failures several times a year, and their worst failures never reach the status page at all. Every team in your organisation wants to call them, each with its own key, its own retry loop and its own idea of a budget.

The pattern the industry converged on is a single internal choke point. Uber calls its build a GenAI Gateway, "a Go service that acts as an encompassing layer around the clients for third-party vendors, complemented by the in-house serving stacks tailored for Uber's LLMs" (July 2024). Grab runs one reverse proxy in front of OpenAI, Azure, AWS Bedrock and Vertex AI. Wealthsimple, a regulated brokerage, built one for audit and reliability before most of its features. Bloomberg co-built Envoy AI Gateway in the open rather than keep its version internal. The CNCF's recap of Bloomberg's KubeCon keynote lists the three pains that forced the layer into existence: providers "need different access patterns, use different ways to manage credentials, and service-specific models have different needs" (CNCF, November 2024).

The surprise in the record, and the reason this guide exists, sits one level down. The layer is sold, internally and commercially, as outage insurance. Yet the only published postmortem by a gateway operator in this corpus, OpenRouter's February 2026 write-up, is about the gateway's own caching dependency, not about any model provider. And the one measured account of fallbacks during a real provider outage, Portkey's, attributes every residual failure to the fallback configuration itself: the 529 status code nobody had mapped, expired keys, targets that no longer existed. The pattern that emerges is blunt: the providers fail often enough to justify the layer, and the layer then fails in the ways you stopped testing. Most of this page is about keeping both halves of that sentence in view at once.

Figure 1 · The landscape: who stands where

Every arrow on the left is a team that would otherwise hold its own provider key and retry loop; the gateway is the one place policy can exist. Reconstructed from Uber, Grab and the Envoy AI Gateway founding proposal.

Diagram source

Scope. This guide covers the consumer side: the layer an organisation that buys model capacity puts in front of providers, and what that layer does when providers fail, throttle or degrade. It deliberately does not cover the provider-side serving stack (the 2026-08-28 dig on inference capacity and routing), prompt change management (2026-09-15), sandboxing what agents may do (2026-10-02), or how to detect silent model regressions in depth (2026-09-08); the Anthropic postmortem appears here for what it says about failover signals, not as an evaluation method.

How it is actually built

Seven published systems share one shape: a thin translating proxy with a synchronous gate in front and an asynchronous ledger behind.

Lay the published builds side by side, Uber's, Grab's, Wealthsimple's, Bloomberg's Envoy AI Gateway, LiteLLM, BricksLLM and OpenRouter, and the same six components appear in every column. The differences are real but they are at the edges; the core recurs often enough to call it a reference architecture.

Figure 2 · Reference architecture of the consumer-side gateway

Everything above the request line must answer before the provider is called, so it is the gateway's entire latency budget; everything below is deliberately moved off the request path. Reconstructed from Uber, Grab, BricksLLM and the Envoy AI Gateway proposal.

Diagram source

The load-bearing design fact is the split between the two halves. A spend or rate check only works before the provider is called, so it is synchronous by definition; BricksLLM's build log prices that pre-flight Redis lookup at about 30 ms and notes it "can not be made async since the rate limit is a feature that needs to happen before the request is forwarded" (2023). Accounting, by contrast, needs the response, because output tokens are unknown until generation ends, and every published build pushes it off the request path: BricksLLM moved its event writes to a channel-based bus and cut gateway overhead to roughly 30 ms total; Grab computes cost "for each request once the response is received from the provider" and archives it "in the data lake along with an audit trail"; Envoy AI Gateway charges token quota on a stream-done callback (proposal 009). The transfer: whatever you add to this layer, decide first which side of the line it lives on, because the synchronous side is a tax on every model call your company makes.

One consequence of the split is reported rather than inferred: enforcement lags usage. A quota charged after the stream completes cannot stop the request that exhausts it, so a burst can overshoot every budget by one in-flight generation per concurrent caller. Envoy AI Gateway's quota design accepts this and reconciles afterwards; Grab prices async workloads with a separate daily job. Nobody in this corpus claims to have closed the gap, which is worth knowing before you promise finance a hard cap.

The dialect boundary

Every build exposes one request schema to the whole company and translates outward. In practice that schema is OpenAI's: Uber mirrors it for ecosystem compatibility, and the Envoy proposal states as an axiom that "the MVP heavily assumes that the requests are sent using the OpenAI spec". The adapter owns provider quirks so callers never see them.

Runs this way at: Uber, Envoy AI Gateway, LiteLLM

The credential boundary

Provider keys live only in the gateway; callers get internal identities. Grab issues two classes, short-lived rate-limited "exploration keys" via a Slack bot and long-term service keys for production, which turns key hygiene into a workflow instead of a wiki page. BricksLLM existed because of "horror stories regarding people losing thousands of dollars from stolen API keys".

Runs this way at: Grab, Bloomberg, BricksLLM

The divergence points

Where the columns differ: whether self-hosted models sit behind the same API (Uber and Wealthsimple, yes; Grab's post centres on external providers), whether the gateway is in-house, open source or a hosted aggregator, and whether response caching exists at all. Uber reports its PII redaction and caching fight each other: redacted placeholders make cached responses hard to map back to entities.

Sources: Uber, Wealthsimple, OpenRouter

The decisions that matter

Five forks, each with the condition that flips it. The failover-trigger decision carries the most operational weight and the least published agreement.

What triggers failover, and who decides?

Chosen

Automated, status-code-driven: retry, then cool the failing deployment down, then fall back to the next target. LiteLLM ships this as the default; OpenRouter runs provider-level failover on by default with cross-model fallback opt-in.
Assembled got here the hard way: manual on-call switching failed because routing is per use case, switches took minutes, and humans could not reliably tell a full outage from partial degradation (June 2025).

Rejected

Manual provider switching via config. Rejected by Assembled after operating it: minutes of failover and misclassified partial outages.
Health-check-only failover. No build in this corpus trusts probes alone, and the Anthropic postmortem shows why: for a month the unhealthy state produced normal status codes and normal latency.

Flips when

There is nowhere to go. LiteLLM exempts single-deployment model groups from cooldown ("by default we should avoid cooldowns on single deployment model groups"): a breaker in front of the only backend converts partial failure into total failure.
The response streams. Once the first token has reached the user the switch window has closed; the Twilio talk's framing is that the fallback opportunity is before streamed output commits.

Which API dialect does the company write against?

Chosen

Mimic the OpenAI API. Uber chose it because "open-source libraries like LangChain and LlamaIndex" already speak it (InfoQ, 2024); the Envoy proposal makes the same assumption an axiom; LiteLLM's entire product is this translation.

Rejected

A provider-neutral schema of your own design. Nobody in this corpus shipped one; the ecosystem gravity is one-directional.
Letting each team use provider-native SDKs directly, which re-scatters keys, retries and budgets.

Flips when

A capability does not fit the dialect. Envoy AI Gateway grew a vendor-specific-fields mechanism, and Grab found "certain SDK functionalities didn't work as expected due to missing paths or configurations" behind its reverse proxy. The dialect is a floor, not a ceiling; budget for escape hatches.

Build it, adopt open source, or route through a hosted aggregator?

Chosen

In-house thin service at Uber, Grab, Wealthsimple (all 2023–24), where PII handling, audit and data-residency rules are the binding constraint.
Shared open source at Bloomberg, which co-built Envoy AI Gateway publicly rather than maintain a private fork.

Rejected

Routing everything through a hosted aggregator. None of the four named enterprises did, and the OpenRouter postmortem names the cost: when the aggregator's own cache failed, every provider behind it became unreachable at once, for 38 and then 35 minutes (February 2026).

Flips when

You are small, model diversity is the product, and operating a gateway is overhead you cannot staff: 25 trillion tokens a week say plenty of teams take that trade knowingly.
Even then, OpenRouter's own reliability page concedes internal failover cannot protect against a fault in the routing layer itself; teams that need independence keep a second direct route.

Rate-limit in requests or in tokens?

Chosen

Tokens. The Envoy AI Gateway founding proposal treats this as the reason the project had to extend Envoy at all: "Rate Limiting for LLM workload is based on tokens, we extend Envoy Gateway to support generic cost based rate limiting as envoy so far only supports request based rate limiting."

Rejected

Request counting, the default of every general-purpose gateway. One request can cost one token or a hundred thousand; the unit does not correlate with spend or provider capacity.

Flips when

It does not flip back, but it is only half-enforceable: input tokens are known pre-flight, output tokens only at stream end, so token limits are charged retrospectively (proposal 009) and a hard real-time cap is not available at this layer.

Where does reserved capacity sit in the routing order?

Chosen

Provisioned throughput first, on-demand as overflow, refuse only when every pool is dry. Envoy AI Gateway's quota-aware routing proposal routes "to Provisioned Throughput (PT) backends while quota is available", falls back to on-demand, and "Returns 429 only when all candidate backends are exhausted" (2026).

Rejected

Spilling to on-demand while paid-for reserved capacity idles, which is the behaviour you get from naive weighted routing that ignores quota state.

Flips when

The quality of the reserved pool differs from on-demand (an older snapshot, a different region): then cheapest-first is quietly a quality decision, and the routing order needs a product owner, not just a platform owner.

Figure 3 · The failover-trigger decision, as the published builds resolve it

The tree the LiteLLM defaults, the OpenRouter layering and the Assembled account jointly encode; every terminal is an action a gateway can take. Sources: LiteLLM router source, OpenRouter, Assembled.

Diagram source

Decision	Chosen	Rejected	Because	Evidence
Failover trigger	Automated on status codes + cooldown	Manual switching; probe-only health	Humans misclassify partial outages; probes miss silent degradation	Assembled, 2025
Breaker granularity	Per model deployment	Per provider	One model's overload should not drain healthy siblings; LiteLLM reworked this in PR #5358	LiteLLM PR, 2024
API dialect	OpenAI-compatible	House schema; native SDKs everywhere	Ecosystem libraries already speak it	InfoQ, 2024
Operate it	In-house or shared OSS	Hosted aggregator as sole route	Aggregator is a correlated failure domain across all providers	OpenRouter postmortem, 2026
Rate-limit unit	Tokens / cost	Requests	Requests do not correlate with spend or provider capacity	Envoy ADR 001
Observability identity	Preserve the client's requested model alongside the served one	Log only the rewritten name	Model overrides otherwise destroy attribution of usage and cost	Envoy ADR 005

What broke in production

Four failure classes cover the published record: the provider's correlated failure, the provider's silent failure, the gateway's own dependencies, and rot in the fallback path itself.

Sorting the incidents by failure class rather than by company makes the pattern visible. Class one, the provider's whole-platform failure, is what the gateway is sold against, and it is real: OpenAI lost everything for four and a half hours in December 2024. Class two is the provider failing while every signal a gateway routes on stays green; Anthropic's August 2025 month is the canonical account. Class three is the gateway's own auxiliary dependencies, which is what the only gateway-operator postmortem in this corpus is actually about. Class four is the quietest and, per the one measured account, the most common residual: the fallback path decays because nothing exercises it. Notice what is missing: no published postmortem in this corpus describes a consumer-side gateway failing over badly during a provider outage and taking the product down. Either the pattern works, or its failures are not written up; both readings argue for drilling the path rather than trusting it.

Postmortem

The provider lost its control plane, and every customer with one provider lost everything

AssumptionA hyperscale AI provider is internally redundant enough that customer-side failover is optional.

What happenedOpenAI deployed a telemetry service at 3:12 PM PST on 11 December 2024; its configuration made every node run resource-intensive Kubernetes API operations, overwhelming the control planes of the largest clusters. DNS caching masked the damage until caches expired, by which time the rollout was fleet-wide. Staging had not reproduced large-cluster behaviour.

Blast radiusAll OpenAI services significantly degraded or unavailable, 3:16 PM to 7:38 PM PST, about 4 hours 22 minutes.

FixOpenAI's remediations are provider-side (staged rollouts, control-plane load protection). The customer-side fix is structural: a second provider wired in before the outage, because nothing you do during it helps.

Design ruleTreat each provider as one failure domain regardless of its internal architecture; diversity you have not already configured is diversity you do not have.

SourceOpenAI postmortem, 2024-12

Postmortem

The provider degraded for a month and returned 200s the whole time

AssumptionProvider failure is visible in status codes, latency or error rates, the only signals a gateway routes on.

What happenedThree Anthropic infrastructure bugs degraded Claude's output quality between August and early September 2025. A routing bug sent short-context requests to servers configured for a 1M-token window, and "sticky" routing kept affected users pinned to the wrong pool; a TPU misconfiguration corrupted output; an XLA compiler bug miscompiled approximate top-k token selection. Anthropic's own evaluations "simply didn't capture the degradation users were reporting".

Blast radiusFrom 0.8% of Sonnet 4 requests on 5 August to 16% at the worst hour on 31 August after a routine load-balancing change amplified it; about 30% of Claude Code users had at least one misrouted message. Full rollback across first-party, Vertex and Bedrock took until 18 September.

FixAnthropic fixed routing, rolled back the TPU change, switched to exact top-k, and added output-quality detection to deployment. Nothing a consumer gateway monitors would have fired.

Design ruleStatus-code failover covers only the loud failure class; the quiet one needs sampled quality evaluation per route, owned by whoever owns the routing table.

SourceAnthropic postmortem, 2025-09-17

Postmortem

The aggregator's cache failed, and every provider behind it vanished at once

AssumptionA multi-provider gateway is more available than any single provider because it can route around failures.

What happenedOpenRouter's third-party cache for API-key lookups dropped its database connections and returned errors; on recovery, the invalidated cache stampeded the database. Auth-path timeouts surfaced to customers as 401 Unauthorized, sending them to debug perfectly valid keys. The first outage coincided with a DoS attack, which got the blame until the identical failure recurred two days later without one.

Blast radius38 minutes from 5:27 UTC on 17 February 2026 and 35 minutes from 7:36 UTC on 19 February; roughly 20% of requests failing initially, across every model and provider on the platform simultaneously.

FixInfrastructure failures during authorization now return 503 rather than 401, and a fallback cache was planned for extended cache downtime.

Design ruleThe gateway's availability is the product of its auxiliary dependencies, auth and quota state first; and an error code that misattributes blame multiplies the outage by your customers' debugging time.

SourceOpenRouter postmortem, 2026-02

Postmortem

The fallback was configured once and decayed silently until the outage audited it

AssumptionA fallback chain configured at launch still works months later.

What happenedDuring an Anthropic API incident Portkey describes as roughly 12 hours of instability, customers with fallbacks configured saw 99.86% of requests succeed. Portkey's listed causes for the 0.14%: fallbacks not configured for the 529 overload status code (the code that spiked most that day), "expired keys, non-existent targets", and rare failures of the fallback target itself. Vendor-reported; no independent verification found. Anthropic's status history separately records 529-class incidents such as "Elevated 529 errors on Sonnet 4" (July 2025).

Blast radius0.14% of gateway-fronted requests failed where zero was the design intent; for teams without fallbacks, the full incident duration.

FixEnumerate provider-specific failure signals (529 is not 429) in the trigger set; treat fallback credentials and targets as production config with expiry alerts and scheduled drills.

Design ruleAn untested fallback is a guess with a config syntax; the outage is the first test only if you decline to run one earlier.

SourcePortkey analysis (vendor), 2025; Anthropic status, 2025-07

Source

The fallback code path itself threw, which no outage was needed to expose

AssumptionThe library's fallback branch is exercised and correct because the happy path is.

What happenedA LiteLLM change from fallback.get("model") to fallback.pop("model") made the fallback invocation pass the model argument twice; the fallback path, and only the fallback path, raised a TypeError. Dudu Lasry hit it in production use, traced it, and filed the issue and fix upstream. LiteLLM's own cooldown machinery needed a rework the same year: PR #5358 moved cooldowns to per-model granularity and fixed retry-after ignoring provider-supplied values.

Blast radiusEvery request that needed a fallback, for anyone on the affected versions; invisible until a primary actually failed.

FixUpstream patch; the durable fix is failure-injection tests that force the fallback branch in CI.

Design ruleThe error-handling path is the least-executed code in the gateway and inherits none of the happy path's test coverage; coverage there has to be bought deliberately.

SourceLasry, n.d.; LiteLLM PR #5358, 2024

Source

The gateway added 8.3 seconds to streamed responses, from its own tokenizer

AssumptionA proxy in front of a 60-second model call is latency-free by comparison.

What happenedBricksLLM initialised a BPE tokenizer core per streaming chunk to count tokens; each initialisation took about 83 ms, so a 100-chunk response accumulated roughly 8.3 seconds of gateway-added latency. "Due to insufficient testing, we didn't realize that BricksLLM's latency increased significantly as the length of prompts and responses increased"; a prospect demanding sub-100 ms exposed it.

Blast radiusEvery streamed response through the gateway, scaling with response length; self-reported, 2023.

FixInitialise the tokenizer once; move post-response accounting onto an async event bus. Gateway overhead fell to about 30 ms.

Design ruleMeter the gateway's own added latency per chunk, not per request; token counting is the hidden hot loop of this layer.

SourceBricksLLM build log, 2023

Figure 4 · The fallback that fails during the failover: the class-four sequence

The primary's outage is survivable; the expired fallback credential is what reaches the user, and it was expired for months before this request. Assembled's account and Portkey's failure taxonomy, combined. Sources: Portkey, Assembled.

Diagram source

Figure 5 · A silent degradation, month by month: why no gateway signal fired

Five weeks separate the first bad deploy from full rollback, and the amplifying event in the middle was a routine load-balancing change; availability, latency and error-rate dashboards stayed green throughout. Source: Anthropic postmortem, 2025.

Diagram source

Numbers you can plan against

Provider downtime is the budget line; gateway overhead is the tax; the cost spread between models is why routing pays for itself.

Metric	Value	At	Context	As of	Source
Provider status-page uptime	99.80% / 99.58%	OpenAI / Anthropic	Over the months before the post; over 3 hours/month of downtime at the lower figure	2025-06	Assembled
Whole-platform outage duration	4 h 22 m	OpenAI	All services, telemetry deploy overwhelmed K8s control planes	2024-12-11	OpenAI
Silent degradation, peak share	16%	Anthropic	Of Sonnet 4 requests at the worst hour, up from 0.8% at introduction; ~30% of Claude Code users touched	2025-08-31	Anthropic
Gateway-operator outage	38 + 35 min	OpenRouter	Two incidents, same caching dependency; ~20% of requests failing initially	2026-02	OpenRouter
Fallback success in a real outage	99.86%	Portkey customers	Vendor claim, no independent check; residual 0.14% all config faults	2025	Portkey
Failover switch time	5+ min → ~100s of ms	Assembled	Manual config switching vs automated routing, self-measured	2025-06	Assembled
Gateway added latency	~30 ms	BricksLLM	After fixing an 83 ms-per-chunk tokenizer bug that added 8.3 s to 100-chunk streams; self-measured on an M1 laptop	2023	BricksLLM
Default breaker settings	5 s / 50% / 5 req	LiteLLM	Cooldown seconds; failure-rate threshold; minimum requests per minute before the rate applies	2026-10 (repo head)	LiteLLM source
Internal gateway volume	16M queries/mo	Uber	~30 teams, one year after launch	2024-09	InfoQ
Internal gateway volume	72k requests total	Wealthsimple	OpenAI + Cohere, from April 2023 launch to the post; the layer starts small	2023	Wealthsimple
Hosted-aggregator volume	25T tokens/wk	OpenRouter	5× growth in six months, per Series B announcement	2026-05	Pulse2
Price spread across providers	~100×	API market	"fees that can differ by two orders of magnitude" for similar tasks; cascades claimed to cut cost up to 98% at matched quality	2023-05	FrugalGPT
Learned-routing saving	>2×	RouteLLM (benchmarks)	Cost reduction from preference-trained strong/weak routing without quality loss, authors' measurement	2024-06	RouteLLM

Measured, claimed, derived

Measured by the party that operated the system: the Anthropic and OpenAI incident figures, LiteLLM's defaults (read from source), BricksLLM's latency (on a laptop, not a fleet). Claimed with an interest in the result: Portkey's 99.86%, Assembled's 99.97% effective uptime, FrugalGPT's 98% (a research benchmark, not production). Derived here: the "3+ hours a month" reading of 99.58% uptime, which is arithmetic on a status page that self-reports. No source in this corpus publishes the number an architect wants most, the end-to-end availability of a dual-provider setup measured over a year; that remains open.

The evidence wall

Every source behind this page, graded. Filter by kind.

Collection note for honesty's sake: this guide was researched from a build container whose network policy allowed direct fetches only to anthropic.com and GitHub's raw file and git endpoints. The Anthropic postmortem was read in full; LiteLLM, Envoy AI Gateway and BricksLLM were read from complete local clones; every other source below was read through server-side search extraction rather than a direct page fetch, and quotes from those sources are reproduced as the extraction returned them. The ledger shipped beside this page (sources.md) marks which is which, row by row.

Postmortem Anthropic2025-09-17

A postmortem of three recent issues

The fullest public account of a provider degrading without erroring: three overlapping infrastructure bugs, a routine load-balancing change as amplifier, sticky routing concentrating harm, and the provider's own evals missing what users reported.

Carry forward Everything your failover triggers on stayed green for a month; quality failure needs its own detector.

anthropic.com/engineering/a-postmortem-of-three-recent-issues

Postmortem OpenAI2024-12-12

API, ChatGPT & Sora facing issues: root-cause write-up

A telemetry deployment overwhelmed Kubernetes control planes in the largest clusters; DNS caching deferred the symptom until the rollout was everywhere. All services down or degraded for about four and a half hours.

Carry forward One provider is one failure domain; its internal redundancy is not your redundancy.

status.openai.com/incidents/ctrsv3lwd797

Postmortem OpenRouter2026-02

OpenRouter outages on February 17 and 19, 2026

The gateway's key-lookup cache failed twice; recovery stampeded the database; auth timeouts reached customers as 401s. The DoS attack blamed on day one was exonerated by the recurrence on day three.

Carry forward Audit which auxiliary dependencies sit on your gateway's request path, and what error code their failure wears.

openrouter.ai/announcements/openrouter-outages-on-february-17-and-19-2026

Postmortem Anthropic status2025-07

Elevated 529 errors on Sonnet 4

A status-history entry documenting the 529 overload code as a live, recurring provider signal, distinct from the per-customer 429. Cited here because 529 is exactly the code Portkey found missing from customers' fallback triggers.

Carry forward Build the trigger set from each provider's actual error vocabulary, not from the HTTP RFC.

status.anthropic.com/incidents/tm5t7t243nvx

Source BerriAI / LiteLLM2026-10 head

router_utils/cooldown_handlers.py and constants.py

The most widely deployed open implementation of the failover decision, read at commit 5338edb7: 5-second default cooldown, 50% failure threshold over a 5-request minimum, immediate cooldown on 429, and an explicit exemption for single-deployment groups.

Carry forward Never open a breaker in front of the only backend; the exemption is in the defaults because people got this wrong.

github.com/BerriAI/litellm/.../cooldown_handlers.py

Source BerriAI / LiteLLM2024-08

PR #5358: refactor cooldown logic to CooldownCache

The argument in code form: cooldowns moved to per-model granularity, and router retry-after fixed to honour the provider's own numbers instead of a local guess.

Carry forward Breaker granularity and retry timing both belong to the provider's semantics, not the gateway's convenience.

github.com/BerriAI/litellm/pull/5358

ADR Envoy AI Gateway2024

Proposal 001: founding design

The requirements document for the category, written by Bloomberg and Tetrate engineers: unified API across providers, upstream credential management, and token-based rate limiting called out as the extension Envoy itself lacked.

Carry forward If the unit of your rate limiter is requests, you have not built an LLM gateway yet.

github.com/envoyproxy/ai-gateway/.../001-ai-gateway-proposal

ADR Envoy AI Gateway2026

Proposal 009: quota-aware routing

Reserved-capacity-first routing: send traffic to provisioned-throughput pools while quota remains, overflow to on-demand, return 429 only when every candidate is exhausted; actual usage charged on a stream-done callback.

Carry forward Pre-flight checks see only input tokens; budget enforcement at this layer is always one generation behind.

github.com/envoyproxy/ai-gateway/.../009-quota-aware-routing

ADR Envoy AI Gateway2025

Proposal 005: original model tracking

When the gateway rewrites model names, metrics lose what the client asked for; the proposal adds an OriginalModel dimension so usage aggregates by requested and served model separately.

Carry forward Every rewrite the gateway performs must be recorded twice: what was asked, what was done.

github.com/envoyproxy/ai-gateway/.../005-original-model

Eng blog Uber2024-07-11

Navigating the LLM landscape: Uber's GenAI Gateway

A Go service wrapping vendor clients and in-house serving behind one OpenAI-style API, with PII redaction, caching and auth. Unusually honest about the limit: gateway availability is tied to vendor status, and redaction fights caching.

Carry forward The gateway centralises policy; it does not by itself add a single nine.

uber.com/blog/genai-gateway

Eng blog InfoQ2024-09

Uber builds GenAI gateway mirroring OpenAI API

Secondary coverage carrying the numbers Uber's post does not: about 30 teams, about 16 million queries a month, and the reasoning behind adopting the OpenAI dialect.

Carry forward Adoption, not architecture, is the gateway's success metric; the dialect decision drives adoption.

infoq.com/news/2024/09/uber-genai-gateway-llm-openai

Eng blog Grabn.d., checked 2026-10

Grab AI Gateway: connecting Grabbers to multiple GenAI providers

One reverse proxy before OpenAI, Azure, Bedrock and Vertex; short-lived exploration keys issued by Slack bot against long-term service keys; per-request cost computed at response time and archived with an audit trail.

Carry forward Make experimentation keys cheap to get and quick to die; key lifecycle is the real access-control surface.

engineering.grab.com/grab-ai-gateway

Eng blog Wealthsimple2023

Get to know our LLM Gateway

A regulated-finance build whose first justifications were audit and reliability, with retries and fallbacks in the programmatic API; 72,000 requests to OpenAI and Cohere in its first months.

Carry forward In regulated settings the gateway is the compliance artefact; reliability features ride along.

engineering.wealthsimple.com/get-to-know-our-llm-gateway...

Eng blog Assembled2025-06-12

Your LLM provider will go down, but you don't have to

The best published account of operating failover rather than designing it: manual switching failed on per-use-case routing, minutes of delay and misclassified partial outages; automation brought switching to hundreds of milliseconds.

Carry forward The on-call human is the slowest and least accurate component in a failover path.

assembled.com/blog/your-llm-provider-will-go-down-but-you-dont-have-to

Eng blog BricksLLM2023

How we built a highly scalable LLM gateway with Go

A build log with its own failure in it: per-chunk tokenizer initialisation added 8.3 seconds to streamed responses; the fix plus an async event bus brought gateway overhead to about 30 ms. Also the clearest statement of the sync/async split.

Carry forward Token counting is the gateway's hot loop; measure added latency per chunk.

github.com/bricks-cloud/BricksLLM/blob/main/blog/...

Eng blog Dudu Lasryn.d., checked 2026-10

Implementing an LLM fallback mechanism

A practitioner hits a TypeError that lives only in the library's fallback branch, traces it to a one-line change, and lands the fix upstream. The fallback path broke while the happy path was fine.

Carry forward Force the fallback branch in CI; it has no organic test coverage.

dudulasry.substack.com/p/implementing-an-llm-fallback-mechanism

Eng blog TechCrunch2025-09-10

Anthropic reports outages: Claude and Console impacted

Users on GitHub and Hacker News noticed the failure around 12:20 p.m. ET; the status update followed eight minutes later. A small, dated data point on detection order.

Carry forward Your error rates will know before the status page does; alert on your own signal.

techcrunch.com/2025/09/10/anthropic-reports-outages...

Eng blog Pulse22026-05

OpenRouter raises $113M as weekly volume reaches 25 trillion tokens

The scale figure that turns the aggregator from a convenience into shared infrastructure: 25 trillion tokens a week, five times the volume of six months earlier.

Carry forward Concentration you adopt for convenience becomes concentration risk at someone else's growth rate.

pulse2.com/openrouter-113-million-raised...

Eng blog CNCF2024-11-14

KubeCon NA 2024 day-2 keynote recap

Names the three pains that produced the shared gateway: divergent access patterns, divergent credential management, and models with service-specific needs.

Carry forward The gateway's first customer is the platform team drowning in provider differences, not the reliability engineer.

cncf.io/blog/2024/11/14/kubecon-cloudnativecon-north-america-2024...

Paper Chen, Zaharia, Zou2023-05

FrugalGPT (arXiv:2305.05176)

Documents the economic asymmetry that makes routing worth building: API fees differing by two orders of magnitude, and a learned cascade matching GPT-4 benchmarks at up to 98% lower cost.

Carry forward The router's budget case rests on price spread, which is a market fact that can narrow; date your assumptions.

arxiv.org/abs/2305.05176

Paper Ong et al., ICLR 20252024-06

RouteLLM: learning to route LLMs with preference data (arXiv:2406.18665)

Routes before generation rather than cascading after it: a router trained on preference data picks strong or weak per query, cutting cost by more than half on the authors' benchmarks without quality loss.

Carry forward Pre-generation routing is the only kind that also saves latency; cascades pay twice on escalation.

arxiv.org/abs/2406.18665

Talk Bloomberg @ KubeCon NA2024-11

Centralizing & simplifying enterprise AI workflows with Envoy AI Gateway

The end-user keynote in which Bloomberg presented the gateway as shared infrastructure: unified model access, usage limiting and upstream authorization, co-built in the open with Tetrate rather than kept as a fork.

Carry forward If every large company needs the same layer, the differentiation is in the routing policy, not the proxy.

youtube.com/watch?v=do1viOk8nok

Talk Twilio @ AI Engineer WF2026

Productionizing LLM gateways

Framing from the talk's published page: a fallback policy needs compatible behaviour, tested capacity and a switch point before streamed output commits; set timeouts from p99 measured per model and route. No in-video timestamps were reachable from this session.

Carry forward A fallback target you have not load-tested is capacity you are assuming, not capacity you have.

ai.engineer/talks/zrZ1amZBSPw-productionizing-llm-gateways-architecture

Vendor OpenRoutern.d., checked 2026-10

Reliability and automatic failover

The two-layer failover model: provider-level failover on by default within a model, cross-model fallback opt-in; and a concession worth the page, that internal failover cannot protect against faults in the routing layer itself.

Carry forward Ask any gateway vendor what happens when the gateway is the fault; the honest answer is "keep a second route".

openrouter.ai/blog/insights/reliability-failover

Vendor Portkey2025

Anthropic's 12-hour outage: customer traffic analysis

The one measured account of fallbacks under a real outage: 99.86% success for configured customers, with every listed residual failure a configuration fault (529 unmapped, expired keys, dead targets). Self-interested source; no independent check exists.

Carry forward Fallback config is production code with an expiry date; drill it on a schedule.

new.portkey.ai/announcements/anthropics-12-hour-outage

Vendor Cloudflare2026, checked 2026-10

The AI engineering stack we built internally

The gateway vendor routing its own internal AI traffic through its gateway product, 20 million requests reported, for routing and cost attribution. Dogfooding evidence, with the usual discount.

Carry forward Cost attribution, not failover, is the feature that makes finance fund this layer.

blog.cloudflare.com/internal-ai-engineering-stack

Build a miniature, then productionise it

Six rungs. The line from toy to real is crossed at rung four, where you start injecting the failures instead of waiting for them.

One endpoint, two providers

An HTTP service exposing the OpenAI chat-completions shape, translating to two real providers chosen by a model prefix. Streaming passthrough included, which is where the first evening goes.

Done when: the same client code, pointed at your base URL, gets answers from both providers without edits. Teaches: how much of the dialect is convention and how much (tool calls, stop reasons, usage fields) genuinely differs per provider.

Keys in, keys out

Issue your own API keys backed by a store; provider keys live only server-side. Add a per-key spend cap checked before forwarding, with token counting and cost accounting written after the response on a queue, not inline.

Done when: a key with a $1 budget is refused once spent, and the refusal arrives before the provider is called. Teaches: the synchronous gate / asynchronous ledger split, and why hard caps still leak one in-flight generation.

The breaker, at the right granularity

Classify provider errors (429, 5xx, provider-specific overload codes like 529, timeouts) and implement retry with provider-supplied retry-after, per-deployment cooldown, and fallback ordering. Copy LiteLLM's exemption: no cooldown when a model group has a single deployment.

Done when: killing one mock provider moves traffic within one request, and killing the only provider does not trip a breaker, it retries. Teaches: why breaker granularity and the trigger vocabulary are the whole design.

The streaming commit point

Make failover work for streamed responses: a TTFT timeout per route derived from measured p99, with fallback allowed only before the first token is forwarded. After that, fail visibly.

Done when: a provider that accepts the request and then stalls produces a fallback answer, and one that stalls mid-stream produces a clean error, not a spliced response. Teaches: that the switch window closes at the first committed token, which no retry policy survives.

The scheduled fallback drill

This is the rung the Portkey data argues for. A weekly job forces the primary to return overload codes for a sampled slice of synthetic traffic and asserts the fallback chain completes: valid keys, live targets, compatible parameters (max_tokens tuned for one model truncates on another).

Done when: the drill has caught one planted fault (an expired fallback key) without a human noticing beforehand. Teaches: config rot, the dominant residual failure in the one measured outage account.

The quality canary

Sample a fixed slice of production prompts, run them against each route daily, score with a cheap grader against reference outputs, and alert on drift per route. This is the detector the Anthropic incident proved status codes cannot replace.

Done when: swapping a route's backing model to a weaker one fires the alert within a day while error-rate dashboards stay green. Teaches: that the worst provider failure mode is invisible to every signal the previous five rungs route on.

Keep hunting

The queries that found the material above, so the reader can extend the search when this page goes stale.

First-person gateway builds

"LLM gateway" "we built" engineering blog provider fallback
<company> engineering "AI gateway" OR "GenAI gateway" blog
"exploration keys" OR "virtual keys" LLM gateway rate limit
"how we built" "LLM gateway" Go OR proxy latency

Incidents and outages

OpenRouter OR Portkey OR LiteLLM incident postmortem provider outage
status.openai.com OR status.anthropic.com incident "write-up" root cause
Anthropic postmortem infrastructure "degraded" quality routing
"fallbacks" "529" OR "overloaded" "expired keys" outage analysis

The argument layer: code and design records

repo:envoyproxy/ai-gateway path:docs/proposals
litellm cooldown "allowed_fails" router github
"provider fallback" aigateway.envoyproxy.io retry BackendTrafficPolicy
litellm github pull request cooldown CooldownCache retry-after

Routing economics and talks

FrugalGPT cascade cost arxiv
RouteLLM preference data router arxiv
kubecon "AI gateway" keynote Bloomberg Envoy
"LLM gateway" talk production ai.engineer OR qcon lessons

References

Anthropic, A postmortem of three recent issues anthropic.com, 2025-09-17. Checked 2026-10-13 (fetched in full).
OpenAI, API, ChatGPT & Sora Facing Issues: incident write-up status.openai.com, 2024-12-12. Checked 2026-10-13 (via search extraction).
OpenRouter, OpenRouter Outages on February 17 and 19, 2026 openrouter.ai, 2026-02. Checked 2026-10-13 (via search extraction).
Anthropic, Elevated 529 errors on Sonnet 4 status.anthropic.com, 2025-07. Checked 2026-10-13 (via search extraction).
BerriAI, LiteLLM router cooldown handlers (source) github.com, read at commit 5338edb7, 2026-10-13 (full local clone).
BerriAI, LiteLLM PR #5358: refactor cooldown logic github.com, 2024-08. Checked 2026-10-13 (via search extraction).
Envoy AI Gateway, Proposal 001: founding design github.com, 2024. Checked 2026-10-13 (full local clone).
Envoy AI Gateway, Proposal 009: quota-aware routing github.com, 2026. Checked 2026-10-13 (full local clone).
Envoy AI Gateway, Proposal 005: original model tracking github.com, 2025. Checked 2026-10-13 (full local clone).
Uber, Navigating the LLM Landscape: Uber's Innovation with GenAI Gateway uber.com, 2024-07-11. Checked 2026-10-13 (via search extraction).
InfoQ, Uber Builds GenAI Gateway Mirroring OpenAI API infoq.com, 2024-09. Checked 2026-10-13 (via search extraction).
Grab, Grab AI Gateway: Connecting Grabbers to Multiple GenAI Providers engineering.grab.com, n.d. Checked 2026-10-13 (via search extraction).
Wealthsimple, Get to know our LLM Gateway engineering.wealthsimple.com, 2023. Checked 2026-10-13 (via search extraction).
Assembled (John Wang), Your LLM provider will go down, but you don't have to assembled.com, 2025-06-12. Checked 2026-10-13 (via search extraction).
BricksLLM, How we built a highly scalable LLM gateway with Go github.com, 2023. Checked 2026-10-13 (fetched in full).
Dudu Lasry, Implementing an LLM fallback mechanism dudulasry.substack.com, n.d. Checked 2026-10-13 (via search extraction).
Portkey, Anthropic's 12-hour outage new.portkey.ai, 2025. Checked 2026-10-13 (via search extraction; vendor claim).
OpenRouter, Reliability & Automatic Failover openrouter.ai, n.d. Checked 2026-10-13 (via search extraction).
Cloudflare, The AI engineering stack we built internally blog.cloudflare.com, 2026. Checked 2026-10-13 (via search extraction).
Chen, Zaharia, Zou, FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance arXiv:2305.05176, 2023-05-09; TMLR 2024. Checked 2026-10-13 (via search extraction).
Ong et al., RouteLLM: Learning to Route LLMs with Preference Data arXiv:2406.18665, 2024-06-26; ICLR 2025. Checked 2026-10-13 (via search extraction).
Griffith (Bloomberg), Centralizing & Simplifying Enterprise AI Workflows with Envoy AI Gateway, KubeCon NA 2024 End User Keynote youtube.com, 2024-11. Checked 2026-10-13 (announcement text read from repository clone).
CNCF, KubeCon + CloudNativeCon North America 2024 Day 2 recap cncf.io, 2024-11-14. Checked 2026-10-13 (via search extraction).
Manuja (Twilio), Productionizing LLM Gateways, AI Engineer World's Fair ai.engineer, 2026. Checked 2026-10-13 (via search extraction).
TechCrunch, Anthropic reports outages: Claude and Console impacted techcrunch.com, 2025-09-10. Checked 2026-10-13 (via search extraction).
Pulse2, OpenRouter: $113 Million Raised As Weekly Volume Reaches 25 Trillion Tokens pulse2.com, 2026-05. Checked 2026-10-13 (via search extraction).
ZenML LLMOps database, summary of Grab's AI Gateway zenml.io, n.d. Checked 2026-10-13 (via search extraction; secondary to ref 12).