Evidence ledger
26 sources
Checked 13 Oct 2026
Evidence ledger
One row per claim in When the model provider goes down: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic: how organisations route application traffic to LLM providers they do not operate, the gateway layer they build to do it, and what provider outages revealed about both.
Checked date for every row: 2026-10-13.
Collection method note. This session ran inside a cloud container whose egress policy
allowed direct fetches only to anthropic.com, github.com git transport and
raw.githubusercontent.com. The Anthropic postmortem was fetched in full; LiteLLM, Envoy AI
Gateway and BricksLLM material was read from full local clones of the repositories (LiteLLM at
commit 5338edb7, 2026-10-13); every other source was read through the session's web-search
tool, which fetches pages server-side and returns extracts. Quotes below marked (extract)
were captured that way rather than from a direct page fetch. Link liveness for the blocked
hosts could not be re-verified locally; the verifier run records them as unreachable from the
build container.
| # | Org | Title | Tier | Published | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|
| 1 | Anthropic | A postmortem of three recent issues | postmortem | 2025-09-17 | https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues | Three overlapping infrastructure bugs degraded Claude output quality Aug–early Sep 2025 while serving stayed up | "Between August and early September, three infrastructure bugs intermittently degraded Claude's response quality." (full fetch) |
| 1b | Anthropic | same | postmortem | 2025-09-17 | same | A routine load-balancing change amplified a routing bug from 0.8% to 16% of Sonnet 4 requests | "This bug initially affected 0.8% of requests. On August 29, a routine load balancing change unintentionally increased the number of short-context requests routed to the 1M context servers. At the worst impacted hour on August 31, 16% of Sonnet 4 requests were affected." |
| 1c | Anthropic | same | postmortem | 2025-09-17 | same | Sticky routing concentrated damage on individual users | "once a request was served by the incorrect server, subsequent follow-ups were likely to be served by the same incorrect server." Also: "Approximately 30% of Claude Code users who made requests during this period had at least one message routed to the wrong server type, resulting in degraded responses." |
| 1d | Anthropic | same | postmortem | 2025-09-17 | same | The provider's own evals did not catch what users reported | "The evaluations we ran simply didn't capture the degradation users were reporting, in part because Claude often recovers well from isolated mistakes." |
| 1e | Anthropic | same | postmortem | 2025-09-17 | same | Quality-for-capacity trade denied on the record | "We never reduce model quality due to demand, time of day, or server load." |
| 2 | OpenAI | API, ChatGPT & Sora Facing Issues — write-up | postmortem | 2024-12-12 | https://status.openai.com/incidents/ctrsv3lwd797 | A telemetry deployment overwhelmed Kubernetes control planes and took all OpenAI services down ~4.5h | Deployed 3:12 PM PST Dec 11; outage 3:16–7:38 PM PST; new telemetry service caused every node to run resource-intensive K8s API operations; DNS caching masked the failure until caches expired; tested in staging where the largest-cluster behaviour did not reproduce. (extract) |
| 3 | OpenRouter | OpenRouter Outages on February 17 and 19, 2026 | postmortem | 2026-02 | https://openrouter.ai/announcements/openrouter-outages-on-february-17-and-19-2026 | The gateway's own caching dependency, not any model provider, took the gateway down; auth-path failure surfaced as 401s | Third-party cache used "for fast API key lookups against our database" dropped connections; ~20% of requests failed initially; invalidated cache entries stampeded the database; timeouts surfaced to users as 401 errors; 38 min from 5:27 UTC Feb 17 and 35 min from 7:36 UTC Feb 19; DoS attack initially blamed on Feb 17 until the Feb 19 recurrence without one. Fix: infrastructure failures now return 503 rather than 401; fallback cache planned. (extract) |
| 4 | OpenRouter | OpenRouter Reliability & Automatic Failover | vendor | n.d. (checked 2026-10-13) | https://openrouter.ai/blog/insights/reliability-failover/ | Two failover layers: provider-level on by default, model-level opt-in | Provider-layer failover is default within one model; model-layer fallbacks are opt-in via a models array; the write-up acknowledges internal failover cannot protect against a fault in the routing layer itself. (extract) |
| 5 | Uber | Navigating the LLM Landscape: Uber's Innovation with GenAI Gateway | blog | 2024-07-11 | https://www.uber.com/blog/genai-gateway/ | The gateway is a thin Go layer around vendor clients plus in-house serving, mirroring one API | "a Go service that acts as an encompassing layer around the clients for third-party vendors, complemented by the in-house serving stacks tailored for Uber's LLMs" (extract) |
| 5b | Uber | same | blog | 2024-07-11 | same | Indirection does not buy availability; Uber says so | Gateway availability "is closely tied to the operational status of these vendors". (extract) |
| 5c | Uber | same | blog | 2024-07-11 | same | PII redaction and response caching conflict | LLM caching "faces a dilemma": the PII redactor's placeholders make cached results hard to tie back to original entities. (extract) |
| 6 | InfoQ | Uber Builds GenAI Gateway Mirroring OpenAI API | blog | 2024-09 | https://www.infoq.com/news/2024/09/uber-genai-gateway-llm-openai/ | Scale and the OpenAI-format decision | ~30 customer teams, ~16 million queries per month at time of writing; OpenAI API format chosen because of wide adoption and open-source library compatibility (LangChain, LlamaIndex). (extract) |
| 7 | Grab | Grab AI Gateway: Connecting Grabbers to multiple GenAI providers | blog | n.d. (checked 2026-10-13) | https://engineering.grab.com/grab-ai-gateway | One reverse proxy in front of OpenAI, Azure, Bedrock and Vertex with two key classes | "exploration keys ... short-lived with a validity period of a few days, stricter rate limit restrictions, and access limited to only the staging environment", requested through a Slack bot; service keys for production. (extract) |
| 7b | Grab | same | blog | n.d. | same | Cost is computed per request at the gateway and archived | "The gateway performs cost calculations for each request once the response is received from the provider. The cost is archived in the data lake along with an audit trail." (extract) |
| 7c | Grab | same (via ZenML LLMOps database summary) | blog | n.d. | https://www.zenml.io/llmops-database/building-a-multi-provider-genai-gateway-for-enterprise-scale-llm-access | Reverse-proxying provider SDKs breaks on SDK edge cases | "edge cases where certain SDK functionalities didn't work as expected due to missing paths or configurations", addressed through integration testing with SDKs before deployments. (extract) |
| 8 | Wealthsimple | Get to know our LLM Gateway | blog | 2023 (n.d.; checked 2026-10-13) | https://engineering.wealthsimple.com/get-to-know-our-llm-gateway-and-how-it-provides-a-secure-and-reliable-space-to-use-generative-ai | A regulated-finance org built the gateway for audit and reliability first | Launched "our own internal LLM gateway which addresses two pain points" (security, reliability); over 72,000 requests to OpenAI and Cohere since April 2023 launch; retry and fallback in the programmatic API. (extract) |
| 9 | Cloudflare | The AI engineering stack we built internally — on the platform we ship | vendor | 2026 (n.d.; checked 2026-10-13) | https://blog.cloudflare.com/internal-ai-engineering-stack/ | Even the vendor of an AI gateway routes its internal traffic through one, for routing and cost tracking | 20 million requests reported routed through AI Gateway internally. (extract) |
| 10 | BricksLLM | How we built a highly scalable LLM gateway with Go | blog | 2023 (n.d.; checked 2026-10-13) | https://github.com/bricks-cloud/BricksLLM/blob/main/blog/how-we-built-a-highly-scalable-LLM-gateway-with-go.md | The gateway's own token counting was the latency bug | "we tried to initialize the core BPE for go-tiktoken for each streaming chunk ... Each initialization took about 83ms ... 100 streaming chunks ... would add 8.3 seconds of latency." (full fetch via raw.githubusercontent) |
| 10b | BricksLLM | same | blog | 2023 | same | The synchronous floor of a gateway is the pre-flight budget check | Redis usage lookup for rate limiting "is about 30ms" and "can not be made async since the rate limit is a feature that needs to happen before the request is forwarded"; post-response accounting moved to an async event bus, total overhead reduced "to about 30ms". (full fetch) |
| 10c | BricksLLM | same | blog | 2023 | same | Key theft and unbounded spend motivated the gateway | "horror stories regarding people losing thousands of dollars from stolen API keys"; first feature was rate-limiting OpenAI spending via API keys. (full fetch) |
| 11 | BerriAI (LiteLLM) | litellm/router_utils/cooldown_handlers.py + litellm/constants.py | source | repo head 2026-10-13 (commit 5338edb7) | https://github.com/BerriAI/litellm/blob/main/litellm/router_utils/cooldown_handlers.py | The default circuit breaker: one 429 cools a deployment for 5s; >50% failure rate over ≥5 requests/min cools it; single-deployment groups are exempt | DEFAULT_COOLDOWN_TIME_SECONDS = 5, DEFAULT_FAILURE_THRESHOLD_PERCENT = 0.5, DEFAULT_FAILURE_THRESHOLD_MINIMUM_REQUESTS = 5, SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD = 1000; code comment: "by default we should avoid cooldowns on single deployment model groups". (local clone) |
| 12 | BerriAI (LiteLLM) | PR #5358: refactor cooldown logic | source | 2024-08 | https://github.com/BerriAI/litellm/pull/5358 | Cooldown had to be reworked to per-model granularity; retry-after was not honouring provider values | PR introduces a CooldownCache, "updates cooldown logic to cooldown individual models", and fixes "retry after on router was not using azure / openai numbers". (extract) |
| 13 | Envoy AI Gateway | Proposal 001: Envoy AI Gateway (founding design) | adr | 2024 (in-repo) | https://github.com/envoyproxy/ai-gateway/blob/main/docs/proposals/001-ai-gateway-proposal/proposal.md | Token-denominated rate limiting was a founding requirement, not an add-on | "Rate Limiting for LLM workload is based on tokens, we extend Envoy Gateway to support generic cost based rate limiting as envoy so far only supports request based rate limiting." Axiom: "The MVP heavily assumes that the requests are sent using the OpenAI spec". (local clone) |
| 14 | Envoy AI Gateway | Proposal 009: Quota-Aware Routing | adr | 2026 (in-repo) | https://github.com/envoyproxy/ai-gateway/blob/main/docs/proposals/009-quota-aware-routing/proposal.md | Reserved capacity first, on-demand as overflow, refusal only when every pool is dry | Route to "Provisioned Throughput (PT) backends while quota is available and fall back to On-Demand capacity when PT quota is exhausted"; the quota service "Returns 429 only when all candidate backends are exhausted". (local clone) |
| 15 | Envoy AI Gateway | Proposal 005: Original Model Tracking | adr | 2025 (in-repo) | https://github.com/envoyproxy/ai-gateway/blob/main/docs/proposals/005-original-model/proposal.md | Model rewriting at the gateway destroys attribution unless the original request is preserved explicitly | "When model name overrides are configured, however, the original client request is not preserved"; introduces OriginalModel so usage can be aggregated by what the client asked for vs what was sent. (local clone) |
| 16 | Chen, Zaharia, Zou (Stanford) | FrugalGPT | paper | 2023-05-09 (arXiv:2305.05176; TMLR 2024) | https://arxiv.org/abs/2305.05176 | Provider prices for similar capability differ by orders of magnitude, which is the economic case for routing | API fees "can differ by two orders of magnitude"; the LLM cascade "can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction". (extract) |
| 17 | Ong et al. (UC Berkeley/Anyscale/Canva) | RouteLLM: Learning to Route LLMs with Preference Data | paper | 2024-06-26 (arXiv:2406.18665; ICLR 2025) | https://arxiv.org/abs/2406.18665 | Pre-generation routing trained on preference data, vs post-generation cascades | Framework for "learning efficient router models that dynamically select between a stronger and weaker LLM during inference"; authors report cost reduction "by over 2 times" without sacrificing response quality. (extract) |
| 18 | Portkey | Anthropic's 12-hour outage (customer-traffic analysis) | vendor | 2025 (n.d.; checked 2026-10-13) | https://new.portkey.ai/announcements/anthropics-12-hour-outage | During a provider outage, configured fallbacks succeeded 99.86%; the 0.14% failed from fallback misconfiguration, not the outage | Failure causes listed: fallbacks not configured for the 529 status code; "expired keys, non-existent targets"; rare cases where the fallback target itself failed. Fallback targets: OpenAI, Azure, Gemini, hosted Llama models. (extract; vendor-reported, no independent verification found) |
| 19 | Assembled | Your LLM provider will go down, but you don't have to | blog | 2025-06-12 | https://www.assembled.com/blog/your-llm-provider-will-go-down-but-you-dont-have-to | Manual provider switching fails for three operational reasons; automation cut failover from minutes to sub-second | OpenAI and Anthropic status pages "reported 99.80% and 99.58% uptime respectively over recent months" (as of 2025-06); manual switching failed on per-use-case routing, response delays, and poor outage classification; automated failover dropped switch time from 5+ minutes to hundreds of milliseconds; claims 99.97% effective uptime across provider outages. (extract) |
| 20 | TechCrunch | Anthropic reports outages (Claude and Console impacted) | blog | 2025-09-10 | https://techcrunch.com/2025/09/10/anthropic-reports-outages-claude-and-console-impacted/ | Provider outages are noticed by users before the status page | Users on GitHub and Hacker News "first noticed problems at around 12:20 p.m. ET", Anthropic's status update came eight minutes later; APIs, Console and Claude down. (extract) |
| 21 | Anthropic | Status incident: Elevated 529 errors on Sonnet 4 | postmortem | 2025-07 | https://status.anthropic.com/incidents/tm5t7t243nvx | 529 (overloaded) is a distinct provider-capacity signal, separate from per-customer 429 | Incident record titled "Elevated 529 errors on Sonnet 4". (extract; title and classification only) |
| 22 | Bloomberg / Envoy AI Gateway | End User Keynote at KubeCon NA 2024: Centralizing & Simplifying Enterprise AI Workflows | talk | 2024-11-14 | https://www.youtube.com/watch?v=do1viOk8nok | Bloomberg centralises model access, usage limiting and upstream authorization in a shared gateway | Keynote by Alexa Griffith (Bloomberg); project described as "a collaborative open source effort led by engineers from Bloomberg and Tetrate", providing "unified, scalable solution for model access, usage limiting, and upstream authorization". (blog text in repo clone; no claim timestamped within the video) |
| 23 | CNCF | KubeCon NA 2024 day 2 keynote recap | blog | 2024-11-14 | https://www.cncf.io/blog/2024/11/14/kubecon-cloudnativecon-north-america-2024-day-2-keynotes-announcements-and-more/ | The three pains that drove a shared gateway | The gateway "aims to solve 3 major pain points common to LLMs: different providers need different access patterns, use different ways to manage credentials, and service-specific models have different needs." (extract) |
| 24 | Twilio (AI Engineer World's Fair 2026) | Productionizing LLM Gateways | talk | 2026 | https://www.ai.engineer/talks/zrZ1amZBSPw-productionizing-llm-gateways-architecture | A fallback needs compatible behaviour, tested capacity, and a switch point before streaming commits | Talk abstract: a fallback policy needs more than a second provider: compatible behaviour, tested capacity, and an opportunity to switch before streamed output commits the response; measure p99 latency per model and route and set timeouts to match. (extract from talk page; no in-video timestamp available to this session) |
| 25 | Bloomberg | Tetrate and Bloomberg Release Open Source Envoy AI Gateway | vendor | 2025 (n.d.; checked 2026-10-13) | https://www.bloomberg.com/company/press/tetrate-and-bloomberg-release-open-source-envoy-ai-gateway-built-on-cncfs-envoy-gateway-project/ | The gateway was released as shared open-source infrastructure rather than kept internal | Press release announcing the open-source release, built on CNCF's Envoy Gateway. (extract) |
| 26 | Pulse2 | OpenRouter: $113 Million Raised As Weekly Volume Reaches 25 Trillion Tokens | blog | 2026-05 | https://pulse2.com/openrouter-113-million-raised-for-ai-model-exchange-platform-as-weekly-volume-reaches-25-trillion-tokens/ | The hosted-gateway aggregation point is now very large, which is what makes it a concentration risk | Volume "surged to 25 trillion tokens per week", "a 5X increase from the 5 trillion tokens processed per week just six months ago" (as of the May 2026 Series B). (extract) |
| 27 | Dudu Lasry | Implementing an LLM fallback mechanism | blog | 2024 (n.d.; checked 2026-10-13) | https://dudulasry.substack.com/p/implementing-an-llm-fallback-mechanism | The fallback path is code, and it breaks like code | Author hit TypeError: litellm.main.acompletion() got multiple values for keyword argument 'model' in the fallback path itself, traced it to a fallback.get("model") to fallback.pop("model") change, and filed the issue and fix upstream. (extract) |
Gaps the hunt could not close
- No rejected, closed-unmerged pull request on fallback or routing semantics was reachable
from this session (GitHub's web and API surfaces were blocked by the container's egress
policy; only git transport and raw file access worked). The argument layer that rejected PRs
usually provide is represented here by the Envoy AI Gateway proposals and LiteLLM PR #5358
instead. For the retry-budget argument, the sibling dig
distributed-systems/2026-08-29-retry-storms-and-metastable-failurealready documents three Envoy PRs closed unmerged. - No consumer-side postmortem names its LLM gateway as the component that failed during a provider outage. The gateway postmortem that exists (OpenRouter, #3) is about the gateway's own auxiliary dependency. Treated in the page as an open finding, not as evidence the consumer-side pattern works.
- Portkey's 99.86% figure (#18) has no independent corroboration. Labelled as a vendor claim everywhere it is used.
- Talk claims (#22, #24) could not be timestamped to video positions from this session; both are cited to their published pages.