Evidence ledger
34 sources
Checked 14 Sep 2026
Evidence ledger
One row per claim in The bill is not a brake: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic: how production systems bound their own cloud spend when the platform converts demand
into capacity automatically, and the only native feedback signal — billing data — arrives hours
to days after the money is spent. Research date: 2026-09-14. Retrieval note: this session
ran inside a network-restricted environment; raw.githubusercontent.com was fetched directly,
and all other hosts were read through the search tool's server-side retrieval, which returns
page content and verbatim passages. Quotes marked (retrieval summary) are the retrieval
tool's close paraphrase rather than a hand-copied sentence; every other quote was returned
verbatim by retrieval. URLs are reproduced exactly as returned by search or fetch; none are
reconstructed from memory.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Milkie Way | We Burnt $72K testing Firebase + Cloud Run and almost went Bankrupt [Part 1] | postmortem | 2020-12 | 2026-09-14 | https://blog.tomilkieway.com/72k-1/ | A \(7 budget and a free-tier plan coexisted with a \)72,000 overnight spend; the budget constrained nothing | "a test with a \(7.00 billing budget and a free database plan on Google Cloud platform (GCP) generated a \)72,000 invoice overnight" (retrieval summary) |
| 2 | Milkie Way | Part 2: the mechanism | postmortem | 2020-12 | 2026-09-14 | https://blog.tomilkieway.com/72k-2/ | The amplifier was a platform default: max-instances preset to 1,000, concurrency 80; billing and dashboards lagged by ~a day | "The max-instances is preset to 1,000, and concurrency set to 80 … Firebase Reads were at one point about 1 Billion Requests per minute!"; "made 116 Billion reads and 33 Million Writes to Firestore"; "Billing takes about a day to be synced, and the Firebase Dashboard took more than 24 hours to update" |
| 3 | The Register | Google Cloud (over)Run: how a free trial experiment ended with a $72,000 bill overnight | casestudy | 2020-12-10 | 2026-09-14 | https://www.theregister.com/2020/12/10/google_cloud_over_run/ | Independent arithmetic on the same incident: the cost was dominated by per-operation read pricing | "Most of the cost was down to Firebase read operations, even at just \(0.06 per 100,000. Multiply that by 116 billion and you get \)69,600" (retrieval summary) |
| 4 | Troy Hunt / HIBP | How I Got Pwned by My Cloud Costs | postmortem | 2022-01 | 2026-09-14 | https://www.troyhunt.com/how-i-got-pwned-by-my-cloud-costs/ | An experienced operator with alerts available but unconfigured found out from the invoice; cause was a CDN cache limit quietly exceeded | Azure Blob Storage egress after Cloudflare stopped caching a >15 GB archive; storage usage fee AUD 4,557.13, total communication cost ~AUD 11,448; spike began 2021-12-20, noticed on the monthly bill (retrieval summary; figures as reported by GIGAZINE's contemporaneous account of the post) |
| 5 | Netlify customer | Netlify billing horror story (support forum) | postmortem | 2024-02 | 2026-09-14 | https://answers.netlify.com/t/netlify-billing-horror-story/113392 | A free-tier static site accrued $104,500 in four days of bandwidth billing before any human intervened | Bandwidth "peaked at 60.7TB on February 16th. Netlify charges \(55 for every 100GB of excess bandwidth, resulting in the \)104,500 bill" (retrieval summary); support first reduced the bill to $5,225 |
| 6 | Netlify (CEO) | HN comment by biilmann on the $104k thread | source | 2024-02 | 2026-09-14 | https://news.ycombinator.com/item?id=39521986 | The provider's stated control at the time was social, not technical: forgive after the fact rather than stop during | "It's currently our policy not to shut down free sites during traffic spikes that don't match attack patterns but instead forgive any bills from legitimate mistakes after the fact." |
| 7 | Netlify community | Add automated kill switch or rate limiting to prevent excess billing from DDOS attacks (feature thread, 50+ replies) | source | 2024-02 → 2025-10 | 2026-09-14 | https://answers.netlify.com/t/add-automated-kill-switch-or-rate-limiting-to-prevent-excess-billing-from-ddos-attacks/113359 | The provider's remedy arrived in layers over 18 months: guaranteed-free suspend (Nov 2024), usage notifications at 50/75/100% plus account-wide pause (Sep 2025), function rate limiting and firewall rules (Oct 2025) | Thread timeline: "a new guaranteed 'Free' plan, where instead of being charged overages your site is suspended once any monthly limits are exhausted" (Nov 2024); "if one project exceeds its limits, all projects on your account will be paused" (Sep 2025); rate limiting for functions/edge functions (Oct 2025) (retrieval summary) |
| 8 | Maciej Pocwierz | How an empty S3 bucket can make your AWS bill explode | postmortem | 2024-04 | 2026-09-14 | https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1 | Anyone who knew a bucket's name could bill its owner: ~100M unauthorized PUTs in a day cost ~$1,300 in two days | "nearly 100,000,000 S3 PUT requests had been executed against this newly created bucket in a single day"; AWS charged "the same amount for PUTs that result in a 403 authentication error" (retrieval summary) |
| 9 | AWS (Jeff Barr) | X post acknowledging the S3 unauthorized-request billing problem | vendor | 2024-04-30 | 2026-09-14 | https://x.com/jeffbarr/status/1785386554372042890 | AWS conceded the billing semantics were wrong, in public, within days | "We agree that customers should not have to pay for unauthorized requests that they did not initiate." |
| 10 | AWS | Amazon S3 will no longer charge for several HTTP error codes | vendor | 2024-05-13 | 2026-09-14 | https://aws.amazon.com/about-aws/whats-new/2024/05/amazon-s3-no-charge-http-error-codes/ | The fix was a billing-plane rule change, applied globally with no customer action | "bucket owners will never incur request or bandwidth charges for requests that return an HTTP 403 (Access Denied) error response if initiated from outside their individual AWS account or AWS Organization" (retrieval summary); effective across all S3 APIs by 2024-08 |
| 11 | AWS | Amazon CloudFront no longer charges for requests blocked by AWS WAF | vendor | 2024-11 | 2026-09-14 | https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-cloudfront-charges-requests-blocked-aws-waf | The same class of fix repeated six months later on a second service: stop metering traffic the customer rejected | Title claim; announcement of no charges for WAF-blocked CloudFront requests |
| 12 | InfoQ | Vercel's serverless scale expenses (Cara) | casestudy | 2024-06 | 2026-09-14 | https://www.infoq.com/news/2024/06/vercel-serverless-scale-expenses | Organic growth is billing-indistinguishable from an attack: 40k→650k users in a week, 56M function invocations/day | "Function usage spiked at 56 million function invocations per day on June 3"; user base "from 40,000 to 650,000 in just a week" (retrieval summary) |
| 13 | Cara (Jingna Zhang) | X posts on the Vercel bill | postmortem | 2024-06 | 2026-09-14 | https://x.com/zemotion/status/1800959534057529394 | The founder's own accounting of the bill and the follow-up optimization work | "I've been working on that \(100k bill with vercel, getting our code optimized, and making sure that the app can be more stable moving forward"; earlier post reports the bill as \)96,280 for one week (second post: https://x.com/zemotion/status/1798558292681343039) |
| 14 | TechCrunch | Cara grew from 40k to 650k users in a week | casestudy | 2024-06-06 | 2026-09-14 | https://techcrunch.com/2024/06/06/a-social-app-for-creatives-cara-grew-from-40k-to-650k-users-in-a-week-because-artists-are-fed-up-with-metas-ai-policies/ | Corroboration of the growth numbers and of Vercel's response that warning emails were sent but missed | "Vercel's vice president of product Lee Robinson replied publicly, claiming that his team attempted to reach out ahead of time" (retrieval summary) |
| 15 | Vercel | Spend Management: Realtime Usage Alerts, SMS Notifications, and Project Pausing | vendor | 2023-10-05 | 2026-09-14 | https://vercel.com/blog/introducing-spend-management-realtime-usage-alerts-sms-notifications | The platform's brake is a pause actuator wired to a spend threshold, plus webhooks to build your own | "automatic project pausing when you reach your defined spend amount … flexible webhooks to instantly take action on real-time usage updates … new APIs to programmatically pause and resume your projects" (retrieval summary) |
| 16 | Vercel | Spend Management now pauses production deployments by default | vendor | undated changelog (retrieved 2026-09) | 2026-09-14 | https://vercel.com/changelog/spend-management-now-pauses-production-deployments-by-default | The default flipped from observe to enforce: reaching the spend amount now pauses production | Title claim; changelog entry (see also https://vercel.com/changelog/improved-hard-caps-for-spend-management) |
| 17 | bex.co | Vercel's Spending Cap Doesn't Cap Your Spending: The Bandwidth Math Behind $23K Bills | blog | 2026-07-08 | 2026-09-14 | https://bex.co/blog/2026/07/08/vercel-no-spending-cap | Even the enforced cap trails the meter: pausing takes minutes, and the trade is bill shock for outage risk | "The pausing feature allows the meter to run for several minutes past the limit before stopping service, meaning teams are trading bill shock for outage risk rather than eliminating either" (retrieval summary); Functions billed on five axes (Active CPU, Provisioned Memory, Invocations, Fast Data Transfer, Edge Requests) |
| 18 | Google Cloud | Create, edit, or delete budgets and budget alerts | vendor | current docs | 2026-09-14 | https://docs.cloud.google.com/billing/docs/how-to/budgets | A budget is a notification object, not a control: it does not stop usage | "Setting a budget does not automatically cap Google Cloud usage or spending … they don't automatically prevent the use or billing of services when budget thresholds are met or exceeded" (retrieval summary) |
| 19 | Firebase (Google) | Avoid surprise bills | vendor | current docs | 2026-09-14 | https://firebase.google.com/docs/projects/billing/avoid-surprise-bills | The alert itself is late by design | "Budgets and budget alerts do not cap your usage or charges."; "There can be a delay between incurring costs and receiving a budget alert (depending on the service, up to a few days)." |
| 20 | Google Cloud | Disable billing usage with notifications | vendor | current docs | 2026-09-14 | https://docs.cloud.google.com/billing/docs/how-to/disable-billing-with-notifications | The documented hard stop is terminal: disabling billing kills every service in the project | "When you disable billing on a project, you terminate all Google Cloud services in the project, including Free Tier services … following these steps doesn't guarantee you won't spend more than your budget" (retrieval summary) |
| 21 | Cyclenerd (community) | poweroff-google-cloud-cap-billing (Terraform kill switch) | source | maintained repo, fetched at head | 2026-09-14 | https://github.com/Cyclenerd/poweroff-google-cloud-cap-billing | The community productized the nuclear option, and its own README warns about lag and irrecoverability | "The resources may not shut down gracefully and be irretrievably deleted. There is no gracefully recovery if you disable Cloud Billing."; "Recommendation: If you have a hard funds limit, set your maximum budget below your available funds to account for billing delays." (verbatim, raw README) |
| 22 | Google (support, via groups thread) | Spending LImits Going Away :( — official response | adr | 2019–2020 thread; feature shut down 2023-07-01 | 2026-09-14 | https://groups.google.com/g/google-appengine/c/dqm8JH2eAo4 | The recorded rationale for removing the only hard cap: the cap's coverage did not keep up with the platform | "While App Engine has evolved, the spending limit functionality has not. Google is replacing this feature because it doesn't cover App Engine costs related to newer capabilities, or services like App Engine Flex or Cloud Build." (retrieval summary of the official support response); deprecated 2020-07-24, shut down 2023-07-01 (per https://cloud.google.com/appengine/docs/standard/managing-costs) |
| 23 | Google Cloud | Manage spend cap budgets (Preview) | vendor | 2026-07 | 2026-09-14 | https://docs.cloud.google.com/billing/docs/how-to/budgets-spend-caps | The cap that finally shipped enforces on estimates, covers only eligible API-based services, and exempts persistent resources | "Spend caps typically use estimated costs to trigger alerts and caps, enforcing a cap much faster than the actual costs are processed and appear on billing reports" (retrieval summary); eligible: Google AI Studio, Gemini Enterprise Agent Platform, Cloud Run, Cloud Run Functions, Maps; alerts at 50% and 80%, enforcement at 100%; paused until manually lifted |
| 24 | Google Cloud | New early anomalies and spend caps on Google Cloud Budgets | vendor | 2026-07-28 | 2026-09-14 | https://cloud.google.com/blog/topics/cost-management/new-early-anomalies-and-spend-caps-on-google-cloud-budgets | The stated driver for shipping caps after two decades is AI-priced usage breaking intuition | "Today, we are announcing two native features in the Google Cloud Billing console: early anomalies on AI services and spend caps on Google Cloud Budgets."; framing that traditional metrics like requests per second no longer predict the bill for AI workloads (retrieval summary) |
| 25 | AWS | AWS Lambda now detects and stops recursive loops in Lambda functions | vendor | 2023-07-13 | 2026-09-14 | https://aws.amazon.com/about-aws/whats-new/2023/07/aws-lambda-detects-recursive-loops-lambda-functions/ | The platform inserted a loop breaker into the invocation path, on by default | "If a function is invoked by the same triggering event more than 16 times, Lambda will stop the next invocation"; "This feature is turned on by default" (retrieval summary); extended to function-level config 2024-08 and Lambda↔S3 loops 2024-10 (https://aws.amazon.com/about-aws/whats-new/2024/10/aws-lambda-detects-stops-recursive-loops-lambda-s3/) |
| 26 | AWS | aws-sdk-go-v2 recursion detection middleware (source) | source | fetched at head | 2026-09-14 | https://github.com/aws/aws-sdk-go-v2/blob/main/aws/middleware/recursion_detection.go | The loop breaker's mechanism is a propagated trace header stamped by SDK middleware, i.e. admission control in the request path | "RecursionDetection detects Lambda environment and sets its X-Ray trace ID to request header if absent to avoid recursion invocation in Lambda" (verbatim code comment; header X-Amzn-Trace-Id, env AWS_LAMBDA_FUNCTION_NAME) |
| 27 | AWS Solutions (community reporter) | innovation-sandbox-on-aws issue #92: Lease budget monitoring has a 24-hour+ detection blind spot due to Cost Explorer data latency | source | 2025–2026, open | 2026-09-14 | https://github.com/aws-solutions/innovation-sandbox-on-aws/issues/92 | AWS's own reference sandbox inherits the billing lag: budget enforcement built on Cost Explorer cannot see spend for up to 24h | "Cost Explorer data can be delayed up to 24 hours from when costs are incurred and refreshes at most 3 times per day … sandbox costs can accumulate well beyond configured budget thresholds before ISB can see them"; "$800+ per-incident exposure during the 24-hour blind spot" (retrieval summary); proposes event-driven detection via CloudTrail/EventBridge |
| 28 | AWS | Analyzing your costs and usage with AWS Cost Explorer (docs) | vendor | current docs | 2026-09-14 | https://docs.aws.amazon.com/cost-management/latest/userguide/ce-what-is.html | The pipeline cadence in the vendor's own words | "Cost Explorer refreshes your cost data at least once every 24 hours" (verbatim per retrieval); AWS Budgets "is updated three times a day, and updates usually occur between eight and 12 hours after the last update" (retrieval summary) |
| 29 | Duckbill Group (Corey Quinn) | Ask Me Anything About AWS Billing | blog | retrieved 2026-09 | 2026-09-14 | https://www.lastweekinaws.com/blog/ask-me-anything-about-aws-billing/ | Practitioners who live in the bill treat days-old data as the freshest trustworthy data | Quinn "tends to disregard bill data that's newer than two or three days old because there's an eventual consistency issue within the billing system" (retrieval summary) |
| 30 | InfoQ | AWS Billing Bug Shows Customers Trillion-Dollar Estimates While Its Own Cost Alarms Fail to Act | postmortem | 2026-07 | 2026-09-14 | https://www.infoq.com/news/2026/07/aws-billing-estimates-incident/ | The alert plane is itself a failure domain: bad estimates fired customers' budget alarms, and AWS disabled budget and anomaly alerts platform-wide during mitigation | "a configuration change in the bill computation system introduced a unit pricing error into the estimated billing pipeline" (2026-07-16 19:46 PDT); "AWS's own alarms detected the anomalies but failed to halt bill generation or page engineers; customer escalations alerted the company 4.5 hours later"; "Budget and cost anomaly alerts were disabled platform-wide during mitigation" (retrieval summary); one dashboard showed $7.1T month-to-date |
| 31 | NUI Galway (Kelly, Glavin, Barrett) | Denial of Wallet — Defining a Looming Threat to Serverless Computing | paper | 2021-04 (arXiv), journal 2021 | 2026-09-14 | https://arxiv.org/abs/2104.08031 | The academic definition of the attack class: financial exhaustion without availability impact | DoW defined as "the intentional mass and continual invocation of serverless functions, resulting in financial exhaustion of the victim in the form of inflated usage bills" (retrieval summary of the paper's definition) |
| 32 | La Trobe University (Dorsett, Mann, Chowdhury, Mahmood) | A Comprehensive Review of Denial of Wallet Attacks in Serverless Architectures | paper | 2025-08 (arXiv) | 2026-09-14 | https://arxiv.org/abs/2508.19284 | The literature has moved from defining the attack to ML-based detection; taxonomy includes blast, inconspicuous, and background-chained variants | "key developments including the categorization of attack types—such as Blast DDoW, Continual Inconspicuous DDoW, and Background Chained attacks"; "six of the ten most recent papers incorporating ML/AI approaches" (retrieval summary) |
| 33 | BerriAI / community (sammcj) | litellm PR #9658: fails to block requests over end-user budget (allows unlimited spending) — closed unmerged | source | 2025, closed unmerged | 2026-09-14 | https://github.com/BerriAI/litellm/pull/9658 | Enforcing a budget in the request path is hard enough that a widely deployed LLM gateway shipped budgets that did not block; the fix attempt was closed without merging | "litellm fails to block requests over end-user budget (allows unlimited spending)" (PR title); follow-up bug https://github.com/BerriAI/litellm/issues/11083 records the enforcement gap persisting |
| 34 | LiteLLM | Proxy config: budget reservation and fail-closed enforcement | source | current docs | 2026-09-14 | https://docs.litellm.ai/docs/proxy/config_settings | The working design: reserve estimated cost before the provider call, reject on insufficient headroom, optionally fail closed (503) when spend cannot be verified | "If the reservation would exceed the budget, LiteLLM rejects the request before sending it to the provider"; fail_closed_budget_enforcement "rejects with 503 when spend cannot be verified" (retrieval summary) |
| 35 | Supabase | Spend cap (billing docs) | vendor | current docs | 2026-09-14 | https://supabase.com/docs/guides/platform/cost-control | A counter-example platform: cap on by default, over-quota usage disallowed until the next cycle | "After exceeding the quota for a usage item, further usage of that item is disallowed until the next billing cycle" (retrieval summary); Pro plan ships with the spend cap enabled by default |
| 36 | USENIX SREcon26 Americas (Praval Panwar, Microsoft) | Infinity Is Not a Strategy: Right-Sizing the Cloud | talk | 2026-03 | 2026-09-14 | https://www.usenix.org/conference/srecon26americas/presentation/panwar | Practitioner framing: reason about capacity, cost and performance as one signal set instead of oscillating between over-provisioning and panic | Abstract: applies capacity-planning frameworks "from airlines, power grids, and logistics" to cloud systems; argues against "oscillating between over-provisioning and panic-scaling when something spikes" (retrieval summary; timestamped video claims could not be verified from the research environment) |
| 37 | AWS (Werner Vogels) / InfoQ | The Frugal Architect keynote, re:Invent 2023 | talk | 2023-11/12 | 2026-09-14 | https://www.infoq.com/news/2023/12/frugal-architect-werner-vogels/ | The design-side doctrine from the platform's own CTO: cost is a non-functional requirement and cost-aware architectures implement cost controls | Laws include "Make Cost a Non-functional Requirement" and "Cost Aware Architectures Implement Cost Controls"; "Cost awareness is a lost art" (Vogels, as quoted in coverage) (retrieval summary; keynote video timestamps could not be verified from the research environment) |
| 38 | FinOps Foundation | FOCUS: FinOps Open Cost and Usage Specification | adr | active spec, fetched at head | 2026-09-14 | https://github.com/FinOps-Open-Cost-and-Usage-Spec/FOCUS_Spec | The industry's answer to unreadable bills is a standard schema for billing data — evidence the raw feed was bad enough to need one | "FOCUS establishes a consistent, vendor-neutral vocabulary for billing and usage data." (verbatim, raw README) |
| 39 | OpenCost (CNCF) | OpenCost Specification v0.1 | adr | fetched at head | 2026-09-14 | https://github.com/opencost/opencost/blob/develop/spec/opencost-specv01.md | Real-time cost must be computed from the control plane's own resource accounting, not the invoice, in Kubernetes environments | "The OpenCost Specification is a vendor-neutral specification for measuring and allocating infrastructure and container costs in Kubernetes environments." (verbatim, raw file) |
| 40 | ServerlessHorrors | Incident collection: $104,500.123 (Netlify) and others | blog | ongoing collection | 2026-09-14 | https://serverlesshorrors.com/all/netlify-104k/ | A curated index of runaway-bill incidents exists and keeps growing — used here as an index, with primary write-ups cited directly | Collection entry for the Netlify $104.5k bill; site catalogues serverless billing incidents |