Cloud platforms convert demand into capacity in milliseconds and into an invoice in days. This guide reconstructs, from six published billing incidents and the platform mechanisms built in response, how production systems actually bound their own spend; and why every brake that works runs on estimated cost in the request path, never on the bill.
A control problem with the feedback wire cut: the actuator responds in milliseconds, the sensor reports tomorrow.
State the problem without naming a cloud. A system that converts incoming demand into capacity automatically has no intrinsic mechanism that converts money already spent back into a constraint. Autoscaling closes the loop between load and resources in seconds. Nothing closes the loop between resources and budget, because the only native signal on that path, billing data, is produced by a batch pipeline that runs hours to days behind the meter. AWS documents that Cost Explorer "refreshes your cost data at least once every 24 hours" and that Budgets updates three times a day, eight to twelve hours apart (docs, checked 2026). Firebase's billing documentation says plainly that "budgets and budget alerts do not cap your usage or charges" and that the alert itself can lag the cost "up to a few days" (Firebase docs, 2026). Corey Quinn, whose consultancy lives inside AWS bills, reports discarding billing data newer than two or three days as untrustworthy (Duckbill Group).
Meanwhile the spend side moves at request speed. Milkie Way's test deployment reached roughly one billion Firestore reads per minute within hours (postmortem, 2020). The gap between those two clocks, milliseconds of actuation against days of sensing, is the whole subject of this page. Every incident below is that gap monetised, and every mechanism below is an attempt to close it from one side or the other.
Who has faced this in production and written it down: startups that scaled into a bill (Milkie Way 2020, Cara 2024), individual operators billed by strangers or by their own configuration (an empty S3 bucket in 2024, Troy Hunt's uncached archive in 2022, a Netlify free-tier site in 2024), the platforms that then changed their billing semantics or shipped brakes (AWS 2023 to 2024, Vercel 2023, Netlify 2024 to 2025, Google Cloud 2026), platforms that shipped hard caps from the start (Supabase), and the LLM-gateway ecosystem now rebuilding admission-time budget enforcement from scratch (LiteLLM). Security research named the adversarial version of the problem "denial of wallet" in 2021 (Kelly, Glavin and Barrett).
The hard spending cap, the single most requested billing feature across every major cloud's forums, is not something the industry never built. Google built it (App Engine's daily spending limit), removed it between 2019 and 2023 because its coverage could not keep up with the platform, and shipped it again in July 2026 in a deliberately narrower form: enforced on estimated costs, for an eligible-services list, with persistent resources exempt. The reintroduction concedes the thesis of this page in the vendor's own design: a cap wired to actual billing data cannot work, because the bill is structurally late. Details and sources in section 3.
Scope. This guide covers runaway spend: cost that accumulates faster than the organisation can observe it. It deliberately does not cover steady-state cost optimisation, rightsizing, commitment purchasing or cloud repatriation (the 2026-08-30 guide on owning hardware covers that ground), and it does not cover FinOps organisational practice beyond where it touches enforcement.
Reconstructed across AWS, Google Cloud, Vercel, Netlify, Supabase and the LLM-gateway ecosystem: four planes, distinguished by how long they take to act.
No published system bounds spend with one mechanism. Across every account in the evidence wall, the same four planes recur, and the honest way to draw them is by time to act, because that is what decided each incident's blast radius.
The admission plane acts in milliseconds and knows nothing about money.
It is quotas, rate limits, concurrency caps and loop breakers, enforced in the request
path. Lambda's recursive loop detection is the clearest specimen: since July 2023 the
platform stops a function invoked by the same triggering event more than 16 times, on by
default (AWS).
The mechanism is not in the billing system at all; it is an SDK middleware that stamps the
X-Ray trace header onto outbound requests so the platform can count hops, visible in
aws-sdk-go-v2's
recursion_detection.go. Google Cloud's equivalent advice for App Engine after removing
spending limits was max_instances
(docs), which
is also the parameter whose default of 1,000 amplified Milkie Way's test into $72k
(postmortem, 2020). The admission plane
is the only plane fast enough to stop the worst incidents in this corpus. It is also the
least used, because its knobs are per-service, unglamorous, and default to permissive.
The estimation plane acts in minutes. It multiplies the real-time usage meter by the price list to approximate spend before the billing pipeline confirms it, and wires the result to an actuator. Vercel's Spend Management (October 2023) pauses projects when a spend amount is reached and exposes webhooks so teams can wire their own actions (Vercel); a later change made pausing production deployments the default behaviour (changelog). Google Cloud's spend cap budgets, shipped in preview July 2026, are explicit about the design: caps trigger on estimated costs precisely because estimates arrive "much faster than the actual costs are processed and appear on billing reports" (docs). LiteLLM, an LLM gateway, goes one step further into the request path: it reserves the estimated cost of a call against the budget and rejects the request before it reaches the provider if headroom is insufficient, with an optional fail-closed mode that returns 503 when spend cannot be verified (LiteLLM docs).
The billing plane acts in hours to days and cannot be made faster from outside. Metering records are batched, rated, aggregated and only then compared against budgets. Everything built directly on it inherits its latency: AWS's own Innovation Sandbox solution, which leases sandbox accounts against a budget, carries an open issue documenting a "24-hour+ detection blind spot" because its enforcement reads Cost Explorer, with an estimated "$800+ per-incident exposure" during the window (issue #92). The billing plane is where the money is authoritative and where control is impossible. Its proper role is reconciliation and anomaly detection over days, and the industry's current work there is standardising the data itself: the FinOps Foundation's FOCUS specification for billing data, and OpenCost computing cost in real time from the Kubernetes control plane's own resource accounting rather than waiting for the invoice.
The social plane acts in days to weeks, and it is load-bearing whether or not anyone admits it. Netlify's CEO described the operating control in 2024 as policy, not software: "it's currently our policy not to shut down free sites during traffic spikes that don't match attack patterns but instead forgive any bills from legitimate mistakes after the fact" (HN comment). AWS refunded the S3 empty-bucket bill "as an exception" (Pocwierz, 2024). Google refunded Milkie Way in full (2020). Refund-on-outcry works for the customer who goes viral; it is not a control an architect can design against, which is why every one of these platforms later moved enforcement into the planes above.
Counts causal hops in the request path via a propagated trace header and stops the chain at a threshold (16 on Lambda). Runs before any money is counted. Shipped default-on, which briefly broke teams who used recursion deliberately.
Runs this way at: AWS Lambda, mechanism in aws-sdk-go-v2
Real-time usage times price, compared to a target, wired to "pause new work". Google Cloud enforces at 100% of target with alerts at 50% and 80%; paused services stay paused until a human lifts the cap. Persistent resources keep accruing.
Runs this way at: Google Cloud (preview, 2026), Vercel, Supabase (default on)
A budget notification triggers a function that unlinks billing from the project, terminating every service in it. The community productised it with Terraform; its README warns the stop is not graceful, may delete resources irretrievably, and still trails spend because the trigger rides the billing pipeline.
Runs this way at: Cyclenerd's kill switch, per Google Cloud's own documented pattern
Five forks, each with the condition that flips it. The central one is twenty years old and was answered three different ways by the same company.
| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Enforcement plane | Request path, on estimates | Billing plane | Billing data is 8h to days late; caps on actuals cannot stop fast spend | GCP docs, 2026; ISB issue #92 |
| Trip behaviour | Pause new usage, keep state | Disable billing (kill all) | Billing disable is documented as non-graceful and possibly unrecoverable | GCP docs; Cyclenerd README |
| Cap default | On for bounded-budget tiers | Forgive-after-the-fact | Refund-on-outcry does not scale and selects for virality | Netlify CEO, 2024; feature thread |
| Rejected-traffic billing | Platform absorbs it | Customer pays for 403s | Unauthorized requests are outside the customer's control entirely | AWS, 2024-05; AWS, 2024-11 |
| Alert consumer | Actuator, then human | Email only | Runaway conditions saturate the humans the email is addressed to | Hunt, 2022; TechCrunch, 2024 |
| Budget data model | Standard schema (FOCUS), control-plane cost feeds (OpenCost) | Per-vendor bill parsing | Cross-vendor enforcement and allocation need one vocabulary | FOCUS spec; OpenCost spec |
Google is the only vendor to have answered the central question three ways. App Engine shipped a daily spending limit; Google closed it to new apps in December 2019, deprecated it in July 2020 and shut it down in July 2023, with the stated reason that "while App Engine has evolved, the spending limit functionality has not", it no longer covered Flex or Cloud Build (official response in the removal thread). Three years of community kill-switch scripts later, spend cap budgets shipped in July 2026, narrower and estimate-based (launch post). The removal reason and the reintroduction design are the same lesson from both sides: a cap is only honest for the services whose usage it can actually see and stop.
Three failure classes cover every published incident found: the loop you built, the meter others can run, and the signal that failed. No class is closed by the fixes shipped so far.
max_instances times unit price times a day is what a bug can spend before anyone is told.max_budget on a gateway key means requests stop when the budget is exhausted.fail_closed_budget_enforcement, rejecting with 503 when spend cannot be verified against the store.What the three classes share: in no incident above did the failure begin in the billing system. It began in the request path (a loop, a hot file, a name collision, a growth spike), and the billing system's only role was to be too slow to matter. The corollary an architect should carry: the denial-of-wallet literature's mitigations, rate limiting, concurrency caps, execution timeouts (Kelly et al., 2021), are identical to the accidental-runaway mitigations. You get the adversarial defence and the self-inflicted defence with the same controls, which is the best cost-benefit line in this whole domain.
The two columns that matter for design are the spend rates and the detection delays; your exposure is always their product.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Peak read amplification | ~1B reads/min | Milkie Way | Cloud Run at default max 1,000 instances hitting Firestore | 2020 | postmortem |
| Spend rate, self-amplification | $72,000/day | Milkie Way | 116B reads at $0.06 per 100k, plus compute | 2020 | Register |
| Spend rate, bandwidth | ~$26,000/day | Netlify customer | Derived: $104,500 over 4 days; 60.7 TB peak day at $55/100GB | 2024 | thread |
| Spend rate, adversarial requests | ~$650/day | Pocwierz | Derived: ~$1,300 in 2 days from ~100M unauthorized PUTs/day | 2024 | postmortem |
| Spend rate, organic growth | $96,280/week | Cara on Vercel | 56M function invocations/day at peak; 40k to 650k users in a week | 2024 | Zhang; InfoQ |
| AWS Cost Explorer refresh | ≥24 h | AWS | "at least once every 24 hours"; some data later | 2026 | docs |
| AWS Budgets evaluation | 3×/day, 8–12 h apart | AWS | Upper bound on billing-plane reaction time | 2026 | docs |
| GCP budget alert lag | up to a few days | Google/Firebase | "delay between incurring costs and receiving a budget alert" | 2026 | docs |
| Practitioner trust horizon | 2–3 days | Duckbill Group | Bill data newer than this treated as unsettled | 2021, standing | Quinn |
| Enforced-cap overshoot | minutes | Vercel | Meter runs past the limit before pause takes effect | 2026 | bex.co analysis |
| GCP spend cap thresholds | 50 / 80 / 100% | Google Cloud | Alerts at 50 and 80; enforcement pauses eligible services at 100% of target, on estimated cost | 2026 | docs |
| Lambda loop threshold | 16 invocations | AWS | Same triggering event; then dropped to DLQ, default on | 2023 | announcement |
| Billing-plane sandbox exposure | $800+/incident | AWS Innovation Sandbox | Estimated loss during the 24 h Cost Explorer blind spot | 2026 | issue #92 |
The incident figures ($72k, $104.5k, $96,280, $1,300, AUD 11,448) are reported by the party that paid them, and in three of five cases were later refunded or reduced; they measure exposure, not final cost. The two "spend rate" derivations marked as derived are this guide's arithmetic from reported totals and durations. Platform cadences (24 h, 3x/day, few days) are vendor documentation, checked September 2026, and are floors rather than guarantees. The Vercel pause-overshoot figure comes from one independent analysis of vendor documentation, not from a measured incident.
Every source behind this page, graded. Filter by kind. Retrieval for this guide ran through search-engine page retrieval plus direct fetches of raw GitHub content; the ledger in sources.md records which quotes are verbatim.
The canonical self-amplification incident, written by the founder with full numbers: default max instances, read rates, the day-late billing sync, the refund.
An operator who teaches security hygiene professionally, billed ~AUD 11,448 because a file outgrew the CDN's 15 GB cache ceiling; available alerts were unconfigured.
First-person forum account of the $104,500 free-tier bill: 60.7 TB peak day, $55/100GB overage pricing, support reducing then the CEO waiving it.
~100M unauthorized PUTs a day against an empty bucket, billed to its owner. The post that forced a global billing-semantics change within three weeks.
The founder's own posts on the bill, the log discrepancies she questioned, and the optimisation work that followed; the vendor's warnings were missed amid the growth.
The estimated-billing pipeline corrupted by a pricing config change; customer budget alarms fired on false data; AWS disabled budget and anomaly alerts platform-wide during mitigation.
The CEO stating the then-current control in public: no shutdown of free sites on non-attack spikes, bills from legitimate mistakes forgiven after the fact.
Fifty-plus replies over 18 months, with Netlify's staged responses landing in the thread: free-plan suspension, usage notifications, account-wide pause, rate limiting.
Terraform-packaged kill switch implementing Google's documented disable-billing pattern, with a custom role so only a billing admin can re-enable. Its README carries the honest caveats: non-graceful, possibly unrecoverable, and still behind the billing lag.
The loop breaker's actual mechanism: SDK middleware stamps the X-Ray trace header on outbound requests from Lambda so the platform can count hops of the same event.
AWS's own budget-leased sandbox solution, enforcement built on Cost Explorer, documented as blind for up to 24 hours with $800+ estimated per-incident exposure; proposes event-driven detection instead.
A recorded argument about enforcement: end-user budgets tracked spend but allowed unlimited overage; the community fix was closed without merging, and the gap was re-reported. The eventual design reserves estimated cost pre-request.
The official response in the removal thread: the limit no longer covered what the platform had become (Flex, Cloud Build), so it was retired rather than extended. Deprecated 2020-07-24, shut down 2023-07-01.
A community specification defining a vendor-neutral schema for cost and usage data across clouds and SaaS; the standardisation effort is itself evidence of how unusable raw billing feeds are for cross-vendor control.
Vendor-neutral spec for measuring and allocating Kubernetes costs from the control plane's own resource accounting, in real time, rather than from the provider invoice.
Independent write-up of the Milkie Way incident with the unit arithmetic: $0.06 per 100k reads times 116 billion reads is $69,600 of the $72k.
Reported figures for the Cara incident: 56M invocations/day peak, the growth curve, and Vercel's response including the claim that warnings were sent ahead of the bill.
Corroborates the growth numbers behind the bill and records Vercel's public response that outreach emails preceded the invoice.
The practitioner's operating rule from inside hundreds of AWS bills: billing data younger than two or three days is treated as unsettled.
Close reading of the spend-management docs: the pause actuator lets the meter run for minutes past the limit, and the five-axis function pricing makes the limit hard to predict from traffic.
Places the watching-cost in context: teams with good observability spend on the order of 20 to 30% of their infrastructure bill to get it, per Honeycomb's leadership.
Practitioner synthesis written directly off the Netlify incident: caps where offered, static-first architectures, and hosting side projects on flat-price infrastructure.
Practitioner reception of the 2026 spend caps from an author who had previously published the DIY kill-switch pattern; documents the enforcement-on-estimates behaviour and the service coverage limits.
A running catalogue of serverless billing incidents, including the Netlify $104,500 entry. Used here as an index; primary write-ups are cited directly above.
Kelly, Glavin and Barrett define DoW as mass, continual invocation causing financial exhaustion rather than unavailability, and enumerate the mitigations: rate limiting, concurrency limits, timeouts, budgets.
Surveys the field's evolution: attack taxonomy (blast, continual-inconspicuous, background-chained) and a shift toward ML-based detection in six of the ten most recent papers reviewed.
Praval Panwar applies capacity-planning frameworks from airlines, power grids and logistics to cloud systems, arguing against oscillating between over-provisioning and panic-scaling; capacity, cost and performance as one signal set.
The platform CTO's doctrine: make cost a non-functional requirement, unobserved systems lead to unknown costs, and cost-aware architectures implement cost controls. "Cost awareness is a lost art."
The cap's semantics in the vendor's own words: enforcement on estimated costs because they arrive faster than actuals; eligible services only; persistent resources keep accruing; paused until manually lifted.
The two sentences that define the problem: budgets and alerts do not cap usage or charges, and the alert can lag the cost by up to a few days.
Sixteen invocations of the same triggering event, then stop and route to the failure destination, on by default across regions; later extended to per-function config and S3 loops.
The billing-semantics fix for the empty-bucket class: unauthorized 403s from outside the account are free, all regions, no application changes.
Realtime usage alerts, SMS notifications, webhooks, APIs, and automatic project pausing at a defined spend amount; a later changelog made pausing production the default.
The counter-example default: the Pro plan ships with the spend cap on; over-quota usage of an item is disallowed until the next billing cycle rather than billed.
Each rung produces a number or a proven behaviour you did not have before. The line from toy to real is crossed at rung four.
Deploy a trivial metered workload (a function calling a paid API in a loop, capped at a few dollars by iteration count). Record when the spend appears in provider metrics, in the billing console, and in a budget alert.
Done when: you have three timestamps and can state your detection latency per surface. Teaches: the lag is real, measurable, and different per tool.
In an isolated project with quotas pre-set low, deploy a service that writes to its own trigger. Watch the platform's loop breaker fire (or fail to exist) and watch instance counts against the configured maximum.
Done when: you can name the exact mechanism that stopped the loop, and what would have stopped it if that mechanism were absent. Teaches: defaults, not budgets, decided the outcome.
Connect a budget notification (Pub/Sub topic, webhook) to a function that actually stops something: pause a project, set a quota to zero, detach billing in a throwaway project. Trip it with synthetic spend.
Done when: synthetic spend causes an automated stop with no human in the loop, and you know the end-to-end trip time. Teaches: the IAM and plumbing between "alert" and "stop" is where these projects stall.
Put a metering gateway (LiteLLM for LLM traffic, or a thin proxy of your own) in front of a paid API. Implement reserve-then-call against a budget: estimate cost, check headroom, reject at exhaustion. Load-test past the budget.
Done when: the overshoot equals your in-flight requests times per-request cost, and you have chosen fail-open or fail-closed for a store outage in writing. Teaches: estimate-based admission control, the only real-time brake there is.
For every service in one production system, list the current quota, concurrency and max-instance settings against measured peak usage. Set each to peak times a deliberate multiplier, and document the multiplier.
Done when: a load test hits a quota before it could hit the budget alert. Teaches: quotas are the hard bound the billing plane pretends to be.
Inject a runaway in a production-like environment: a loop with the breaker disabled, or replayed unauthorized traffic. Measure time-to-detect, time-to-stop, and money burned, with the on-call operating from runbooks only.
Done when: spend MTTR is a number your team has seen and reduced at least once. Teaches: the human and IAM latencies that never show up in architecture diagrams.
Per environment, decide in writing: which services pause at the cap, which are never paused automatically, what the plausibility bounds on the estimate feed are, and who can lift a tripped cap (not the project's own service accounts, per the Cyclenerd role design).
Done when: the policy is reviewed alongside availability SLOs and the pause path is tested quarterly. Teaches: the brake is an availability decision, and availability owners must sign it.
The queries that found this material, grouped by what they surface. The incident vocabulary ("bill", "burnt", "horror") outperforms any technology term.
"we burnt" OR "billing horror" cloud bill postmortemsite:news.ycombinator.com "bill" vercel OR netlify OR firebase"empty S3 bucket" unauthorized requests billserverlesshorrors.com"spend cap" OR "spending limit" site:docs.cloud.google.com"budgets" "do not cap" usage chargeslambda "recursive loop detection" default"spend management" pause production changelog"Cost Explorer" "24 hours" refresh latencyrepo:aws-solutions "Cost Explorer" latency is:issue"budget alert" delay "few days" firebase"denial of wallet" serverless arxivDoW attack taxonomy detection surveylitellm max_budget enforcement is:issue OR is:pr