We Burnt $72K testing Firebase + Cloud Run
A test deployment recursively scaled to ~1,000 Cloud Run instances driving ~1B Firestore reads/minute; budgets alerted but did not stop spend, and billing lagged hours. $72K in an afternoon, March 2020.
In the summer of 2025, 37signals deleted its AWS account and Stack Overflow unracked its last physical server: the two loudest self-hosting exemplars of the 2010s crossed in opposite directions, weeks apart, and both moves were economically right. This guide reconstructs the rent-or-own decision from the documented record, so a reader can put a defensible number on their own answer and name the conditions that would flip it.
The problem, stated without naming any vendor: whether to run steady workloads on computers you rent by the hour, at a premium that buys elasticity, or on computers you own that depreciate whether you use them or not, with the people, spares and lead times that ownership implies.
Start with the surprise, because it disciplines everything after it. On 30 June 2025, 37signals' final S3 contract lapsed and the company finished moving 18PB onto its own flash arrays, completing an exit it had priced at more than $10M of five-year savings (DHH, 2024; The Register, May 2025). Within weeks, Stack Overflow, the company whose nine on-prem web servers were the standing rebuttal to cloud maximalism for a decade, unracked the last of roughly fifty servers in New Jersey and went entirely to Google Cloud and Azure (Stack Overflow, Aug 2025). Neither company was confused. They had different load shapes, different storage gravity, different teams, and, decisively, different forcing events. The lesson of the whole corpus is that direction is not ideology: every documented move in either direction pairs a forcing event with a unit-economics model, and the same arithmetic points different ways for different shops.
The second thing to fix before going further is the statistics, because the debate runs on numbers that measure different things. The famous 83% figure, from a Barclays CIO survey and amplified by a16z, counts companies moving at least one workload back from public cloud; a company relocating one compliance database and a company exiting entirely land in the same bucket (Channelnomics analysis, 2024). IDC's workload survey puts full repatriation plans at 8–9% of companies, while Gartner has public cloud spend growing past $723B in 2025, up more than 20% in a year (The Stack, Nov 2024). All three numbers are simultaneously true. Repatriation in the record is overwhelmingly workload-level, not company-level, and the interesting question for an architect is not "is the herd leaving" but "which of my workloads is on the wrong side of the line, and what would moving one actually cost".
Scope. This guide covers the economics and architecture of moving steady production workloads between rented cloud and owned or colocated hardware, in both directions, plus the middle positions (rented dedicated servers, and self-hosted software on cloud VMs). It deliberately does not cover GPU and AI-training capacity economics, data-sovereignty mandates, desktop virtualisation, or the choice between SaaS and self-hosted applications; and it treats the Broadcom/VMware licensing shock only as a forcing event, not as a hypervisor comparison.
Across 37signals, Dropbox, X, OneUptime and GEICO, the successful moves share one shape, and the parts everyone keeps renting are as instructive as the parts they buy.
The common playbook has six stages, and each is attributable. A forcing event opens a decision window: 37signals' $3.2M annual bill and an expiring S3 commitment (DHH, 2022), GEICO's bill rising 2.5× over roughly a decade of cloud-first (Weekly, QCon SF 2024), Dropbox's unit storage cost at hundreds of petabytes. A unit-economics model follows: Ahrefs prices a server-month, Fastmail prices a 2U slot, Dropbox priced cost-of-revenue for the S-1. Then a portability layer: every workload that moved was already, or was first made, an ordinary container or process with open-source datastores underneath. 37signals wrote its own deployment tool, Kamal, whose README promise is exactly this portability: "Deploy web apps anywhere," from bare metal to cloud VMs (basecamp/kamal). Then a landing zone (two colo sites and an ops partner for 37signals; OCP racks and an in-house platform org for GEICO), a phased dual-run cutover (seven apps in about six months at 37signals, per the June 2023 completion post), and finally a remainder that stays rented.
The remainder deserves more attention than it gets. Dropbox, the canonical exit, still served European customers from AWS and kept under 10% of storage there after Magic Pocket, because building datacenters on another continent was a worse trade than renting them (ARCHITECHT, 2017). 37signals kept its CDN and assorted SaaS. Nobody in this corpus repatriated the edge. Inferred, and worth stating plainly: the exits in the record are exits from compute and storage at steady state, never from the cloud as a category.
Three answers exist in production. Buy arrays: 37signals put 18PB of Pure flash in two DCs for about $1.5M plus under $1M of five-year support. Build: Dropbox wrote Magic Pocket, adopted SMR drives first among major operators, and erasure-codes at exabyte scale. Rent anyway: GitLab kept cloud object storage even while planning a metal move, and that dependency helped reverse the plan.
Sources: DHH 2024, Dropbox, The Register 2016
GEICO is building a Kubernetes-and-OpenStack private cloud across OCP hardware, an enterprise platform for hundreds of teams. 37signals went the other way on purpose: containers pushed over SSH, to, in DHH's words, "dodge the complexity of Kubernetes, and avoid any sort of enterprisey service contract entanglements." Fleet size and team count pick the winner, not taste.
Zerodha, India's largest broker, self-hosts open-source everything (PostgreSQL, ClickHouse and dozens more) on plain EC2, with about 5% external-vendor dependency, and its CTO estimates at least $3M/yr saved against proprietary subscriptions. The repatriation that matters to them is at the software layer; the hardware stays rented for compliance and convenience.
Source: CNCF case study
One mechanism underlies the whole cloudward-to-metal lane, and it is worth naming because the papers wrote it down before the bloggers did. Owned hardware is cheap only at high utilisation, and high utilisation is engineered, not assumed. Google's Borg paper attributes its cluster economics to "admission control, efficient task-packing, over-commitment, and machine sharing" (EuroSys 2015); Facebook's f4 paper drove effective storage replication from 2.8× down to 2.1× with a two-datacenter XOR scheme, saving 53PB of raw disk against 65PB of logical data as of 2014 (OSDI 2014). That is what the rent premium pays someone else to do. The stay-side mirror of the same idea is in AWS's own Karpenter consolidation design, which deletes or replaces underutilised nodes against their price (design doc). Whichever side of the line you sit on, somebody has to do the packing; the decision is who, and at what margin.
The two mirrored decisions of summer 2025, then the recurring forks with the conditions that flip each one.
Read together, the two blocks say the quiet part: the decision inputs are load shape, storage gravity, fleet size, team appetite and the contract calendar. Vendor pricing appears in both models, but neither company moved because a price changed; they moved when a commitment expired or a building closed. Corroborated across the corpus: Dropbox timed against S3 contracts, 37signals against its S3 commitment, Stack Overflow against a lease, and the VMware estates now heading for cloud are timing against renewal dates that arrive with 71% of customers reporting outsized price rises (ETR, Dec 2025). Your real decision windows are dated documents, not architecture reviews.
| Decision | Chosen (by whom) | Rejected | Because | Flips when |
|---|---|---|---|---|
| Storage tier on metal | Buy arrays (37signals: 18PB Pure, ~$1.5M + support) | Building a blob store | 10PB does not amortise a storage-systems team | At exabyte scale it flips hard: Dropbox built Magic Pocket and banked $74.6M in two years |
| Orchestration on metal | Containers over SSH, no Kubernetes (37signals, OneUptime went the k8s route instead) | Kubernetes/OpenStack platform | A handful of apps and one ops team do not need a platform org | Hundreds of teams and mixed workloads: GEICO is building exactly that platform on OCP |
| Who racks the boxes | Ops partner (37signals with Deft) | Hiring a datacenter team | Keeps headcount flat; DHH reports no ops-team growth | Fleet grows past what a partner contract covers, or the partner becomes the single point of failure |
| Exit egress timing | After the 2024 waivers (37signals: ~$250K waived in a 60-day window) | Paying metered egress | Google (Jan 2024) then AWS (Mar 2024) zeroed final-exit transfer, EU-Data-Act driven | Never flips back inside the EU: switching charges are banned outright from 2027-01-12 |
| Commitment posture while renting | Discounts with exit windows (37signals rode its S3 term to the exit date) | Long commitments without options (Snap: $2B + $1B over five years, with pay-the-difference shortfall clauses) | Commitments price like debt: cheaper, and binding | If a plausible exit or big re-architecture sits inside the term, shorter and dearer wins |
| Which workloads move first | Steady, storage-heavy, stateless-friendly apps (all exits) | Spiky, geo-distributed, managed-service-coupled ones | They carry the premium without using the elasticity | A workload that bursts 16x on a viral week (Cara) belongs on elastic capacity, with a budget cap |
One more decision hides in a rejected pull request. Kamal's tracker keeps receiving orchestrator-shaped feature proposals, and keeps declining them: a custom-SSL-path implementation was argued down in favour of reading certificates from secrets and closed unmerged in June 2025 (PR #969, superseded by the merged #1531), and a standby-container failover mode was closed unmerged in June 2026 (PR #1810). The tool's boundary is a decision record in itself: the moment you need pre-warmed failover pools and cluster-level scheduling, you have re-derived the platform you left, and you should either accept that platform or stop needing it.
Three failure classes account for the published incidents: unbounded elastic billing, the lift-and-shift premium, and adopting an ops discipline nobody owned. The fourth finding is an absence, and it matters as much.
No published postmortem in this corpus attributes a production incident to an exit cutover, or to post-exit hardware operations. 37signals moved seven apps and 10PB with no incident write-up; Stack Overflow's move produced retrospectives, not apologies. Either the cutovers work, or their failures go unpublished; both readings point the same way: the risk you cannot price from public evidence is year-two operations, not migration weekend. Only OneUptime has published a multi-year follow-up (two years later, savings grown to $1.2M/yr), and one account is anecdote, not a base rate.
Everything quantitative in one place, dated. Measured means a primary account with arithmetic behind it; claimed means self-reported without methodology; survey means aggregated self-reports.
| Metric | Value | At | Status | As of | Source |
|---|---|---|---|---|---|
| Two-year opex reduction from storage exit | $74.6M | Dropbox | Measured (SEC filing) | 2018 | S-1 |
| Year-one components of the above | −$92.5M / +$53M | Dropbox | Measured: third-party DC spend cut vs own-DC cost added, netting $39.5M | 2016 | GeekWire |
| Annual bill before/after exit year one | $3.2M → $1.3M | 37signals | Measured (self-reported with line items) | 2024 | DHH |
| Exit hardware spend | ~$700K | 37signals | Measured; paid back inside the first year's savings | 2023 | DHH |
| 18PB flash, bought + 5yr support | ~$1.5M + <$1M | 37signals | Measured (self-reported) | 2024 | DHH |
| Final-exit egress waived | ~$250K | 37signals / AWS | Reported; 60-day window under the 2024 policy | 2025 | AWSInsider |
| Colo server-month, all-in | $1,550 | Ahrefs | Claimed; own model, includes space, power, transit, network | 2023 | Ahrefs |
| Cloud-equivalent server-month | $17,557 | Ahrefs | Claimed; EC2 list-price equivalent of hardware clouds do not sell as one instance (2TB RAM, 16×15TB, 2×100Gbps) | 2023 | The Register |
| The $400M headline, checked | 850 × ($17,557−$1,550) × 30 ≈ $408M | Ahrefs | Derived here from their own per-server figures; matches their $39.5M vs $447.7M totals | 2023 | The Register |
| Storage server, 24×61TB NVMe, 2U | ~$190K + ~$3K/yr colo | Fastmail | Measured (self-reported) | 2024 | Fastmail |
| Managed k8s bill pre-exit | >$38K/mo (28 nodes) | OneUptime | Measured (self-reported); savings $230K/yr at exit, >$1.2M/yr two years on | 2023/2025 | OneUptime |
| Cloud + storage cost cuts | −60% / −75% | X (Twitter) | Claimed; no methodology published; Sacramento exit put at $100M/yr, 5,200 racks, 148K servers | 2023 | DCD |
| Bill growth under cloud-first | 2.5× over ~10yrs; >$300M/yr | GEICO | Claimed in talks; consistent across two venues | 2021/2024 | The Stack |
| Committed cloud spend at IPO | $2B + $1B / 5yrs | Snap | Measured (S-1 disclosure), with shortfall clause | 2017 | Recode |
| Cloud share of software COGS | ~50% | 50 public SaaS cos | Derived by a16z from filings; contested, directionally accepted | 2021 | a16z |
| Self-estimated cloud waste / over-budget orgs | 29% / 17% | Flexera respondents | Survey; waste rose for the first time in five years | 2026 | Flexera |
| Companies planning full repatriation | 8–9% | IDC survey | Survey; against 83% moving ≥1 workload (Barclays) | 2024 | The Stack |
| Public cloud end-user spend | $723.4B (from $595.7B) | Worldwide | Forecast (Gartner) | 2025 | The Stack |
| EU switching charges | cost-based now, $0 from 2027-01-12 | All providers serving EU | Regulated fact (Reg. 2023/2854, applicable 2025-09-12) | 2025 | Kemp IT Law |
| Effective storage replication factor | 2.8× → 2.1× | Facebook f4 | Measured (paper); 53PB raw saved on 65PB logical | 2014 | OSDI 14 |
The Ahrefs and X rows are the ones most often quoted and least verifiable: Ahrefs compares against list-price on-demand for a hardware shape no cloud sells as one instance, and X published percentages with no baseline. The Dropbox row is the only figure in this table that passed an audit. The a16z 50% figure is a derivation from public filings by an investor with a thesis; treat it as an upper anchor. Undated cloud list prices change quarterly, which is why none appear here; re-derive any comparison with current pricing before quoting it in a review.
Every source behind this page, graded by kind. Filter it; the postmortems and the two closed-unmerged pull requests are the rows generic coverage never cites.
A test deployment recursively scaled to ~1,000 Cloud Run instances driving ~1B Firestore reads/minute; budgets alerted but did not stop spend, and billing lagged hours. $72K in an afternoon, March 2020.
Files grew past Cloudflare's 15GB cacheable-object cap; downloads silently fell through to Azure origin egress. ~$11K surprise on the December 2021 invoice, found on the invoice.
40K to 650K users in a week; 56M function invocations/day at peak; a ~$96K–$98K Vercel month for a free app. Incident review assembled from the founder's public accounts.
"Deploy web apps anywhere." Containers pushed over SSH with zero-downtime proxy swaps; deliberately not an orchestrator. The configuration surface is a candid map of what leaving managed platforms makes you own.
Custom SSL certificate paths, argued down by maintainer djmb ("I think we should read these directly from the Kamal secrets") and superseded by the merged secrets-based #1531. A complete rejected-design argument, on the record.
Standby containers "in a Created (stopped) state" for near-instant failover, proposed March 2026 (issue #1809), closed June 2026 without a public design rationale. Orchestrator-shaped asks keep arriving; the boundary holds.
Blog-post issue #289, "Why We're Choosing Bare Metal & How We Knew it was Time to Leave the Cloud," drafted during the 2016 metal proposal and left behind when the decision reversed in public.
The stay-side answer to exit math: delete nodes whose pods fit elsewhere, replace nodes with cheaper ones, reasoning explicitly about "what the node we are considering replacing costs."
"A common schema for technology cost and usage data across cloud, SaaS, data center, and other technology categories." The spec exists because rent-vs-own comparisons fail at the data layer first.
$39.5M cost-of-revenue reduction in 2016 (third-party datacenter spend down $92.5M, own-datacenter costs up $53M), a further $35.1M in 2017: $74.6M over two years, disclosed under securities law.
850 colo servers at $1,550/month all-in against $17,557 for an EC2-equivalent: $39.5M vs $447.7M over 30 months. The hardware shape (2TB RAM, 16×15TB drives, 2×100Gbps) is exactly what clouds price worst.
India's largest broker runs self-hosted FOSS on plain EC2 with ~5% external-vendor dependency; CTO Kailash Nadh puts the saving at ≥$3M/yr against proprietary subscriptions. Repatriation at the software layer only.
$2B/5yr to Google plus $1B/5yr to AWS, with the S-1 admitting "If we fail to meet the minimum purchase commitment during any year, we are required to pay the difference."
Bill $3.2M to $1.3M in year one; the remainder was one S3 contract, retired mid-2025 with 18PB of bought flash (~$1.5M plus support). The projection that started at $7M was raised, not walked back.
Sixteen years of racking, cabling and disk swaps by a small SRE team, ended by a datacenter that closed underneath them: out by 2025-07-31, no renewal option, public sites to Google Cloud.
Roughly fifty servers decommissioned in New Jersey; the Colorado DR site had gone that June. The company that taught a generation to run lean on metal now owns no hardware at all.
The public reversal of the 2016 bare-metal proposal, after hundreds of comments and emails warned what owning a Ceph estate would take. The rare exit postmortem written before the exit.
Twenty-five years on owned metal; current fleet: 2U servers with 24×61TB NVMe at ~$190K each, ~$3K/yr per 2U for space, power and cooling. The stayer's argument is knowing your load curve for decades.
The custom multi-exabyte blob store behind the exit: replication plus an erasure coding variant, later SMR drives adopted first among major operators. The $74.6M did not come from racking servers; it came from storage systems engineering.
The only multi-year exit retrospective in the record: a $456K/yr managed-k8s bill became $230K/yr of savings at exit, growing past $1.2M/yr as the fleet grew, with the operational questions answered in public.
Triggered by a 2008 corruption in its own datacenter that stopped DVD shipping for three days; finished January 2016 when the last datacenter closed. Seven-plus years for one committed company: the honest duration anchor for any full migration.
Corey Quinn's standing argument: Dropbox is always the example because a second one is hard to name; his later update concedes nuance while holding that wholesale exits stay rare. The strongest published counterweight to exit triumphalism.
High utilisation from "admission control, efficient task-packing, over-commitment, and machine sharing" across cells of tens of thousands of machines. The mechanism that makes owned hardware cheap, written down by its largest operator.
Erasure coding plus a cross-datacenter XOR scheme cut effective replication from 2.8× to 2.1×, saving 53PB of raw storage against 65PB logical at publication. Cost-driven architecture at the owner's limit.
The enterprise verdict on cloud-all-in: unfinished after a decade, bills up 2.5×, reliability down, "at the cumulative mercy of our clouds." The rebuild is hybrid on OCP hardware with an open-source platform.
The exit's author on the record, in a transcribed long-form interview: early AWS adoption from 2006 ("in the cloud before it was cool"), the premium for elasticity never used, and the decision mechanics behind the posts.
"You're crazy if you don't start in the cloud; you're crazy if you stay on it." Cloud at ~50% of COGS across 50 public SaaS companies, ~$100B of market value suppressed. An investor thesis, and the debate's reference point.
Self-estimated waste up to 29%, the first rise in five years, driven by AI workloads; 17% of orgs over budget; fewer than half using any one commitment discount per provider.
71% of surveyed VMware customers report price rises well beyond the software market since the acquisition; 42% plan partial migration to Microsoft within a year; of those leaving, close to 75% pick public cloud over another hypervisor.
Seven rungs from a spreadsheet you can build this week to an exit (or a documented decision to stay) your CFO and your on-call rota both sign.
Normalise one month of billing into unit economics per workload (FOCUS gives you the schema). Find the three line items that dominate; in the documented exits they were steady compute, storage at rest, and egress.
Done when: you can price one request, one user and one stored TB. Teaches: where the money actually is, which is rarely where the dashboard points.
Rent a dedicated server for the price of a lunch a month, deploy a production-shaped service with Kamal or plain containers, and replay production-shaped load at it.
Done when: p95 matches your cloud baseline and you know the box's actual ceiling. Teaches: what the managed layer was doing for you, item by item.
Fail the disk, reboot mid-traffic, restore the datastore from backup onto a second box, and time every step. This is the rung that separates tourists from residents.
Done when: a written restore runbook beats your stated RTO. Teaches: the operational surface you would be adopting, before it is yours at 3am.
Three-year TCO for your top workload: hardware plus refresh, colo, people or partner, support contracts, the rented remainder, and the migration itself. Beside it, the optimised stay: commitments, rightsizing, consolidation.
Done when: two numbers a CFO accepts, with the crossover condition named. Teaches: your own flips-when, which no blog post can supply.
Apply the stay-side machinery before deciding: commitment coverage, consolidation or its equivalent, storage tiering, egress audit. Flexera's respondents call 29% of spend waste; claim yours either way.
Done when: waste is under ~10% or you hold evidence the premium is structural. Teaches: whether the exit's savings were actually available without moving.
Move one steady, storage-heavy workload to the landing zone with dual-run, reconciliation, and a written, rehearsed rollback. 37signals moved seven apps this way; Stack Overflow rehearsed on Teams before touching the public sites.
Done when: 30 days on the new floor with error budget intact and unit cost measured. Teaches: cutover mechanics and the true shape of day-two work.
Build the renewal-and-lease calendar with decision windows and trigger conditions (bill growth, utilisation, egress share, headcount). Every documented move in this guide happened at a contract boundary; engineer your next one instead of meeting it by surprise.
Done when: the next renewal arrives with an option you designed. Teaches: that rent-or-own is a standing decision with dated inputs, not a one-time debate.
The queries that found this material, grouped by what they surface. The numbers will stale; these will not.
"why we're leaving the cloud" OR "we have left the cloud""why we use our own hardware""moving from AWS to bare-metal" saved"by not going to the cloud" saved"why we are not leaving the cloud""the great unracking" stackoverflow"journey to the cloud" site:stackoverflow.blogVMware Broadcom migration survey "public cloud" percentage"we burnt" OR "we burned" cloud bill postmortem"how I got pwned by my cloud costs"serverless bill "$96,000" OR "96k" vercel"budget alert" lag "billing" postmortem quotais:pr is:closed is:unmerged repo:basecamp/kamalS-1 "cost of revenue" infrastructure savings site:sec.govEU Data Act switching charges "12 January 2027"path:designs consolidation repo:aws/karpenter-provider-awsChecked 2026-08-30. This session ran under an egress allowlist:
github.com URLs were fetched directly; all others were verified live through
search-engine retrieval, several with exact-phrase confirmation. The companion
sources.md ledger records the claim and supporting quote taken from each.