Self-Hosting Versus API Break-Even

The argument for self-hosting usually starts with a division: an 8-GPU node costs $20 an hour, it can emit 18 million tokens an hour, so each million tokens costs $1.11, far below a $3 API price. The division is correct and the conclusion is usually wrong, because it assumes the node is busy every hour of the month at its full throughput and that nobody has to run it.

Unit economics of an AI feature notes that self-hosting converts variable cost to fixed cost and wins above a utilisation threshold. Capex, depreciation and the cost of a GPU-hour builds the hourly rate. This concept computes the threshold, so the decision rests on a number rather than a slogan.

The model

Let \(N\) be the number of serving nodes, \(H\) the monthly cost of one node (rental, or amortised capex plus power and hosting), \(E\) the monthly engineering and operations cost of running the service, \(C\) the tokens one node can serve in a month at full use while meeting the latency target, \(u\) the realised utilisation (served tokens divided by capacity), and \(p\) the API price per token for equivalent quality. Self-hosted cost per token is

and self-hosting breaks even when \(c_{\text{self}} = p\), at utilisation

If \(u^* > 1\), self-hosting cannot win at any load with that fleet. If \(u^*\) is below the utilisation you will actually achieve, it wins.

A worked calculation

Every figure below is an illustrative assumption, not a quoted price; substitute your own.

• Node: 8 GPUs at $2.50 per GPU-hour, so \(H = \$20 \times 730 = \$14{,}600\) per month.
• Throughput: 5,000 blended input and output tokens per second per node at the latency SLO, so \(C = 5{,}000 \times 3{,}600 \times 730 \approx 13.1\) billion tokens per month.
• API price: \(p = \$3.00\) per million blended tokens for a model of comparable quality on the task.
• Engineering: \(E = \$20{,}000\) per month for one node, $40,000 for ten.

Hardware alone gives \(u^* = 14{,}600 / (3.00 \times 13{,}140) \approx 37\) percent. Adding engineering for one node gives \(34{,}600 / 39{,}420 \approx 88\) percent, a utilisation almost no interactive service achieves. With ten nodes, \(u^* = 186{,}000 / 394{,}200 \approx 47\) percent, needing about 62 billion tokens a month.

Two conclusions survive any reasonable change of inputs. Fixed engineering cost dominates at small scale, so self-hosting a single node rarely beats an API on price alone. And the hourly GPU price is not the decisive variable; halving it at ten nodes moves \(u^*\) from 47 to about 29 percent, while realised utilisation can easily vary by a factor of three between workloads.

Why realised utilisation is low

Interactive traffic has peaks, and a fleet sized for peak load runs at roughly mean-to-peak utilisation. With a 3:1 peak-to-mean ratio, \(u \le 33\) percent before failures, deployments and headroom, which in the example above loses at every fleet size. The levers are all about shape: mixing deferrable batch work into troughs, autoscaling on rented capacity so \(N\) follows demand, or routing only the base load to owned capacity and overflow to an API.

Throughput \(C\) is equally movable and is pure engineering. PagedAttention-based serving improved throughput 2-4 times over earlier systems at the same latency (Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP, arXiv:2309.06180). Doubling \(C\) halves the hardware share of \(u^*\), which is why a self-hosting estimate must use throughput measured on your own prompt and output lengths, not a benchmark's.

When it breaks

The API price is a moving target. Epoch AI found the price of reaching a given level of LLM performance fell between 9 and 900 times per year depending on the performance threshold (Cottier et al., 2025, LLM inference prices have fallen rapidly but unequally across tasks, Epoch AI). A one-year hardware commitment justified by today's \(p\) can be underwater within months. Recompute \(u^*\) with next year's plausible price, not this year's.

Equivalent quality is an assumption. If the open-weights model needs longer prompts, more retries or a larger size to match the hosted model on your task, \(C\) falls and effective tokens per request rise. The comparison has to be cost per successful task, not cost per token.

Engineering cost is the contested input. Advocates of self-hosting treat serving as solved by open-source stacks; sceptics count on-call, upgrades, security patching, capacity planning and model evaluation as a standing team. Both sides are describing real organisations, and \(E\) can differ by a factor of five between them.

Price is not the only reason. Data residency, regulatory control, fine-tuned weights, latency co-location, rate limits and freedom from provider deprecations can justify self-hosting above break-even. They should be named as the reason, not hidden inside an optimistic utilisation figure.