Cost & FinOps for AI
Unit economics per request, token accounting, reserved versus spot capacity, and attributing spend to features.
12concepts
105flashcards
90minutes of reading
- 01 Cost per Completed Task Why a list price per million tokens cannot rank models for a workload, how to compute cost per completed task instead, and the measured evidence that the cheaper model per token is often the more expensive one per outcome.
- 02 Non-Token Charges in an AI Bill The line items a cost-per-token model cannot see, covering per-call tool fees, sandbox and session runtime, residency premiums and the token overhead of tool definitions, and why they decide the bill for agentic workloads.
- 03 Per-Request Energy, Water and Carbon How to put an energy, water and carbon figure on a single inference request, why the two most rigorous published per-request figures differ by a factor of 38 on carbon and 173 on water, and which accounting choices produce the gap.
- 04 Reserved, On-Demand and the Shape of Commitment How to choose a commitment mix when demand is uncertain, why the break-even is simply a price ratio, and the option value that makes shorter commitments rational despite costing more.
- 05 Self-Hosting Versus API Break-Even A worked break-even model for serving an open-weights model yourself against paying per token, showing why utilisation, not the hourly GPU price, usually decides the answer, and which assumptions flip it.
- 06 Spend Guardrails and Quotas Why AI spend can rise by orders of magnitude in hours, the layered controls that bound it without blocking legitimate work, and the design of a kill switch that is actually usable.
- 07 The Price Menu: Caching, Batch and Service Tiers The multipliers that sit between a model's list price and what you actually pay, the break-even arithmetic for a cache write, and why the menu exists as price discrimination rather than generosity.
- 08 Unit Economics of an AI Feature How to build a cost-per-request figure that survives scrutiny, why the marginal cost of an LLM feature does not fall with scale the way software's does, and the retry and failure multipliers everyone forgets.
- 09 Cost Variance and the Tail of Agentic Spend Why the budget for an agent workload is set by the tail of its cost distribution rather than its mean, where the multipliers in that tail come from, and the controls that bound a single run without quietly lowering success.
- 10 The Cost of Evaluation and Experimentation Why evaluation is a standing line item that can rival production inference, the multipliers that make a golden set expensive, and how to buy statistical confidence at the lowest price.
- 11 Token Accounting and Cost Attribution Why a single provider invoice cannot be allocated to teams, features or customers without instrumentation, and the tagging discipline that makes AI spend attributable.
- 12 Training Cost Estimation Before You Commit The FLOP arithmetic that converts a model and dataset size into a GPU-hour figure, why achieved utilisation is the term that decides the answer, and the overheads that turn an estimate into a budget.