Training Cost Estimation Before You Commit
The FLOP arithmetic that converts a model and dataset size into a GPU-hour figure, why achieved utilisation is the term that decides the answer, and the overheads that turn an estimate into a budget.
A training run is a large, mostly irreversible commitment, and the compute cost can be estimated within a factor well before any code is written. The estimate is straightforward, the term that dominates its error is one people guess at, and getting it approximately right is what separates a plan from a hope.
The arithmetic
For a dense transformer, the standard approximation is that training consumes about six FLOPs per parameter per token: two for the forward pass and four for the backward.
A 7B model on 2 trillion tokens is therefore about \(6 \times 7\times10^9 \times 2\times10^{12} \approx 8.4 \times 10^{22}\) FLOPs.
Converting that to time requires achieved throughput, not peak. An H100 quotes roughly 990 TFLOP/s for bf16 with sparsity disabled, and well-optimised large-scale training typically achieves somewhere between 35 and 55 percent of peak, a figure usually reported as model FLOPs utilisation. At 40 percent, that is about 400 TFLOP/s per device.
which is roughly 58,000 GPU-hours, or about 10 days on 256 devices. Multiply by the hourly rate for the capacity type to get money.
For a mixture-of-experts model, replace \(N_{\text{params}}\) with active parameters per token, which is why MoE training is cheaper per token than its total parameter count suggests.
Where the estimate goes wrong
Utilisation is the dominant uncertainty. The difference between 30 and 50 percent MFU is a 66 percent difference in cost, and it is determined by parallelism strategy, interconnect, sequence length, and how much engineering effort goes into the kernels. Estimating with a single assumed number, without stating it, is where most of the error enters.
Failures and restarts add real time. At scale, interruptions consume a meaningful fraction of wall-clock. A goodput figure, useful work as a fraction of allocated time, is the honest planning number and is lower than MFU.
The run is not the project. Data preparation, ablations at smaller scale, hyperparameter tuning, evaluation and the inevitable restart after a mistake typically add substantially to the headline number. Budgeting only the final run is the most common underestimate.
Reported costs are not comparable. Published figures such as DeepSeek-V3's widely cited $5.576 million refer, as that paper states plainly, to the final training run at an assumed rental price, explicitly excluding prior research, ablations and data work. Treating such a figure as the total cost of producing the model misreads what was reported.
When it breaks
Inference cost usually exceeds training cost over a model's life. For a widely used model, cumulative inference spend passes the training bill within months. Optimising the training budget while ignoring the inference profile optimises the smaller number.
Sequence length changes the arithmetic. The \(6N\) approximation omits attention, which is quadratic in sequence length and becomes significant at long context. At 32k tokens and beyond the attention term is no longer negligible and the estimate needs the extra term.
Price and availability move together. A cost estimate assumes a rate for capacity that must actually be obtainable for the duration. A cheap rate for hardware you cannot reserve for three weeks is not a usable input.
14 flashcards for this concept
Click a card to reveal the answer.