Dissection · Kaplan et al., 2020 and Hoffmann et al., 2022

Anatomy of Neural Scaling Laws

Kaplan et al. said spend on size; Hoffmann et al. said spend on both. 108 trained miniatures, three estimators and one question: what does the answer depend on?

PapersKaplan et al. (2020); Hoffmann et al. (2022)

Model11 sizes, 1.5k to 590k parameters, live

Reading29 min · 30 steps

Published2026-10-23

01Overview

How should a fixed budget of compute be split between a bigger model and more data?

In 2020 Kaplan and colleagues trained language models of many sizes and found that the loss fell as a power law in model size, in data and in compute [1]. Their advice followed: for a given budget, spend most of it on a bigger model. Two years later Hoffmann and colleagues trained over 400 language models, from 70 million to over 16 billion parameters on 5 to 500 billion tokens, measured again, and advised the opposite: for every doubling of model size the number of training tokens should also be doubled [2]. Before the second paper the largest models had been trained on about the same number of tokens however big they were: most were trained for approximately 300 billion tokens.

This page rebuilds the disagreement at a scale a laptop can run. On the table are training runs of character-level Transformers, in eleven sizes from 1.5 thousand to 590 thousand parameters, each trained on a different amount of text. Every block's height is the loss it reached (taller is better). The tubes are the compute-optimal size found three ways, and beside them two straight lines with the slopes the papers recommend. Scroll to be taken through the model, the runs, the three estimators, and the experiment I ran on them.

It is a research dissection. The last part pre-registers three questions the papers left open, runs them, and reports what came out, including where the answer is no.

02Overview

What a scaling law claims, and what a miniature can test

A scaling law says that loss L depends on the number of parameters N and the number of training tokens D through a smooth formula, and that the formula keeps holding as both grow. Kaplan et al. fitted it over eight in C, six in N, two in D and, from it, derived how to allocate compute C [1]. Hoffmann et al. fitted three different estimators to their own runs and got a different allocation [2].

A miniature cannot say what happens at 70 billion parameters. It can say what changes when a choice changes, because every choice can be made on the same runs. That is the question here: given one set of trained models, how much does the recommended allocation depend on what counts as a parameter, which estimator is used, and how the models were trained?

The miniature's sizes and corpus are in the last two columns. The corpus is this site's own learning library, 10.4 million characters of technical prose, split by document so that no validation sentence sits next to a training sentence. Every symbol the models can output appears in the training text at least 155 times.

03The model family

What is the model whose size is being scaled?

A decoder-only Transformer [3] with a few hyperparameters: the number of layers, the width of the residual stream, the width of the feed-forward layer and of attention, and the number of heads. Kaplan et al. fix the last three in proportion to the second and report that the exact shape hardly matters: an aspect ratio varied forty-fold changes the loss by only a few percent [1]. So a model is one number, its size.

The family on the stage has eleven members, from 8 wide and 2 layers to 128 wide and 3 layers. Each uses a feed-forward width of 4d and an attention width of d, so the count of the matrices is

N \approx 12 n_{layer} d_{model}^{2}

which is the paper's formula with biases and LayerNorm gains left out. The stage's bottom-left block is a ledger: its columns are the eleven models, and the first row is log₁₀ N, from 1,536 to 589,824.

Kaplan et al. note that one-layer models sit off the trend and leave them out of the fits [1]. Every model here has at least two layers.

04The model family

Why does Kaplan et al.'s N leave the embeddings out?

They define N to exclude all vocabulary and positional embeddings, because this produces significantly cleaner scaling laws. Hoffmann et al. do the opposite: they count the embedding matrices in the parameters and in the FLOPs. At the scale of their models the difference is nothing. At the scale of the early Kaplan runs it is not.

The second and third rows of the ledger show why. A model's embedding table has (v + n_ctx)d entries, here (96 + 64)d, which grows like d while the rest grows like d². For the smallest model of the parameters are embeddings; for the largest, . A count that leaves them out therefore grows faster than one that includes them, and the two counts disagree most where the models are smallest.

That disagreement is where the rest of this page is headed: if the optimal size scales as a power of compute, then which size and which compute you plot decides the power. Pearce and Song argue it decides most of the gap between the two papers: Kaplan counted non-embedding rather than total parameters, at small scale [4].

05The model family

How does one block read a sentence?

One of the eleven, the 13,824-parameter model (24 wide, two layers, two heads), is laid open on the left of the stage with its real weights. Its input is the first 24 characters of your text, one per row. Each character is looked up in the embedding table E and a learned position vector is added: that is the first block, x₀.

The first sub-layer of a block is attention. W_QKV turns each row into a query, a key and a value; each head compares every query with the keys of the characters before it (the triangles: later characters are masked) and the softmax of those comparisons weights the values. The result is added back into the residual stream. This is the model of [3] with the encoder removed, as in Kaplan et al.'s decoder-only setup, and the site has a full dissection of the Transformer for the mechanism. Here the point is only that every block on the stage is a number computed from your text and the trained weights.

Concepts behind this step: weight tying, the anatomy of a block, attention.

06The model family

Where do most of the parameters sit?

The second sub-layer is a position-wise feed-forward layer: W_in widens each row from 24 to 96 features, a GELU keeps the positive ones, and W_out (drawn transposed so the hidden units line up) narrows it again; the result is added to the residual stream. A block has 4d² weights in attention and 8d² in the feed-forward layer, and the second dominates: that is the factor 12 in N ≈ 12 n_layer d².

The second layer, above, repeats the pattern. Nothing here depends on the particular sentence except the activations.

07The model family

What is the model's answer, and how is it scored?

A final LayerNorm and the transpose of E turn the last residual stream into a score for each of the 96 characters; a softmax makes them probabilities. The row for the character t is the model's belief about character t + 1. The loss that is trained and reported is the cross-entropy of what actually came next,

L = - \frac{1}{T} t = 1 \sum T ln p_{θ} (x_{t + 1} ∣ x_{\leq t})

in nats per character; dividing by ln 2 gives bits. A model that guesses uniformly among 96 characters scores ; the unigram entropy of the training text is 4.72 bits.

Concepts behind this step: next-token prediction and cross-entropy, perplexity.

08The model family

Does a larger model predict your text better?

The strip at the bottom scores the same text with all eleven models: one row per size, one column per character being predicted, height in nats. Its right-hand column is each row's mean.

Kaplan et al. report the same for text from distributions the models were not trained on: the loss on other data distributions improves smoothly with model size, in direct parallel with the improvement on WebText2 [1]. A sentence of the kind the models were trained on follows the law; a string of random symbols does not, because there is nothing to learn about it.

09The model family

How is the compute of a training run counted?

Kaplan et al. count the non-embedding training compute as C ≈ 6N floating-point operations per training token, so a run of D tokens costs C = 6ND (a forward pass is N multiply-adds, or 2N operations, and the backward pass costs twice that). Their Table 1 itemises the forward cost: the embedding, the three attention projections, the attention over the context, the output projection, the feed-forward layers and the de-embedding, 2 d_model n_vocab. They keep only the terms that scale with N and give the condition for dropping the context term: d_model > n_ctx/12.

All eleven models here satisfy it (8 > 64/12), some barely, and the last row of the ledger shows how much the rule still leaves out: for the smallest model the exact count is more than twice 6N, because the output matrix and the attention over the context are large next to a 1.5-thousand-parameter body. Porian et al. find the same thing at their scale, that not accounting for the model head under-approximates compute by roughly 10% at larger models and roughly 90% at smaller ones [5].

So there are at least four defensible ways to count: Kaplan's (N and C without embeddings), with the head, Hoffmann's (everything, C = 6ND with all parameters), and the full forward count of Table 1. The last is exploratory here; the first three are the pre-registered conventions of the research desk and are called K, H and T.

Concept links: counting parameters, the FLOPs of a forward pass.

10Training the grid

How were the models trained?

Every run uses AdamW [6] (the optimiser of Chinchilla; Kaplan et al. used Adam [7]) with a peak learning rate that falls with size as lr = 0.02 (N/3456)^−1/2, capped at 0.03. Kaplan et al. also used a rule of the form 0.003239 − 0.0001395 log N; mine was fitted by hand to a sweep at three sizes (research/pilot/). Batches are 16 sequences of 64 characters, so a step is 1,024 tokens and a run of 4 million tokens is 3,906 steps. The batches of Kaplan et al. are 512 sequences of 1,024 tokens: a fixed 2.5×10⁵ steps with a batch size of 512 sequences of 1024 tokens.

What differs between the studies is the schedule. Kaplan et al. trained every model on a 3000-step linear warmup followed by a cosine decay to zero over the same fixed number of steps and read the loss along the way. Hoffmann et al. objected that one fixed schedule prevents modelling the impact of these hyperparameters, and set the cosine so that each run decays by a factor of 10 over its own horizon. The figure shows both on the page's runs: the fixed-horizon protocol (arm K) and the matched one (arm M), plus a matched protocol with a fixed 100-step warmup (arm W) to separate the two things Porian et al. say differ.

The stage shows arm K: eleven long runs, one per size. The rows are sizes, the columns are 24 checkpoints along the run, and the height is how far the loss is below the worst loss in the study.

11Training the grid

Which runs make an IsoFLOP grid?

Pick a compute budget C. For each model size N train for D = C/6N tokens, with a schedule that decays over exactly those tokens. The runs at one budget lie on a diagonal of the landscape: small models with much data at one end, large models with little at the other. Hoffmann et al. did this for nine budgets spanning a factor of 500. Here there are 18 budgets, from 2×10⁹ to 5×10¹² FLOPs, and the 108 runs that fall between 10⁵ and 1.03×10⁷ tokens (a hundred steps at the least, one pass over the corpus at the most), each trained under three seeds.

The stage's measured landscape shows the result. Size grows to the right and tokens grow into the distance, in quarter-octave bins; height is how far the loss is below the worst run. The diagonals appear in order of budget as the step plays.

The corpus is read once, as in Hoffmann et al., who note their runs have all been trained on less than an epoch of data. Up to four repeated epochs are close to free in practice [8]; that is outside this page.

12Training the grid

What does a larger model buy during training?

Kaplan et al.'s first summary is that large models are more sample-efficient: they reach the same loss with fewer examples. The fixed-horizon runs show it directly. Read along any column of the curve landscape: at the same number of tokens a larger model is nearly always higher (lower loss).

Plotted against compute instead of tokens, the curves cross, and each size is best over a range of budgets. The black line is the lower envelope. Taking the lowest curve at each budget is the idea of Hoffmann et al.'s Approach 1; applied to one fixed schedule per size it is what Kaplan et al. did, with a difference that matters later: their curves come from a fixed batch size for all models and one schedule, so the early part of a long curve is the loss of a model whose learning rate has not yet fallen.

13Kaplan's laws

Is the loss at fixed data a power law in model size?

For models limited only by their number of parameters, trained to convergence on sufficiently large datasets, Kaplan et al. write

L (N) = (\frac{N _{c}}{N})^{α_{N}}

and report α_N = 0.076 [1], so that doubling the parameters multiplies the loss by 2^−0.076 = 0.95. Others had seen power laws before; Hestness et al. found power-law generalization error scaling across a breadth of factors [10], and the same group extended Kaplan's law to other domains with a constant added to the power law [11].

On this page every model has the same 4-million-token schedule, which is not convergence, so what is fitted is the loss at that horizon. The figure fits the pure power law and a power law with a floor, and lets you drop the smallest models from the fit.

A pure power law cannot fit a floor. Text has non-zero entropy, so the loss must flatten eventually; the fitted exponent therefore depends on how far from the floor the fit starts. Kaplan et al. say the same of their own range: non-zero entropy They add that their laws must break by about 10⁴ PF-days, 10¹² parameters and 10¹² tokens, at a loss near 1.7 nats per token, a point they call highly uncertain.

14Kaplan's laws

What is the joint law?

Kaplan et al. combine the two single-variable laws into one with three requirements: a change of vocabulary should only rescale the loss, the formula should reduce to L(N) as D → ∞ and to L(D) as N → ∞, and it should be analytic in 1/D. The result is

L (N, D) = [(\frac{N _{c}}{N})^{α_{N} / α_{D}} + \frac{D _{c}}{D}]^{α_{D}}

with the fitted values α_N 0.076, α_D 0.103, N_c 6.4×10¹³, D_c 1.8×10¹³ [1]. They call it a conjecture for other generative models: three principles. A year later Rosenfeld et al. proposed a similar joint form [12].

The law gives a data requirement. Since the model term scales as N^α_N/α_D, every time the model grows 8×, only roughly 5× more data is needed to avoid a penalty, which I can check: 8^0.076/0.103 = 4.6.

On the stage the second block is the fitted law of Hoffmann et al. (their Approach 3, below) evaluated on the same cells, and the third is what is left over. Kaplan's form has no additive floor, and this grid reaches nothing like zero loss; the exploratory table in the research desk compares the two forms by residual.

15Where to spend compute

What did Kaplan et al. conclude about allocating a budget?

With a learning-curve law in model size and steps, L(N, S_min) = (N_c/N)^α_N + (S_c/S_min)^α_S, fixed compute and the critical batch size, they minimise over N and find a compute-efficient frontier. For a budget grown tenfold, a figure caption puts it this way: optimal model size increases by 5×, and the data by only 2×. In their table of compute-efficient values the exponents are 0.73 for the model size, 0.27 for the data and 0.03 for the number of serial steps. They draw the conclusion in their summary, train very large models and stop significantly short of convergence, and in the discussion, big models over big data.

Hoffmann et al. restate the consequence for a tenfold budget as 5.5× model, 1.8× tokens, which differs from a direct reading of the exponents: 10^0.73 = 5.4, and 5.5 is 10^0.74.

The blue tube on the stage has slope 0.73 (model size against compute), the gold tube slope 0.50. Both pass through the same point, this page's own optimum at 10¹¹ FLOPs, so that only the slopes differ. The constants of the papers belong to their data and tokens, not to a 96-character vocabulary.

16Where to spend compute

How does Approach 1 find the optimum from training curves?

Hoffmann et al. train four models per size over horizons spanning 16×, smooth each loss curve, interpolate it as a function of FLOPs, and at 1,500 log-spaced FLOP values pick the run with the lowest loss. The chosen runs are the ones within the last 15% of training, which is what makes the scheme sound: a run that has decayed to its floor is an honest sample of what that size and that budget can do. They fit power laws to the picked sizes and tokens and report the exponents 0.50 and 0.50.

The page does the same on arm M. Each size has runs at about nine horizons; the final loss against ln D is interpolated by a quadratic through the three nearest runs and never extrapolated; at each of 60 budgets between 2×10⁹ and 5×10¹² the best of the supported sizes is picked. A budget counts only if at least five sizes are supported and the winner is not the smallest or largest of them. The teal tube on the stage is that frontier under the count you choose.

Because the sizes are discrete, the frontier is a staircase: the winner jumps from one size to the next. On the test-suite surface, exactly Chinchilla-form with a floor of 1.2 nats, this estimator leans high by about 0.02. I report such biases with the results rather than correct for them.

17Where to spend compute

What does fixing the budget and varying the model show?

Approach 2 is the experimental design the grid was built for. At a fixed budget, the loss against model size is a valley: too small and the model cannot use the compute, too large and it has seen too little data. Hoffmann et al. fit a parabola to each IsoFLOP curve to find the minimum, then fit power laws to the minima across the nine budgets: 0.49 for the model and 0.51 for the data.

The figure draws one of the 18 profiles from the page's own runs. Move the budget; switch the count. The valley's floor is the optimum, the highlighted diagonal on the stage is the budget, and when the count changes the same runs regroup (a run's compute is 6ND with the N of that count), so the minimum moves.

Hoffmann et al. report intervals "estimated via bootstrapping data (80% of the dataset is sampled 100 times)". The page's intervals redraw the seeds of every cell with replacement, 200 times.

18Where to spend compute

Can one formula for the whole surface give the frontier in closed form?

Hoffmann et al. model every final loss with L̂(N, D) = E + A/N^α + B/D^β. The three terms are the entropy of the text, the loss of an ideal generative process on the data distribution; what is lost by having only N parameters; and what is lost by training on only D tokens. Under the constraint 6ND = C the minimum has a closed form,

N_{opt} = G (\frac{C}{6})^{a}, D_{opt} = G^{- 1} (\frac{C}{6})^{b}, a = \frac{β}{α + β}, b = \frac{α}{α + β}

with G = (αA/βB)^1/(α+β) (their section 3.3). They fit by minimising a Huber loss of the log loss with δ = 10⁻³ from a grid of initialisations, and get a = 0.46, b = 0.54 with a very narrow interval.

The figure fits the same function to the page's cells by L-BFGS from a grid of 243 starting points, shows the residuals as a heatmap (one cell per run, amber where the run is worse than the law) and gives the exponent. Choose a different Huber δ:

19Where to spend compute

What do the three estimators recover from the same runs?

The stage now shows all five lines over the measured landscape: teal (envelope), orange (IsoFLOP), magenta (parametric), blue (Kaplan's slope) and gold (equal scaling). The figure plots the optimal size against compute for each, with the exponent as the slope, under whichever count you choose.

The generated results in the research desk say what the pre-registered counts and estimators gave. Before them, one caution about what this comparison can and cannot say. Hoffmann et al.'s three approaches yield comparable predictions (their three exponents are within 0.04 of each other); three estimators that agree on a surface that is exactly a Chinchilla law (as the test-suite surface is) differ from each other by a few hundredths because each has its own bias. Agreement to within 0.08, the pre-registered bar, is a modest one.

20Where to spend compute

What would each rule train for a budget you choose?

Move the budget in the figure: for each rule it gives the model size and the number of tokens, the tokens per parameter, the nearest of the eleven models and the loss the page's parametric law predicts. The first rule has Kaplan et al.'s slope and the second the 20 tokens per parameter that Hoffmann et al.'s tables imply (for example 20.2 billion tokens for a 1 billion-parameter model). The last two are fitted to the grid.

At budgets inside the grid the highlighted diagonal on the stage shows which runs were trained at that budget. Outside it the table extrapolates, which is exactly what the papers warn about for their own fits.

The tokens-per-parameter rule is not a law of nature. Sardana et al. show that if a model will serve many requests the best choice is smaller and longer than Chinchilla-optimal, and that quality keeps improving as tokens per parameter rise to extreme ranges [14]. Training compute-optimality answers one question, cheapest training for a loss, not the question of total cost.

21Results and costs

Did the prescription work at scale?

Hoffmann et al. tested the prediction with one expensive pair of runs. For the compute of Gopher, 5.76×10²³ FLOPs, they predicted that an optimal model should be 4 times smaller, trained on 4 times more tokens. They trained that model: 70B parameters, 1.4T tokens, with 80 layers, 64 heads and d_model 8,192, AdamW in place of Adam because it improves the loss and the downstream tasks, and compared it with Gopher and larger models on language modelling, reading comprehension, common sense and MMLU, where it reached 67.6%, 7.6 points above Gopher.

The test is one pair at one budget, as the authors say: two comparable runs. They also show a head-to-head at 10²¹ FLOPs, where Kaplan et al.'s recipe gave 4.68 billion parameters and theirs 2.86; the 2.8 billion model did better. (Kaplan et al.'s adjusted power law, written as 1.3×10⁹ (C / 8.64×10¹⁹)^0.73, gives 7.8 billion parameters at 10²¹ FLOPs, not the 4.68 billion of the comparison; the paper does not say how the 4.68 billion was obtained.)

The result is about training compute only. Sardana et al. add the cost of serving the model, and find the best choice then moves away from the training optimum, to smaller models trained on more tokens, as the step on budgets notes.

22Results and costs

How many tokens does each parameter get on the frontier?

The ratio of tokens to parameters at the optimum is the practical summary of an allocation law. For Chinchilla-style scaling it is roughly constant; for Kaplan's slopes it falls as compute rises, because D/N ∝ C^1−2a and 1 − 2 × 0.73 is negative. Hoffmann et al.'s Table 3 stays between 20 and 22 across four orders of magnitude of model size: 8.0 billion tokens for 400 million parameters, 20.2 billion for one billion, 205.1 billion for ten billion, 21.2 trillion for one trillion.

On this page the same ratio measured from the IsoFLOP optima, in tokens per total parameter, is shown in the results of the research desk. Two cautions apply. The characters of this corpus are cheaper than subword tokens, so the ratio is not comparable with the papers'. And a page of ratios from 108 runs is not a replication of a law.

For the case where data is the limit, Kaplan et al. have a different formula, D ≳ 5×10³ N^0.74 to avoid overfitting, and Muennighoff et al. show that up to four epochs of repeated data cost almost nothing [8]. Neither applies to a single pass over this corpus, where no run overfits.

23Scholarship

Which numbers in the two papers do not agree with each other?

Reading both sources closely turns up small inconsistencies. None changes a conclusion; each shows how a number moves between a derivation and a table.

Kaplan et al., the loss–data exponent. The summary says α_D = 0.095; the joint fit that produces the data-overfitting rule gives 0.103. The paper explains the difference: fitting the full L(N, D) That is a reasonable account of a fitted parameter. It does mean the quoted 8×-to-5× rule uses 0.103 and the headline exponent 0.095.
Kaplan et al., the compute exponent. Three values (0.050, 0.052 and 0.054) for one quantity, described in the earlier step.
Kaplan et al., seed noise. The seed-to-seed variation is given as roughly 0.02 in one section and as about 0.05 in the caption of the learning-rate figure; the second is the spread across schedules, a different quantity.
Hoffmann et al., the range of tokens. The abstract says the 400 models were trained on 5 to 500 billion tokens; the introduction says 5B to over 400B.
Hoffmann et al., MMLU. 67.5% in the abstract and 67.6% in the body.
Hoffmann et al., the Approach 3 exponent. a = 0.46 follows only from the unrounded exponents in a LaTeX comment; the printed 0.34 and 0.28 give 0.45. The reported interval for Approach 3 is (0.454, 0.455), a width that Besiroglu et al. argue is implausibly narrow [13].
Hoffmann et al., the compute of Gopher. 5.76×10²³ FLOPs by their count; 6ND with 280 billion parameters and 300 billion tokens gives 5.04×10²³; their appendix says a more accurate calculation gives 6.3×10²³.

These are small. They matter for a reason that returns in the research desk: a scaling exponent is the difference of two logarithms, and the third digit is not the paper's measurement.

24Scholarship

What did later work find behind the gap between 0.73 and 0.5?

Hoffmann et al.'s own account is the schedule: the Kaplan et al. protocol used many models under 100M parameters and one schedule for all. Three papers tested accounts since.

Counting parameters. Pearce and Song take the Chinchilla law, convert it to non-embedding parameters and compute, evaluate it over Kaplan's size range, and fit a local power law. They get 0.74 to 0.78 where Kaplan reported 0.73, and note that the relation between the non-embedding optimum and non-embedding compute is not a power law; they check it with five models of 0.8M to 4.6M parameters on BookCorpus, getting 0.49 with total parameters and 0.74 with non-embedding [4]. They recommend total parameters and compute.

Three causes. Porian et al. reproduce Kaplan's exponent in a modern setup (their 0.88 before Kaplan's batch-size correction) and then identify last layer computational cost, warmup duration and scale-dependent optimizer tuning. A fixed warmup is "too long": 3,000 steps at every scale. They also find that careful learning-rate decay is not essential, which contradicts Hoffmann et al.'s hypothesis, and that AdamW β₂ = 0.95 is suboptimal at small batches [5].

The refit. Besiroglu et al. find the third approach inconsistent with the other two, and their refit moves a from 0.454 to 0.5126.

The stage compares the IsoFLOP frontier under the non-embedding count (blue) with the total count (orange), on the same runs. Each of these accounts predicts something this miniature can measure, and the research desk does.

25Research desk

What was predicted, before any run?

The hypotheses, the design and the numeric verdict rules were written in research/README.md before any exponent was computed. Two things came earlier: a learning-rate sweep on a seed the grid never uses, and a check of the estimators on synthetic data that changed the grid (described below, and recorded in the README's amendments). Three arms were run: M (matched cosine schedules, 3 seeds), K (one fixed 4M-token schedule per size, 2 seeds) and W (matched, fixed 100-step warmup, 2 seeds).

E1 claim

Pearce and Song: the Kaplan–Chinchilla gap comes mostly from counting non-embedding parameters at small scale. Porian et al.: counting the head's compute lowers the exponent further.

Hypothesis

H1a: counting K instead of T raises the Approach 2 exponent by at least 0.05. H1b: counting the head lies between: a(K) > a(H) > a(T).

Design

The same 108 cells × 3 seeds regrouped under four counts; Approaches 1, 2 and 3; 200 seed-bootstraps.

E2 claim

Hoffmann et al.: the three approaches "yield comparable predictions"; 0.50, 0.49, 0.46. Besiroglu et al.: the third is off.

Hypothesis

H2a: with total parameters the three agree within 0.08. H2b: the IsoFLOP exponent is within [0.40, 0.60], with its interval inside [0.35, 0.65].

E3 claim

Hoffmann et al.: the fixed Kaplan protocol raises the exponent. Porian et al.: a fixed long warmup does. Pearce and Song: the schedule matters the other way.

Hypothesis

H3a: a(K) − a(M) ≥ 0.05 (envelope estimator). H3b: a(W) − a(M) ≥ 0.03 (Approach 2). Total parameters, seeds 0 and 1.

Verdict rule

Supported if the point difference reaches its threshold and the bootstrap 10th percentile is above 0; partly if the difference is positive but short of that; not supported if zero or negative. H1b: ordered. H2a: spread ≤ 0.08 supported, ≤ 0.15 partly. H2b: as stated.

26Research desk

Does the count change the exponent?

The stage draws the three IsoFLOP frontiers: non-embedding in blue, with the head in gold, total in orange.

27Research desk

Do the three estimators agree on one set of runs?

The three tubes on the stage are the envelope (teal), the IsoFLOP optimum (orange) and the parametric frontier (magenta), all under the total count.

28Research desk

Do the schedule and the warmup change the exponent?

The stage shows the matched frontier in orange, the fixed-warmup one in violet and the fixed-horizon envelope in blue.

29Research desk

What else is in the data?

30Research desk

What does this not show, and what next?

What this page does not show

Scale and vocabulary. The models are 1.5 thousand to 590 thousand parameters on 96 characters of technical prose. An embedding share of 45% down to 3% is what makes the count matter here; with a 50,000-token vocabulary the share is far higher at every size, and the effect should be larger. That is a prediction, not a result.
Training protocol. The batch is 1,024 tokens throughout, β₂ is 0.99, the learning-rate rule was fitted at three sizes and no run uses dropout. Porian et al. find that β₂ = 0.95 is suboptimal at small batches and that per-size tuning moves the exponent; none of that is tuned per cell here.
Early training. The cheapest cells are a hundred steps long and have barely left the plateau of the initial loss. They are in every fit. A grid that started later would have narrower intervals and a different answer for Approach 3.
Seeds. Three seeds for the headline arm, two for the others. The Approach 2 intervals are wide partly for that reason.
No batch-size correction. Kaplan et al. adjust their compute for training away from the critical batch size [9]; this page does not.

Next experiments

Repeat the grid with a 50,000-entry vocabulary (or a byte-pair encoding of the same corpus) and test whether the K − T gap grows with the embedding share, as the control predicts. Related library page: vocabulary-size scaling laws.
Tune the learning rate per cell instead of per size, and sweep warmup as a fraction of steps at every budget, to see whether the fixed-warmup effect of Porian et al. appears when the plateau is escaped in every run.
Add the critical-batch-size correction of Kaplan et al. to the compute of every run [9] and refit under the four counts.
Continue past one epoch with the same four counts, using the repeated-data results of Muennighoff et al. as the prediction [8].
Replace the objective: choose the model that minimises training plus a fixed number of inference requests, as in Sardana et al. [14], and see how far the optimum moves from the training-compute frontier on this family.

The pre-registration, every run, the estimators in two languages and the tests are in this piece's directory in the site's repository.

References

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J. and Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361v1 The first subject of this page. Quotes are from the v1 LaTeX source (the only version).
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O. and Sifre, L. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556v1 The second subject of this page ("Chinchilla"). Quotes are from the v1 LaTeX source, including a comment in that source that holds unrounded fit values.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł. and Polosukhin, I. (2017). Attention Is All You Need. NIPS 2017; arXiv:1706.03762 The architecture the family is built from; dissected on this site.
Pearce, T. and Song, J. (2024). Reconciling Kaplan and Chinchilla Scaling Laws. Transactions on Machine Learning Research; arXiv:2406.12907 (latest version, fetched 2026-10-23) Attributes much of the Kaplan/Chinchilla gap to counting non-embedding parameters at small scale; includes a tiny-scale experiment. Quotes from the LaTeX source.
Porian, T., Wortsman, M., Jitsev, J., Schmidt, L. and Carmon, Y. (2024). Resolving Discrepancies in Compute-Optimal Scaling of Language Models. arXiv:2406.19146 (latest version, fetched 2026-10-23) Reproduces Kaplan's law and finds three causes: last-layer cost, warmup, and optimiser tuning. Quotes from the LaTeX source; the source marks some of its numbers as tentative, and none of those is used here.
Loshchilov, I. and Hutter, F. (2019). Decoupled Weight Decay Regularization. ICLR 2019; arXiv:1711.05101 AdamW, the optimiser of Hoffmann et al.'s Chinchilla and of every run here.
Kingma, D. P. and Ba, J. (2015). Adam: A Method for Stochastic Optimization. ICLR 2015; arXiv:1412.6980 The optimiser of Kaplan et al.; dissected on this site.
Muennighoff, N., Rush, A. M., Barak, B., Le Scao, T. et al. (2023). Scaling Data-Constrained Language Models. NeurIPS 2023; arXiv:2305.16264 What changes when the data runs out. Quoted from its abstract only.
McCandlish, S., Kaplan, J., Amodei, D. and the OpenAI Dota Team (2018). An Empirical Model of Large-Batch Training. arXiv:1812.06162 The critical batch size that Kaplan et al. use to correct their compute estimates. Quoted from its abstract only.
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y. and Zhou, Y. (2017). Deep Learning Scaling is Predictable, Empirically. arXiv:1712.00409 The closest earlier work, by Kaplan et al.'s account. Quoted from its abstract only.
Henighan, T., Kaplan, J., Katz, M., Chen, M. et al. (2020). Scaling Laws for Autoregressive Generative Modeling. arXiv:2010.14701 Kaplan's group extends the law to other domains with a constant added to the power law. Quoted from its abstract only.
Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y. and Shavit, N. (2019). A Constructive Prediction of the Generalization Error Across Scales. ICLR 2020; arXiv:1909.12673 Appeared as Kaplan et al. finished; a similar joint form in model and data size. Quoted from its abstract only.
Besiroglu, T., Erdil, E., Barnett, M. and You, J. (2024). Chinchilla Scaling: A replication attempt. Epoch AI; arXiv:2404.10102 (v2) Refits Hoffmann et al.'s Approach 3 on data digitised from their figure and traces the published estimate to rounding and an early-stopped optimiser. Quotes from the LaTeX source.
Sardana, N., Portes, J., Doubov, S. and Frankle, J. (2023). Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws. ICML 2024; arXiv:2401.00448 Adds inference cost to the objective. Quoted from its abstract only.

Fact ledger

Every number from the literature on this page, with the sentence it comes from. Hover any underlined number in the text to see its entry.

Value	What	Source
over 400 models, 70M to over 16B parameters, 5 to 500 billion tokens	scale of the study	[2] Abstract By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens
double the model, double the tokens	the headline result	[2] Abstract for every doubling of model size the number of training tokens should also be doubled.
about 300 billion	tokens of the largest models before Chinchilla	[2] Table 1 caption Other than LaMDA, most models are trained for approximately 300 billion tokens.
eight in C, six in N, two in D	orders of magnitude covered	[1] §1.2 These relations hold across eight orders of magnitude in C_min, six orders of magnitude in N, and over two orders of magnitude in D.
an aspect ratio varied by a factor of 40 changes the loss only slightly	depth against width	[1] Fig. 5 caption Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance
N ≈ 12 n_layer d_model²	parameter count formula	[1] §2.1 = 12 n_layer d_model^2 with the standard d_attn = d_ff/4 = d_model
non-embedding parameters	definition of N	[1] §1.3 N -- the number of model parameters, excluding all vocabulary and positional embeddings
cleaner laws	why embeddings are excluded	[1] §2.1 we do not include these when discussing the model size N; we will see that this produces significantly cleaner scaling laws.
in parameters and FLOPs	embeddings counted	[2] App. F We include all training FLOPs, including those contributed to by the embedding matrices, in our analysis. Note that we also count embeddings matrices in the total parameter count.
non-embedding versus total parameters, at small scale	Pearce and Song's claim	[4] Abstract This paper finds that much of this discrepancy can be attributed to Kaplan counting non-embedding rather than total parameters, combined with their analysis being performed at small scale.
tied embeddings, perhaps	why the head was left out of the compute	[5] §3.2 Perhaps because their model head and embeddings had tied weights, this led them to also discount the contribution of the model head to the model's FLOPs per token
exponent falls by more than 0.1	effect of counting the head	[5] §3.2 switching to our model size count also reduces the exponent a by more than 0.1
1,024 tokens	context length	[1] §2 We optimize the autoregressive log-likelihood (i.e. cross-entropy loss) averaged over a 1024-token context, which is also our principal performance metric.
improve in parallel with WebText2	other text distributions	[1] §3.2.2 We see that the loss on these other data distributions improves smoothly with model size, in direct parallel with the improvement on WebText2.
6N	compute per training token	[1] §2.1 we then define the estimated non-embedding compute as C ≈ 6 N floating point operators per training token.
2 d_model n_vocab	de-embedding FLOPs	[1] Table 1 De-embed & --- & 2 d_model n_vocab
d_model > n_ctx/12	when the context term is negligible	[1] §2.1 For contexts and models with d_model > n_ctx / 12, the context-dependent computational cost per token is a relatively small fraction of the total compute.
10% at large models, 90% at small	the head's share of FLOPs	[5] §3.2 not accounting for the model head leads to under-approximation that grows smoothly as model size decreases, from roughly 10% at larger models to roughly 90% at smaller models.
0.003239 − 0.0001395 log N	learning-rate rule	[1] App. D.6 LR(N) ≈ 0.003239 + -0.0001395 log(N)
Adam, 2.5×10⁵ steps, 512 sequences of 1,024 tokens	training recipe	[1] §2.2 Unless otherwise noted, we train models with the Adam optimizer for a fixed 2.5 × 10^5 steps with a batch size of 512 sequences of 1024 tokens.
3,000-step warmup, cosine to zero	learning-rate schedule	[1] §2.2 all training runs included in our data used a learning rate schedule with a 3000 step linear warmup followed by a cosine decay to zero.
one schedule for all models	the first difference from Kaplan	[2] §2 First, the authors use a fixed number of training tokens and learning rate schedule for all models; this prevents them from modelling the impact of these hyperparameters on the loss.
4 horizons per size, 16× range	Approach 1 design	[2] §3.1 For each parameter count N we train 4 different models, decaying the learning rate by a factor of 10× over a horizon (measured in number of training tokens) that ranges by a factor of 16×.
9 FLOP counts from 6×10¹⁸ to 3×10²¹	Approach 2 design	[2] §3.2 (ranging from 6 ×10^18 to 3 ×10^21 FLOPs)
less than one epoch	single epoch	[2] §6 Finally, the training runs for our analysis have all been trained on less than an epoch of data
10% dropout and early stopping	regularisation for the data-limited runs	[1] §4.2 We regularize all our models with 10% dropout, and by tracking test loss and stopping once it is no longer decreasing.
poor for the smallest D	small-data fits	[1] App. C We did not thoroughly investigate the small data regime, and our fits for L(N,D) were poor for the smallest values of D
all models, not truly optimal	fixed batch size	[1] §3.2 Note that in these results the batch size B remains fixed for all models, which means that these empirical results are not truly optimal.
noise scale rises as the loss falls	the critical batch size	[9] Abstract We find that the noise scale increases as the loss decreases over a training run and depends on the model size primarily through improved model performance.
trained to convergence on sufficiently large datasets	L(N) is defined at convergence	[1] §1.2 For models with a limited number of parameters, trained to convergence on sufficiently large datasets:
0.076	exponent of L(N)	[1] App. A, key parameters alpha_N=0.076 & N_c=8.8×10^13 params (non-embed)
power-law generalization error scaling	power laws before Kaplan et al.	[10] Abstract Our empirical results show power-law generalization error scaling across a breadth of factors
non-zero entropy	the trends must level off	[1] §6.3 Our trends must eventually level off, though, since natural language has non-zero entropy.
C* ~ 10⁴ PF-days, N* ~ 10¹², D* ~ 10¹², L* ~ 1.7 nats	where the laws must break	[1] §6.3 C^* sim 10^4 PF-Days, N^* sim 10^12 parameters, D^* sim 10^12 tokens, L^* sim 1.7 nats/token
an order of magnitude either way	uncertainty of that point	[1] §6.3 though the numerical values are highly uncertain, varying by an order or magnitude in either direction depending on the precise values of the exponents from the power-law fits.
α_N 0.076, α_D 0.103, N_c 6.4×10^13, D_c 1.8×10^13	fit of L(N, D)	[1] §4.2 Value & 0.076 & 0.103 & 6.4 × 10^13 & 1.8 × 10^13
three principles	the L(N, D) ansatz	[1] §4.1 We conjecture that this functional form may also parameterize the trained log-likelihood for other generative modeling tasks.
8× the model, about 5× the data	data needed per model growth	[1] §1.1 every time we increase the model size 8x, we only need to increase the data by roughly 5x to avoid a penalty.
5×; data 2×	model growth per 10× compute	[1] Fig. 14 caption Optimal model size grows very rapidly with C_min, increasing by 5x for each 10x increase in compute. The number of data examples processed makes up the remainder of the increase, growing relatively modestly by only 2x.
0.73	optimal model size exponent	[1] App. A, compute-efficient training N_opt = N_e·C_min^p_N & p_N = 0.73 & N_e = 1.3·10^9 params
0.27	optimal data exponent	[1] App. A, compute-efficient training D_opt = D_e·C_min^p_D (1 epoch) & p_D = 0.27 & D_e = 2·10^10 tokens
0.03	optimal steps exponent	[1] App. A, compute-efficient training S_min = S_e·C_min^p_S (lower bound) & p_S = 0.03 & S_e = 5.4·10^3 steps
stop significantly short of convergence	convergence is inefficient	[1] §1.1 we attain optimal performance by training very large models and stopping significantly short of convergence
big models over big data	the paper's conclusion	[1] §8 Big models may be more important than big data.
5.5× model, 1.8× tokens	what Kaplan implies for 10× compute	[2] §1 Specifically, given a 10× increase computational budget, they suggests that the size of the model should increase 5.5× while the number of training tokens should only increase 1.8×.
0.050	exponent of L(C_min)	[1] App. A, key parameters alpha_C^min=0.050 & C_c^min=3.1×10^8 PF-days
0.054	exponent predicted from L(N, S_min)	[1] §6.2 α_C^min ≡ 1/(1/α_S + 1/α_B + 1/α_N) ≈ 0.054
0.052	the same exponent in the appendix	[1] App. B.2 α_C = 1/(1/α_S + 1/α_B + 1/α_N) ≈ 0.052
Figure X	unresolved figure reference	[1] App. B.4 The result is shown in Figure X.
1,500 log-spaced FLOP values	Approach 1 evaluation points	[2] §3.1 At 1500 logarithmically spaced FLOP values, we find which model size achieves the lowest loss of all models along with the required number of training tokens.
within the last 15% of training	where the envelope points sit	[2] §3.1 footnote Note that all selected points are within the last 15% of training.
a = 0.50, b = 0.50	Approach 1 exponents	[2] Table 2 1. Minimum over training curves & 0.50 (0.488, 0.502) & 0.50 (0.501, 0.512)
a parabola per budget	IsoFLOP minima	[2] §3.2 We fit a parabola to each IsoFLOPs curve to directly estimate at what model size the minimum loss is achieved
a = 0.49, b = 0.51	Approach 2 exponents	[2] Table 2 2. IsoFLOP profiles & 0.49 (0.462, 0.534) & 0.51 (0.483, 0.529)
bootstrap, 80% resampled 100 times	how the intervals were made	[2] Table 2 caption The 10th and 90th percentiles are estimated via bootstrapping data (80% of the dataset is sampled 100 times) and are shown in parenthesis.
about 0.02	seed-to-seed spread	[1] §4.2 We estimate that the variation in the loss with different random seeds is roughly 0.02
E + A/N^α + B/D^β	parametric form	[2] §3.3 L̂(N,D) ≜ E + A/N^α + B/D^β
entropy of natural text	meaning of E	[2] §3.3 The first term captures the loss for an ideal generative process on the data distribution, and should correspond to the entropy of natural text.
a = β/(α+β), b = α/(α+β)	closed-form frontier	[2] §3.3 a = β/(α+β), and b = α/(α + β)
δ = 10⁻³	Huber threshold	[2] §3.3, App. D.2 We use delta = 10^-3 for the Huber loss.
a = 0.46, b = 0.54	Approach 3 exponents	[2] Table 2 3. Parametric modelling of the loss & 0.46 (0.454, 0.455) & 0.54 (0.542, 0.543)
0.33917084 for the parameter slope	unrounded exponent, left in a TeX comment	[2] main.tex, after App. D.2 'param_slope': array(0.33917084, dtype=float32),
0.2849083 for the token slope	unrounded token exponent, left in a TeX comment	[2] main.tex, after App. D.2 'token_slope': array(0.2849083, dtype=float32)
β = 0.2849, reported as 0.28	rounding in the body of the paper	[13] §3 The true value of the data exponent beta should be around 0.2849 according to the comments in the TeX source code
averaged, not summed, Huber loss	why the original fit stopped early	[13] §3.1 they averaged the Huber loss values over different data points instead of summing them during loss minimization and the high loss scale of their optimizer caused their optimization to terminate early
E 1.8172, A 482.01, B 2085.43, α 0.3478, β 0.3658	their refit	[13] §3 L(N, D) = 1.8172 + 482.01/N^0.3478 + 2085.43/D^0.3658
comparable predictions	the three approaches agree	[2] §3.4 We find that the three approaches, despite using different fitting methodologies and different trained models, yield comparable predictions for the optimal scaling in parameters and tokens with FLOPs
treated as outliers by the Huber loss	low-compute points	[2] §3.4 The fitted model places increased weight on the points with more FLOPs---automatically considering the low-computational budget points as outliers due to the Huber loss.
concave at high compute	curvature of the frontier	[2] §6 we observe some concavity in log(N_opt) at high compute budgets
left as future work	curvature not taken into account	[2] App. E In this work, we do not take this in to account and we leave this as interesting future work
20.2 billion tokens	tokens per parameter at 1B	[2] Table 3 1 Billion & 1.21e+20 & 1/4,761 & 20.2 Billion
smaller and longer than Chinchilla-optimal	inference-aware sizing	[14] Abstract LLM researchers expecting reasonably large inference demand (~1B requests) should train models smaller and longer than Chinchilla-optimal.
quality keeps improving up to 10,000	extreme tokens per parameter	[14] Abstract model quality continues to improve as we scale tokens per parameter to extreme ranges (up to 10,000)
5.76×10²³ FLOPs	Gopher's compute	[2] Fig. 2 caption based on the number of FLOPs used to train Gopher (5.76 × 10^23)
4× smaller, 4× the tokens	Gopher against Chinchilla	[2] §1 we predict that for the compute budget used to train Gopher, an optimal model should be 4 times smaller, while being training on 4 times more tokens.
70B parameters, 1.4T tokens	Chinchilla	[2] §4 We test this hypothesis by training a model on the larger end of this range---70B parameters---for 1.4T tokens
80 layers, 64 heads, d_model 8,192	Chinchilla architecture	[2] Table 4 Chinchilla 70B & 80 & 64 & 128 & 8,192 & 1 × 10^-4 & 1.5M → 3M
AdamW	optimiser change	[2] §4.1 We use AdamW for Chinchilla rather than Adam as this improves the language modelling loss and the downstream task performance after finetuning.
67.6%	MMLU in the body	[2] §4.2.2 with an average accuracy of 67.6% (improving upon Gopher by 7.6%)
two comparable runs	validation at large scale	[2] §6 Due to the cost of training large models, we only have two comparable training runs at large scale (Chinchilla and Gopher), and we do not have additional tests at intermediate scales.
4.68B against 2.86B recommended	head-to-head at 10²¹ FLOPs	[2] App. D.3 From Kaplan et al. (2020), we find that the optimal model size should be 4.68 billion parameters. From our approach 1, we estimate a 2.86 billion parameter model should be optimal.
1.3×10⁹ (C / 8.64×10¹⁹)^0.73	Kaplan's adjusted power law, as Porian et al. write it	[5] §3.5 = 1.3e9 (C / 8.64e19)^0.73 which Kaplan et al. (2020) obtain by estimating the compute required to reach the same results at a sufficiently low batch size
8.0 billion tokens	Table 3, 400 million parameters	[2] Table 3 400 Million & 1.92e+19 & 1/29,968 & 8.0 Billion
205.1 billion tokens	Table 3, 10 billion parameters	[2] Table 3 10 Billion & 1.23e+22 & 1/46 & 205.1 Billion
21.2 trillion tokens	Table 3, 1 trillion parameters	[2] Table 3 1 Trillion & 1.27e+26 & 221.3 & 21.2 Trillion
D ≳ 5×10³ N^0.74	data needed to avoid overfitting	[1] §4.2 D gtrsim (5 × 10^3) N^0.74
up to 4 epochs behave like unique data	repeated data	[8] Abstract We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data.
0.095	exponent of L(D)	[1] App. A, key parameters alpha_D=0.095 & D_c=5.4×10^13 tokens
fitting the full L(N, D)	why the joint fit differs	[1] §4.2 as here we are fitting the full L(N,D) rather than just L(N, ∞) or L(∞, D).
about 0.05 in the loss	run-to-run variation, learning-rate figure	[1] App. D.6, Fig. caption Run-to-run variation is at the level of ~0.05 in the loss, so averaging multiple runs is necessary to validate performance changes smaller than this level.
5B to over 400B tokens	the same range, in the introduction	[2] §1 and trained on 5B to over 400B tokens
67.5%	MMLU in the abstract	[2] Abstract Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark
Approach 3's estimates inconsistent with Approaches 1 and 2	Besiroglu et al.'s finding	[13] Abstract we find that their reported estimates are inconsistent with their first two estimation methods, fail at fitting the extracted data, and report implausibly narrow confidence intervals
6.3×10²³	recounted Gopher compute	[2] App. F we use a slightly more accurate calculation giving a slightly different value (6.3 × 10^23 compared to 5.76 × 10^23)
most Kaplan runs under 100M	the second difference	[2] §2 in contrast the majority of runs in Kaplan et al. (2020) are significantly smaller---many being less than 100M parameters.
0.74 to 0.78 against 0.73	the simulated local exponent	[4] §3, Step 4 which are close to the Kaplan coefficient of 0.73.
only locally	the relation is not a power law	[4] §3, Step 3 This shows that in general the relationship between N^*_∖E & C_∖E is not a power law.
five models, 0.8M to 4.6M, BookCorpus	their tiny models	[4] §4, Experiment 1 We trained five models of sizes, N_T in [ 0.8M, 1.6M, 2.1M, 3.3M, 4.6M] on the BookCorpus dataset.
0.49 with total parameters	their tiny-scale result	[4] §4, Result 1 Ours, N_T & C_T & 0.49 & 0.51
0.74 with non-embedding parameters	their tiny-scale result, non-embedding	[4] §4, Table 1 Ours, N_∖E & C_∖E & 0.74 & 0.26
total parameters and compute	their recommendation	[4] Abstract These findings lead us to recommend that future scaling studies use total parameters and compute.
0.88	Kaplan's unadjusted exponent	[5] §3.1 we obtain a compute-optimal exponent a and power law fits close to the power law 1.6e9 (C / 8.64e19)^0.88 obtained by Kaplan et al. (2020)
three factors	Porian et al.'s explanation	[5] Abstract identifying three factors causing the difference: last layer computational cost, warmup duration, and scale-dependent optimizer tuning.
3,000 steps at every scale	Kaplan's warmup is too long	[5] §3.3 this warmup period is too long: for smaller-scale models, the optimal number of tokens as a function of compute is less than or close to the number of warmup tokens, and therefore these models are suboptimally trained.
contra Hoffmann et al.	learning-rate decay is not essential	[5] Abstract Counter to a hypothesis implied in Hoffmann et al. (2022), we find that careful learning rate decay is not essential for the validity of their scaling law.
0.95 is suboptimal at 128 sequences or fewer	AdamW β₂ at small batch	[5] §1 we observe that setting the AdamW beta_2 parameter to be 0.95 is suboptimal at smaller batch sizes (128 and below)

How this was made

The eleven models are PyTorch-trained float16 weights run forward in plain JavaScript and checked against PyTorch (logits, attention, per-character loss). The estimators (power-law and parabola fits, Approaches 1, 2 and 3, Kaplan's L(N, D)) are the same code that produced the research results and are checked against numpy and scipy, against a noiseless surface with a known optimum, and against every derived number in the text. Literature numbers are sourced to verbatim quotes from the arXiv LaTeX; the hypotheses and verdict rules were written before the grid ran.

Built with the dissection skill. The visual grammar of tensors as blocks of cells owes much to Brendan Bycroft's LLM Visualization. The eleven models and the estimators are checked against PyTorch, numpy and scipy. Corpus, training scripts, the pre-registration, every run and the tests: anatomy/ai-ml/2026-10-23-scaling-laws/ in the site's repository.

Cite this page

@misc{sigmoid-scaling-laws,
  title  = {Anatomy of Neural Scaling Laws},
  author = {Praveen T N},
  year   = {2026},
  note   = {Sigmoid Anatomy, interactive dissection},
  url    = {https://praveentn.live/anatomy/scaling-laws}
}