Evaluating Quantised Models

Because accuracy is a marginal and hides swaps. The compressed model can newly fail 6% of the questions the baseline got right while newly passing 5.7% it got wrong, and the aggregate moves by 0.3.

Dutta et al. (2024) found exactly this: aggregate accuracy within 2 points across six quantisation schemes and seven tasks, while all but GPTQ W8A16 showed large flip counts and underperformed on MT-Bench generative evaluation.

Users experience the flips, not the average.

A flip is a question whose correctness changes between baseline and compressed model, in either direction.

Directional reporting matters because incorrect-to-correct flips are not a bonus. A compression scheme is meant to preserve behaviour, so a newly correct answer is still evidence that the model changed, and the same underlying perturbation that produced it will produce correct-to-incorrect flips elsewhere.

Counting only the net effect reproduces exactly the blindness that accuracy already had.

Because it compares the full next-token distributions of baseline and quantised model rather than only the arg-max, so it detects degradation long before any answer changes. It is continuous, requires no labels, and correlates strongly with flip rate.

It also matches the actual objective: a quantised model is not supposed to be capable, it is supposed to be the same as the model you already accepted.

That reframing, from an accuracy question to a distance question, is the main contribution of the flips literature.

Because it reads a single token position. Quantisation error compounds across hundreds of autoregressive steps, so long-form generation degrades earlier and far more visibly than any single-token score suggests.

Structured output and long context amplify it further: JSON validity, format adherence and function-call correctness routinely break before benchmark accuracy moves at all.

Add a generative comparison against the baseline (pairwise preference or a judge model) and segment-level breakouts for long prompts, non-English, code and structured output.

Because different batch sizes and kernel paths change floating-point reduction order, so two runs of the same model can produce different outputs. Compare a baseline at batch 1 against a quantised model at batch 32 and you are measuring batch non-invariance as well as quantisation.

Fix seed, batch shape, kernel selection and sequence lengths on both sides, then attribute the difference.

The same discipline applies to speedup claims: report at your batch size, sequence lengths and hardware, since a 3x decode gain at batch 1 says little at batch 64.

1. Fix a corpus of a few thousand real production prompts, held out from calibration.
2. Generate greedily from both models with identical seed, batch shape and kernel configuration.
3. Report token-level KL, output exact-match rate, directional flip rate on the labelled subset, and a pairwise preference score.
4. Break out by segment: long prompts, non-English, code, structured output.
5. Only then look at latency and memory.

Re-run the whole gate after any weight change, including a LoRA merge.

Because accuracy is a marginal and hides swaps. The compressed model can newly fail 6% of the questions the baseline got right while newly passing 5.7% it got wrong, and the aggregate moves by 0.3.

Dutta et al. (2024) found exactly this: aggregate accuracy within 2 points across six quantisation schemes and seven tasks, while all but GPTQ W8A16 showed large flip counts and underperformed on MT-Bench generative evaluation.

Users experience the flips, not the average.

A flip is a question whose correctness changes between baseline and compressed model, in either direction.

Directional reporting matters because incorrect-to-correct flips are not a bonus. A compression scheme is meant to preserve behaviour, so a newly correct answer is still evidence that the model changed, and the same underlying perturbation that produced it will produce correct-to-incorrect flips elsewhere.

Counting only the net effect reproduces exactly the blindness that accuracy already had.

Because it compares the full next-token distributions of baseline and quantised model rather than only the arg-max, so it detects degradation long before any answer changes. It is continuous, requires no labels, and correlates strongly with flip rate.

It also matches the actual objective: a quantised model is not supposed to be capable, it is supposed to be the same as the model you already accepted.

That reframing, from an accuracy question to a distance question, is the main contribution of the flips literature.

Because it reads a single token position. Quantisation error compounds across hundreds of autoregressive steps, so long-form generation degrades earlier and far more visibly than any single-token score suggests.

Structured output and long context amplify it further: JSON validity, format adherence and function-call correctness routinely break before benchmark accuracy moves at all.

Add a generative comparison against the baseline (pairwise preference or a judge model) and segment-level breakouts for long prompts, non-English, code and structured output.

Because different batch sizes and kernel paths change floating-point reduction order, so two runs of the same model can produce different outputs. Compare a baseline at batch 1 against a quantised model at batch 32 and you are measuring batch non-invariance as well as quantisation.

Fix seed, batch shape, kernel selection and sequence lengths on both sides, then attribute the difference.

The same discipline applies to speedup claims: report at your batch size, sequence lengths and hardware, since a 3x decode gain at batch 1 says little at batch 64.

1. Fix a corpus of a few thousand real production prompts, held out from calibration.
2. Generate greedily from both models with identical seed, batch shape and kernel configuration.
3. Report token-level KL, output exact-match rate, directional flip rate on the labelled subset, and a pairwise preference score.
4. Break out by segment: long prompts, non-English, code, structured output.
5. Only then look at latency and memory.

Re-run the whole gate after any weight change, including a LoRA merge.