NF4 and Double Quantisation

A 65-billion-parameter model needs roughly 130 GB in BFloat16. A single H100 SXM holds 80 GB. Without quantisation, fine-tuning that model is physically impossible on one GPU. NF4, introduced in the QLoRA paper (Dettmers et al., 2023), closes that gap not by approximating the model crudely, but by exploiting a structural fact about how pretrained weights are distributed: they are almost always approximately Gaussian.

Why the weight distribution matters for quantisation

Standard 4-bit integer formats (INT4) map values to equally spaced bins across a fixed range. If your weights are clustered near zero with light tails, most of those 16 bins are wasted on the extremes where almost no weight values live. You are paying 4 bits per weight but using the resolution of perhaps 2 or 3 effective bits.

A better strategy is to place quantisation levels where the data actually is, which is what quantile quantisation does. Given a distribution, you find the 16 quantiles and use those as the bin boundaries. Each bin then represents an equal fraction of the probability mass, which is the information-theoretically optimal arrangement for a fixed bit-width.

The problem with naive quantile quantisation is that you need to estimate the quantiles from data at runtime, which is slow and per-tensor. NF4 sidesteps this by pre-computing the optimal quantiles once, assuming the weights follow a zero-mean unit-variance normal distribution after normalisation:

# Conceptual sketch
quantiles = [Q(p) for p in linspace(0, 1, 17)]   # 17 boundaries => 16 bins
nf4_codebook = [(quantiles[i] + quantiles[i+1]) / 2 for i in range(16)]
nf4_codebook = nf4_codebook / max(abs(nf4_codebook))  # scale to [-1, 1]

At quantisation time, each weight tensor is scaled so that its absolute maximum maps to 1.0, then each value is rounded to the nearest entry in the fixed 16-point NF4 codebook. The scale factor (one float16 per block of 64 weights by default) is stored separately. Dequantisation multiplies the looked-up codebook value by the stored scale. The total storage per weight is 4 bits for the index plus 16 bits/64 = 0.25 bits for the scale, giving roughly 4.25 bits per parameter.

Double quantisation: quantising the quantisation constants

The scale factors themselves are full float16 values. With a block size of 64, you need one scale per 64 weights, which is 16 bits / 64 = 0.25 bits per weight. That sounds cheap, but across a 65B model it accumulates to roughly 512 MB of scale-factor overhead.

Double quantisation applies a second round of quantisation to those scale constants:

1. Collect all block-level scale factors across a larger "super-block" (typically 256 weights wide).
2. Quantise those scales to 8-bit integers, storing one float32 super-scale per super-block.
3. The super-scale is tiny: one float32 per 256 weights is 32/256 = 0.125 bits per weight.

The net result: scales go from 16-bit (0.25 bits/weight) to 8-bit (0.0625 bits/weight from the quantised scale) plus 0.125 bits/weight from the super-scale, totalling roughly 0.1875 bits/weight. The paper reports this saves approximately 0.4 bits per parameter compared to not double-quantising, which at 65B parameters is roughly 3 GB.

QLoRA: putting NF4 and double quantisation to work

NF4 and double quantisation alone only reduce memory for storing and loading the frozen model. To actually fine-tune, you still need gradients. QLoRA threads those gradients through the quantised weights without ever dequantising in the backward pass storage sense: the NF4 model is treated as a frozen constant, and only the LoRA adapter weights (typically rank-8 to rank-64 matrices in BFloat16) accumulate gradients.

The forward pass dequantises each NF4 block on-the-fly into BFloat16 for the matrix multiply, then the activations flow into the LoRA branch. Because dequantised values are never stored between micro-steps (only computed transiently), the memory footprint stays close to the 4-bit storage size. The practical upshot: a 65B model (originally ~130 GB in BFloat16) fits in roughly 41 GB in NF4 with double quantisation, enabling fine-tuning on a single A100 80 GB or two consumer 24 GB GPUs.

QLoRA also introduced paged optimisers (using NVIDIA unified memory to spill optimiser states to CPU RAM during memory spikes) as a third complementary technique.

When it falls down

Quantisation error is not uniform. NF4 assumes weights are normally distributed, but certain layers diverge from this, particularly embedding tables and the final language model head. These are typically kept in BFloat16 or skipped during quantisation. Getting this wrong silently degrades perplexity.

Throughput vs. memory trade-off. Dequantising on-the-fly during the forward pass adds compute overhead. On hardware with ample memory bandwidth (e.g., H100s with HBM3), the dequantisation cost can make NF4 inference slower than BFloat16 inference on the same GPU, even though it fits a larger model. NF4 is a memory-budget tool first; it is not always the fastest option.

Double quantisation interacts badly with very small tensors. The super-block structure assumes you have enough weights to form multiple 256-weight super-blocks. For tiny projection layers or bespoke small modules, the quantisation overhead can exceed the savings.

Calibration data matters less than in PTQ, but still matters. Because NF4 uses a fixed codebook derived from the normal distribution assumption rather than data-calibrated quantiles, unusual weight distributions (e.g., from models trained with aggressive weight decay or layer-wise learning rate schedules) may see larger-than-expected quantisation error.

Not hardware-accelerated universally. INT4 tensor core operations are natively supported on Ampere and later NVIDIA GPUs. NF4 is a lookup-table data type, not a native hardware integer type, so compute kernels must perform a table lookup plus dequantise step. On GPUs without custom kernels (or non-NVIDIA hardware), NF4 falls back to software emulation, which is considerably slower.

Fine-tuning with NF4 is not the same as NF4 inference. After QLoRA training, the standard workflow is to merge the LoRA adapter into the BFloat16 base model weights, not to deploy the NF4 checkpoint directly. Deploying the unmerged quantised checkpoint with adapters introduces the on-the-fly dequantisation overhead at inference time.

Further reading

• Dettmers, T. et al. (2023). "QLoRA: Efficient Finetuning of Quantized LLMs." arXiv:2305.14314. The primary source for NF4, double quantisation, and the QLoRA training procedure. https://arxiv.org/abs/2305.14314
• Dettmers, T. et al. (2022). "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale." NeurIPS 2022. arXiv:2208.07339. The predecessor work on mixed-precision 8-bit quantisation that motivated the bitsandbytes library. https://arxiv.org/abs/2208.07339
• Hugging Face. "Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA." Official blog post with practical usage of NF4 and double quantisation in the bitsandbytes + transformers stack. https://huggingface.co/blog/4bit-transformers-bitsandbytes
• Hu, E. et al. (2021). "LoRA: Low-Rank Adaptation of Large Language Models." arXiv:2106.09685. Required background for understanding the adapter component of QLoRA. https://arxiv.org/abs/2106.09685