Low-Bit Number Formats and Microscaling

Integer quantisation forces a uniform grid onto a distribution that is anything but uniform. Neural network weights and activations are roughly bell-shaped with long tails, so most values cluster near zero where a uniform grid wastes resolution, and the tail values that matter get clipped. Low-bit floating-point formats attack this directly by making the grid logarithmic, and the interesting engineering is in how few bits a usable float needs.

Splitting eight bits

FP8 exists in two standard variants, both defined in the OCP 8-bit floating point specification and both implemented in NVIDIA Hopper and later, and in comparable hardware from other vendors.

E4M3 gives 4 exponent bits and 3 mantissa bits, with a maximum magnitude around 448 and about 2 decimal digits of precision. Its dynamic range is narrow and its resolution is comparatively good.

E5M2 gives 5 exponent bits and 2 mantissa bits, reaching about 57,344 with roughly 1 decimal digit of precision. It is a truncated bfloat16 in structure and covers a much wider range.

The division of labour follows from what each tensor needs. Weights and activations have limited dynamic range and benefit from precision, so they use E4M3. Gradients span many orders of magnitude and tolerate coarse precision, so they use E5M2. Training recipes that use FP8 throughout typically apply exactly this split, with fp32 accumulation and a per-tensor scaling factor updated dynamically.

Why per-tensor scaling fails below 8 bits

At 4 bits there are 16 levels. Any single scale factor covering a whole tensor, or even a whole channel, leaves most values quantised to two or three distinct levels. The response is to shrink the scope of the scale until the values it covers are genuinely similar.

Microscaling formats standardise this. An MX format assigns a shared 8-bit power-of-two exponent to each block of 32 consecutive elements, with each element carrying a small mantissa: MXFP8, MXFP6 and MXFP4 differ in the element width. MXFP4 stores 4 bits per element plus 8 bits per 32-element block, giving 4.25 bits per value.

The shared exponent is the key idea. Within 32 adjacent elements the dynamic range is small, so one exponent describes them all well, and the per-element bits are spent entirely on mantissa. It is the same reasoning as per-group integer quantisation, expressed in a format the hardware can decode natively.

When it breaks

Hardware support decides, not numerics. A format is only fast if the tensor cores read it directly. Blackwell adds native MXFP4 and MXFP6 support; earlier generations do not, so an MXFP4 tensor there must be unpacked to a supported type before the matmul, which costs more than it saves. Format choice is therefore a deployment-target question first and a numerical one second.

FP8 training needs dynamic scaling and it is fragile. The per-tensor scale is maintained from recent history of observed maxima, so a sudden spike overflows before the scale catches up. Delayed scaling using a window of previous maxima is standard, and the window length is a real hyperparameter: too short and the scale is noisy, too long and it lags a genuine distribution shift.

Reduced precision changes what "the same run" means. FP8 training is not bitwise reproducible across hardware or library versions, and small numerical differences compound over long runs. Comparing an FP8 run against a bf16 baseline requires matching loss curves over many thousands of steps rather than checking a single checkpoint.

Block size is a genuine tradeoff, not a constant. Thirty-two elements per block is a standardisation choice balancing metadata overhead against how well one exponent fits a block. Smaller blocks track the distribution better and cost more metadata; larger blocks are cheaper and reintroduce the outlier problem. The standard picked one point on that curve, and it is not optimal for every tensor.

Accumulate in something wider, always. Whatever the storage format, the dot-product accumulator must be fp32 or at least fp16 with careful ordering. Accumulating thousands of terms in a low-bit format loses small contributions to rounding entirely, and the resulting error is systematic rather than random, so it does not average out across a batch.