Ternary and Extreme Low-Bit Models

In quantisation-aware training, the forward pass quantises weights so the network experiences rounding error and learns around it. But the quantisation function is a step function with zero gradient almost everywhere, so the backward pass would receive nothing.

The straight-through estimator replaces its derivative with the identity, passing the gradient through unchanged.

It is wrong by construction and works because coarse rounding behaves like noise the optimiser averages over. It breaks when quantisation is coarse enough that the error becomes systematic rather than noise-like.

\(\log_2 3 \approx 1.58\): three states per weight, \(\{-1, 0, +1\}\).

The arithmetic is the real point. With those weights a matmul contains no multiplications at all, only adds, subtracts, and skips. Multiplication dominates the energy budget of a matmul unit, so removing it changes the hardware equation rather than just the memory equation. The zero state also gives structured sparsity for free.

BitNet b1.58 claims parity with a full-precision transformer at matched size and training tokens.

Because it is a pretraining recipe, not a compression method. The model must experience ternary constraints during training so its representations distribute themselves across the states available.

Post-training quantisation below about three bits fails outright: per-layer reconstruction error exceeds what the next layer can absorb, and no calibration data or rounding scheme recovers it.

The practical consequence is that adopting ternary costs a full training run up front, which is why uptake sits with labs training their own models rather than with deployment teams.

8-bit weights and activations: routinely lossless post-training.

4-bit weight-only: standard practice on large models, small measurable loss.

4-bit weights and activations: workable with rotation methods (QuaRot, SpinQuant), near-lossless on large models.

2-bit and below, post-training: not viable as a drop-in.

1.58-bit trained from scratch: viable, with an author-reported parity claim not yet independently reproduced at frontier scale.

Every line in this list is size-dependent; smaller models sit one step to the left.

That the realised speedup is far below the paper's arithmetic argument, because kernel support lags. Ternary weights only pay off if the kernels actually exploit add-subtract-skip; without them the weights are unpacked to bf16 and multiplied conventionally.

You then get the memory saving and none of the compute or energy saving, which was the whole point.

The same gap appears for int4: without int4 tensor-core support on your hardware, a 4-bit method silently falls back to dequantise-then-bf16.

In quantisation-aware training, the forward pass quantises weights so the network experiences rounding error and learns around it. But the quantisation function is a step function with zero gradient almost everywhere, so the backward pass would receive nothing.

The straight-through estimator replaces its derivative with the identity, passing the gradient through unchanged.

It is wrong by construction and works because coarse rounding behaves like noise the optimiser averages over. It breaks when quantisation is coarse enough that the error becomes systematic rather than noise-like.

\(\log_2 3 \approx 1.58\): three states per weight, \(\{-1, 0, +1\}\).

The arithmetic is the real point. With those weights a matmul contains no multiplications at all, only adds, subtracts, and skips. Multiplication dominates the energy budget of a matmul unit, so removing it changes the hardware equation rather than just the memory equation. The zero state also gives structured sparsity for free.

BitNet b1.58 claims parity with a full-precision transformer at matched size and training tokens.

Because it is a pretraining recipe, not a compression method. The model must experience ternary constraints during training so its representations distribute themselves across the states available.

Post-training quantisation below about three bits fails outright: per-layer reconstruction error exceeds what the next layer can absorb, and no calibration data or rounding scheme recovers it.

The practical consequence is that adopting ternary costs a full training run up front, which is why uptake sits with labs training their own models rather than with deployment teams.

8-bit weights and activations: routinely lossless post-training.

4-bit weight-only: standard practice on large models, small measurable loss.

4-bit weights and activations: workable with rotation methods (QuaRot, SpinQuant), near-lossless on large models.

2-bit and below, post-training: not viable as a drop-in.

1.58-bit trained from scratch: viable, with an author-reported parity claim not yet independently reproduced at frontier scale.

Every line in this list is size-dependent; smaller models sit one step to the left.

That the realised speedup is far below the paper's arithmetic argument, because kernel support lags. Ternary weights only pay off if the kernels actually exploit add-subtract-skip; without them the weights are unpacked to bf16 and multiplied conventionally.

You then get the memory saving and none of the compute or energy saving, which was the whole point.

The same gap appears for int4: without int4 tensor-core support on your hardware, a 4-bit method silently falls back to dequantise-then-bf16.