Binary and Int8 Embedding Quantisation

It keeps only the sign of each component of an L2-normalised vector, discarding magnitude within every dimension. Geometrically it projects the vector onto the nearest corner of a hypercube, retaining only which orthant it occupies.

Retrieval survives because in high-dimensional learned embedding spaces much of the discriminative signal lives in the sign pattern. It fails when two documents share an orthant but differ in how strongly, which is corpus-dependent and must be measured rather than assumed.

float32: 1024 dims, 8,192 bytes.

int8: still 1024 dims, 2,048 bytes, a 4x reduction; similarity remains a dot product in int8 with int32 accumulation.

binary: 1024 bits, 128 bytes, a 32x reduction; similarity becomes Hamming distance (XOR plus popcount), giving retrieval speedups of a similar order.

With rescoring, binary preserves up to roughly 96% of total retrieval performance. The reference Wikipedia pipeline goes from about 200GB of vectors to roughly 5.2GB in memory plus 47.5GB on disk.

Search the binary index for a shortlist of the top 100 to 1000, fetch the original float32 vectors for just those candidates, recompute exact similarity, and re-order before returning the top k.

It turns quantisation from a lossy shortcut into a memory hierarchy: the compact index stays in RAM for the approximate first pass, the full-precision vectors live on disk and are touched a few hundred times per query.

It does not raise the recall ceiling. If the answer misses the binary shortlist, rescoring cannot recover it.

They multiply, because they compress different axes: Matryoshka cuts the number of dimensions, quantisation cuts bits per dimension. 1024 float32 dims truncated to 256 and binarised is 128x smaller before rescoring.

They degrade differently. Matryoshka truncation degrades gracefully, since the model was explicitly trained so that prefixes are usable representations. Binarisation degrades abruptly once the sign pattern stops separating your corpus, and that threshold is a property of the data, not the model.

Scalar quantisation maps a value range onto 256 buckets, and that range is estimated from a sample of your embeddings. If the corpus distribution shifts, the calibration no longer matches, and quality degrades with no error and no obvious signal.

Binary quantisation has no such parameter: the threshold is zero for normalised vectors, so there is nothing to go stale. That statelessness, rather than the compression ratio, is often the stronger operational argument.

Below a few million vectors, where float32 fits in memory comfortably and the added shortlist-plus-rescore machinery buys complexity you pay for during debugging.

Also when your bottleneck is elsewhere: if recall is limited by chunking or by the embedder itself, a 32x smaller index does not help. And when quality retention on your own corpus has not been measured, since the 96% figure is a benchmark result on specific data, not a guarantee.

It keeps only the sign of each component of an L2-normalised vector, discarding magnitude within every dimension. Geometrically it projects the vector onto the nearest corner of a hypercube, retaining only which orthant it occupies.

Retrieval survives because in high-dimensional learned embedding spaces much of the discriminative signal lives in the sign pattern. It fails when two documents share an orthant but differ in how strongly, which is corpus-dependent and must be measured rather than assumed.

float32: 1024 dims, 8,192 bytes.

int8: still 1024 dims, 2,048 bytes, a 4x reduction; similarity remains a dot product in int8 with int32 accumulation.

binary: 1024 bits, 128 bytes, a 32x reduction; similarity becomes Hamming distance (XOR plus popcount), giving retrieval speedups of a similar order.

With rescoring, binary preserves up to roughly 96% of total retrieval performance. The reference Wikipedia pipeline goes from about 200GB of vectors to roughly 5.2GB in memory plus 47.5GB on disk.

Search the binary index for a shortlist of the top 100 to 1000, fetch the original float32 vectors for just those candidates, recompute exact similarity, and re-order before returning the top k.

It turns quantisation from a lossy shortcut into a memory hierarchy: the compact index stays in RAM for the approximate first pass, the full-precision vectors live on disk and are touched a few hundred times per query.

It does not raise the recall ceiling. If the answer misses the binary shortlist, rescoring cannot recover it.

They multiply, because they compress different axes: Matryoshka cuts the number of dimensions, quantisation cuts bits per dimension. 1024 float32 dims truncated to 256 and binarised is 128x smaller before rescoring.

They degrade differently. Matryoshka truncation degrades gracefully, since the model was explicitly trained so that prefixes are usable representations. Binarisation degrades abruptly once the sign pattern stops separating your corpus, and that threshold is a property of the data, not the model.

Scalar quantisation maps a value range onto 256 buckets, and that range is estimated from a sample of your embeddings. If the corpus distribution shifts, the calibration no longer matches, and quality degrades with no error and no obvious signal.

Binary quantisation has no such parameter: the threshold is zero for normalised vectors, so there is nothing to go stale. That statelessness, rather than the compression ratio, is often the stronger operational argument.

Below a few million vectors, where float32 fits in memory comfortably and the added shortlist-plus-rescore machinery buys complexity you pay for during debugging.

Also when your bottleneck is elsewhere: if recall is limited by chunking or by the embedder itself, a 32x smaller index does not help. And when quality retention on your own corpus has not been measured, since the 96% figure is a benchmark result on specific data, not a guarantee.