Vector Index Quantisation
also called Product Quantisation, Scalar Quantisation, Vector Compression
Storing embeddings in fewer bits so a large index fits in memory, trading a measurable amount of recall for a multiple-times reduction in RAM and a rescoring stage to win most of it back.
A corpus of 200 million chunks embedded at 768 dimensions in 32-bit floats is about 614 GB of raw vectors, before an approximate index adds its graph links. That does not fit in one machine's memory, and spilling the graph to disk turns every search into random I/O. The choice is to shard across many memory-heavy machines, or to store each vector in fewer bits.
Quantisation takes the second route. Scalar quantisation maps each dimension from a float to an 8-bit integer, cutting memory four times. Binary quantisation keeps one bit per dimension, cutting it 32 times. Product quantisation splits the vector into sub-vectors and replaces each with the index of its nearest codebook entry — 768 dimensions in 96 one-byte sub-vectors is 96 bytes against 3,072.
None of this is free, and the cost is paid in a currency that raises no alarm: recall falls, and a search returning slightly worse neighbours looks exactly like a search that works.
Why it matters
Memory is the dominant cost line of a large vector service, so the compression ratio goes straight to the hardware bill: 32x is the difference between a rack and a machine.
Latency follows. Comparison over compressed codes is faster as well as smaller, because more vectors fit in cache and the arithmetic is cheaper — binary codes compare with an XOR and a population count.
Implementation patterns
- Quantise then rescore. Search the compressed index for 10 to 20 times the final
k, then recompute exact distances for those candidates from full-precision vectors held on SSD and re-rank. This recovers most of the lost recall for one extra read phase, and it is what makes aggressive compression safe. - Train the codebook on a representative sample. Product quantisation is data-dependent, so retraining belongs on the same schedule as re-embedding.
- Keep full-precision vectors somewhere durable. Codes cannot be inverted, so the uncompressed vectors or the source text must survive for the next rebuild.
- Measure recall against exact search on a fixed query set, not against the previous index. Brute force over a 10,000-document sample is the only honest reference.
Industry example
The open-source libraries document this directly: FAISS ships inverted-file indexes combined with product quantisation so billion-scale collections fit in memory, and its published benchmarks are the reference for what each setting costs in recall. Graph indexes follow the HNSW paper, where the link budget M adds a per-node overhead on top of the vectors. The shape is consistent: a cheap compressed scan, then an exact rescoring pass over a small candidate set.
Failure scenarios
- Silent recall loss. Compression ships, p99 improves, memory halves, and nobody notices that the correct passage now ranks 14th instead of 3rd. Nothing errors.
- Score thresholds break. Distances computed over codes are not on the same scale as the float distances a cutoff was tuned against, so an abstention threshold either rejects everything or nothing.
- Codebook drift. A corpus growing into new subject areas is no longer described by the trained codebook, and recall decays over months.
- Rescoring becomes the bottleneck. Fetching 2,000 full vectors per query from disk can cost more than the memory saved.
Trade-offs
| Choose | Gains | Pays |
|---|---|---|
| Scalar 8-bit | 4x memory, usually one or two points of recall | A quantisation step and recalibrated thresholds |
| Product quantisation | Up to 32x memory, fast scans | A trained codebook to maintain and real recall loss |
| Binary plus rescoring | Largest reduction, cheapest comparisons | A second read phase and full vectors kept nearby |
When not to use it
Below a few million vectors, do not. Ten million at 768 dimensions is about 30 GB, which fits on an ordinary machine, so quantisation buys nothing and adds a recall risk plus a tuning surface nobody will revisit.
Avoid it also where recall is contractual rather than best-effort — legal discovery, safety screening, patent search. There, prefer an exact search over a narrowed subset (filter by metadata, then brute force) to an approximate search over everything. The decision flips on one question: can you state the recall you are shipping, measured against exact search? If not, you are not trading recall, you are losing it.
Interview question
Q: You have 200M vectors at 768 dimensions and a 64 GB memory budget per node. Walk me through getting the index to fit, and tell me how you would know what it cost you.
What a strong answer covers: the arithmetic first (614 GB, so roughly 10x compression or 10 nodes); that sharding and compressing are usually combined; quantisation plus a rescoring stage with a stated candidate multiplier; recall measured against exact brute force before and after, at the k the product uses; the score thresholds that must be refitted; and where the full-precision vectors live for the next rebuild.
Quick check
Quiz: Binary quantisation drops recall at 10 from 0.94 to 0.71. Cheapest fix? — Retrieve several hundred candidates instead of tens and rescore them with full-precision vectors; the compressed index only has to get the right passage into the candidate set.
Flashcard: Memory for 200M 768-dimension float32 vectors, and what 8-bit quantisation changes? — About 614 GB raw; 8-bit coding cuts it near 154 GB for one or two points of recall, recoverable by exact rescoring.