Index Memory Budget

Sizing a vector index is arithmetic, and getting it wrong by an order of magnitude is routine because people reach for the wrong term.

For 200 million embeddings of 1024 dimensions in 32-bit floats:

vectors:  200,000,000 × 1024 × 4 bytes      ≈ 819 GB
graph:    2 × M × 4 bytes per node, M = 32
          = 256 bytes × 200,000,000          ≈  51 GB
total                                        ≈ 870 GB

The vectors are 94% of it. Which is why lowering the connectivity parameter M from 32 to 16 - the first instinct of most teams - saves about 26 GB, or 3% of the total, and costs recall. The levers with a factor of four behind them are element width and dimension.

Why it matters

Vector search is one of the few workloads where the capacity question is settled before any benchmarking: either the index fits in memory on an affordable number of machines or it does not, and that single fact determines the architecture. A team that estimates 80 GB when the answer is 870 GB designs for a tenth of the problem and discovers it during the first full load.

It also redirects tuning effort. Hours spent on efConstruction and M move recall and latency by percentages; moving from fp32 to int8 moves memory by a factor of four, which is what decides the machine count and the bill.

And the arithmetic inverts at the bottom. Under binary quantisation the vectors fall to about 26 GB and the 51 GB graph becomes the larger term - so the right lever genuinely does change, and knowing where the crossover sits is the point of doing the sum.

Implementation patterns

• Compute vectors and graph separately and look at the ratio before tuning anything.
• Reach for int8 scalar quantisation with full-precision rescoring of the top 100 to 200 candidates. Four times less memory, and recall@10 typically within about a percentage point of fp32. The decision rule: if measured recall at int8 is inside your tolerance, take the saving - this is the highest-value change available and the most often skipped.
• Measure recall yourself against an exact brute-force baseline on a sample. Vendor recall figures are from other people's data.
• Question the dimension. Many embedding models support truncation (Matryoshka-style training), so 1024 dimensions can often become 512 with small quality loss - another factor of two, and it compounds with quantisation.
• Shard for rebuild time and availability, not only memory. An 870 GB index on one machine takes hours to rebuild and has no failover.
• Consider disk-resident approximate search when the corpus grows faster than the budget: a few milliseconds of latency for an order of magnitude less memory.
• Ask whether one index is the requirement. A per-tenant index of 50,000 vectors is a different and far cheaper problem, and the single global index is frequently a default rather than a decision.

Industry example

Published material from vector database vendors and from the FAISS and HNSW implementations converges on the same ordering - quantisation first, then dimension, then graph parameters - with sizing formulas that make the vector term explicit. The DiskANN line of work from 2019 onward exists precisely because the memory term binds at billion-scale corpora, trading a small latency increase for keeping most of the index on solid-state storage.

Failure scenarios

• Order-of-magnitude sizing error, usually from using the wrong element width or forgetting a factor of ten in the count, producing a capacity plan that fails at load.
• Tuning M to save memory, trading recall for 3%.
• Quantising without rescoring, which loses several points of recall and gets quantisation blamed for a mistake in how it was applied.
• Quantisation accepted on an aggregate metric while a subpopulation - short documents, a minority language - degrades badly, because aggregate recall was the only thing measured.
• Forgetting the payload. Stored metadata and original text alongside the index are often larger than the vectors; the index budget is not the storage budget.
• No headroom for rebuild. Building a new index while serving the old one needs roughly twice the memory, which is discovered during the first re-embedding.

Trade-offs

When not to use it

Below roughly a million vectors, all of this is noise: a million 768-dimension fp32 vectors is about 3 GB, fits anywhere, and the interesting questions are filtering semantics and freshness instead.

It is also the wrong frame for a managed service priced by stored vectors or by capacity units rather than by memory - compute the cost in the vendor's units, while keeping the same ordering of levers, since dimension and precision usually drive their pricing too.

And where the corpus changes continuously, the binding constraint is often rebuild and insert throughput rather than steady-state memory; size for the rebuild, which needs roughly double.

Interview question

Q: You need similarity search over 200 million 1024-dimension embeddings with a 50 ms p95 budget. Size it, and tell me the first change you would make.

What a strong answer covers: the arithmetic, vectors and graph computed separately, giving roughly 819 GB plus 51 GB · the observation that vectors are 94% of it, so element width and dimension are the levers and connectivity is not · int8 with full-precision rescoring of the top candidates as the first change, with recall measured against an exact baseline rather than assumed · dimension truncation as a compounding second lever · sharding justified by rebuild time and availability as well as memory · roughly double memory needed during a rebuild · the payload and metadata sized separately from the index · and the question of whether one global index is required at all, since per-tenant indexes change the problem entirely.

Quick check

Quiz: Why is lowering the HNSW connectivity parameter the wrong way to cut index memory? Because the graph is only a few hundred bytes per node - about 6% of the total at typical dimensions - so halving it saves a few percent and costs recall, while quantisation saves a factor of four.

Flashcard: What is the memory for 200 million 1024-dimension fp32 vectors with an HNSW graph at M = 32, and where does the arithmetic invert? - About 819 GB of vectors plus 51 GB of graph, so about 870 GB; under binary quantisation the vectors drop to roughly 26 GB and the graph becomes the larger term.