Operating a Vector Store in Production
The operational facts that decide whether a vector search system works, why memory is the binding constraint, and the migration nobody plans for.
Vector search is chosen for its recall and latency and is operated as a memory problem with a maintenance schedule. The properties that determine whether a deployment succeeds are mostly not the ones the benchmarks measure.
Memory is the constraint
A graph index holds the vectors plus the graph. For 100 million vectors at 768 dimensions in fp32, the raw vectors alone are about 300 GB, and HNSW's neighbour lists add substantially on top, often another 30 to 50 percent depending on the connectivity parameter.
This is why quantisation is not an optimisation in this setting but a precondition. Scalar quantisation to int8 cuts vector memory fourfold with small recall loss. Product quantisation cuts it far more with a larger loss, and is usually paired with a reranking pass over full-precision vectors for the top candidates, which recovers most of the accuracy at a fraction of the memory. Binary quantisation with Hamming distance and reranking is the aggressive end, and works better than intuition suggests for high-dimensional embeddings.
Dimensionality is the other lever and is decided upstream: a 1,536-dimensional embedding costs twice a 768-dimensional one in memory and bandwidth forever, and Matryoshka-style embeddings that permit truncation make that a deployment-time choice rather than a model-selection one.
The migration nobody plans for
Embedding models improve, and adopting a better one means re-embedding the entire collection and rebuilding every index. For a large collection this is a multi-day operation requiring the old system to keep serving throughout, which means capacity for two full copies and a plan for how documents added during the migration are handled.
Teams that treat the embedding model as fixed discover this when a materially better model appears. Designing for it, versioned indexes, a documented dual-write path, and capacity headroom, converts a crisis into a scheduled project.
What to monitor
Recall against ground truth, measured periodically on a fixed sample with exact search as the reference. This is the only signal that catches gradual index degradation, and it is the one most often absent because it requires maintaining a reference.
Latency at the tail, since p50 hides the fan-out and compaction effects that users experience.
Index size and segment count, which predict when compaction is falling behind before latency shows it.
Filter selectivity distribution, since recall under filtering is the failure mode that unfiltered benchmarks cannot see.
When it breaks
Benchmarks use static clean collections. Published latency and recall numbers come from a built index over unchanging data with no filters, no deletions, no concurrent writes and no multi-tenancy. Production has all five, and each degrades the numbers.
Parameters are set once and never revisited. HNSW's construction and search parameters trade build time, memory and recall, and the right point moves as the collection grows and the query mix changes. They are almost always left at their initial values.
Cost is dominated by memory, not compute. Sizing by query throughput underestimates the requirement, because the collection must be resident regardless of how many queries arrive. A vector store's bill is a function of collection size far more than of traffic.
A vector store may not be needed. Below a few hundred thousand vectors, exact search over a matrix is fast enough, simpler, has perfect recall, and supports arbitrary filtering trivially. Adopting a specialised system before that threshold buys operational complexity for a problem that does not exist yet.
12 flashcards for this concept
Click a card to reveal the answer.