A retrieval-augmented system serves millions of documents with sub-100 ms latency while embeddings are regenerated nightly. How should index rebuilds, hybrid retrieval, filtering and embedding versions be handled?

The versioning problem is the central one

Embeddings from different models are not comparable. A query embedded with model version 2 and searched against an index built with version 1 returns results that are not merely worse — they are semantically meaningless, because the vector spaces are unrelated.

So a model change requires re-embedding the entire corpus, and during that process the index is inconsistent. The resolution:

• Version the index explicitly, with the embedding model version as part of its identity.
• Build the new version alongside the old, in full.
• Validate it — recall against a labelled query set, latency, index size, and a comparison of results for a fixed set of probe queries.
• Swap the serving pointer atomically, so no query ever mixes vector spaces.
• Retain the previous version, making rollback a pointer flip rather than a multi-hour rebuild.

The query embedding and the index must always come from the same model version, and this should be enforced in code rather than by convention — it is the failure that produces silently poor results with no error.

Incremental updates alongside the nightly rebuild

Documents change continuously. The nightly rebuild plus streaming updates has the same structure as any derived-data pipeline:

• Track the point in the change stream that the rebuild's snapshot corresponds to, so incremental updates resume against the new index from the correct position. Without this, every swap silently discards a day of updates.
• Apply incremental updates to the in-progress index as well as the live one, so the new version is not born stale.
• Support deletion, which many vector indexes handle poorly — usually via a tombstone set consulted at query time, with periodic compaction.

Hybrid retrieval, because vectors alone are insufficient

Dense retrieval is weak on exact matches — product codes, error identifiers, names, rare terms — which keyword search handles trivially. Production systems run both and combine.

• Both retrievers run in parallel, and their results are fused, typically by reciprocal rank fusion, which needs no score calibration between the two systems.
• A reranker over the fused top-k — a cross-encoder scoring query and document together — which is far more accurate than either retriever and too expensive to run over the whole corpus. Retrieve broadly, rerank narrowly.
• This layering is where most retrieval quality actually comes from, and teams that tune the vector index while omitting the reranker are optimising the smaller term.

Filtering, which is harder than it appears

Permission filters, tenant filters and date ranges must be applied, and there are two wrong ways:

• Post-filtering — retrieve k, then filter — can return nothing when the user has access to a small subset, because all k results are filtered away.
• Pre-filtering by scanning the permitted subset defeats the index for large subsets.

The workable approaches are filtered search within the index structure, where the index supports it, and partitioning the index by the dominant filter dimension — usually tenant — so the filter becomes index selection rather than a predicate. Partitioning by tenant is the single most useful structural decision in a multi-tenant retrieval system, and it also bounds the blast radius of a permission bug.

Latency

Sub-100 ms end-to-end includes query embedding, retrieval, fusion, reranking and filtering — and the query embedding is a model call that is frequently forgotten in the budget. Approximate nearest neighbour parameters trade recall against latency and should be tuned against a labelled set rather than by intuition; caching embeddings for repeated queries is a large and easy saving in products where queries repeat.