Continuous Delivery for Retrieval Indexes

A retrieval-augmented application gets worse overnight. No code shipped, no prompt changed, no model version moved. The nightly ingest re-embedded the 8 percent of the corpus that had changed, using the embedding model the serving config now pointed at, and wrote those vectors into an index whose other 92 percent were still in the previous model's space. Cosine similarity between vectors from two different embedding spaces is a well-formed float with no meaning, so retrieval degraded smoothly and silently, which is the worst way for it to fail.

The index is an artefact, and building it is a pipeline stage

Rebuilds are not a side effect of writing rows. Building an HNSW index over 1 million 1536-dimensional OpenAI embeddings in pgvector, with m = 16 and ef_construction = 200, took about 1 hour 27 minutes single-threaded; the parallel builds introduced in pgvector 0.6.0 brought the same build down to roughly 9.5 minutes on a 16-core, 64 GB instance, with equal measured accuracy and queries per second (Supabase, 2024, pgvector 0.6.0: 30x faster with parallel index builds).

The footprint is as concrete. A million float32 vectors at 1536 dimensions is \(10^6 \times 1536 \times 4 \approx 6.1\) GB before a single graph edge, and HNSW stores neighbour lists on top of that, on the order of \(2m\) identifiers per node at the base layer (see ANN indexes). Scale the corpus by ten and both numbers scale at least linearly, which is why index builds end up on dedicated hardware with their own schedule.

The version that has to travel with it

The deployable unit is not the index binary. It is the tuple of corpus snapshot, chunker version, embedding model identifier, index parameters and the built index, and changing any element invalidates the rest. Most importantly, the encoder used at query time must be the one the index was built with, bit for bit, which means the index metadata has to record it and the serving configuration has to be checked against that record at startup rather than trusted.

Incremental, tombstoned, or rebuilt

HNSW has no true delete. Engines mark a vector deleted and reclaim the space later: Weaviate's maintainers rejected flag-only deletion as unworkable given unpredictable delete order and volume, implemented asynchronous cleanup instead, and found a run of concurrency bugs around graph entry points in the process (hnswlib discussion, deleting vectors from the index). Operators see the consequence as storage that does not shrink until compaction has run several times. Milvus takes a comparable route, buffering deletes into a delta segment that later compaction merges down.

The consequence for delivery is the one teams miss. An index that has absorbed six months of inserts, tombstones and compactions is not the index a clean rebuild from the same corpus produces. The graph topology differs, so recall differs, and nothing in a code review will show it. Periodic full rebuilds are a correctness practice, not housekeeping.

Swap, do not mutate

Build into a new index under a new name, run the gate against it while the old one serves, flip the alias, and keep the previous index for the rollback window. This is build-once-promote-the-artefact applied to a thing most teams treat as a database (see build once, promote the artefact). Mutating the live index in place means the rollback path is a rebuild, which at the build times above is measured in hours.

The gate itself is a fixed query set with known relevant documents, scored against the previous index on the same queries: recall@k and nDCG@10 as a paired per-query difference, not two independently computed averages. Alongside it, two cheap assertions catch most real incidents: document count and a checksum over the set of ingested identifiers, and a startup check that the query encoder matches the index metadata.

When it breaks

A half-built index passes a health check. Liveness asks whether the service answers. An empty or partially populated index answers quickly and wrongly, and cheerfully returns its three nearest neighbours out of the 4 percent of the corpus that loaded. Assert cardinality against the expected count before the alias flips.

Rebuild and ingest overlap. A long build reading a corpus that is being written produces an index that matches no snapshot. Build from a pinned snapshot, not from the live table.

Chunker changes are invisible to every version check. Same embedding model, same index parameters, different chunk boundaries, materially different retrieval. The chunker's version belongs in the tuple above and in the index metadata.

The evaluation set leaks into the corpus. Queries whose gold documents were built from the same pipeline that now indexes them produce a gate that cannot fail. Keep the gold set's provenance separate and re-derive it when the corpus definition changes.

Re-embedding cost pushes teams into the opening failure. Using an illustrative $0.02 per million tokens, re-embedding 10 million chunks of 500 tokens is 5 billion tokens for roughly $100, which is affordable; the wall-clock of doing it through a rate-limited endpoint is usually what actually blocks. That pressure is why incremental re-embedding is the default, and why a hard refusal to write vectors from a different model version into an existing index is the one guard rail worth building first.