A team wants a dedicated vector database for a RAG system over 200,000 documents. Is it necessary?

What the interviewer is testing

Whether you evaluate a new datastore against what the estate already runs, rather than adopting the category-appropriate technology.

The scale question

200,000 documents chunked at, say, five chunks each is about a million vectors. That is modest. Vector extensions in relational databases, and vector support in existing search engines, handle this comfortably.

Dedicated vector databases earn their place at much larger scale — tens or hundreds of millions of vectors — or where specialised index tuning, very high query throughput or particular filtering behaviour is required.

The case for using what you already run

Operational simplicity: no new system to deploy, secure, back up, monitor, patch and learn.

Transactional consistency: the vectors sit alongside the source records, so a document update and its embedding update commit together. With a separate store they are a dual write, which drifts.

Filtering with real predicates: enterprise RAG almost always needs metadata filtering — by permissions, date, document type, department — and doing that in a database that already holds the metadata is straightforward. Combining vector search with filtering is where standalone vector stores have historically been weakest.

Hybrid retrieval: lexical search alongside vector search is consistently one of the largest quality gains available, and an existing search engine or database gives you both in one place.

When the answer changes

Very large corpora, sustained high query volume, a requirement for specific index types, or multi-tenant vector isolation at scale.

What a strong answer adds

Where the effort should actually go. Retrieval quality is dominated by chunking, hybrid search, reranking and query rewriting — not by which vector store holds the embeddings. Teams routinely optimise the store and leave fixed-size chunking that cut tables from their headers in place.

And the migration point: if the corpus later outgrows the existing store, moving vectors is straightforward because they are regenerable from source.

Common weak answers

Adopting a dedicated store because the workload involves vectors. Rejecting it without stating the scale at which the answer changes.