A live assistant serves 1.8M documents chunked at a fixed 512 tokens with no overlap. You need to move to structure-aware chunks of about 900 tokens with 15% overlap and a newer embedding model, with no downtime and no quality regression. What is the sequence?

Why this is not an in-place edit

The chunker and the embedding model are one versioned unit. New chunks embedded with the new model are not comparable to old chunks embedded with the old one, and a single index holding both returns nonsense rankings for any query that straddles them. So the migration is a second index, not an update.

The sequence

1. Freeze a labelled evaluation set first. Two to three hundred real queries with the passage that should be retrieved, drawn from production logs. Without it you cannot tell a regression from a rebuild, and everything below is unfalsifiable.
2. Version the tuple. Write chunker_version + embedding_model + index_name into the retrieval config and log it on every query. This is what makes the rollback a config change, and without it a partial rollout fails in a way nobody can attribute, because the query log does not say which index answered.
3. Build the new index alongside the old, in a separate collection or namespace. Both are live; only the old one serves.
4. Backfill with a bounded worker pool. At 1.8M documents and, say, 2.5 chunks each after re-chunking, that is roughly 4.5M embeddings. At 1,000 embeddings per second across the pool the backfill runs about 75 minutes of pure compute; in practice budget a day for retries and provider rate limits.
5. Dual-write from the point the backfill starts. Every document create or update writes to both indexes. This is the step teams skip, and it is the one that makes divergence permanent.
6. Shadow-evaluate. Run the frozen query set against both indexes and compare recall at k, not answer quality. Answer quality mixes retrieval and generation and will not tell you which one moved.
7. Route a small share of live traffic to the new index, watch the same metrics plus latency, then ramp.
8. Cut over by flipping the config value, keeping the old index warm for a rollback window.
9. Delete the old index only after the window closes. That is the point of no return.

Where data diverges and how you would know

Documents changed during the backfill are the hazard: the crawler may have read the old version while the dual-write path wrote the new one, leaving the new index with a stale chunk. Reconcile by comparing a document checksum and a chunk count per document id between the two indexes after the backfill finishes, and re-embed the mismatches. A count mismatch is the cheap signal; the checksum catches same-count edits.

What is likely to regress and what to do about it

Larger chunks with overlap usually raise recall and lower precision: the right passage is retrieved more often and carries more irrelevant text with it. Expect answer quality to dip even when retrieval improves, because the generator now sees more distractor text per chunk. Compensate by reducing the number of chunks sent to the model, from say 8 to 5, since each one is now nearly twice as long.

How long it really takes

The embedding compute is hours. The evaluation set, the dual-write path and the reconciliation are the schedule, and two weeks is a realistic figure for a corpus this size with a team that does not already have a labelled query set. The costs people forget: double index storage for the whole rollback window, and double write throughput on the ingestion path, which has been enough to saturate a modestly sized cluster in production more than once.

When not to bother

If the current retrieval recall at 10 is already above roughly 0.9 on the evaluation set, re-chunking will not be where the remaining quality lives. Measure before you migrate; the common outcome is that the generation prompt, not the chunker, was the constraint.