Embedding Pipeline Service  ·  View 06 of 22  ·  People and journeys

Journey — Upgrade the Embedding Model

The journey that justifies the whole architecture: adopting a better model across 4.8 billion chunks without a regression and without a project.

Editable source SVG draw.io All views
Platform engineer on call for the fleet Goal — Adopt a better embedding model across 4.8 billion chunks without a quality regression or a quarter-long project Trigger — A new open-weights model benchmarks 6 points better on the corpus's own query set Done when — Every tenant served by the new contract, the old index retired, and a rollback that was never needed but stayed possible 1 · Evaluate on a sample 2 · Build beside dual-write 3 · Gate ◆ moment of truth 4 · Cut over ◆ moment of truth 5 · Retire What they do Pins a model digest Prices the migration Starts the shadow build Reads the recall diff Flips tenants in waves Reclaims the old index What the platform does New contract registered GPU hours estimated Both indexes live-fed Oldest unmigrated chunk Recall@10 vs outgoing Alias write per tenant 21-day rollback window How it feels In control Exposed Cornered Where it hurts Storage doubled for 3 weeks Scores not comparable across models A half-flipped estate during the wave What answers it Contract is identity, so the build is separate Dual-index overhead is a budgeted capacity state Gate on the frozen set, not on raw scores No query ever spans two contracts Rollback is an alias write, not a rebuild Journey — Upgrade the Embedding Model The trough is cut-over, and it is where the critical design decision earns itself: a per-tenant alias flip is safe only because a query never spans contracts. v 1.0 · owner Data & AI Platform Architecture · date 2026-10

The trough, and what answers it

  • Cut-over is the dip, and it is where the critical design decision earns itself: a per-tenant alias flip is only safe because a query never spans two contracts.
  • Similarity scores from two models are not comparable, so query-time fan-out across contracts is not an option — a fact worth establishing before someone attempts it.
  • The storage doubling during the overlap is not an incident. It is a budgeted capacity state with a declared 21-day window.

The numbers that bound this journey

  • Background re-embedding 4,000 chunks/s gives a 14-day rebuild; surge 18,500 chunks/s gives 72 hours at about 4.5× the hourly cost.
  • A full-corpus re-embed is budgeted at ≤ $2,200 in GPU time and ≤ 60% additional storage (stated assumptions).
  • Cutover gate: recall@10 against the frozen set must not fall more than 2 percentage points against the outgoing contract.

Risks

  • A frozen query set that does not represent the corpus will pass a contract that is worse in production. The gate is only as good as the set.
  • Progressive cut-over means a half-flipped estate for days. Cross-tenant comparisons of retrieval quality during that window are meaningless and must not be made.
◀  Journey — Ship a New Corpus All views Layered Architecture  ▶
Solution Architecture v1.0 · open source, self-hosted on Kubernetes · Data & AI Platform Architecture · 2026-10 · 22 views · 17 architecture decision records06-journey-upgrade-the-model