Embedding Pipeline Service  ·  View 11 of 22  ·  Data

Storage Zones

Four zones by ownership and rebuildability — and the reason a rebuildable store still gets snapshots.

Editable source SVG draw.io All views
Not ours — systems of record for content and permissions Corpus owners Document service content + ACLs File store bytes Connected SaaS tenant-authorised Ours and authoritative — must not lose a write Chunk ledger Chunk ledger RPO 0, RTO 15 min Control Contract registry RPO 0 Index catalogue aliases Evidence Change + deletion log append-only, 13 mo Audit log immutable Ours and recoverable — re-derivable at a cost Retained input Normalised text 30 d, RPO 1 h Quality record Evaluation + drift 13 months Ours and disposable — rebuildable projections, never backed up as truth Serving indexes Vector indexes one per contract Lexical indexes Index snapshots RTO 4 h, not truth Caches Vector cache cost, not correctness Query cache rebuilds re-chunk input replay before serving Storage Zones — Ownership and Rebuildability External / third party Data store batch failure / alternate Snapshots of a rebuildable store exist for recovery time, not durability: a from-scratch rebuild is 3 to 14 days and satisfies no RTO. v 1.0 · owner Data & AI Platform Architecture · date 2026-10

The zone that decides the recovery plan

  • Authoritative and ours: the chunk ledger, the contract registry, the index catalogue and the deletion log. RPO 0, RTO 15 minutes. These are backed up as truth.
  • Disposable and ours: every vector index, lexical index and cache. Rebuildable from the ledger plus retained text plus a pinned contract.
  • Not ours: content and permissions. The platform cannot restore a corpus and does not try.

Why snapshot a rebuildable store

  • A from-scratch rebuild takes 3 to 14 days, which satisfies no recovery-time objective anyone would sign.
  • So index snapshots exist for recovery time, not durability: RTO 4 hours from the latest snapshot versus days from the ledger.
  • A restored snapshot is not trusted until the deletion log has been replayed over it — otherwise a restore resurrects an erased document.

Retention (assumptions)

  • Normalised text 30 days. Chunk ledger for the life of the document. Deletion log 13 months.
  • A superseded index 21 days past cutover for rollback, then reclaimed within 7 days.
  • Evaluation runs and drift series 13 months, so two consecutive migrations can be compared.
◀  Data Flow — Edit to Retrievable All views Data Model  ▶
Solution Architecture v1.0 · open source, self-hosted on Kubernetes · Data & AI Platform Architecture · 2026-10 · 22 views · 17 architecture decision records11-storage-zones