Enterprise Generative Search — Azure and Open Source  ·  View 17 of 41  ·  Data

Storage Zones

Four zones, and the only one of them that has a recovery point objective.

Editable source SVG draw.io All views
Zone 1 — Raw and immutable · source bytes as received Content Raw content ADLS Gen2, 12 TB Rendered pages page images for citation Why it exists Reparse without refetching source throttles are real Citation anchors to a page not to a moving document Zone 2 — Derived and rebuildable · deletable at any time, cost is only time Artefacts Parsed text Delta on ADLS Chunk table 320 M rows Vectors and index Azure AI Search Graph projection Apache AGE Rebuild economics Full rebuild 14 hours 1,900 USD of GPU Single source rebuild under 2 hours Zone 3 — Authoritative to the platform · nothing else can recreate it Records Document and version ledger PostgreSQL Answer provenance claims and evidence ids Feedback and labels the golden set grows here Audit record who asked, what was shown Consequence Backed up, geo-redundant RPO 5 minutes The only zone with an RPO everything else rebuilds Zone 4 — Ephemeral · correctness must not depend on it Caches and state Answer and embedding cache Redis, TTL by class Conversation state 30 days, then gone Traces Langfuse, 90 days Rule Flushing it degrades latency only never an answer Source systems keep owning the truth the platform holds no original Storage Zones — What Can Be Rebuilt, and What Cannot Deleting zones 1, 2 and 4 costs money and hours. Deleting zone 3 loses evidence that no source system can return, which is why only it is replicated. v 1.0 · owner Data and AI Global Practice

The decision this view makes

  • Zone 3 — the ledger, provenance, feedback and audit — is the only authoritative data the platform holds. Everything else is either a copy of a source or a derivative that can be rebuilt.
  • That is why only zone 3 is geo-replicated with an RPO. Backing up an index that can be rebuilt in 14 hours costs money and buys nothing.
  • Zone 4 is defined by a rule rather than a technology: flushing it may cost latency, never correctness. Any cache whose loss changes an answer is misplaced.

Numbers

  • 12 TB raw content, about 2.1 TB of parsed text, 320 M chunks, index footprint about 1.4 TB.
  • Full rebuild 14 hours and about 1,900 USD; single-source rebuild under 2 hours.
  • RPO 5 minutes and RTO 30 minutes, both applying to zone 3 only.

Assumptions

  • Raw content is retained for 90 days under an immutability policy so a reparse never needs to refetch from a throttled source.
  • Rendered page images exist for citation anchoring; they are storage-cheap and remove a whole class of citation drift when a source document is edited.
◀  Knowledge Ingestion Pipeline All views Chunking and Representation  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 41 views · Microsoft Azure and open source17-storage-zones