Enterprise Generative Search — Azure and Open Source  ·  View 16 of 41  ·  Data

Knowledge Ingestion Pipeline

From a change in a source system to a chunk that can be retrieved and cited.

Editable source SVG draw.io All views
Detect Connector delta token per source Change Event Event Hubs Work Queue Service Bus, per class Deletion Detected tombstone, not a gap Acquire Fetch Content throttle-aware Content Hash skip if unchanged Dedup and Canonicalise SimHash near-duplicates Raw Store ADLS Gen2, immutable Parse Text and Office Parse Apache Tika Layout OCR Document Intelligence Transcribe Whisper, timestamped Table and Figure Extract kept as units Enrich Entity Extraction to the graph Taxonomy Classify domain and doc type PII and Sensitivity Presidio plus labels Injection Screen at index, not only at query Segment Structure-Aware Chunking per type, view 18 Context Header title and heading path Overlap Policy 15% on prose only Represent Dense and Sparse Vectors BGE-M3, versioned Multimodal Vectors figures and frames Chunk Summary for long tables only Publish Idempotent Upsert Azure AI Search Graph Upsert entities and supersession Ledger Commit PostgreSQL, the receipt Cache Invalidate by document and ACL delete then invalidate no change Knowledge Ingestion — From a Change in a Source to a Retrievable Chunk Application we own Queue / topic Decision point Data store Security / platform failure / alternate batch Every stage writes a receipt to the ledger. A document that fails at any stage is visible to its steward rather than silently absent. v 1.0 · owner Data and AI Global Practice

Decisions

  • Every stage writes a receipt to the ledger, including the stage that decides nothing has changed. An absent document and an unchanged one must be distinguishable, or view 08 has no answer.
  • A deletion is a tombstone plus a cache invalidation, never a silently missing row. Deleted content that keeps answering questions is the failure mode this exists to prevent.
  • Injection screening runs at index time as well as at query time, so a poisoned document is known about before it is ever retrieved.

Numbers

  • About 240,000 changed items a day; P95 from source change to retrievable is 5 minutes for collaboration content and 24 hours for the archive.
  • OCR is the expensive path: 26 M archive pages at Document Intelligence rates dominated the one-off build cost and is why re-OCR is never triggered by a re-embedding.
  • Near-duplicate collapse removes about 11% of chunks, which is a direct saving on index size and on retrieved redundancy.

Risks

  • Parse failures are the largest hidden quality problem in any corpus of this age. They are surfaced per document to stewards rather than counted in an operations dashboard nobody reads.
  • Canonicalisation can merge two documents that are genuinely different versions. The rule is conservative: near-duplicates are collapsed for ranking, never for storage.
◀  Model Portfolio All views Storage Zones  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 41 views · Microsoft Azure and open source16-knowledge-ingestion-pipeline