Enterprise Generative Search — Azure and Open Source

Architecture Views

41 views, in reading order. Every view ships three ways: an HTML page, an SVG that re-opens in diagrams.net fully editable, and draw.io source.

A generative search platform is not an LLM in front of a search box. It is a retrieval system with a language model attached to the end of it, and almost every failure it has is a retrieval, authorisation or evidence failure wearing a model's clothes. Read the set in order: the boundary first, then the contract that decides everything after it, then who the platform is for and what they get to do, and only then the structure that has to justify itself against them. One decision carries the whole design — the evidence contract in view 03: nothing enters the context window that the caller could not open themselves, every sentence carries an evidence id the retriever minted, and retrieved content is data rather than instruction. That contract is why the ACL fingerprint is part of the cache key, why the model holds no index credential, why citations are bound rather than requested, and why the downstream CRM assistant needed no security design of its own.

Context and scope

What sits inside the boundary, the contract everything else follows from, and how a question is routed before a model is ever called.
01 Unstructured content sources Confluence and Jira engineering knowledge Microsoft 365 SharePoint, Teams, OneDrive ServiceNow Knowledge support articles Document Archive 12 TB scanned PDF People it serves Knowledge Worker 9,000 daily active Research Analyst 400 seats Field Engineer 1,200 mobile Knowledge Steward 85 content owners Structured, external and downstream Databricks Lakehouse governed Delta tables Downstream Applications CRM and intranet assistants Regulator and Standards Feeds public web Generative Search Platform grounded answers over 40 M documents Platform services it depends on Microsoft Entra ID identity and groups Azure AI Foundry Azure OpenAI models Azure AI Content Safety Prompt Shields Azure Monitor and Langfuse traces and telemetry asks multi-hop photo, voice curates REST delta KB API nightly SQL API crawl OIDC, OBO inference screening OTLP Enterprise Generative Search — System Context External / third party Person or role Security / platform synchronous batch event / async Read-only against every source. The platform indexes and cites content; it never edits or republishes it. v 1.0 · owner Data and AI Global Practice · date 2026-09 System Context Who asks, what the platform reads, and what it deliberately never writes to. HTML page SVG draw.io
02 Ask Web and Mobile UI streamed answers Answer API OpenAPI, SSE Search Gateway FastAPI on AKS Understand Query Understanding intent, entities, filters Entitlement Resolver Entra group set Conversation State PostgreSQL Plan Query Planner LangGraph Retrieval Policy per source and class Budget Governor tokens, hops, time Retrieve Hybrid Index Azure AI Search Structured Retrieval governed SQL views Knowledge Graph Apache AGE Answer Cache Redis, ACL-keyed Rank Fusion reciprocal rank Cross-Encoder Rerank bge-reranker-v2-m3 Evidence Selection diversity and authority Generate Context Builder fenced, budgeted Model Gateway LiteLLM proxy Synthesiser structured answer Assure Grounding Verifier claim-level NLI Injection and Safety Filter Prompt Shields Answer with Citations or search fallback cache probe replan High-Level Architecture — Ask to Assured Answer Application we own Interface / broker Security / platform Data store Decision point synchronous failure / alternate Ingestion is view 16. Identity, observability, evaluation and cost control are acts 6 and 7. v 1.0 · owner Data and AI Global Practice High-Level Architecture The shape of a request in one picture: understand, plan, retrieve, rank, generate, assure. HTML page SVG draw.io
03 Rule 1 — Nothing enters the context that the caller could not open themselves Enforced here Entitlement Resolver group set per request Index-side security filter never post-filtered ACL fingerprint in cache key SHA-256 of group set Therefore forbidden Model holding an index credential it requests, never queries One answer cache for all callers cross-principal reuse Rule 2 — Every sentence carries an evidence id the retriever minted, or it is not shown Enforced here Evidence ids minted at retrieval chunk plus version Claim to evidence binding verified, not requested Provenance persisted with every answer Therefore forbidden Model-written citations a string it can invent Unbound sentence rendered dropped or flagged Rule 3 — Retrieved content is data. Only the system prompt and the caller turn are instructions Enforced here Structural fencing evidence in a typed slot Injection screening at index and at query Tool broker with allowlist policy per query class Therefore forbidden Tool call sourced from a document never honoured Free-form model egress no open network Consequence — the CRM assistant needed no access model of its own Consequence — any answer replays and re-derives from its evidence ids The Evidence Contract — The Three Rules Everything Else Follows From The test of the contract: an answer that cannot be re-derived from its evidence ids is a defect, not a matter of style. v 1.0 · owner Data and AI Global Practice The Evidence Contract The three rules the rest of the architecture is a consequence of. HTML page SVG draw.io
04 Recognised by Retrieval plan Generation mode Budget and target Lookup Short, entity-bearing 38% of traffic Hybrid, depth 30 single pass Extractive answer one passage, quoted 4k tokens, 2.5 s P95 cacheable Explanatory How or why phrasing 31% of traffic Hybrid, depth 80 rewrite and expand Synthesis over 6-10 chunks claim-bound 14k tokens, 6 s P95 cacheable Multi-hop or comparative Several entities, contrast 11% of traffic Decomposed sub-queries graph-assisted Agentic loop, 3 hops max contradiction check 45k tokens, 18 s P95 never cached Structured or numeric Aggregation verbs, metrics 9% of traffic NL to SQL on allowlist semantic layer only Table plus narrative figures never restated 8k tokens, 9 s P95 result cached Multimodal Image or audio attached 6% of traffic Cross-modal plus text figure and transcript Answer with a visual crop page-anchored 20k tokens, 12 s P95 never cached Ambiguous or uncovered Low retrieval confidence 5% of traffic Broad recall probe then stop No generation at all clarify or list results Under 1 s, no model cost the cheapest answer Query Classes and the Routing Policy The class is decided before any model is called, so the expensive path is a routing outcome rather than the default. v 1.0 · owner Data and AI Global Practice Query Classes and Routing Six classes of question, and what each is allowed to spend before anything is generated. HTML page SVG draw.io

People and journeys

Who the platform is for, and the three journeys whose worst moments the rest of the set has to answer.
05 The people who ask Knowledge Worker 9,000 daily, 26 queries a week Goal — Give me an answer I can paste into a client mail without opening three documents to check it first. Core journeys Ask and act on one answer 180k queries a month Follow up in the same thread Open the source that convinced me Research Analyst 400 seats, regulated work Goal — Show me every place we said something about this, including the places that disagree, and let me cite it. Core journeys Answer a multi-hop question 9k a month Compare positions across sources Export an evidence pack Field Engineer 1,200 mobile, low bandwidth Goal — Let me photograph the panel in front of me and get the right procedure for that exact revision. Core journeys Search from a photo Ask by voice, hands busy Pin a procedure offline The people who keep it honest Knowledge Steward 85 content owners Goal — Tell me why my document is not being found, in terms I can act on without raising a ticket. Core journeys Diagnose a document that never surfaces the trust journey Retire and supersede content Read my domain coverage report Search Platform Engineer 6 engineers, on call Goal — When an answer is wrong, tell me in one trace whether it was retrieval, ranking, context or the model. Core journeys Triage a bad answer from its trace Ship a retrieval change behind a gate Rebuild an index without downtime Security and Compliance Officer 2 named owners Goal — Prove to an auditor that no answer ever contained a passage its reader could not have opened. Core journeys Replay an answer for an audit Attest the entitlement path Review injection and safety events Machines that act without being asked Source Connectors 6 sources, delta-driven Goal — Notice every change in my source within its freshness class and never lose one silently. Core journeys Pull a delta and enqueue it Reconcile a missed change Entitlement Sync 60-second target Goal — Make a permission change true in the index before the person it affects asks their next question. Core journeys Apply an ACL delta Invalidate affected cache keys Nightly Evaluation Run 1,400 golden queries Goal — Fail the build before a regression reaches a person who is going to trust the answer. Core journeys Score retrieval and generation Block a release on a gate Re-embed and Compact model-version driven Goal — Move the corpus to a new embedding model without a day where the index answers from two. Core journeys Shadow-build a new index Cut over by alias Systems that consume answers CRM Assistant on-behalf-of the seller Goal — Ask on my user's behalf and inherit exactly what they may see, with no access model of my own. Core journeys Ask with a delegated token Render citations inline Intranet Answer Widget read-only embed Goal — Answer the six questions people ask the intranet every day without a second content pipeline. Core journeys Serve a cached common answer Fall back to a result list Actors and Their Core Journeys Person or role Journey / task Security / platform External / third party Every actor here has a goal in their own words. Three of these journeys are mapped in views 06 to 08; the rest are served by views later in the set. v 1.0 · owner Data and AI Global Practice Actors and Their Core Journeys Who the platform is for, in their own words, and what each of them gets to do with it. HTML page SVG draw.io
06 Knowledge Worker mid-task, in a hurry Goal — Get one answer good enough to act on without checking it myself Trigger — A client question they cannot answer from memory Done when — They paste the answer and the link, and are still right a week later 1 · Notice in Teams or a doc 2 · Ask 3 · Wait ◆ moment of truth 4 · Read and judge ◆ moment of truth 5 · Act What they do Hits a gap mid-task Types one sentence Adds no filters Watches the stream Skims the answer Checks a citation Pastes it, or moves on Where Teams side panel Search box Streaming pane Answer with citations Their own document Behind the glass Nothing yet Query understanding Entitlement resolve Hybrid retrieve, rerank First token at 1.2 s Claim-level binding Source preview Feedback captured How it feels Confident Fine Doubtful Where it hurts Does not know what is indexed Blank pane reads as broken Confident answer, retired policy No way to tell how sure it is Re-checks it anyway What answers it Scope chips from the plan Retrieval shown while waiting Version and as-of on each cite Abstain when unsupported One-click open at the passage Journey — Ask a Question and Act on the Answer The trough is not latency. It is that a fluent answer and a wrong answer look identical, which is why views 19, 21 and 26 exist. v 1.0 · owner Data and AI Global Practice Journey — Ask and Act What happens in the first four seconds, and why the trough is not latency. HTML page SVG draw.io
07 Research Analyst regulated deliverable, named reviewer Goal — Assemble a defensible position from sources that do not agree with each other Trigger — A regulatory change, or a question from counsel Done when — A reviewer accepts the evidence pack without re-doing the search 1 · Frame 2 · Decompose 3 · Gather 4 · Reconcile ◆ moment of truth 5 · Cite ◆ moment of truth 6 · Deliver What they do States the real question Accepts or edits the plan Scans evidence, adds a source Weighs conflicting passages Checks every citation Exports the pack Where Research workspace Visible query plan Evidence tray Comparison table Citation panel Signed export Behind the glass Intent and scope model Planner, 3 sub-queries Graph expansion Per-hop retrieval Diversity selection Contradiction detector Authority and recency Claim to evidence binding Provenance record persisted How it feels In control Working Losing trust Where it hurts Plan is invisible, cannot steer Six near-identical passages Draft averaged two positions False consensus reads as fact Citation points at the document What answers it Editable plan before it runs Redundancy collapsed by MMR Disagreement is a stated state Both positions kept, dated Cite anchors to the passage Pack replays from evidence ids Journey — Answering a Multi-Hop Question That Has to Survive Review An averaged answer is worse than no answer here. Reconciliation is a first-class output, which is why contradiction is detected rather than smoothed. v 1.0 · owner Data and AI Global Practice Journey — Multi-Hop Under Review Where an averaged answer is worse than no answer at all. HTML page SVG draw.io
08 Knowledge Steward owns a domain, not the platform Goal — Make my domain answerable, and know when it is not Trigger — A colleague says the platform never finds our standard Done when — The document is retrievable, and the steward knows why it was not 1 · Publish 2 · Assume it works 3 · Hear a complaint ◆ moment of truth 4 · Diagnose ◆ moment of truth 5 · Fix 6 · Verify What they do Publishes to SharePoint Moves on Tries the query themselves Opens the document report Fixes title and metadata Supersedes the old version Re-runs the probe query Where SharePoint library Nowhere The search box Steward console Source system Coverage report Behind the glass Graph delta notices it Parse, chunk, embed, index Retrieval returns rivals Per-document diagnostics Retrieval explain probe Incremental re-index Coverage metric recomputed How it feels Trusted Neutral Powerless Where it hurts No signal it was indexed Blamed for a platform fault Scanned PDF, no text layer Nothing tells them that What answers it Indexed receipt per document Self-service probe query Parse, chunk, ACL and OCR state Named cause, not a ticket OCR path re-queued on demand Domain coverage trend Journey — Finding Out Why a Document Never Surfaces A steward with no diagnostics blames the platform, and the platform blames the content. The diagnostics in view 34 exist to end that argument. v 1.0 · owner Data and AI Global Practice Journey — Why Is My Document Not Found The journey that decides whether content owners trust the platform or blame it. HTML page SVG draw.io

Structure

The layering rule, the deployable units, how a sentence becomes a plan, and the four retrieval backends behind one contract.
09 Experience Web Search UI streamed, citation-first Teams App side panel Mobile Client photo and voice Answer API SSE, OpenAPI Classic Search API the fallback surface Answer orchestration Query Understanding intent, entities, scope Query Planner LangGraph state machine Agentic Controller 3 hops, hard budget Answer Assembler claims and citations Budget Governor tokens, time, hops Retrieval and ranking Retriever Facade one contract, four backends Lexical Retriever BM25 and filters Vector Retriever HNSW, BGE-M3 Structured Retriever governed SQL only Graph Retriever entity and supersession Fusion and Rerank RRF, cross-encoder Grounding and assurance Claim Binder sentence to evidence id Groundedness Verifier NLI over evidence Contradiction Detector pairwise on claims Safety and Injection Filter Prompt Shields Policy Engine OPA, source and tool rules Model and tools Model Gateway LiteLLM, routing and budgets LLM Pool Azure OpenAI and vLLM Embedding Service TEI on GPU Rerank Service TEI cross-encoder Tool Broker allowlisted, audited Knowledge platform Connectors delta and CDC Parsing and OCR Tika, Document Intelligence Enrichment entities, taxonomy, PII Chunking per content type Embedding Pipeline batch, versioned Indexer idempotent upsert Data and index Hybrid Index Azure AI Search Knowledge Graph PostgreSQL and AGE Metadata and Provenance PostgreSQL Content Lake ADLS Gen2 and Delta Cache Azure Cache for Redis Trace Store Langfuse on PostgreSQL Platform and control Entra ID and Key Vault identity and secrets AKS and Terraform CPU and GPU pools OpenTelemetry one trace per request Evaluation Harness Ragas and IR metrics Cost Control per-query accounting one entry point retrieval requests filtered query routed call upsert replan Layered Architecture — What Depends on What Application we own Interface / broker Decision point Security / platform Data store synchronous batch failure / alternate Dependencies point down. The single upward call is the verifier asking the planner to retrieve again; it is bounded by the budget governor and is drawn rather than hidden. v 1.0 · owner Data and AI Global Practice Layered Architecture Eight layers, dependencies pointing down, and the single upward call drawn rather than hidden. HTML page SVG draw.io
10 Azure landing zone — AKS, private networking only Answer plane (namespace: answer) Search Gateway FastAPI, SSE, 12 pods Query Understanding Python, 6 pods Planner and Controller LangGraph, 8 pods Answer Assembler Python, 8 pods Verifier NLI, 4 GPU pods Retrieval plane (namespace: retrieve) Retriever Facade Python, 10 pods Structured Retriever SQL guard, 4 pods Graph Retriever openCypher, 4 pods Embed Service TEI, 6 GPU pods Rerank Service TEI, 6 GPU pods Knowledge plane (namespace: ingest) Airflow KubernetesExecutor Connector Workers one per source Parse and OCR Workers Tika, autoscaled Chunk and Embed Workers batch GPU Indexer idempotent upsert Shared control services (namespace: platform) Model Gateway LiteLLM, 6 pods Tool Broker allowlist, 4 pods Policy Engine OPA sidecar Langfuse self-hosted traces flagd retrieval flags Managed data services — private endpoints Retrieval stores Azure AI Search S3, 12 partitions PostgreSQL Flexible metadata and AGE graph Azure Cache for Redis answer and embedding cache Content and events ADLS Gen2 raw and rendered, 12 TB Event Hubs change and telemetry Service Bus ingestion work queues Azure OpenAI synthesis models Document Intelligence layout OCR Content Safety Prompt Shields Databricks SQL governed tables Microsoft Entra ID OBO and groups plan retrieve hybrid query allowlisted SQL routed inference upsert work items Container Architecture — The Deployable Units Application we own Interface / broker Security / platform Data store Queue / topic External / third party synchronous batch event / async GPU pods carry the embedding, reranking and verification models. No pod holds a source credential; connectors use workload identity against Key Vault. v 1.0 · owner Data and AI Global Practice Container Architecture The deployable units, their technology, and which of them hold GPUs or credentials. HTML page SVG draw.io
11 Receive Raw ask text, voice or image Language detect fastText, 14 languages PII scrub on the query Presidio Interpret Class and intent 6 classes, view 04 Entities and filters product, date, org Temporal scope as-of resolution Contextualise Coreference resolve from turn history Conversation state PostgreSQL, 30 turns Scope chips shown to the user Authorise On-behalf-of token Entra OBO Group set expansion cached 60 s ACL fingerprint SHA-256, cache key Rewrite Query rewrite Qwen3-8B on vLLM Multi-query expansion up to 4 variants Domain synonyms curated, not learned Emit QueryPlan object typed, logged, replayable Cache probe exact then semantic Low confidence exit clarify or list confidence below 0.45 Query Understanding — From a Sentence to an Executable Plan Application we own Security / platform Decision point Data store Risk / gap failure / alternate Authorisation is resolved here, before any retrieval, because the group set is an input to both the query filter and the cache key. v 1.0 · owner Data and AI Global Practice Query Understanding How a sentence becomes a typed, authorised, replayable plan. HTML page SVG draw.io
12 The contract every backend honours Inputs it must accept Principal group set mandatory, never optional Classification ceiling per request Freshness and as-of declared, not implied Outputs it must return Evidence id chunk plus version Score and its scale comparable after fusion Retrieval reason why this was returned Candidate generation Lexical BM25 over Azure AI Search field-boosted Analyzer and synonym map curated per domain Exact identifier match part and policy numbers Semantic Dense HNSW BGE-M3, 1024-d Learned sparse terms same model, lexical bridge Multimodal vectors figures and frames Structured Semantic SQL views allowlisted only NL to SQL guard parse, bound, explain Row to evidence adapter figures become citable Graph Entity and relation store Apache AGE on PostgreSQL Supersession edges replaces and replaced-by Hop expansion 2 hops, capped fan-out Retrieval cache Redis, ACL-fingerprinted Per-source ranking policy authority and recency weights Retrieval trace every backend call recorded Retrieval Fabric — Four Backends, One Contract Adding a fifth backend means implementing the contract, not changing the planner. That is the whole reason the facade exists. v 1.0 · owner Data and AI Global Practice Retrieval Fabric Four backends behind one contract, and what that contract obliges each of them to accept and return. HTML page SVG draw.io
13 Consumers Web and Teams UI Mobile Client CRM Assistant Intranet Widget Published interfaces Answer API POST /v1/answer, SSE Search API GET /v1/search, ranked list Admin and Steward API diagnostics, coverage Ingestion API push for systems without a delta feed Dependencies Content Source Systems Graph, Confluence, ServiceNow Databricks SQL Azure AI Services OpenAI, Safety, OCR Microsoft Entra ID SSE REST OBO token embed delta SQL inference OIDC Integration Surface — Every Way This Platform Touches Another System Application we own External / third party Interface / broker Security / platform synchronous Four interfaces are published and versioned; everything else is an internal call. The six content sources are one box here because view 14 carries them one by one. v 1.0 · owner Data and AI Global Practice Integration Surface Four published interfaces, and every system the platform depends on. HTML page SVG draw.io
14 Change detection Authorisation model Freshness class Parse path Microsoft 365 Graph delta query 8.1 M items Item ACL to Entra groups expanded at index 5 minutes P95 collaboration class Native text plus Tika OCR only when scanned Confluence and Jira CQL updated-since 1.4 M pages Space and page restrictions mapped to groups 15 minutes collaboration class Storage-format HTML macro-aware ServiceNow Knowledge sys_updated_on poll 220 k articles User criteria to groups role-derived 15 minutes support class HTML plus attachments article as one unit Document archive Blob change feed 26 M scanned pages Folder ACL, inherited coarse by design 24 hours archive class Document Intelligence layout and tables Databricks governed tables Not indexed at all queried live Unity Catalog grants caller passed through Query time no staleness No parse, rows are evidence adapter, not chunker Public web and regulators Sitemap and ETag crawl 40 domains Public, tenant-wide no ACL to carry 24 hours external class Readability plus Tika boilerplate stripped Source Connector Matrix — How Each Source Is Read, Authorised and Refreshed Databricks is deliberately not indexed. A number that can be computed correctly on demand should never be embedded and allowed to go stale. v 1.0 · owner Data and AI Global Practice Source Connector Matrix How each of the six sources is read, authorised, refreshed and parsed. HTML page SVG draw.io
15 Model chosen Runs on Cost and latency Fallback Intent and rewrite Qwen3-8B-Instruct open weights vLLM on AKS GPU 2 x A10, shared 0.0002 USD, 90 ms per query Rules and the raw query no rewrite is safe Embedding BGE-M3 dense and sparse TEI on AKS GPU batch and online 0.00004 USD, 12 ms per query Lexical only retrieval recall drops, still serves Reranking bge-reranker-v2-m3 cross-encoder TEI on AKS GPU top 50 in one batch 0.0006 USD, 110 ms per query Fusion order only quality gate records it Answer synthesis Azure OpenAI, large tier quality-routed Azure AI Foundry PTU plus pay-go spill 0.014 USD, 2.4 s P50 explanatory Smaller tier, then extractive never no answer Grounding verification NLI cross-encoder open weights, fine-tuned TEI on AKS GPU one call per claim 0.0009 USD, 180 ms per answer Content Safety groundedness managed, slower Multimodal Azure AI Vision embeddings plus Whisper for audio Managed plus AKS GPU ingest-time mostly 0.003 USD, 400 ms per query with media Caption text retrieval degraded but useful Model Portfolio — Which Model Does Which Job, and What It Costs Every task names a model and a fallback. The gateway can move a task between them by configuration; no service holds a model name in its code. v 1.0 · owner Data and AI Global Practice Model Portfolio Six model tasks, where each runs, what each costs, and what happens when it is unavailable. HTML page SVG draw.io

Data

How content becomes retrievable, what can be rebuilt and what cannot, the provenance model, and how permissions travel with the data.
16 Detect Connector delta token per source Change Event Event Hubs Work Queue Service Bus, per class Deletion Detected tombstone, not a gap Acquire Fetch Content throttle-aware Content Hash skip if unchanged Dedup and Canonicalise SimHash near-duplicates Raw Store ADLS Gen2, immutable Parse Text and Office Parse Apache Tika Layout OCR Document Intelligence Transcribe Whisper, timestamped Table and Figure Extract kept as units Enrich Entity Extraction to the graph Taxonomy Classify domain and doc type PII and Sensitivity Presidio plus labels Injection Screen at index, not only at query Segment Structure-Aware Chunking per type, view 18 Context Header title and heading path Overlap Policy 15% on prose only Represent Dense and Sparse Vectors BGE-M3, versioned Multimodal Vectors figures and frames Chunk Summary for long tables only Publish Idempotent Upsert Azure AI Search Graph Upsert entities and supersession Ledger Commit PostgreSQL, the receipt Cache Invalidate by document and ACL delete then invalidate no change Knowledge Ingestion — From a Change in a Source to a Retrievable Chunk Application we own Queue / topic Decision point Data store Security / platform failure / alternate batch Every stage writes a receipt to the ledger. A document that fails at any stage is visible to its steward rather than silently absent. v 1.0 · owner Data and AI Global Practice Knowledge Ingestion Pipeline From a change in a source system to a chunk that can be retrieved and cited. HTML page SVG draw.io
17 Zone 1 — Raw and immutable · source bytes as received Content Raw content ADLS Gen2, 12 TB Rendered pages page images for citation Why it exists Reparse without refetching source throttles are real Citation anchors to a page not to a moving document Zone 2 — Derived and rebuildable · deletable at any time, cost is only time Artefacts Parsed text Delta on ADLS Chunk table 320 M rows Vectors and index Azure AI Search Graph projection Apache AGE Rebuild economics Full rebuild 14 hours 1,900 USD of GPU Single source rebuild under 2 hours Zone 3 — Authoritative to the platform · nothing else can recreate it Records Document and version ledger PostgreSQL Answer provenance claims and evidence ids Feedback and labels the golden set grows here Audit record who asked, what was shown Consequence Backed up, geo-redundant RPO 5 minutes The only zone with an RPO everything else rebuilds Zone 4 — Ephemeral · correctness must not depend on it Caches and state Answer and embedding cache Redis, TTL by class Conversation state 30 days, then gone Traces Langfuse, 90 days Rule Flushing it degrades latency only never an answer Source systems keep owning the truth the platform holds no original Storage Zones — What Can Be Rebuilt, and What Cannot Deleting zones 1, 2 and 4 costs money and hours. Deleting zone 3 loses evidence that no source system can return, which is why only it is replicated. v 1.0 · owner Data and AI Global Practice Storage Zones Four zones, and the only one of them that has a recovery point objective. HTML page SVG draw.io
18 Unit of meaning How it is segmented What travels with it How it is represented Policy and Office documents A clause or subsection Heading path, 800 tokens 15% overlap Title, heading trail, version effective dates Dense plus sparse BGE-M3 Scanned PDF A page region Layout blocks from OCR reading order restored Page number and bounding box citation anchor Dense plus page image vector Wiki and HTML A section under a heading DOM section split boilerplate stripped Space, labels, last editor Dense plus sparse Support articles The whole article Never split symptom needs the fix Product, version, resolution state Dense over article, sparse over title Tables and spreadsheets A row group with its header Header repeated per shard no orphan rows Sheet, units, as-of date Serialised text plus summary Slide decks A slide with its notes Per slide, never merged Deck title, slide number Text plus slide image vector Audio and video A topical passage Transcript segmented at pauses 90 to 150 seconds Timecode, speaker, asset id Transcript vector plus keyframes Chunking and Representation — One Strategy per Content Type Chunking is the design, not a parameter. A support article split in half retrieves the symptom without the fix, which is the failure this table exists to prevent. v 1.0 · owner Data and AI Global Practice Chunking and Representation Seven content types, seven strategies, and why one setting for all of them fails. HTML page SVG draw.io
19 source source_id PK kind m365 | confluence | snow | archive | sql | web freshness_class ranking_policy document document_id PK source_id FK -> source external_id steward_upn classification retired_at document_version version_id PK document_id FK -> document content_hash effective_from superseded_by FK -> document_version indexed_at acl_grant grant_id PK document_id FK -> document principal_oid effect allow | deny synced_at chunk chunk_id PK version_id FK -> document_version ordinal heading_path page_no bbox modality representation rep_id PK chunk_id FK -> chunk model_name model_version kind dense | sparse | image index_alias entity entity_id PK canonical_name type aliases entity_mention mention_id PK chunk_id FK -> chunk entity_id FK -> entity confidence query query_id PK principal_oid acl_fingerprint class asked_at conversation_id query_plan plan_id PK query_id FK -> query sub_queries jsonb backends jsonb budget jsonb retrieval_result evidence_id PK plan_id FK -> query_plan chunk_id FK -> chunk backend raw_score rank_after_fusion reason answer answer_id PK query_id FK -> query model_name tokens_in tokens_out groundedness state answered | abstained | fallback claim claim_id PK answer_id FK -> answer sentence_ordinal verdict supported | unsupported | contradicted nli_score citation citation_id PK claim_id FK -> claim evidence_id FK -> retrieval_result quote_span feedback feedback_id PK answer_id FK -> answer signal useful | wrong | unclear note promoted_to_golden 1 : N 1 : N 1 : N 1 : N 1 : N 1 : N 1 : N 1 : 1 1 : N 1 : N 1 : N 1 : N 1 : N 1 : N 1 : N Evidence Data Model — Provenance as a First-Class Structure A citation joins a claim to a retrieval_result, never straight to a chunk. That single join is what makes an answer replayable and an audit answerable. v 1.0 · owner Data and AI Global Practice Evidence Data Model Fifteen entities, and the one join that makes an answer auditable. HTML page SVG draw.io
20 At the source Native ACL item and folder Group membership Entra, nested Sensitivity label Purview At ingest ACL extraction with the content, not after Principal normalisation everything to an object id Deny capture deny beats allow Classification mapped label to ceiling In the index acl_allow[] object ids acl_deny[] evaluated first classification filterable tenant and region hard partition At query On-behalf-of token never a service identity Group set expanded transitive, cached 60 s Filter composed index-side, pre-scoring At render Citation re-check at display time Cache key fingerprint principal group set Audit record query, evidence, principal On change ACL delta job 60 second target Targeted re-index fields only, not content Cache invalidation by principal and document field-level update Permission Propagation — How Authorisation Travels With the Data Security / platform Application we own Decision point Data store event / async Post-filtering is banned. If the filter is applied after scoring, the ranked set already reveals what exists, and the top-k is quietly wrong. v 1.0 · owner Data and AI Global Practice Permission Propagation How authorisation travels from a source system to a filter, a cache key and an audit record. HTML page SVG draw.io
21 Change detected delta, CDC or crawl Prioritised by freshness class Reprocessed only what changed Republished idempotent upsert Probed canary query per source Superseded old version demoted Staleness published as-of on every answer Freshness Loop per source class queued 5 min or 24 h parse and embed alias swap found or alarmed graph edge written back to watching Freshness Lifecycle — Keeping the Index Honest About Time Application we own Decision point Data store Security / platform Staleness is never hidden. Where a source is behind its class, the answer says so rather than reading as current. v 1.0 · owner Data and AI Global Practice Freshness Lifecycle The loop that keeps the index honest about time, and publishes it when it is not. HTML page SVG draw.io

Runtime

One request followed end to end, the bounded agentic loop, fusion and reranking, context assembly, verification, tools, multimodal and graceful degradation.
22 Knowledge Worker Search Gateway Query Understanding Planner Retriever Facade Azure AI Search Model Gateway Verifier 1. POST /v1/answer 2. validate OBO token, start trace 3. interpret and authorise 4. class, entities, group set, fingerprint 5. QueryPlan 6. retrieve, depth 80 7. hybrid query with ACL filter 8. 94 candidates 9. fuse, rerank, diversify 10. 8 evidence items 11. synthesise over fenced evidence 12. first token, 1.2 s 13. draft with claim markers 14. verify claims against evidence 15. one claim unsupported 16. targeted re-retrieval, hop 2 17. 2 further items 18. re-verify 19. all claims supported 20. answer, citations, provenance 21. stream completed, 4.1 s Ask to Answer — The Critical Path, One Request The unsupported claim on message 15 is the normal case, not the exception. Budget allows one extra hop; a second failure abstains rather than guessing. v 1.0 · owner Data and AI Global Practice Ask to Answer — The Critical Path One request, twenty-one messages, and the re-retrieval that is the normal case rather than the exception. HTML page SVG draw.io
23 Understand class and entities Plan sub-queries, backends Retrieve policy-filtered Evaluate evidence coverage and conflict Refine or stop budget decides Synthesise claims, not prose Verify grounding and safety Bounded Loop 3 hops, 18 s typed plan retrieval requests candidates and reasons gap named, not felt one more hop, or enough draft with claim markers answer, abstain or fall back The Bounded Agentic Loop — Where the Model Is Allowed to Decide Application we own Data store Decision point Security / platform The model chooses what to ask for. It never chooses what it may see, how long it may run, or whether its answer ships. v 1.0 · owner Data and AI Global Practice The Bounded Agentic Loop Where the model is allowed to decide, and the three things it never decides. HTML page SVG draw.io
24 Prepare Plan and filters ACL, source, freshness Query variants up to 4 Query embedding BGE-M3, cached Generate candidates Lexical BM25 top 100 Dense vector top 100 Learned sparse top 60 Graph expansion top 40 Fuse Reciprocal rank fusion k = 60 Source authority weights policy, not tuning Near-duplicate collapse SimHash on passages Rerank Cross-encoder top 50 scored Score floor 0.32, tuned on golden set Freshness adjustment class-aware, bounded Diversify Maximal marginal relevance lambda 0.7 Two passages per document hard cap Sub-query coverage check each hop represented Select evidence Token budget fit 8 passages, 6k tokens Evidence ids minted chunk plus version Retrieval reason recorded for the trace re-plan dropped Hybrid Retrieval and Fusion — From Four Candidate Sets to Eight Passages Application we own Data store Decision point Security / platform failure / alternate batch Vector-only retrieval failed the golden set on part numbers and policy identifiers, which is why the lexical arm is not an option in the configuration. v 1.0 · owner Data and AI Global Practice Hybrid Retrieval and Fusion Three hundred candidates down to eight passages, and what each stage is allowed to remove. HTML page SVG draw.io
25 Where it comes from Trust level Token budget When the window is short System policy Versioned prompt registry reviewed like code Trusted the only instructions 700 tokens fixed Never trimmed request fails first Task and output schema Query class template one per class Trusted server-composed 400 tokens fixed Never trimmed schema is the contract Conversation state Last 6 turns, summarised PostgreSQL Semi-trusted the caller wrote it 800 tokens rolling summary Summarise harder oldest turns first Retrieved evidence Evidence set, 8 passages each with an id Untrusted data fenced, never obeyed 6,000 tokens the majority share Drop lowest-ranked passage and say the set was cut Tool results Broker output, typed SQL rows, calculations Untrusted data same fence as evidence 1,200 tokens rows truncated with a count Aggregate, never sample a sample misleads The caller turn This question, verbatim plus resolved scope Trusted as intent not as instruction to tools 300 tokens truncated with notice Never trimmed it is the question Context Assembly — What Occupies the Window, and Under What Rules Compression happens on evidence and history, never on policy or the question. A window that cannot hold the fixed slots is a routing failure, not a prompt problem. v 1.0 · owner Data and AI Global Practice Context Assembly Six slots, their trust levels, their budgets, and what is sacrificed when the window is short. HTML page SVG draw.io
26 Draft Model output structured, claim-marked Schema validation reject and retry once Decompose Claim segmentation one assertion per claim Numeric and date extraction checked literally Quote span capture for the citation anchor Align Claim to evidence NLI cross-encoder, per pair Numeric equality check deterministic, not model Cross-source agreement pairwise on claims Judge Supported score above 0.72 Unsupported no evidence covers it Contradicted sources disagree Act Cite and keep evidence id bound Re-retrieve once targeted at the gap Drop or abstain budget exhausted Show both positions dated and attributed Present Claim-level citations anchored to a passage Confidence and as-of stated, not implied Disagreement banner when sources conflict Provenance persisted replayable gap conflict Grounding and Verification — Turning a Draft Into Claims That Hold Application we own Decision point Risk / gap Data store failure / alternate Groundedness is measured against the evidence actually supplied, not against the world. A claim that is true but unsupported here is still dropped. v 1.0 · owner Data and AI Global Practice Grounding and Verification How a fluent draft becomes claims that either hold, get re-retrieved, or are withdrawn. HTML page SVG draw.io
27 Planner Tool Broker Policy Engine SQL Guard Databricks SQL Answer Assembler Audit Log 1. tool request: revenue by region, FY26 2. may this class call this tool? 3. allowed: structured class, read-only 4. reject if the request came from a document 5. compile against the semantic layer 6. allowlisted views, bound parameters, row cap 7. parameterised SQL as the caller 8. 42 rows, Unity Catalog applied 9. typed result plus the SQL executed 10. tool call, principal, statement 11. rows as citable evidence 12. figures rendered from rows, never retyped Structured Retrieval and Tool Invocation — The Model Asks, the Broker Decides The model never emits SQL that reaches a database. It names an intent; the guard compiles it against views a data owner published. v 1.0 · owner Data and AI Global Practice Structured Retrieval and Tools The model names an intent; the broker and the guard decide what actually runs. HTML page SVG draw.io
28 Represented at ingest Matched at query What is cited Known limit Figures in documents Image vector plus caption and surrounding text Text query to image space cross-modal Cropped figure, page anchored Dense schematics under-retrieve caption carries the recall Photo from the field Not indexed, it is the query Image to image plus OCR text asset tags read from the plate Matching procedure and revision Poor lighting drops OCR falls back to visual only Slides Slide image plus notes text Text first, image as a tie-break Slide thumbnail and number Design-heavy decks are noisy notes weighted higher Audio and video Transcript plus keyframe vectors timecoded Transcript retrieval, frame confirm Player deep-link at the timecode Speaker attribution is weak never used as authority Tables and forms Row groups plus a summary line Lexical on headers, dense on summary The row group, with its header Merged cells still mis-parse flagged to the steward Multimodal Retrieval — Text, Image, Table, Audio and Video Every modality names its known limit. A modality with no stated limit has not been measured, and would be more dangerous than one that is simply weak. v 1.0 · owner Data and AI Global Practice Multimodal Retrieval Five modalities, how each is represented and matched, and the limit each one has. HTML page SVG draw.io
29 Healthy All backends up nothing failing Full generative answer claim-bound citations 99.9% of minutes the SLO Rung 1 — model pressure Azure OpenAI throttled 429 or latency spike Route to smaller tier gateway policy Shorter answer, same citations user sees a notice Rung 2 — ranking loss Rerank service down GPU pool unavailable Serve fusion order quality flag recorded Answer with wider evidence confidence lowered Rung 3 — semantic loss Vector arm unavailable index or embed service Lexical-only retrieval recall drops to 0.71 Extractive answer only no synthesis over gaps Rung 4 — index loss Search service degraded region or partition Serve the cache, then fail over paired region alias Cached answers and links staleness stated Rung 5 — floor Generation unavailable safety or policy stop Classic search API always deployable alone Ranked list of documents the product still works search SLO is separate, 99.95% Degradation Ladder — What the User Still Gets When a Dependency Fails Application we own Decision point Risk / gap Journey / task failure / alternate The classic search API is deployable with no model, no GPU and no vector index. That is why the floor of this ladder is a working product rather than an error page. v 1.0 · owner Data and AI Global Practice Degradation Ladder Five rungs down from a full answer, and why the bottom rung is still a working product. HTML page SVG draw.io

Operations

Where it runs, how a prompt reaches production through the same gates as code, what is measured, what is traced, and what it costs.
30 Azure — West Europe (primary region) Hub VNet — edge and shared services Front Door and WAF TLS, bot rules API Management quota, keys, versions Azure Firewall egress allowlist Key Vault workload identity Bastion no public admin AKS spoke — application node pools, 3 availability zones Answer plane 38 pods, D8s v5 Retrieval plane 26 pods, D8s v5 Ingest plane autoscaled 4 to 60 Platform plane gateway, policy, traces AKS spoke — GPU node pools Embed and rerank 4 x A10, always on Verifier and small LLM 2 x A10 Batch embedding 6 x A10 spot, queue-driven Data services — private endpoints only Azure AI Search S3, 12 partitions, 3 replicas PostgreSQL Flexible zone-redundant HA Cache for Redis premium, zone-redundant ADLS Gen2 ZRS, immutable raw Event Hubs and Service Bus zone-redundant Azure — North Europe (paired region, warm) Standby capacity AKS minimal scales on failover AI Search replica index indexer writes both PostgreSQL read replica promotable Recovery targets RTO 30 minutes search first, then answers RPO 5 minutes ledger and provenance only Azure OpenAI two regions, one gateway Microsoft Entra ID global Databricks governed SQL warehouse Content sources read-only TLS 1.3 mTLS query second index build async replication Deployment Architecture — Where It Runs and What Survives a Failure Security / platform Interface / broker Application we own Data store Queue / topic Decision point External / third party synchronous batch event / async The secondary is not a copy of the primary: the indexer builds both, so failover does not depend on replicating a search index that Azure AI Search does not replicate for us. v 1.0 · owner Data and AI Global Practice Deployment Architecture Two regions, four node pools, and a failover that does not depend on replicating a search index. HTML page SVG draw.io
31 Source Application code GitHub, trunk-based Prompt registry versioned, reviewed Index schema declared, migrated Golden set 1,400 queries Build Container build GitHub Actions SBOM and signature Syft and Cosign Unit and contract tests retriever contract Evaluate Retrieval metrics Recall@50, nDCG@10 Generation metrics groundedness, citations Adversarial suite injection and leakage Cost per query budget regression Stage Shadow index build new alias, not swapped Shadow traffic 10% mirrored, not served Side-by-side scoring same queries, both stacks Gate Thresholds met no metric below baseline Human approval prompt and policy changes Change record what moved and why Release Argo CD sync GitOps, per environment Canary 5% 30 minutes Alias swap index cutover is atomic Watch 48-hour watch online quality signals Auto rollback alias back, pods back Baseline updated new reference scores revert CI/CD — Shipping Code, Prompts, Index Schemas and Models Through the Same Gates Application we own Data store Security / platform Decision point failure / alternate A prompt change and a code change take the same path. Treating prompts as configuration is how a quality regression reaches production without a review. v 1.0 · owner Data and AI Global Practice CI/CD and Index Migration Code, prompts, index schemas and models through one path with the same gates. HTML page SVG draw.io
32 Golden set Measured by Gate Runs when Retrieval 620 queries, judged pools graded 0 to 3 Recall@50, nDCG@10, MRR trec_eval semantics Recall@50 at least 0.90 blocks release Every pull request fixture corpus Ranking Same pools, reranked pairwise preferences nDCG@10 delta vs fusion must be positive Gain at least 0.06 or the reranker is off Nightly and on model change full corpus Generation 480 queries with answers human-written references Groundedness, citation correctness Ragas plus NLI judge Groundedness at least 0.95 citations at least 0.97 Nightly, and every prompt change no exceptions End to end 300 task scenarios simple, multi-hop, long-tail Task success, abandonment and reformulation rate Task success at least 0.82 trend, not a single run Nightly, plus weekly online read from real traffic Safety and access 220 adversarial cases injection, leakage, PII Attack success rate and false refusal rate Zero access leaks a single failure blocks Every build, and on new sources never sampled Cost and latency Traffic-weighted replay yesterday's mix Cost per answered query, P95 per class Within 10% of baseline or approved explicitly Nightly reported with quality Evaluation Harness — What Is Measured, Against What, and What It Blocks Quality and cost are gated together. A change that improves groundedness by two points and doubles the bill is a decision, not an improvement. v 1.0 · owner Data and AI Global Practice Evaluation Harness Six layers of measurement, what each blocks, and how often each runs. HTML page SVG draw.io
33 Observe traces and feedback Classify the failure which stage broke Reproduce replay by evidence id Fix at the stage chunking, weights, prompt Evaluate offline against the golden set Ship behind a flag canary, then all Promote to golden the case becomes a test RAGOps weekly cadence a wrong answer, with its trace retrieval, rank, context or model the exact evidence set returns one change, one stage gate passes or the fix is wrong measured online for 48 hours it can never regress silently RAGOps — How a Bad Answer Becomes a Fixed System Security / platform Decision point Application we own Data store The loop closes only when the failing query joins the golden set. A fix that is not testable tomorrow is a patch, not an improvement. v 1.0 · owner Data and AI Global Practice RAGOps Loop How one bad answer becomes a fixed system and a permanent test. HTML page SVG draw.io
34 Span recorded Metric What it proves Alerts when Query understanding class, entities, rewrite and confidence class distribution vs the 7-day baseline The question was read correctly or was not class mix shifts 20% usually a UI change Authorisation group set size, fingerprint never the group names expansion latency P95 and cache hit rate The filter was applied and to what any query without a filter page immediately Retrieval per backend: query, depth, latency and every id returned recall proxy, zero-result rate per source The right document was found or never was zero-result rate above 4% per source Ranking fusion order, rerank scores before and after rank of the cited passage distribution It was found and then buried the commonest cause cited rank drifts past 6 ranking regression Context assembly slots, tokens, what was dropped and why evidence truncation rate The model was given enough or was starved truncation above 8% budget is wrong Model call model, tokens, cost, retries prompt version TTFT, total latency, spend It had what it needed and still failed a model problem TTFT P95 above 2.5 s or spend above budget Verification and answer claims, verdicts, citations abstentions recorded groundedness, abstain rate and feedback What the user was actually shown abstain rate above 6% corpus gap, not a bug Observability — One Trace That Answers Which Stage Was Wrong One trace id spans all seven rows. Without it, every quality question becomes an argument between the retrieval team and the model team. v 1.0 · owner Data and AI Global Practice Observability and Tracing One trace id across seven stages, so a wrong answer has a cause rather than an argument. HTML page SVG draw.io
35 Demand 62,000 queries a day peak 34 per second Class mix from view 04 11% agentic 40 M documents 320 M chunks Where money goes LLM tokens 58% of run cost GPU node pools 19% Azure AI Search 14% Storage, egress, ops 9% What each scales with Tokens: class mix and hops not user count GPU: queries per second plus reindex bursts Search: chunk count and replica count Levers Semantic cache 35% hit, 0 tokens Class routing the largest single saving Rerank depth 50 not 200 Spot GPU for batch embed 62% cheaper Guardrails Budget governor per query, hard stop Per-tenant quota daily token ceiling Cost in the eval gate view 32 Outcome 0.021 USD blended per answered query 0.004 USD lookup 0.180 USD agentic About 39k USD a month at current volume removes the token line Cost Model — What a Query Costs and Which Levers Move It Application we own Data store Decision point Security / platform batch The agentic class is 45 times the cost of a lookup. Routing, not model choice, is what keeps the blended figure where it is. v 1.0 · owner Data and AI Global Practice Cost Model What a query costs, what each line scales with, and which lever actually moves the total. HTML page SVG draw.io

Assurance

Trust boundaries, the identity path, six concrete attacks on untrusted content, sensitive data, governance evidence, and every failure mode named.
36 Internet Browser and mobile managed devices Public web sources crawled, untrusted Attacker or a poisoned document Edge Front Door and WAF TLS 1.3, bot rules API Management token validation, quota DDoS protection standard tier Application Search Gateway no source credentials Planner and retrieval acts as the caller Policy Engine OPA, decisions logged Tool Broker egress allowlist Model and GPU Self-hosted models no egress, no internet Azure OpenAI private endpoint Content Safety screens both directions Data Azure AI Search private endpoint only PostgreSQL CMK, no public route ADLS Gen2 immutable raw Key Vault workload identity only HTTPS, Entra token private link filtered query fenced context crawled, screened injection attempt short-lived credential Security Zones — Trust Boundaries and What Crosses Them External / third party Risk / gap Security / platform Interface / broker Application we own Data store synchronous batch failure / alternate The only inbound path is Front Door. Every data service refuses public traffic, and no application pod holds a long-lived secret. v 1.0 · owner Data and AI Global Practice Security Zones Five trust zones, and every flow that crosses one. HTML page SVG draw.io
37 User Client App Entra ID API Management Search Gateway Entitlement Resolver Azure AI Search Audit Log 1. opens search 2. authorisation code with PKCE 3. access token, 60 min 4. POST /v1/answer with the user token 5. validate signature, audience, quota 6. forwarded with the caller identity 7. on-behalf-of exchange 8. downstream token for Graph 9. resolve entitlements 10. transitive group membership 11. object ids, cached 60 s 12. group set and ACL fingerprint 13. query with acl filter and classification ceiling 14. only permitted chunks 15. principal, query, evidence ids 16. answer the caller was entitled to Identity and Authorisation — Who Proves What, to Whom, in What Order There is no service identity that can read content on a user's behalf. If the on-behalf-of exchange fails, the request fails; it never falls back to an application identity. v 1.0 · owner Data and AI Global Practice Identity and Authorisation Sixteen messages from a login to a filtered query, with no application identity anywhere in them. HTML page SVG draw.io
38 Where it enters What it attempts Control Residual risk Instruction inside a document Any indexed source white text, alt text, notes Be obeyed as a system rule ignore prior instructions Fenced evidence slot plus screening at index Novel phrasings evade detection the fence still holds Poisoned public page Crawled web source 40 allowed domains Rank above internal policy and be cited Authority weight, source policy external never outranks internal A legitimate external page can still mislead citation makes it visible Malicious tool argument Model output after reading a document Reach a system or exfiltrate arbitrary SQL or URL Broker allowlist, compiled SQL no free-form statements An allowlisted tool misused within its scope audited per call Exfiltration via a link Answer rendering markdown image or link Send context to an outside host as a URL parameter Links only to cited sources no model-authored URLs None known; the renderer is the control Scope escalation by asking The user turn itself Retrieve what the caller may not open act as another principal Filter is server-composed the query cannot set it Inference from the absence of results accepted and documented Jailbreak in the user turn The user turn itself Bypass safety or the answer format Prompt Shields plus schema validation output must parse Over-refusal annoys real users false refusal rate is a gate Untrusted Content — Six Attacks and What Stops Each The controls are structural, not lexical. A filter that looks for the phrase ignore previous instructions has already lost; the evidence slot is never read as instruction in the first place. v 1.0 · owner Data and AI Global Practice Untrusted Content and Prompt Injection Six concrete attacks, the control for each, and the residual risk stated rather than implied. HTML page SVG draw.io
39 Detected once, at ingest Detectors Presidio recognisers names, ids, accounts Purview label read authored by the business Internal identifier patterns curated regexes Written to the chunk classification public to restricted pii_kinds[] for redaction rules Enforced at retrieval and display Retrieval Classification ceiling per request from the caller, not the query Restricted sources excluded unless the class allows them Display Redaction at render never in the index Reveal is an audited action with a reason Constrained in generation Model handling No tenant data in prompts to shared models beyond the evidence slot No training or fine-tuning on content contractual and technical Abuse-monitoring retention disabled approved exemption Logging Traces store ids, not text passages by reference Prompt capture is opt-in and time-boxed Isolation between business units and regions Boundaries tenant field on every chunk filter, never a convention Separate index for regulated units physical, not logical Customer-managed keys per unit revocable Subject rights Erasure removes chunks and traces within 30 days Provenance keeps the reference not the content Source systems remain the record for a subject access request Sensitive Data and Tenant Isolation Classification is decided once, at ingest, and enforced everywhere afterwards by a filter. Deciding it at query time would make every new consumer a new security review. v 1.0 · owner Data and AI Global Practice Sensitive Data and Tenant Isolation Classified once at ingest, enforced everywhere afterwards by a filter. HTML page SVG draw.io
40 Policy Source policy who may index what Model policy which models, which data Tool policy per query class Retention policy traces, answers, feedback Expressed as OPA rules in git reviewed, versioned Index schema fields classification, tenant Gateway configuration model routing, budgets Enforced at Index-side filter before scoring Model Gateway model and region choice Tool Broker call-by-call Verifier what may be shown Recorded as Audit log append-only, 7 years Answer provenance claims and evidence ids Decision log every policy evaluation Evaluation history every gate, every release Reviewed by Source onboarding review before a source is indexed Quarterly architecture board policy and drift Red team twice a year Evidence produced Replay any answer from its evidence ids Prove entitlement at the time fingerprint plus ACL history Show the quality record gates at that release findings change policy Governance and Audit — From a Written Policy to Evidence an Auditor Accepts Security / platform Application we own Data store failure / alternate Explainability here means naming the evidence that produced an answer, not interpreting the model. That is answerable; the other is not. v 1.0 · owner Data and AI Global Practice Governance and Audit From a written policy to evidence an auditor accepts, without a single interpretive step. HTML page SVG draw.io
41 Retrieval — the answer was wrong before the model saw anything How it fails Document never indexed parse failed silently Entitlement stale index behind the source Answer split across chunks neither is retrievable Embedding drift two model versions live Detected and answered by Per-document receipt steward console, view 08 60-second ACL delta plus cache invalidation Structure-aware chunking and coverage probes Alias cutover, never mixed view 31 Generation — the evidence was right and the answer was not How it fails Unsupported claim fluent and wrong Contradiction smoothed false consensus Over-refusal abstains on a good answer Schema break citations unparseable Detected and answered by Claim-level NLI drop or re-retrieve Contradiction detector both positions shown False refusal rate gate view 32 Validate and retry once then fall back Security — the boundary held, or it did not How it fails Cross-principal cache hit the highest-severity bug Injection reaches a tool document as instruction Prompt log holds content logging becomes a copy Detected and answered by Fingerprint in every key plus a nightly assertion Broker provenance check and egress allowlist Traces store ids only capture is opt-in Operations — it worked yesterday How it fails GPU pool exhausted reindex meets peak Source throttles the connector freshness silently slips Cost runaway one agentic loop pattern Model deprecated by the provider with notice, still disruptive Detected and answered by Separate pools and priorities batch is preemptible Freshness SLO per source staleness is published Budget governor and quota hard stop per query Gateway abstraction swap and re-evaluate Accepted risk — a correct answer from a stale but authorised document answered by publishing as-of, not by blocking Failure Modes — Named, Detected and Answered Every failure here is named, has a detector and has an owner. A failure mode with no detector is the one that becomes an incident. v 1.0 · owner Data and AI Global Practice Failure Modes Fifteen named failures across four planes, each with a detector and an owner. HTML page SVG draw.io

Architecture Decision Record

Twenty-two decisions carry this architecture. Everything else in the set is a consequence of one of them. Each record below states the forces that made the decision necessary, what was chosen, what was rejected and why, and what the choice costs — because a decision recorded without its price is a preference, not an architecture. Read a record's Consequences before its Decision if you want to judge it honestly.

ContextThe forces in play — requirement, constraint, volume, failure mode. If the context is not real, the decision is decoration.
DecisionWhat was chosen, stated so that a reviewer can disagree with it precisely.
RejectedThe credible alternative and the specific reason it lost. An alternative with no reason means the option was never considered.
ConsequencesWhat this costs — in money, latency, complexity or freedom lost later. Every decision has some.
ADR-001The evidence contract is the governing constraint ADR-002Separate planning, retrieval, generation and verification ADR-003Azure AI Search as the hybrid retrieval substrate ADR-004Hybrid retrieval, never vector-only ADR-005Self-hosted BGE-M3 for embeddings ADR-006Cross-encoder reranking at depth 50 ADR-007LangGraph for orchestration ADR-008LiteLLM as the model gateway ADR-009Managed frontier models for synthesis, open weights for everything else ADR-010Apache AGE on PostgreSQL for the knowledge graph ADR-011Structured retrieval through an allowlisted semantic layer ADR-012Permissions projected at index time, enforced as an index-side filter ADR-013The ACL fingerprint is part of every cache key ADR-014Claim-level binding verified by a separate NLI model ADR-015Structure-aware chunking, one strategy per content type ADR-016One PostgreSQL for the ledger, the graph, conversation state and traces ADR-017Langfuse self-hosted for LLM tracing, Azure Monitor for everything else ADR-018Evaluation is a release gate, built before launch ADR-019The classic search API is a separately deployable product ADR-020Airflow on AKS for ingestion orchestration ADR-021A warm secondary region with a dual index build ADR-022Self-hosted GPU for embedding, reranking and verification
ADR-001

The evidence contract is the governing constraint

Accepted
PrincipleSecurityGrounding

Context

A generative search system fails in three ways that a search engine cannot: it can show a passage the reader was never entitled to open, it can assert something no retrieved passage supports, and it can be instructed by the content it retrieves. All three are architectural, not behavioural, and none is fixed by prompt wording.

Decision

Three rules bind every component. One: nothing enters the context window that the caller could not open themselves, so authorisation is an input to retrieval rather than a filter on its output. Two: every sentence carries an evidence id minted by the retriever, or it is not shown. Three: retrieved content occupies a typed slot that is never read as instruction.

Considered and rejected

  • Prompt-level guardrails — instruct the model to respect permissions, cite sources and ignore embedded instructionsMakes the security boundary a matter of wording, which cannot be tested, cannot be proved to an auditor, and degrades silently with every model change.
  • Post-hoc filtering — generate first, then remove anything the caller should not have seenThe content has already entered the model's context; the leak has already happened, and the ranked set was computed over documents the caller cannot see.

Consequences

  • +Positive the downstream CRM assistant needed no access model of its own, and any answer can be replayed and re-derived from its stored evidence ids months later.
  • +Positive a poisoned document becomes a ranking problem rather than a compromise.
  • −Negative claim-level binding costs about 180 ms and 0.0009 USD per answer, and is the largest single quality expense in the design.
  • −Negative the contract erodes silently — a cache key missing its ACL fingerprint breaks it with no test failing, which is why three specific assertions exist to catch exactly that.
Drawn in03 · The Evidence Contract20 · Permission Propagation26 · Grounding and Verification
ADR-002

Separate planning, retrieval, generation and verification

Accepted
OrchestrationQuality

Context

The dominant industry pattern is a single agent with tools that searches, reads and answers as it sees fit. It is faster to build and it makes latency unbounded, cost unpredictable, and the failure classification impossible — nobody can say whether a wrong answer came from retrieval, ranking, context or the model.

Decision

Four stages with typed contracts between them, executed by a LangGraph state machine. The model proposes retrieval; a facade executes it under policy; a budget governor decides when the loop stops; a verifier decides whether the answer ships.

Considered and rejected

  • Single agentic loop with tool callingAuthorisation becomes prompt-shaped, cost becomes unbounded, and each stage cannot be measured or replaced independently.
  • Classic linear RAG with no loop at allCannot answer multi-hop or comparative questions, which are 11% of traffic and the ones the analyst population judges the platform by.

Consequences

  • +Positive each stage has its own evaluation, its own latency budget and its own failure alert — which is what makes view 34 diagnostic rather than descriptive.
  • +Positive the model and the retrieval stack can be replaced independently.
  • −Negative more services, more contracts, more deployment surface than a single agent would need.
  • −Negative one backward edge — the verifier asking the planner to retrieve again — has to be bounded explicitly, or it becomes the unbounded loop this decision was meant to avoid.
Drawn in02 · High-Level Architecture09 · Layered Architecture23 · The Bounded Agentic Loop
ADR-003

Azure AI Search as the hybrid retrieval substrate

Accepted, review scheduled
RetrievalAzureBuy

Context

The platform needs BM25 and dense vector search over 320 M chunks in one query, filterable on ACL arrays and classification, with faceting, synonym maps and analyzer control, at 34 queries per second with a 450 ms budget. It also needs security trimming to be an index-side filter rather than an application concern.

Decision

Azure AI Search at S3 with 12 partitions and 3 replicas. Hybrid retrieval with reciprocal rank fusion is native, ACL arrays are filterable fields evaluated before scoring, and the operational burden of a sharded search cluster stays with the platform provider.

Considered and rejected

  • OpenSearch or Elasticsearch self-hosted on AKSFully open source and entirely capable, but a 320 M document cluster is a full-time operational commitment — shard rebalancing, JVM tuning, upgrade windows — for a six-engineer team whose scarce skill is retrieval quality, not cluster operations.
  • Qdrant or Milvus for vectors plus OpenSearch for lexicalTwo systems means two consistency stories, two ACL projections and a fusion layer that must reconcile two definitions of a document version. The correctness risk sits exactly where this design cannot afford it.
  • pgvector on the existing PostgreSQLExcellent to about 10 M vectors and outside its comfort zone at 320 M; it would also couple the query path to the ledger's availability.

Consequences

  • +Positive hybrid search, security filters and index aliasing are one managed service, and the alias swap in view 31 is atomic.
  • −Negative real vendor coupling. It is contained by the retriever facade in view 12, which every backend implements, so a migration is a new backend rather than a rewrite.
  • −Negative cross-region index replication is not provided, which is why the indexer writes both regions in view 30.
  • ·Review revisit if corpus growth passes about 500 M chunks or if the index line exceeds 20% of run cost.
Drawn in12 · Retrieval Fabric30 · Deployment Architecture31 · CI/CD and Index Migration
ADR-004

Hybrid retrieval, never vector-only

Accepted
RetrievalQuality

Context

Vector search is the default assumption in most generative search designs. On this corpus it fails on precisely the queries that dominate traffic: part numbers, policy identifiers, product codes and exact names — the lookup class, 38% of all questions.

Decision

Four candidate generators — BM25, dense HNSW, learned sparse and graph expansion — fused with reciprocal rank fusion at k=60, then reranked by a cross-encoder. The lexical arm is load-bearing and is not a configuration option.

Considered and rejected

  • Dense vector retrieval onlyMeasured Recall@50 of 0.71 against 0.92 for the fused set, with almost all the loss on identifier-bearing queries.
  • Score normalisation instead of rank fusionBM25 scores, cosine similarities and graph walk scores are not on a common scale, and normalising them fabricates a comparability that does not exist.

Consequences

  • +Positive Recall@50 of 0.92 and nDCG@10 of 0.68 on the golden set.
  • −Negative four backends mean four failure modes on the query path, which is why the degradation ladder in view 29 exists.
  • −Negative fusion weights per source are a governed policy rather than a tuning knob, which adds process to a change an engineer could otherwise make in an afternoon.
Drawn in12 · Retrieval Fabric24 · Hybrid Retrieval and Fusion29 · Degradation Ladder
ADR-005

Self-hosted BGE-M3 for embeddings

Accepted
ModelsOpen sourceCost

Context

Embeddings are computed for 320 M chunks once, for about 240,000 changed items a day, and for every query. The cost scales with corpus size and queries per second rather than with answer tokens, and the model must be pinned for the life of an index because changing it invalidates the whole index.

Decision

BGE-M3 served on Hugging Face Text Embeddings Inference on an AKS GPU pool, producing dense and learned sparse representations from one model. Batch embedding runs on preemptible spot nodes.

Considered and rejected

  • Azure OpenAI text-embedding-3-largeStrong quality and about 3.4 times the cost at this volume, with a per-call dependency on the critical path and a version lifecycle we do not control for an artefact that must stay pinned for years.
  • A smaller open model such as all-MiniLMCheaper and materially worse on multilingual content, which is 14 languages here.

Consequences

  • +Positive one model produces both the dense and the sparse representation, so the lexical bridge needs no second model to run or evaluate.
  • +Positive the embedding version is ours to pin, and a provider deprecation cannot force an unplanned reindex.
  • −Negative we now operate GPU nodes, with capacity planning, driver upgrades and spot interruption handling.
  • −Negative about 4,100 USD a month of standing GPU cost across embedding, reranking and verification.
Drawn in15 · Model Portfolio18 · Chunking and Representation30 · Deployment Architecture
ADR-006

Cross-encoder reranking at depth 50

Accepted
RetrievalQualityLatency

Context

Fusion produces a good candidate ordering and a poor top-10. The passages that end up in the context window are chosen from the top of that list, so ranking quality translates almost directly into groundedness.

Decision

bge-reranker-v2-m3 as a cross-encoder over the top 50 fused candidates, self-hosted on the same GPU pool as embeddings. Depth 50, not 200.

Considered and rejected

  • Azure AI Search semantic ranker aloneUseful and cheap, and it cannot be evaluated or tuned against our golden set the way a model we host can. It remains the fallback.
  • Reranking depth of 200The nDCG gain was inside the noise band on the golden set; the added latency was not.
  • No reranking, fusion order onlynDCG@10 falls from 0.68 to 0.61, and the passages the model is handed get measurably worse.

Consequences

  • +Positive 0.07 nDCG@10 for 110 ms and 0.0006 USD — the best quality-per-millisecond trade in the whole pipeline.
  • −Negative a GPU dependency on the query path, mitigated by rung 2 of the degradation ladder, which serves fusion order and records a quality flag.
Drawn in15 · Model Portfolio24 · Hybrid Retrieval and Fusion29 · Degradation Ladder
ADR-007

LangGraph for orchestration

Accepted
OrchestrationOpen source

Context

The answer path is a bounded state machine with conditional edges, a hard budget, checkpointing and a requirement that every transition appears in a trace. It is not a linear chain and it is not an open-ended agent.

Decision

LangGraph in Python, with checkpoints in PostgreSQL, running as a service on AKS. Graph nodes map one-to-one onto the stages in view 09 and onto spans in the trace.

Considered and rejected

  • Semantic KernelA capable orchestrator, and its planner abstractions pull towards the agent-decides-everything shape that ADR-002 rejects. It would be the stronger choice for a .NET estate.
  • Azure AI Agent ServiceFastest to a working demo and it puts retrieval, tools and the loop inside a managed boundary, which is exactly where this design needs to enforce its own policy.
  • Bespoke orchestration codePerfectly viable, and it re-implements checkpointing, retries and streaming for no gain over a library the team already knows.

Consequences

  • +Positive state, retries and streaming come for free, and the graph is the same artefact the trace shows.
  • −Negative a fast-moving open-source dependency on the critical path; the version is pinned and upgrades run through the same evaluation gates as a prompt change.
Drawn in09 · Layered Architecture10 · Container Architecture23 · The Bounded Agentic Loop
ADR-008

LiteLLM as the model gateway

Accepted
ModelsOpen sourceCost

Context

Six model tasks call four different providers — Azure OpenAI, self-hosted vLLM, self-hosted TEI and Azure AI Vision. Each needs routing, fallback, per-tenant budgets, token accounting and a way to move a task between providers without a code change.

Decision

LiteLLM proxy on AKS as the single egress point for every model call. No service holds a model name in its code; the gateway maps task to model, enforces budgets and emits token and cost telemetry per request.

Considered and rejected

  • Direct provider SDKs in each serviceModel names spread through the codebase, and cost accounting becomes a reconciliation exercise against a bill rather than a per-request fact.
  • Azure API Management as the model gatewayStrong on quota and keys, weak on model-aware concerns — token accounting, semantic caching, per-task fallback. APIM stays at the edge, where those are not needed.

Consequences

  • +Positive a provider deprecation becomes a configuration change plus an evaluation run — the whole migration plan in ADR-009.
  • +Positive per-query cost is a measured fact rather than a monthly estimate, which is what makes the cost gate in view 32 possible.
  • −Negative one more hop on the critical path, about 6 ms, and a component that must be as available as the models behind it.
Drawn in10 · Container Architecture15 · Model Portfolio35 · Cost Model
ADR-009

Managed frontier models for synthesis, open weights for everything else

Accepted
ModelsCostAzure

Context

Treating the LLM as one decision is how a platform ends up paying frontier prices to classify a question. The six model tasks have different quality requirements, different cost curves and different rates of change.

Decision

Azure OpenAI for answer synthesis, where quality requirements move faster than we can chase them. Open-weight models self-hosted for intent, rewriting, embedding, reranking and verification, where the workload is high-volume, well-defined and stable.

Considered and rejected

  • One frontier model for every taskWould raise per-query model cost from 0.0161 to about 0.049 USD for no measured quality gain on the small tasks.
  • Open weights for synthesis as wellEvaluated and close on the explanatory class, still behind on the multi-hop and comparative questions that carry the analyst population. It is the fallback and is re-evaluated each quarter.

Consequences

  • +Positive 0.014 USD of the 0.0161 blended model cost is synthesis; everything else is rounding.
  • −Negative a dependency on a provider's roadmap for the most visible component, mitigated only by the gateway abstraction and by keeping a tested open-weight fallback warm.
  • −Negative PTU capacity has to be reserved before go-live, which is a commercial commitment made on a traffic forecast.
Drawn in15 · Model Portfolio29 · Degradation Ladder35 · Cost Model
ADR-010

Apache AGE on PostgreSQL for the knowledge graph

Provisional
DataOpen source

Context

Multi-hop questions need entity and relationship expansion, and version supersession needs to be a first-class edge so a retired document can be retrieved and labelled rather than quietly outranked. The graph is a few hundred million edges, read-mostly, and reached only after a first retrieval pass.

Decision

Apache AGE, the open-source graph extension, on the PostgreSQL Flexible Server that already holds the ledger. openCypher queries, two-hop cap, bounded fan-out.

Considered and rejected

  • Neo4jThe better graph database by some distance, and it adds a licensed product, a cluster and a second operational model for a component whose contribution to answer quality is not yet proven.
  • Azure Cosmos DB with GremlinManaged and scalable, with a query model the team does not use elsewhere and a cost profile that does not suit a read-mostly workload of this size.
  • No graph at allSupersession and entity expansion would have to be simulated with metadata filters, which handles version chains poorly and multi-hop expansion not at all.

Consequences

  • +Positive no new operational surface — it is an extension on a server that already exists.
  • −Negative AGE is materially less mature than Neo4j, and its query planner is weaker on deep traversals. The two-hop cap is as much a containment as a design choice.
  • ·Provisional the graph arm's contribution is measured separately in view 32. If it does not pay for itself within two quarters it is removed, and if it proves central it is promoted to a purpose-built store.
Drawn in12 · Retrieval Fabric19 · Evidence Data Model21 · Freshness Lifecycle
ADR-011

Structured retrieval through an allowlisted semantic layer

Accepted
DataSecurityTools

Context

Nine per cent of questions are numeric or aggregate, and answering them from indexed text would mean embedding figures that go stale. Natural language to SQL is also the least reliable component available and the most dangerous one to get wrong.

Decision

The model names an intent and parameters. A SQL guard compiles that intent against views a data owner published, with bound parameters and a row cap, and executes as the caller so Unity Catalog grants apply. No model-authored SQL ever reaches a database, and figures are rendered from returned rows rather than retyped by the model.

Considered and rejected

  • Model-generated SQL with a read-only accountRead-only prevents damage, not disclosure or a wrong answer; and a hallucinated join produces a number nobody can trace.
  • Indexing the tables as textFreezes figures at index time and reintroduces staleness into exactly the answers where staleness is least acceptable.

Consequences

  • +Positive the executed statement is stored with the answer, so a disputed number is re-run rather than argued about.
  • −Negative the semantic layer becomes a dependency of answer quality — a renamed view breaks a class of question, so the allowlist is version-pinned and its owners are on the change list.
  • −Negative questions outside the allowlist cannot be answered numerically at all, which is a deliberate and visible limit.
Drawn in14 · Source Connector Matrix27 · Structured Retrieval and Tools38 · Untrusted Content and Prompt Injection
ADR-012

Permissions projected at index time, enforced as an index-side filter

Accepted
SecurityRetrieval

Context

Permissions must travel with the data through the entire retrieval pipeline. There are three ways to do it, and two of them are wrong in ways that are hard to see from the outside.

Decision

Source ACLs are extracted with the content, normalised to Entra object ids, and written onto every chunk as acl_allow and acl_deny arrays. At query time the resolved group set becomes a filter the search service evaluates before scoring. Deny is evaluated before allow.

Considered and rejected

  • Post-filtering — retrieve, then drop what the caller cannot seeThe ranked set was computed over documents the caller cannot see, so the visible top-k is quietly wrong even when nothing leaks.
  • Checking entitlement against the source system per resultCorrect and far too slow at retrieval depth 80 across four backends; it is retained only for the few sources where a 60-second projection lag is unacceptable.

Consequences

  • +Positive authorisation is a filter, which means it is testable, fast, and applied before ranking.
  • −Negative an entitlement change takes up to 60 seconds to become true in the index — a real window, stated rather than hidden.
  • −Negative the platform now holds a projection of every source's permission model, which must be kept correct and is itself sensitive.
Drawn in14 · Source Connector Matrix20 · Permission Propagation37 · Identity and Authorisation
ADR-013

The ACL fingerprint is part of every cache key

Accepted
SecurityCostCaching

Context

Caching is the single largest cost lever available — 35% of queries repeat closely enough to serve from cache, and a cache hit costs no tokens at all. It is also the fastest way to turn an authorisation system into a leak with a hit rate.

Decision

Every cache key includes a SHA-256 fingerprint of the caller's resolved group set alongside the normalised query. Two people with different entitlements never share a cached answer, whatever they asked.

Considered and rejected

  • Cache by query text aloneThe highest-severity defect this platform can have: an answer built from documents the second caller cannot open.
  • Cache only public-classified contentSafe and it discards most of the benefit, because the repeated questions are internal ones.

Consequences

  • +Positive a 35% hit rate that is safe to enable, worth roughly 40% of the token line.
  • −Negative hit rate falls as entitlement sets fragment — a person in an unusual set of groups effectively has a private cache.
  • −Negative an entitlement change must invalidate by principal as well as by document, which is why the ACL delta job in view 20 does both.
Drawn in11 · Query Understanding20 · Permission Propagation35 · Cost Model
ADR-014

Claim-level binding verified by a separate NLI model

Accepted
GroundingQuality

Context

Asking a model to cite its sources produces citation-shaped strings whose accuracy is unmeasured. Citation correctness is a stated requirement at 0.97, and a requirement that cannot be measured cannot be met.

Decision

The draft is segmented into claims; each claim is aligned against the supplied evidence by a cross-encoder NLI model from a different family than the synthesiser; numbers and dates are checked deterministically. A claim with no supporting evidence is re-retrieved once, then dropped or abstained on.

Considered and rejected

  • Trusting model-written citationsA citation the model composes is a string it can invent, and measurement shows it does.
  • Azure AI Content Safety groundedness detection aloneA good managed signal at answer level, and it does not bind a specific sentence to a specific passage, which is what an auditor and a careful reader both need. It is retained as the fallback.

Consequences

  • +Positive measured hallucination rate of 1.1% against a 1.5% ceiling, and citations that anchor to a passage rather than a document.
  • −Negative 180 ms and 0.0009 USD per answer, plus a second model to operate and evaluate.
  • −Negative over-refusal becomes a real risk, which is why false refusal rate is itself a release gate.
Drawn in03 · The Evidence Contract26 · Grounding and Verification32 · Evaluation Harness
ADR-015

Structure-aware chunking, one strategy per content type

Accepted
DataQuality

Context

Chunking decides what can be retrieved at all, and no single setting fits a policy document, a support article, a spreadsheet and a video transcript. A support article split in half retrieves the symptom without the fix.

Decision

Seven strategies, one per content type, each declaring its unit of meaning, its segmentation rule, the context header that travels with it, and its retrieval representation. Overlap applies to prose only.

Considered and rejected

  • Fixed 512-token splitting with overlap everywhereMeasured 0.06 lower Recall@50, and it duplicates evidence in tables and articles, inflating apparent corroboration.
  • Model-driven semantic chunking of the whole corpusAttractive on quality and it costs a model call for every one of 40 M documents, and it re-runs on every reprocessing.

Consequences

  • +Positive retrievable units correspond to units of meaning, and every chunk carries the version and heading path that make it interpretable alone.
  • −Negative seven strategies to maintain and evaluate, and a new content type is a design task rather than a configuration change.
  • −Negative a chunking change invalidates the whole index, so it is always a shadow build with an alias swap.
Drawn in16 · Knowledge Ingestion Pipeline18 · Chunking and Representation31 · CI/CD and Index Migration
ADR-016

One PostgreSQL for the ledger, the graph, conversation state and traces

Accepted, review scheduled
DataOperations

Context

Four workloads need a transactional store: the document and provenance ledger, the AGE graph, conversation state, and Langfuse traces. Each could justify its own service, and each would bring its own backup, patching, network and on-call story.

Decision

One PostgreSQL Flexible Server with zone-redundant high availability, four schemas, one operational model. The ledger is the only workload with an RPO.

Considered and rejected

  • A managed service per workloadFour services, four failure modes and four bills for a platform whose transactional volume is modest — this is not a scale problem yet.
  • Putting traces in Azure Monitor onlyExcellent for infrastructure signals and poor at the LLM-specific structure — prompts, evidence sets, claim verdicts — that the trace in view 34 depends on.

Consequences

  • +Positive one backup, one patching window, one network path, one set of credentials.
  • −Negative a noisy-neighbour risk between trace writes and ledger reads, contained by separate connection pools and monitored per schema.
  • ·Review split traces out first if write volume or retention pressure grows; the ledger is the workload that must never be moved for convenience.
Drawn in10 · Container Architecture17 · Storage Zones30 · Deployment Architecture
ADR-017

Langfuse self-hosted for LLM tracing, Azure Monitor for everything else

Accepted
ObservabilityOpen source

Context

The question the platform must answer about any wrong answer is which stage was at fault. That needs a trace whose spans carry evidence ids, rank positions, token counts, claim verdicts and prompt versions — structure that generic APM tooling does not model.

Decision

OpenTelemetry instrumentation throughout, with LLM spans to a self-hosted Langfuse on the existing PostgreSQL and infrastructure signals to Azure Monitor, Prometheus and Grafana. Traces store identifiers, never passage text.

Considered and rejected

  • Azure Monitor and Application Insights aloneWould force LLM structure into custom dimensions and lose the evidence-level detail that makes failure classification possible.
  • A commercial LLM observability SaaSWould send prompts and evidence outside the tenant, which conflicts directly with the sensitive-data controls in view 39.

Consequences

  • +Positive one trace id spans all seven stages in view 34, and 90 days of traces is about 60 GB.
  • −Negative another self-hosted component to run and upgrade.
  • −Negative storing identifiers rather than text makes some debugging slower; prompt capture exists but is opt-in and time-boxed, deliberately.
Drawn in10 · Container Architecture34 · Observability and Tracing39 · Sensitive Data and Tenant Isolation
ADR-018

Evaluation is a release gate, built before launch

Accepted
QualityProcess

Context

A generative search system with no golden set cannot distinguish an improvement from a regression, which makes every change after launch a guess. Building evaluation afterwards means the first six months of changes are unmeasured.

Decision

A 1,400-query golden set across six layers, run in the pipeline with hard thresholds that block a release. Retrieval and generation are scored separately. Access safety has a zero-tolerance gate with no waiver path. Cost is gated alongside quality.

Considered and rejected

  • Post-hoc dashboards and manual spot checksDetects regressions after users do, and provides no mechanism to stop one shipping.
  • End-to-end scoring onlyCannot say whether the answer was wrong because the document was never found or because the model ignored it — the distinction the whole set is organised around.

Consequences

  • +Positive a prompt change and a code change take the same path, and a quality regression cannot reach production unnoticed.
  • −Negative about two annotator-days a month to maintain reference answers, plus the compute for a nightly full run.
  • −Negative gates can be gamed by a golden set that drifts towards what the system is already good at, which is why it grows from real failures and is reviewed quarterly.
Drawn in31 · CI/CD and Index Migration32 · Evaluation Harness33 · RAGOps Loop
ADR-019

The classic search API is a separately deployable product

Accepted
AvailabilityProduct

Context

Generative answers depend on a model, a GPU pool, a vector index and a verifier. Every one of those is a dependency the search results themselves do not have, and an availability target that includes all of them is lower than one that does not.

Decision

A ranked-results Search API that deploys with no model, no GPU and no vector dependency, carrying its own 99.95% target against 99.9% for generative answers. It is the floor of the degradation ladder and a product in its own right.

Considered and rejected

  • One endpoint with a generative flagCouples the availability of the simple path to the dependencies of the complex one, which is the opposite of what a degradation ladder needs.
  • Returning an error when generation is unavailableDiscards a working product to protect a feature.

Consequences

  • +Positive the bottom rung of the ladder is a useful product rather than an error page, and the availability difference is priced deliberately.
  • −Negative two surfaces to document, version and keep behaviourally consistent.
Drawn in13 · Integration Surface29 · Degradation Ladder36 · Security Zones
ADR-020

Airflow on AKS for ingestion orchestration

Accepted
DataOpen sourceOperations

Context

Ingestion is six connectors, four parse paths, backfills, per-source retries, priority by freshness class and a 14-hour full rebuild. It needs dependency-aware scheduling, backfill semantics and per-task observability, and it runs on the same cluster as its workers.

Decision

Apache Airflow with the Kubernetes executor on the existing AKS cluster, workers autoscaling from 4 to 60 pods on queue depth.

Considered and rejected

  • Azure Data FactoryStrong for managed data movement and awkward for custom Python parse and embed steps, which is what most of this pipeline is.
  • Azure Logic Apps or Durable FunctionsSuited to event-driven glue rather than to a dependency graph with backfills and per-source priority.

Consequences

  • +Positive backfill and per-task retry come as standard, and the workers run beside the models they call.
  • −Negative Airflow is an operational commitment of its own — an upgrade path, a metadata database and a scheduler to keep healthy.
Drawn in10 · Container Architecture16 · Knowledge Ingestion Pipeline21 · Freshness Lifecycle
ADR-021

A warm secondary region with a dual index build

Accepted
AvailabilityAzureCost

Context

The recovery target is 30 minutes, and a full index rebuild takes 14 hours. Azure AI Search does not replicate an index across regions, so a secondary must be built rather than copied.

Decision

The indexer writes both regions on every ingestion run. PostgreSQL replicates asynchronously to a promotable replica, and AKS in the secondary runs minimal and scales on failover. Search returns first, generative answers second.

Considered and rejected

  • Rebuild on demand in the secondary14 hours against a 30-minute RTO.
  • Active-active across both regionsRoughly doubles run cost for a workload whose availability requirement is 99.9%, and adds a cross-region consistency problem the design does not otherwise have.

Consequences

  • +Positive the failover path is exercised continuously by normal ingestion rather than annually by a test.
  • −Negative a standing cost for an event that may never happen, accepted explicitly against the RTO.
  • −Negative it does not protect against a regional Azure OpenAI capacity event; the gateway holds deployments in two regions, which is a mitigation, not a guarantee.
Drawn in17 · Storage Zones29 · Degradation Ladder30 · Deployment Architecture
ADR-022

Self-hosted GPU for embedding, reranking and verification

Accepted
ModelsCostOperations

Context

Three model tasks run on every query or every chunk. Their cost scales with queries per second and corpus size, not with answer tokens, and their quality requirement is stable — which is the opposite profile to synthesis.

Decision

One AKS GPU pool, always on, for online embedding, reranking and verification; a second preemptible spot pool for batch embedding, so a reindex can never take capacity from the query path.

Considered and rejected

  • Managed per-call endpoints for all threeAbout 3.4 times the cost at this volume, and a per-call dependency on the critical path for tasks that never change.
  • One shared GPU pool for online and batchA reindex would compete with live queries at exactly the moment both matter.

Consequences

  • +Positive about 4,100 USD a month for three always-on model services, and 62% saved on the batch embedding path by using spot.
  • −Negative GPU capacity planning, driver upgrades and spot interruption handling become the team's problem.
  • −Negative a GPU pool outage degrades two rungs of the ladder at once, which is why rungs 2 and 3 are exercised together in game days.
Drawn in15 · Model Portfolio29 · Degradation Ladder30 · Deployment Architecture

Technology Selection, Component by Component

Every capability in the platform, the technology chosen for it, whether that technology is an Azure service or open source, why it won, and what was considered against it. Where a row names an ADR, the reasoning is above in full.

Experience and access

CapabilityChosenWhy it wonConsidered instead
Edge, TLS and WAF AzureAzure Front Door One inbound path for the whole platform, with bot and rate rules applied before anything else runs. Application Gateway alone — no global anycast or edge caching.
API management AzureAzure API Management Per-consumer quota, key management and versioning for four published interfaces, so the intranet widget cannot consume the tenant's token budget. Gateway-level rate limiting in FastAPI — no per-consumer product model.
Search gateway and APIs Open sourceFastAPI on AKS, SSE streaming Streaming, typed contracts and the same language as the retrieval and ingestion code. Azure Functions — cold starts on a latency-sensitive streaming path.
Identity AzureMicrosoft Entra ID, OIDC and on-behalf-of The estate's identity provider, and the source of the group memberships every source system's ACLs are normalised to. See ADR-012. A platform-local authorisation model — a second source of truth for permissions.

Orchestration and models

CapabilityChosenWhy it wonConsidered instead
Answer orchestration Open sourceLangGraph on AKS A bounded state machine whose nodes are also the trace spans. See ADR-007. Semantic Kernel; Azure AI Agent Service; bespoke code.
Model gateway Open sourceLiteLLM proxy One egress point, per-task routing, budgets and token accounting. See ADR-008. Provider SDKs in each service; API Management as the model gateway.
Answer synthesis AzureAzure OpenAI via Azure AI Foundry The one task where quality requirements move faster than we can chase them. See ADR-009. Self-hosted open weights — close on explanatory questions, behind on multi-hop.
Intent, classification and rewriting Open sourceQwen3-8B-Instruct on vLLM A 90 ms task on every query; frontier pricing here is the commonest cost mistake in this pattern. The synthesis model — three times the blended model cost for no measured gain.
Embeddings Open sourceBGE-M3 on Hugging Face TEI Dense and learned sparse from one model, pinned for the life of an index. See ADR-005. Azure OpenAI text-embedding-3-large — 3.4x cost and a version lifecycle we do not control.
Reranking Open sourcebge-reranker-v2-m3 on TEI 0.07 nDCG@10 for 110 ms. See ADR-006. Azure AI Search semantic ranker — retained as the fallback, not tunable against our golden set.
Grounding verification Open sourceNLI cross-encoder, open weights Claim-to-passage alignment from a different model family than the synthesiser. See ADR-014. Content Safety groundedness detection — answer-level, not claim-level; kept as the fallback.
Multimodal representation Azure + open sourceAzure AI Vision embeddings, Whisper Managed cross-modal embeddings at ingest, open-weight transcription where volume makes per-call pricing wrong. Open CLIP or SigLIP throughout — weaker on document figures in evaluation.

Retrieval and storage

CapabilityChosenWhy it wonConsidered instead
Hybrid lexical and vector index AzureAzure AI Search, S3 tier BM25, HNSW, RRF fusion and index-side ACL filters in one managed service at 320 M chunks. See ADR-003. OpenSearch or Elasticsearch self-hosted; Qdrant plus OpenSearch; pgvector.
Knowledge graph Open sourceApache AGE on PostgreSQL Entity expansion and supersession edges with no new operational surface. Provisional — see ADR-010. Neo4j; Cosmos DB Gremlin; no graph at all.
Structured retrieval Azure + open sourceDatabricks SQL over governed Delta tables Figures computed on demand under Unity Catalog grants, never embedded and left to go stale. See ADR-011. Indexing the tables as text; model-authored SQL against a read-only account.
Ledger, provenance, state, traces AzureAzure Database for PostgreSQL Flexible Server Four workloads, one operational model, at a transactional volume that does not yet justify four. See ADR-016. A managed service per workload; traces in Azure Monitor only.
Content lake Azure + open sourceADLS Gen2 with Delta Lake Immutable raw content for 90 days so a reparse never refetches from a throttled source, plus parsed text as Delta. Blob storage without Delta — no schema or time travel on parsed artefacts.
Caching AzureAzure Cache for Redis, premium Exact and semantic answer caching keyed by query plus ACL fingerprint. See ADR-013. In-process caching — cannot be invalidated by principal on an entitlement change.
Eventing and work queues AzureAzure Event Hubs and Service Bus Kafka-protocol change events and per-freshness-class work queues, both zone-redundant. Self-hosted Kafka — an operational commitment with no benefit at this volume.

Knowledge pipeline

CapabilityChosenWhy it wonConsidered instead
Ingestion orchestration Open sourceApache Airflow, Kubernetes executor Dependency-aware scheduling, backfills and per-source priority beside the workers it schedules. See ADR-020. Azure Data Factory; Logic Apps or Durable Functions.
Document parsing Open sourceApache Tika and Unstructured Broad format coverage for native text, and cheap enough to re-run across 40 M documents. Document Intelligence for everything — an order of magnitude more expensive on documents with a text layer.
OCR and layout AzureAzure AI Document Intelligence Reading order, tables and bounding boxes on 26 M scanned pages — the anchors citations depend on. Tesseract — materially worse reading order and no table structure.
PII detection Open sourceMicrosoft Presidio Extensible recognisers for internal identifier patterns, run at ingest so classification is decided once. See view 39. Azure AI Language PII — capable, and a per-call cost across the whole corpus.
Chunking Open sourceCustom, per content type Seven strategies because one setting cannot fit a policy document and a spreadsheet. See ADR-015. Fixed-size splitting; model-driven semantic chunking of the whole corpus.

Assurance and operations

CapabilityChosenWhy it wonConsidered instead
Prompt injection and content safety AzureAzure AI Content Safety, Prompt Shields A managed screen at index and at query time, behind the structural fence that does the real work. See view 38. Regex and heuristic filters alone — a lexical control against a semantic attack.
Policy enforcement Open sourceOpen Policy Agent Source, model and tool policy as versioned rules in git rather than as a document people are asked to remember. Policy in application code — invisible to review and untestable in isolation.
LLM observability Open sourceLangfuse, self-hosted Spans that model evidence ids, rank positions and claim verdicts, inside the tenant. See ADR-017. Application Insights alone; a commercial LLM observability SaaS.
Infrastructure telemetry Azure + open sourceAzure Monitor, Prometheus, Grafana Cluster, GPU and service signals where the operations team already looks. A second stack duplicating what the estate already runs.
Evaluation Open sourceRagas plus IR metrics in Airflow Retrieval and generation scored separately, as a release gate rather than a dashboard. See ADR-018. Azure AI Foundry evaluation — a reasonable managed alternative; rejected to keep gates and golden set in the same pipeline as the code.
Delivery Open sourceGitHub Actions and Argo CD GitOps to AKS, with prompts, index schemas and code passing the same gates. See view 31. Direct pipeline deployment — no declarative cluster state to roll back to.
Infrastructure as code Open sourceTerraform The estate standard, and it covers Azure, Databricks and Kubernetes resources in one workflow. Bicep — Azure-only, and this platform is not Azure-only.
Secrets AzureAzure Key Vault with workload identity No pod holds a long-lived secret; connectors exchange for a source token per run. Kubernetes secrets — long-lived material at rest in the cluster.
Open svg/<view>.svg or drawio/<view>.drawio in draw.io Desktop or at app.diagrams.net to edit. The SVG carries the diagram inside it, so it is both the picture and the source. This folder is self-contained — copy it whole and every link still resolves.
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 41 views · Microsoft Azure and open source41 views · HTML / SVG / draw.io