Enterprise Generative Search — Azure and Open Source  ·  View 10 of 41  ·  Structure

Container Architecture

The deployable units, their technology, and which of them hold GPUs or credentials.

Editable source SVG draw.io All views
Azure landing zone — AKS, private networking only Answer plane (namespace: answer) Search Gateway FastAPI, SSE, 12 pods Query Understanding Python, 6 pods Planner and Controller LangGraph, 8 pods Answer Assembler Python, 8 pods Verifier NLI, 4 GPU pods Retrieval plane (namespace: retrieve) Retriever Facade Python, 10 pods Structured Retriever SQL guard, 4 pods Graph Retriever openCypher, 4 pods Embed Service TEI, 6 GPU pods Rerank Service TEI, 6 GPU pods Knowledge plane (namespace: ingest) Airflow KubernetesExecutor Connector Workers one per source Parse and OCR Workers Tika, autoscaled Chunk and Embed Workers batch GPU Indexer idempotent upsert Shared control services (namespace: platform) Model Gateway LiteLLM, 6 pods Tool Broker allowlist, 4 pods Policy Engine OPA sidecar Langfuse self-hosted traces flagd retrieval flags Managed data services — private endpoints Retrieval stores Azure AI Search S3, 12 partitions PostgreSQL Flexible metadata and AGE graph Azure Cache for Redis answer and embedding cache Content and events ADLS Gen2 raw and rendered, 12 TB Event Hubs change and telemetry Service Bus ingestion work queues Azure OpenAI synthesis models Document Intelligence layout OCR Content Safety Prompt Shields Databricks SQL governed tables Microsoft Entra ID OBO and groups plan retrieve hybrid query allowlisted SQL routed inference upsert work items Container Architecture — The Deployable Units Application we own Interface / broker Security / platform Data store Queue / topic External / third party synchronous batch event / async GPU pods carry the embedding, reranking and verification models. No pod holds a source credential; connectors use workload identity against Key Vault. v 1.0 · owner Data and AI Global Practice

Decisions

  • Four namespaces — answer, retrieve, ingest, platform — with separate node pools and separate identities. The ingest plane's burst load must never take capacity from the query path.
  • Embedding, reranking and verification run self-hosted on GPU because they are called on every query and their cost scales with traffic, not with tokens. Synthesis is managed because its quality requirement changes faster than we can chase it.
  • No pod holds a source credential. Connectors use workload identity against Key Vault and exchange for a source token per run.

Numbers

  • 64 application pods plus 12 GPU pods steady state; the ingest plane autoscales from 4 to 60 on queue depth.
  • Azure AI Search at S3 with 12 partitions and 3 replicas holds the 320 M chunk index with headroom to about 500 M.
  • PostgreSQL Flexible Server carries the ledger, the graph via Apache AGE, conversation state and Langfuse traces — four workloads, one operated service.

Assumptions and risks

  • One PostgreSQL instance for four workloads is a deliberate consolidation for a platform of this size. It is the first thing to split if the ledger's write rate or the graph's read rate becomes the constraint.
  • GPU capacity in the chosen region is assumed available for A10-class nodes; PTU capacity for Azure OpenAI is assumed reserved before go-live.
◀  Layered Architecture All views Query Understanding  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 41 views · Microsoft Azure and open source10-container-architecture