Enterprise Generative Search — Azure and Open Source  ·  View 15 of 41  ·  Structure

Model Portfolio

Six model tasks, where each runs, what each costs, and what happens when it is unavailable.

Editable source SVG draw.io All views
Model chosen Runs on Cost and latency Fallback Intent and rewrite Qwen3-8B-Instruct open weights vLLM on AKS GPU 2 x A10, shared 0.0002 USD, 90 ms per query Rules and the raw query no rewrite is safe Embedding BGE-M3 dense and sparse TEI on AKS GPU batch and online 0.00004 USD, 12 ms per query Lexical only retrieval recall drops, still serves Reranking bge-reranker-v2-m3 cross-encoder TEI on AKS GPU top 50 in one batch 0.0006 USD, 110 ms per query Fusion order only quality gate records it Answer synthesis Azure OpenAI, large tier quality-routed Azure AI Foundry PTU plus pay-go spill 0.014 USD, 2.4 s P50 explanatory Smaller tier, then extractive never no answer Grounding verification NLI cross-encoder open weights, fine-tuned TEI on AKS GPU one call per claim 0.0009 USD, 180 ms per answer Content Safety groundedness managed, slower Multimodal Azure AI Vision embeddings plus Whisper for audio Managed plus AKS GPU ingest-time mostly 0.003 USD, 400 ms per query with media Caption text retrieval degraded but useful Model Portfolio — Which Model Does Which Job, and What It Costs Every task names a model and a fallback. The gateway can move a task between them by configuration; no service holds a model name in its code. v 1.0 · owner Data and AI Global Practice

Decisions

  • Six model tasks, six independent choices. Treating the LLM as one decision is what produces a system that pays frontier prices to classify a question.
  • No service holds a model name in its code. The gateway maps task to model, so moving a task between a managed and a self-hosted model is a configuration change with an evaluation run behind it.
  • Every task names a fallback, and every fallback is a degradation the platform can measure rather than an error it returns.

Numbers

  • Per-query model cost, blended across the class mix: 0.0161 USD, of which synthesis is 0.014.
  • Self-hosted embedding, reranking and verification cost about 4,100 USD a month in GPU nodes and would cost roughly 3.4 times that as managed per-call services at this volume.
  • Verification adds 180 ms and 0.0009 USD per answer. That is the measured price of claim-level grounding.

Assumptions and risks

  • Open-weight model licences are reviewed before adoption and re-reviewed on version change; the set assumes permissive licences for BGE-M3 and the reranker.
  • Provider model deprecation is treated as a scheduled event, not an incident: the gateway abstraction plus the evaluation harness is the whole migration plan.
◀  Source Connector Matrix All views Knowledge Ingestion Pipeline  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 41 views · Microsoft Azure and open source15-model-portfolio