Enterprise Generative Search — Azure and Open Source  ·  View 35 of 41  ·  Operations

Cost Model

What a query costs, what each line scales with, and which lever actually moves the total.

Editable source SVG draw.io All views
Demand 62,000 queries a day peak 34 per second Class mix from view 04 11% agentic 40 M documents 320 M chunks Where money goes LLM tokens 58% of run cost GPU node pools 19% Azure AI Search 14% Storage, egress, ops 9% What each scales with Tokens: class mix and hops not user count GPU: queries per second plus reindex bursts Search: chunk count and replica count Levers Semantic cache 35% hit, 0 tokens Class routing the largest single saving Rerank depth 50 not 200 Spot GPU for batch embed 62% cheaper Guardrails Budget governor per query, hard stop Per-tenant quota daily token ceiling Cost in the eval gate view 32 Outcome 0.021 USD blended per answered query 0.004 USD lookup 0.180 USD agentic About 39k USD a month at current volume removes the token line Cost Model — What a Query Costs and Which Levers Move It Application we own Data store Decision point Security / platform batch The agentic class is 45 times the cost of a lookup. Routing, not model choice, is what keeps the blended figure where it is. v 1.0 · owner Data and AI Global Practice

The decision

  • Routing, not model choice, is the dominant cost control. The agentic class costs 45 times a lookup, so the classifier in view 04 is worth more to the bill than any negotiation on token price.
  • Embedding, reranking and verification are self-hosted because their cost scales with queries per second rather than with tokens; synthesis is managed because its quality requirement moves faster than we can.
  • Cost is a release gate, not a monthly review. A change that quietly doubles token usage is caught before it ships, in view 32.

Numbers

  • 0.021 USD blended per answered query; 0.004 lookup, 0.180 agentic. About 39,000 USD a month at 62,000 queries a day.
  • Split: LLM tokens 58%, GPU node pools 19%, Azure AI Search 14%, storage and operations 9%.
  • Semantic cache hit rate 35%, which removes the token line entirely for those queries; spot GPU for batch embedding saves 62% on the reindex path.

Assumptions and risks

  • Costs assume reserved PTU capacity for the synthesis model with pay-as-you-go spill at peak. Pure pay-as-you-go raises the blended figure to about 0.028 USD.
  • The cache hit rate is a function of query repetition, which is high in an enterprise and would not hold on an external corpus. It is measured weekly rather than assumed.
◀  Observability and Tracing All views Security Zones  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 41 views · Microsoft Azure and open source35-cost-model