Enterprise Generative Search — Azure and Open Source · View 02 of 41 · Context and scope
Decisions
- Query planning, retrieval, generation and verification are four separable stages with typed contracts between them. They fail independently, are measured independently, and can be replaced independently.
- The model is invoked twice at most on the common path — once to synthesise, once to verify — and never to decide what it may retrieve.
- Verification can send the request back to the planner. That single backward edge is the whole difference between a RAG demo and a system that can be trusted with a wrong retrieval.
The alternative considered
- A single agent loop with tools, letting the model search, read and answer as it sees fit. Simpler to build, and it was rejected: it makes latency unbounded, cost unpredictable, and the authorisation boundary a matter of prompt wording.
- Chosen instead: the model proposes retrieval, a facade executes it under policy, and the budget governor decides when the loop stops.
Numbers
- Time to first token 1.2 s P50, 2.5 s P95. Complete answer 6 s P95 explanatory, 18 s P95 agentic.
- Retrieval budget 450 ms P95 across all four backends; cross-encoder reranking 110 ms for 50 candidates.
- 99.9% availability for generative answers, 99.95% for the classic search fallback, which has fewer dependencies by design.
