AI Agent Orchestration Platform  ·  View 08 of 32  ·  2 · Structure

Model Abstraction and Routing

How a request for a capability becomes a call to a specific model, and what happens when that model is unavailable.

Editable source SVG draw.io All views
AI Gateway · Azure API Management Request policy chain Identity Check agent identity Token Rate Limit per tenant · TPM Semantic Cache Redis · vector Egress Guard classification Router Capability Match modality · context Route Policy cost · latency · region Weighted Pool load balanced Circuit Breaker 429 · 5xx trip Response policy chain Token Metric Emit per execution Stream Relay SSE passthrough Fallback Chain next healthy route Model backends Azure AI Foundry Reasoning PTU Pool reserved capacity PAYG Spillover burst · 3 regions Embedding Models batch friendly Alternate providers Foundry Models partner catalogue Open-Weight on AKS KAITO · GPU pool Reranker self-hosted Model Registry control plane Provider Health probe · 30 s Cost Meter Event Hubs · ADX route table backend state primary on trip on trip tokens Model Abstraction — AI Gateway, Routing and Fallback Security / platform Data store Decision point Interface / broker External / third party Application we own event / async synchronous failure / alternate No agent holds a provider key. The gateway presents its own managed identity to every backend. v 1.0 · owner Data & AI Global Practice · date 2026-08

Decisions

  • API Management is the AI gateway: token rate limiting, token metric emission, semantic caching, load-balanced backend pools and circuit breaking are policy, not application code
  • Agents request a capability and a policy class, never a deployment name — that is what makes the platform model-agnostic
  • The gateway holds the provider credentials and presents its own managed identity to each backend

Routing policies supported

  • Cheapest model meeting the quality bar; highest quality for a critical task
  • Region-pinned routing for data residency and classification-aware routing for labelled content
  • Latency-aware selection, with the reserved PTU pool preferred and pay-as-you-go used for burst

Risks

  • Fallback to a different model changes behaviour, not just availability; fallback routes are evaluated against the same golden set as the primary
  • Semantic cache hits can serve a stale answer for a time-sensitive question; cacheability is declared per agent, default off
  • PTU under-utilisation is a real cost, so PTU headroom is a tracked operational metric
◀  Agent Runtime Internals All views Tool Plane and Permissions  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-08 · 32 views · Microsoft Azure08-model-gateway-and-routing