AI Code Completion Service · View 08 of 21 · Structure
Decisions
- The request plane is stateless on Container Apps and scales on ordinary signals; the inference plane is session-affine on AKS GPU node pools and scales on admission rejections and queue-wait at percentile.
- Two model servers, not one: a fast single-line model and an escalated multi-line model, so the common case does not pay for the rare one.
- Safety filters are their own component with their own budget, so filter latency is attributable and a filter outage is a declared fail-closed rather than a timeout.
Deliberate omissions
- Per-tier Monitor exporters, private endpoints, the Entra token path and the affinity-map write are omitted here and drawn on views 09, 11, 15, 17, 18 and 20.
- The admin console is a caller, not a component — see view 09.
Risks
- Session affinity makes the inference plane the only tier where scale-in is correctness-adjacent: removing a replica evicts live sessions, so autoscaling and node upgrades need a drain protocol rather than a termination grace period.
