AI Code Completion Service · View 02 of 21 · Context and scope
Decisions
- Admission control sits before inference, not inside it: a request that cannot make its deadline is refused at the door for zero accelerator cost.
- The affinity router is a distinct stage because routing here is placement, not balancing — it asks which replica already holds this session's prefix.
- Safety filtering is on the critical path and budgeted there (p99 ≤ 8 ms), because a filter applied after display is not a filter.
Why the red arrow
- Returning empty is the designed behaviour, not the failure case: under peak load, in a cold region, or with a dependency down, the correct output is a fast silence.
- This is why availability is defined as a displayable-or-explicitly-empty result within deadline, rather than as uptime.
Assumptions
- Time to first displayable token: p50 ≤ 120 ms, p95 ≤ 300 ms, p99 ≤ 500 ms, measured client-side.
- Client hard deadline 2,000 ms, after which the client abandons and any server result is discarded.
