Model Evaluation Service · View 18 of 21 · Operations
Decisions
- A judge whose agreement has drifted is an incident with the same shape as a failing dependency, and it retires a frame.
- Block rate and waiver rate are standing metrics. A gate waived more often than it blocks is a policy defect, escalated as one.
- The cost ceiling halts a run rather than alarming after the money is spent.
Targets
- Slice regression query ≤ 3 s p95 over a 12-month window · guardrail breach to rollback ≤ 5 min p99 (stated assumptions).
- Total evaluation spend ≤ 2.5% of monthly production inference spend.
Risks
- Corpus staleness and the contamination canary gap are slow signals. Both can be breached for weeks before anyone looks, which is why they alarm rather than report.
