Model Evaluation Service · View 14 of 21 · Runtime
Decisions
- The judge is the only lane whose authority is conditional: below 0.65 κ it still scores, but observationally — it cannot block.
- Human adjudication is final on a refusal. A machine may raise a block; a person confirms one.
- The calibration monitor can retire a frame. Silent judge drift is caught here or it is not caught at all.
Assumptions
- Judge-human agreement ≥ 0.75 Cohen's κ to gate, ≥ 0.65 to be admissible at all; calibration set ≥ 1,200 labelled examples per rubric (stated assumptions).
- 40 reviewer-hours per week across audit and adjudication.
Open
- Where human judgment should sit — calibration, adjudication, or continuous audit — is Core Architecture Question 3. This view shows all three, which is the expensive answer.
