Model Evaluation Service  ·  View 02 of 21  ·  Context and scope

High-Level Architecture

Submit, plan, generate, score, store, decide — the six stages every candidate passes through.

Editable source SVG draw.io All views
Submit Candidate Register config digest Policy Repository suites · gates Plan Frame Resolver 4-axis tuple Run Planner cost ceiling Generate Evaluation Harness GKE · fan-out Tool Sandbox no prod route Score Deterministic Scorers Model Judges pinned · calibrated Human Review Queue Store Score Store BigQuery · append-only Artefact Store Cloud Storage Decide Gate Engine per slice Verdict API fail closed Model Evaluation Service — High-Level Architecture Application we own External / third party Security / platform Interface / broker Data store Decision point Online evaluation closes the loop from serving back into targeted regression sets; see the data flow and lifecycle views. v 1.0 · owner Data & AI Global Practice · date 2026-09

Decisions

  • The frame is resolved at plan time, before any inference is bought, so cost is never spent on a comparison that cannot be made.
  • Generation and scoring are separate stages, so a scorer change re-scores stored outputs instead of re-running the model.
  • The verdict API is the only synchronous obligation during a release window.

Why this shape

  • Splitting generate from score is what makes the score cache viable and what keeps a rubric fix cheap.
  • Putting the cost ceiling in the planner rather than in reporting means a runaway run is halted, not merely noticed.

Risks

  • Judge inference is assumed to be ≤ 55% of run cost; if the judge is larger than assumed, the economics of the whole set change.
◀  System Context All views Actors and Their Core Journeys  ▶
Solution Architecture v1.0 · Google Cloud · Data & AI Global Practice · 2026-09 · 21 views · 18 architecture decision records02-high-level-architecture