Model Evaluation Service  ·  View 13 of 21  ·  Runtime

Scoring Pipeline

One example, end to end — and the classifier that decides whether a failure belongs to the model or the harness.

Editable source SVG draw.io All views
Work Example Work Item Pub/Sub Cache Probe temp 0 only Generate Assemble Prompt pinned config Candidate Inference Vertex AI Tool Sandbox gVisor Classify Outcome Classifier model vs harness Retry / Quarantine Score Deterministic Checks Judge Pass 2 orderings Human Sample audit fraction Bind Stamp Frame 4-axis tuple Persist Trace Cloud Storage Aggregate Slice Aggregation Score Store BigQuery Model Evaluation Service — Scoring Pipeline Queue / topic Decision point Application we own External / third party Security / platform Interface / broker Data store The outcome classifier is the load-bearing stage: a sandbox timeout or a provider 5xx leaves the pipeline as a harness error and never as a wrong answer. A run whose harness-error rate exceeds its budget cannot yield a PASS. v 1.0 · owner Data & AI Global Practice · date 2026-09

Decisions

  • The outcome classifier is the load-bearing stage. A sandbox timeout or a provider 5xx leaves as a harness error and is never scored as a wrong answer.
  • The cache is probed only for candidates running at temperature zero. A cached score for a stochastic candidate asserts a determinism it does not have.
  • The frame is stamped at bind time, in the same write as the score.

Assumptions

  • 3 generations per example; 2 judge orderings per pairwise comparison to counter position bias (stated assumptions).
  • A run exceeding its declared harness-error budget yields INSUFFICIENT_EVIDENCE.

Open

  • Whether gating suites can run at temperature zero and still represent production behaviour is Core Architecture Question 6, and it decides the platform's largest cost lever.
◀  Take a Candidate Through the Gate All views Scoring by Family  ▶
Solution Architecture v1.0 · Google Cloud · Data & AI Global Practice · 2026-09 · 21 views · 18 architecture decision records13-scoring-pipeline