# Model Evaluation Service

**Solution Architecture v1.0 · Google Cloud · Data & AI Global Practice · 2026-09 · 21 views · 18 architecture decision records**

The system that decides whether the next version of a consumer AI assistant reaches its 40 million users. Every week the product team produces release candidates — a new base model, a rewritten system prompt, a changed tool definition, a different retrieval configuration — and every one of them is a bet that the assistant got better. This platform turns that bet into evidence: it runs each candidate against versioned datasets, scores the outputs with deterministic checks, a calibrated model judge and human reviewers, compares the result against the incumbent per slice, and emits a release verdict a deployment pipeline is contractually bound by. It is built on GKE for the harness fan-out, Cloud Storage for immutable dataset and trace artefacts, BigQuery as the score store, Pub/Sub as the event spine, and Vertex AI plus an external provider for candidate and judge inference.

The design rests on one rule: **a score is a fact about a comparison frame, not about a model.** The frame is the tuple (dataset version × scorer version × judge version × harness version), it is part of a score's identity rather than metadata beside it, and two scores from different frames are never compared — the incumbent is re-scored instead.

The decisions that carry the design:

- **The frame is part of the score.** Cross-frame comparison is a rejected operation, not a discouraged one. This is the single decision every other expensive property follows from (ADR-01).
- **A frame change re-scores the baseline.** The incumbent's old number is never reinterpreted under a new frame, gating is suspended until the re-score lands, and the cost is a budgeted recurring line rather than an exception (ADR-02).
- **Immutable corpus, append-only scores.** A dataset correction publishes a new version; a score correction supersedes rather than updates. This is what makes a three-year-old number interpretable (ADR-03, ADR-04).
- **The subject is the whole configuration.** Model, prompt, tools, retrieval index, decoding and safety filter together — scoring the model alone confounds a prompt edit with a model upgrade (ADR-05).
- **Generation is separate from scoring.** Traces are stored, so a rubric clarification costs a scoring pass rather than a full generation sweep, and a blocked engineer can read what actually happened (ADR-06).
- **The judge is an instrument under calibration control.** Pinned version, continuous drift detection against a fixed calibration set, and a published agreement threshold below which it may observe but not block (ADR-08, ADR-09).
- **Known judge biases are countered structurally.** Both orderings evaluated, self-preference disqualifying, humans blinded — because a directional bias does not average out (ADR-10).
- **Absence of evidence is not a pass.** `INSUFFICIENT_EVIDENCE` is a distinct verdict: an incomplete run, an unavailable judge or an unevaluated gated slice stops the release (ADR-11).
- **Guardrails veto rather than trade.** Safety, latency and cost block regardless of quality gain, and safety guardrails have no waiver path in code (ADR-13).
- **Fail closed, in the consumer's contract.** A pipeline that cannot reach the gate does not release, and that property lives where an outage of this platform cannot reach it (ADR-14).
- **The harness is an untrusted workload.** It executes text a model wrote, against tools, at volume — so kernel isolation, denied egress and a hard in-harness cost ceiling follow from the classification (ADR-18).

The architecture one-pager (including why the design should still hold up in ten years) and the full decision record appear on the landing page of the diagram set, directly below the index of views. The same content is published as [docs/architecture-one-pager.md](docs/architecture-one-pager.md) (~12 min) and [docs/decision-record.md](docs/decision-record.md) (~54 min).

---

## What is here

| Path | Contents |
|---|---|
| `diagrams/index.html` | The landing page: 21 views in seven acts with every format linked, then the **architecture one-pager** and the **decision record** |
| `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning cards, with copy / PNG / PDF export |
| `diagrams/svg/*.svg` | The same views as SVG with the diagram XML embedded; they re-open fully editable in diagrams.net |
| `diagrams/drawio/*.drawio` | draw.io source |
| `docs/architecture-one-pager.md` | The one-pager as markdown |
| `docs/decision-record.md` | The 18 decision records, the capability-to-technology table and the package glossary as markdown |
| `specs/part-a..f.json` | Diagram specifications, the source of truth for every view |
| `specs/manifest-a..b.json` | Acts, page titles, subtitles and reasoning cards |
| `specs/adr-onepager.json`, `specs/adr-records-a..b.json` | The one-pager, the decision records, the capability-to-technology table and the glossary |
| `scripts/build.sh` | Rebuilds every deliverable from the specs (Node only) |
| `ask.md` | The requirement |

## Rebuilding

```bash
bash scripts/build.sh
```

Node 20+ and nothing else — no draw.io Desktop, no browser, no network. The build assembles
`specs/views.json` and `specs/manifest.json` from the authoring parts, then runs generate →
validate → route check → SVG → draw.io → HTML → link check, and finally renders the one-pager
and decision record into `diagrams/index.html` and `docs/`. Edit the parts, never the
assembled files.

Current state: **0 validator errors, 0 validator warnings, 97 edges routed with 0 routing
errors and 0 clutter warnings, 336 icons embedded across 21 views, all 166 relative links
resolve.**

## The seven acts

| # | View | What it answers |
|---|---|---|
| 01 | System Context | What the platform is asked, by whom, and what it deliberately does not own |
| 02 | High-Level Architecture | The six stages every candidate passes through |
| 03 | Actors and Their Core Journeys | Who this is for, and what each of them gets to do |
| 04 | Journey — Taking a Candidate Through the Gate | The engineer's day, and where it goes wrong |
| 05 | Journey — Chasing a Regression Users Already Met | What happens when the gate missed something |
| 06 | Layered Architecture | Six layers, and what may cross between evidence and decision |
| 07 | Container View | The deployable units and the two planes they split across |
| 08 | Integration Architecture | Every inbound and outbound surface, and the one synchronous obligation |
| 09 | Data Flow | From a user's conversation to a gate-bearing example, and back |
| 10 | Data Architecture and Ownership | Four zones with different rules, and where immutability is broken |
| 11 | Core Data Model | Twelve entities, and the foreign key that carries the architecture |
| 12 | Take a Candidate Through the Gate | The critical flow, including the re-baseline most designs treat as an error |
| 13 | Scoring Pipeline | One example end to end, and the model-versus-harness classifier |
| 14 | Scoring by Family | Five lanes, what each costs, and which may stop a release |
| 15 | Frame Change and Re-Baseline | Why a rubric edit suspends gating and spends a day of compute |
| 16 | Deployment Architecture | Where it runs, and the projects the execution plane cannot write to |
| 17 | Release Pipeline | How the platform ships itself, and where the verdict sits in the model's release |
| 18 | Observability and Operations | Six signal families, including measurement quality as a production signal |
| 19 | The Evidence Loop | Six steps back into the corpus, and the one most platforms skip |
| 20 | Security Trust Zones | Five zones drawn around executing model-written tool calls at volume |
| 21 | Identity and Access | Who may change a gate, and the one request the platform must refuse |

## Evidence and limits

This is a design, not a report on a running system. Every rate, latency, volume, retention and
threshold is a **stated assumption** from `ask.md`, invented to be defensible and arguable
rather than absent, and to be replaced by measured telemetry before build. Four numbers in
particular are the first that real data should overturn: the 0.75 κ judge-admissibility
threshold, the 2 percentage-point detectable regression that sets the 6,500-example suite size,
the 5% false-block rate, and the assumption that judge inference is no more than 55% of run
cost.

Two questions are left genuinely open rather than quietly resolved. Whether gating suites can
run at temperature zero without ceasing to represent production behaviour decides the
platform's largest cost lever (ADR-16). And the collision between immutable dataset versions
and a user's right to erasure is managed by tombstoning rather than solved — aggregate
comparability survives, full reproducibility from artefacts does not, and the record says so
(ADR-17).