# Architecture One-Pager

*Model Evaluation Service · Solution Architecture v1.0 · Google Cloud · Data & AI Global Practice · 2026-09 · 21 views · 18 architecture decision records*

**A score is a fact about a comparison frame, not about a model. Two scores from different frames are never compared — the incumbent is re-scored instead.**

Every week, a team ships a new version of an AI assistant to 40 million people. A new base model, a rewritten system prompt, a changed tool definition, a different retrieval configuration — each one a bet that the assistant got better, and each one capable of making it quietly worse for a cohort nobody checked. The question the organisation needs answered is narrow and hard: is this candidate better than what users have today, and where is it worse? Answering it requires measuring open-ended quality at a volume no human team can reach, which forces a model into the role of judge; and it requires those measurements to stay comparable for years, across corrections to the corpus, clarifications to the rubric, and upgrades to the judge itself. The second requirement is the one that is usually lost. A platform that loses it does not fail loudly — it keeps producing confident numbers that mean nothing, indefinitely, without ever erroring.

A candidate is a fully pinned configuration, identified by its digest. A run is planned against a comparison frame — dataset version, scorer version, judge version, harness version — resolved before any inference is bought. A harness fans out across GKE, generating outputs with tools executed in a kernel-isolated sandbox that has no route to production, and stores a full trace for every generation. A separate scoring stage reads those traces: deterministic checks, a pinned and calibrated model judge evaluating both orderings of every pair, and a sampled human audit. Scores are appended to BigQuery as immutable facts, each stamped with its frame. The gate engine compares candidate against incumbent within a single frame, per declared slice, with an interval on every delta, and returns PASS, BLOCK or INSUFFICIENT_EVIDENCE to a deployment pipeline that will not release without an affirmative pass. Exposure is staged — shadow, canary, progressive rollout — with online guardrails armed to roll back automatically, and production failures flow back into targeted regression sets so that every incident becomes a permanent test.

## What it is, and what it is not

- **A measurement bound to the conditions that produced it** — not a score attached to a model version, compared across years of silently changing corpora and rubrics.
- **A comparison against the incumbent** — not a benchmark leaderboard position, or a score against an absolute threshold that ages out.
- **A judge under calibration control, with published agreement and a revocable licence to block** — not whichever strong model was available, trusted because its rationales read well.
- **A three-valued verdict where absence of evidence stops a release** — not a pass/fail gate that quietly lets an incomplete run through.
- **A small, declared set of gate-bearing slices with a stated error preference** — not gating on every measurable cohort and discovering the false-block rate later.
- **Safety, latency and cost as independent vetoes** — not a composite quality score in which a large enough helpfulness gain can outvote a safety regression.
- **An untrusted workload executing model-generated tool calls in isolation** — not an internal batch job that happens to call some APIs.
- **Offline as a proxy that must earn its place by predicting online outcomes** — not a growing collection of suites nobody can justify or retire.

## The decisions that are the architecture

1. **A score is bound to its frame** (ADR-01) — The tuple (dataset × scorer × judge × harness) version is part of a score's identity. Cross-frame comparison is a rejected operation, not a discouraged one.
2. **Frame changes re-score the baseline** (ADR-02) — The incumbent is re-scored in the new frame; its old number is never reinterpreted. Gating is suspended until that completes, and the cost is a budgeted line.
3. **Immutable corpus, append-only scores** (ADR-03) — A dataset correction publishes a new version; a score correction supersedes rather than updates. This is what makes a three-year-old number interpretable.
4. **The subject is the whole configuration** (ADR-05) — Model, prompt, tools, retrieval index, decoding and safety filter together. Scoring the model alone confounds a prompt edit with a model upgrade.
5. **Generation is separate from scoring** (ADR-06) — Traces are stored, so a rubric clarification costs a scoring pass rather than a full generation sweep — and a blocked engineer can read what actually happened.
6. **The judge is an instrument under calibration control** (ADR-08) — Pinned version, continuous drift detection against a fixed calibration set, and a published agreement threshold below which it may observe but not block.
7. **Absence of evidence is not a pass** (ADR-11) — INSUFFICIENT_EVIDENCE is a distinct verdict. An incomplete run, an unavailable judge or an unevaluated gated slice stops the release.
8. **Guardrails veto rather than trade** (ADR-13) — Safety, latency and cost block regardless of quality gain, and safety guardrails have no waiver path in code.
9. **Fail closed, in the consumer's contract** (ADR-14) — A pipeline that cannot reach the gate does not release. The property lives where an outage of the platform cannot reach it.
10. **The harness is untrusted** (ADR-18) — It executes text a model wrote, against tools, at volume. Kernel isolation, denied egress and a hard in-harness cost ceiling follow from that classification.

## Why this should still be right in ten years

An evaluation platform outlives the models it was built to judge, because its output is the historical record an organisation uses to know whether it is getting better. These are the properties that should survive a change of model, of cloud, and of the people who built it.

- **The frame rule names no technology.** ADR-01 depends on nothing about BigQuery, Google Cloud, or any judge. It is a statement about what a measurement means, and it survives replacing every component underneath it. It is also the property a future team would most easily discard without noticing, which is why it is enforced by a rejected query rather than a convention.
- **Re-baselining gets cheaper, not more painful.** The main cost of the design is re-scoring the incumbent on every frame change. Inference costs fall and caching improves; the discipline therefore becomes easier to sustain over time rather than harder, which is the right direction for a rule that must not be broken.
- **Judge authority is data the gate reads.** Admissibility lives on the scorer row, not in a policy document. A future team swapping in a judge nobody has calibrated finds it cannot block, without anyone having to remember why.
- **The three-valued verdict outlasts its metrics.** Suites, slices and thresholds will all churn. The distinction between 'worse' and 'unknown' will not, and any future gate built on this platform inherits it for free.
- **Untrusted-by-classification ages well.** As agents gain more tools and more autonomy, treating model output as untrusted input moves from prudent to obvious. This decision will read as unremarkable long before it stops mattering.
- **The named compromise is recoverable.** The erasure-versus-immutability gap in ADR-17 is written down rather than hidden. A future team facing tighter obligations can make a different trade knowingly, which is not possible when a compromise was made silently.

## Non-functional targets

Every number here is a stated assumption from the requirement, chosen to be argued with rather than believed. The right-hand column names the view where the mechanism that meets it is drawn.

| Quality | Target | How it is met | View |
|---|---|---|---|
| Gate decision availability | ≥ 99.9% monthly, in release windows | Control plane across two zones; the verdict API is the only synchronous obligation, and fail-closed lives in the pipeline's contract rather than here. | 16 |
| Smoke tier latency | ≤ 8 min p95, 400 examples | A small subset of the gating suite on the same execution pool at interactive priority — the only tier a human waits on. | 04 |
| Gate suite latency | ≤ 90 min p95, ≤ 150 min p99 | 6,500 examples × 3 generations fanned across 0-600 harness pods, each example independently retryable. | 12 |
| Full regression sweep | ≤ 6 h p95, 42,000 examples | Same pool at lower priority, on Spot capacity with checkpointed resume. | 16 |
| Re-baseline window | ≤ 12 h across all gating suites | Queued ahead of candidate runs on a frame change, so a rubric edit lands inside one working day. | 15 |
| Throughput | 1,200 generations/min steady, 4× burst for 30 min | Pub/Sub work items against an autoscaling node pool, rate-limited against shared serving quota. | 13 |
| Slice query latency | ≤ 3 s p95 over 12 months of scores | BigQuery partitioned by run and clustered by slice; ~55 M rows/year. | 18 |
| Judge agreement | ≥ 0.75 κ to gate; < 0.65 κ inadmissible | Calibration against ≥ 1,200 human labels per rubric, stored on the scorer row and read by the gate engine. | 14 |
| Detection power | 2 pp regression at ≥ 90% power, α = 0.05 | The requirement that sets the 6,500-example suite size, evaluated as a paired comparison against the incumbent. | 12 |
| False-block rate | ≤ 5% of candidates | 34 declared gate-bearing slices rather than every measurable cohort, with human adjudication on every block. | 14 |
| Reproducibility | ± 0.5 pp on ≥ 99% of re-runs | Stored (candidate, frame) re-runs against immutable artefacts; bounded by the tombstone case in ADR-17. | 11 |
| Rollback latency | ≤ 5 min p99 from guardrail breach | Continuous guardrail evaluation against serving telemetry, emitting a rollback trigger without waiting for analysis. | 19 |
| Cost per gate run | ≤ $340 fully loaded; judge ≤ 55% | In-harness ceiling that halts the run, plus per-team monthly budgets enforced at admission. | 18 |
| Evaluation spend | ≤ 2.5% of production inference spend | Suite cost reported against measured offline-online correlation, so an unproductive suite is a visible retirement candidate. | 18 |
| Verdict retention | 7 years, immutable | Append-only verdict records in BigQuery with no update or delete path; traces for blocking runs held 3 years. | 10 |
| Recovery | Scores RPO 0 / RTO ≤ 4 h; run state RPO ≤ 5 min | Managed multi-region stores; in-flight runs resume from checkpoint rather than restarting. | 16 |

## Scope

**In scope**

- Candidate registration with full-configuration identity, and maintenance of the incumbent pointer.
- Versioned, immutable datasets in three classes — frozen golden, rolling production sample, targeted regression — with provenance and slice labels.
- Harness execution with tool sandboxing, trace capture, checkpointed resume and harness-error classification.
- Deterministic scorers, a calibrated and drift-monitored model judge, and the human audit and adjudication paths.
- Frame-bound append-only scoring, and a gate engine that refuses cross-frame comparison.
- Per-slice and guardrail gates, the three-valued verdict, and the waiver register.
- Staged online exposure, guardrail-triggered rollback, and the loop from production failure back into targeted regression sets.

**Explicitly out of scope**

- Training, fine-tuning and serving. The platform measures a candidate; it does not produce or run one.
- Choosing which candidate to build — a research decision this platform informs rather than makes.
- General observability of the assistant in production. Serving telemetry is an input, not a product.
- Experiment design and analysis beyond the declared guardrail and rollout metrics.
- Human rater recruitment, payment and workforce management, which the rater console vendor owns.
- Model safety policy itself. The platform enforces safety gates; it does not decide what is safe.

## What a four-week prototype should prove

Four of this architecture's claims are cheap to test and expensive to be wrong about. A prototype that proves these can be built on; one that skips them is a demonstration of running a benchmark, which is the easy half.

1. Frame enforcement end to end: register a frame, score a candidate, change the rubric, and confirm the platform refuses the comparison and enqueues a re-baseline rather than quietly reusing the old number.
2. Judge calibration and drift: measure κ against 1,200 human labels, then re-score the calibration set daily for four weeks and confirm the monitor would detect a deliberately substituted judge version.
3. Cost and latency at scale: 6,500 examples × 3 generations × 2 judge orderings on the target node pool, with wall-clock p95 and fully-loaded cost measured against the $340 and 90-minute assumptions.
4. Caching honesty: run the same deterministic candidate twice and confirm reproducibility within ±0.5 pp; run a stochastic one twice and confirm the cache correctly misses.

- Kill the judge endpoint mid-run and confirm the run pauses and resumes rather than substituting a different judge or scoring the gap as failure.
- Submit a candidate with a deliberately injected 3-point regression in one language slice and confirm the gate blocks on that slice, names the examples, and routes to adjudication.
- Make the evaluation platform unreachable and confirm the deployment pipeline stops rather than proceeding, timing out to pass, or finding a break-glass path.
- Raise a deletion request for a user whose conversation became an example, and confirm the tombstone path executes and the affected verdict is correctly marked as no longer fully reproducible.

## Open risks, carried rather than hidden

| Risk | If it lands | Response |
|---|---|---|
| Re-baselining proves unaffordable | If a frame change costs more than modelled, teams will avoid changing rubrics and datasets — and a corpus nobody improves is the slow version of the problem the frame rule exists to solve. | Measure re-baseline cost in the prototype. The mitigation is score caching for scorer-only changes (ADR-16), which requires the temperature-zero question to be settled. |
| Gating at temperature zero mis-measures production | The cache, and much of the cost model, assumes gating suites can run deterministically. If deterministic runs do not represent live behaviour, the largest cost lever disappears. | Compare deterministic and stochastic runs of the same candidate across all gate-bearing slices before committing to the cache. |
| Judge agreement is lower than assumed | If no available judge reaches 0.75 κ on the rubrics that matter, the gate has no automated blocking authority and human capacity becomes the binding constraint immediately. | Measure κ first, on the real rubrics. The fallback is narrower rubrics with higher agreement, plus deterministic scorers carrying more of the gating load. |
| False blocks erode trust in the gate | A gate people route around is worse than no gate, because it carries authority it no longer earns. The 34-slice set and the 5% target are modelled, not measured. | Track false-block rate from the first week and treat a breach as a policy defect. Reducing the gate-bearing slice set is the available lever. |
| The erasure gap widens | ADR-17 accepts that a tombstoned example makes a verdict not fully reproducible. Tightening obligations could make tombstoning insufficient and force retroactive corpus changes. | Track what fraction of gating examples are production-derived. If it is small, migrating gating suites to synthesised or licensed material removes the conflict rather than managing it. |

The reasoning behind every component and technology choice is in the [Architecture Decision Record](decision-record): 18 records across 6 areas, each with the alternatives that lost and what the choice costs.