# Architecture Decision Record

*Model Evaluation Service · Solution Architecture v1.0 · Google Cloud · Data & AI Global Practice · 2026-09 · 21 views · 18 architecture decision records*

The argument these decisions serve is summarised in the [Architecture One-Pager](architecture-one-pager).

Eighteen decisions make up this architecture. Everything else across the twenty-one views is convention, sizing or consequence. Each record states the forcing question, the context that makes it hard, what was decided, how it is realised on Google Cloud, the options weighed, what the choice buys and costs, the conditions that would flip it, why it should still be right in ten years, and the lesson that transfers beyond this platform.

> **Status of this document.** This is a design, not a report on a running system. The rates, latencies, volumes, retentions and thresholds are the requirement's stated assumptions for a consumer AI assistant — 40 million monthly active users, 25 release candidates per week, 18 suites over 42,000 examples — invented to be defensible and arguable rather than absent. They are to be replaced by measured telemetry before build, and four of them in particular are the first numbers real data should overturn: the 0.75 κ judge-admissibility threshold, the 2 percentage-point detectable regression that sets the 6,500-example suite size, the 5% false-block rate, and the assumption that judge inference is no more than 55% of run cost.

## How to read a record

- **Question:** The forcing question: why a decision was needed at all.
- **Context:** The requirement, the scale and the constraint that make it hard.
- **Decision:** What this architecture does, stated so it can be checked.
- **How it is realised on Google Cloud:** The concrete mechanism: which service or package, configured how, in which project.
- **Options weighed:** Chosen, rejected, deferred, or right elsewhere, with the reason for each.
- **Consequences:** What the choice buys and what it costs, both kept visible.
- **Choose differently when:** The conditions that would flip the decision for your system.
- **Why it holds up over time:** What keeps the decision right as scale, staff and technology change.
- **Lesson:** The principle that transfers beyond this platform.

## Decision map

**Comparability and the frame**: What a score is a fact about, and why two numbers may not be placed side by side.

- ADR-01 · A score is bound to a comparison frame, and cross-frame comparison is a rejected operation
- ADR-02 · A frame change re-scores the incumbent rather than reinterpreting its stored number
- ADR-03 · Datasets are immutable, versioned artefacts; a correction publishes a new version
- ADR-04 · Scores are append-only facts; a correction supersedes rather than updates

**What is measured**: The subject of an evaluation, and the corpus it is evaluated against.

- ADR-05 · The evaluation subject is the full pinned configuration, not the model
- ADR-06 · Generation and scoring are separate stages joined by a stored trace
- ADR-07 · Three dataset classes: a frozen golden set, a rolling set, and targeted regression sets

**The judge as an instrument**: Using a model to score a model, without pretending the ruler is fixed.

- ADR-08 · The judge is pinned to an explicit version and monitored for silent drift
- ADR-09 · A judge may block a release only above a published agreement threshold
- ADR-10 · Known judge pathologies are countered structurally, not by prompt wording

**The decision**: How evidence becomes a verdict, and what happens when there is not enough of it.

- ADR-11 · The verdict is three-valued: PASS, BLOCK and INSUFFICIENT_EVIDENCE
- ADR-12 · Gates are evaluated per slice against a declared gate-bearing set
- ADR-13 · Guardrail gates block regardless of quality gain, and safety guardrails are never waivable
- ADR-14 · Fail closed is a property of the pipeline's contract, not of the platform's availability

**Offline and online**: The proxy, the truth, and how each earns its authority.

- ADR-15 · Offline gates entry to exposure; online guardrails gate rollout and trigger rollback
- ADR-16 · Suites justify themselves by predicting online outcomes, and cached scores must not assert false determinism

**Data, cost and safety**: User data under an immutable corpus, and running generated code at volume without an unbounded bill.

- ADR-17 · Production examples are redacted at admission, and erasure is a tombstone with bounded reproducibility
- ADR-18 · The harness is an untrusted workload, and every run carries a hard cost ceiling

## Technology by capability

Every capability and what it is realised with, the origin of that choice, the alternative that was weighed, and the record that argues it. The requirement stays vendor-neutral throughout; this table is where the architecture commits to Google Cloud.

| Capability | Choice | Origin | Credible alternative | Why this one | Record |
|---|---|---|---|---|---|
| Score store | BigQuery, EU multi-region, partitioned by run and clustered by slice | Google Cloud | ClickHouse on GKE with object storage | The dominant query is a per-slice comparison across tens of millions of frame-bound rows with a 3-second p95 target; a columnar warehouse with no cluster to operate wins on both the query shape and the operational burden. | ADR-01 |
| Dataset, trace and artefact store | Cloud Storage, dual-region, object versioning on, in a separate project | Google Cloud | A content-addressed store built on a filesystem | Immutability and retention are enforced by the platform rather than by application code, and project separation puts the corpus outside every execution-plane identity's reach. | ADR-03 |
| Run, candidate and frame state | Cloud SQL for PostgreSQL, HA, point-in-time recovery | Google Cloud | Spanner | Single-region transactional state at modest volume with an RPO of 5 minutes; Spanner's global consistency buys nothing here and costs materially more. | ADR-02 |
| Execution fan-out | GKE node pool on Spot capacity with an on-demand floor | Google Cloud | Cloud Run jobs, or Batch | 0-600 pods with checkpointed resume makes re-baselining a natural fit for interruptible capacity, while the on-demand floor protects gate runs inside a release window. | ADR-06 |
| Work distribution and events | Pub/Sub, six topics | Google Cloud | Cloud Tasks, or a queue on Redis | Independently retryable per-example work items with at-least-once delivery, and the same spine carries verdict and drift events to consumers. | ADR-06 |
| Candidate inference | Vertex AI endpoints, pinned model versions | Google Cloud | Direct provider APIs for everything | The primary in-house model is served here already; pinning is first-class and quota is manageable against production serving. | ADR-05 |
| Judge inference | External provider API, pinned version, behind an egress allowlist | Third party | A self-hosted open-weight judge | Judge quality currently justifies the dependency; ADR-08 exists precisely because that dependency cannot be trusted to stay fixed, and ADR-18 keeps the egress path narrow. | ADR-08 |
| Tool sandbox | gVisor-isolated pods, egress denied, side-effect-free tool stubs | Open source on Google Cloud | Plain containers with network policy | The workload executes model-generated tool calls at volume; kernel-level isolation is proportionate to running untrusted generated code. | ADR-18 |
| Score cache | Memorystore, keyed by candidate, example and frame | Google Cloud | No cache | The single largest cost lever, admitted only for deterministic candidates — a cached score for a stochastic candidate asserts a determinism it does not have. | ADR-16 |
| Redaction at admission | Sensitive Data Protection, versioned inspection templates | Google Cloud | A bespoke redaction service | Production-derived examples must be de-identified before anything is durable, and the redaction version is recorded so the treatment applied to an example is itself auditable. | ADR-17 |
| Data perimeter | VPC Service Controls around the corpus, label and trace projects | Google Cloud | IAM alone | IAM governs who may call; a service perimeter governs where data may go, which is the control that survives a leaked service-account key. | ADR-18 |
| Workload identity | Workload Identity Federation, short-lived tokens only | Google Cloud | Service account keys | There is no long-lived credential anywhere in the policy path, so a gate change cannot be made by anything holding an old secret. | ADR-14 |
| Policy source of truth | Git repository with enforced two-person review | Open source | A policy table in Cloud SQL | A gate is a production control; it must be reviewable, diffable and impossible to edit in place, which a database row is not. | ADR-14 |
| Platform CI/CD | Cloud Build to Artifact Registry with signed images | Google Cloud | GitHub Actions with Workload Identity Federation | Keeps the build identity inside the same trust domain as the perimeter, and the synthetic regressed candidate runs as a release gate on the platform itself. | ADR-13 |
| Observability | OpenTelemetry into Cloud Monitoring, quality signals into BigQuery | Open source on Google Cloud | A self-hosted Prometheus and Grafana stack | Operational signals and measurement-quality signals have different shapes: one is time series, the other is a statistic over scores, and each goes where it belongs. | ADR-08 |

## The decisions, and the alternatives that lost

### Comparability and the frame

*What a score is a fact about, and why two numbers may not be placed side by side.*

#### ADR-01 · A score is bound to a comparison frame, and cross-frame comparison is a rejected operation

**Status:** Accepted  ·  **Shown on views:** 06, 11, 12, 15

*Is a score a property of a candidate, or a property of a candidate measured a particular way?*

**Context.** The natural schema stores `candidate-7 scored 0.83`. It makes the dashboard trivial and it is wrong in a way that takes about a year to surface. Datasets get corrected. Rubrics get clarified. A judge is upgraded because the old one could no longer separate two good answers. The harness changes how it truncates context. Each change is individually reasonable and individually small, and each silently redefines what the stored number means. A year later the quality chart shows a smooth upward line assembled from measurements never taken the same way, and nobody can say which improvements were real. This is not a data-hygiene issue: it is the failure mode that makes an evaluation platform actively harmful, because it produces confident numbers that mean nothing and produces them indefinitely without ever erroring.

**Decision.** A comparison frame is the tuple (dataset version × scorer version × judge version × harness version). Every score carries its frame_id as part of its identity, not as metadata beside it. The gate engine and the reporting layer refuse — explicitly, with an error — any request that would place two scores from different frames on the same axis.

**How it is realised on Google Cloud.** `frame` is a first-class table in Cloud SQL with the four axes as columns and a `retired_at`. `frame_id` is a required column on both `run` and `score` in BigQuery, and every comparison query joins through it. The frame resolver runs at plan time, before any inference is purchased, so a run that cannot be compared is never paid for.

| Option | Verdict | Reasoning |
|---|---|---|
| Frame as part of score identity, cross-frame comparison refused | Chosen | Comparability by construction. Costs a re-baseline on every frame change and occasionally forces the platform to tell someone their chart cannot be drawn. |
| Frame as metadata, comparison discouraged by convention | Rejected | Every team intends to check the metadata. None does, at 2 a.m., in a quarterly review deck. A convention that is not enforced is a defect with a documentation page. |
| Normalise old scores into the new frame with a correction factor | Rejected | Requires knowing the mapping between frames, which is exactly what has not been measured. It converts an honest 'cannot compare' into a plausible wrong number. |

**What it buys**

- A number produced today can be honestly placed next to one from eighteen months ago, or the platform says why it cannot.
- Every expensive property of this architecture — immutable datasets, append-only scores, a pinned judge — follows from one rule rather than from a list of best practices.
- A scorer defect becomes visible, because fixing it is a frame change that forces a re-baseline rather than a silent shift in the series.

**What it costs**

- Every frame change costs a full re-scoring of the incumbent, which is a standing budget line rather than an exception.
- The platform will sometimes refuse to draw a chart someone wants, which is an unpopular property that has to be defended repeatedly.

**Choose differently when.** If the organisation genuinely never changes a dataset, rubric, judge or harness — which no organisation does past its first year — the frame collapses to a constant and this machinery is overhead. More realistically: if a cheap, validated mapping between frames could be established empirically, normalisation would become defensible and the refusal could soften into a warning.

**Why it holds up over time.** The rule names no technology. It survives replacing BigQuery, Google Cloud, the judge and the harness, because it is a statement about what a measurement means rather than about where it is stored. It is also the property a future team would most easily throw away without noticing, which is why it is enforced by a rejected query rather than by a paragraph in a README.

> **Lesson.** When a system produces numbers that will be compared over time, the conditions of measurement are part of the measurement. A metric store that cannot say how a number was produced will eventually be a very confident liar.

#### ADR-02 · A frame change re-scores the incumbent rather than reinterpreting its stored number

**Status:** Accepted  ·  **Shown on views:** 12, 15, 17

*When the dataset, rubric, judge or harness changes, what happens to the baseline the next candidate will be compared against?*

**Context.** ADR-01 makes cross-frame comparison illegal, which leaves an operational question: a rubric is clarified on Tuesday, and on Wednesday an engineer submits a candidate. The incumbent has no score in the new frame. The tempting answers are to carry its old number forward — which is precisely the comparison ADR-01 forbids, performed by a different route — or to compare the candidate against an absolute threshold, which answers a different question than 'is this better than what users have'.

**Decision.** The incumbent is re-scored in the new frame before gating resumes. Gating is suspended between frame registration and the completion of that re-score, and the suspension is announced rather than discovered. Re-baselining capacity is budgeted as a recurring cost sized to complete across all gating suites within a declared window.

**How it is realised on Google Cloud.** The frame resolver checks for an incumbent score in the resolved frame at plan time. If it is absent, the run planner enqueues the re-baseline ahead of the candidate's own run on the same execution pool, at higher queue priority, and reports both costs against the change that caused them. Target: all gating suites re-scored within 12 hours.

| Option | Verdict | Reasoning |
|---|---|---|
| Re-score the incumbent, suspend gating until done | Chosen | Honest and expensive. Makes the price of a frame change visible at the moment it is approved. |
| Carry the incumbent's old score forward | Rejected | It is the forbidden cross-frame comparison wearing a different name, and it is undetectable once done. |
| Gate against an absolute threshold while re-baselining | Rejected | Answers 'is this good enough' rather than 'is this better', and thresholds drift out of date faster than baselines do. |
| Lazy re-baseline: score the incumbent only on slices the candidate touched | Deferred | Attractive for cost, and correct only if slices are independent. Worth revisiting once per-slice variance is measured. |

**What it buys**

- A frame change is a priced decision. The quality lead approving a rubric edit sees the compute bill and the gating pause before approving it.
- The baseline is always real rather than inferred, so a gate never blocks on a difference that is partly an artefact of changed measurement.

**What it costs**

- Gating stops for up to a working day after a frame change, which is genuinely disruptive on a week with several.
- Re-baselining consumes capacity that would otherwise evaluate candidates, so frame changes compete with releases.

**Choose differently when.** If score caching proves safe at scale (ADR-16), a re-baseline for a scorer-only change becomes a re-score of stored outputs rather than a re-run of the model, which cuts the cost by roughly the generation share and could remove the gating pause entirely.

**Why it holds up over time.** The rule follows from ADR-01 rather than from any implementation, so it holds as long as that does. What will change is the price: cheaper inference and better caching make re-baselining less painful over time, which strengthens the decision rather than weakening it.

> **Lesson.** A rule that is expensive to honour will be quietly broken unless the cost of honouring it is budgeted in advance. Naming the price is what keeps the rule real.

#### ADR-03 · Datasets are immutable, versioned artefacts; a correction publishes a new version

**Status:** Accepted  ·  **Shown on views:** 09, 10, 11

*When an example is found to be wrong — a bad reference answer, a mislabelled slice — is it fixed in place or superseded?*

**Context.** Fixing in place is obviously convenient and quietly catastrophic under ADR-01: a score references a dataset version, and if that version's content can change, the frame no longer identifies anything. Every score computed before the fix silently becomes a measurement of a corpus that no longer exists, and there is no way to detect that from the data.

**Decision.** A dataset version is immutable once published. Corrections, additions and removals produce a new version with a new identifier. Old versions remain readable indefinitely because scores reference them. Every example carries provenance — source, date, consent state, approver — and slice labels, and a gating dataset must have a named, current owner.

**How it is realised on Google Cloud.** Dataset versions are content-addressed objects in Cloud Storage in a separate `mes-corpus` project, with object versioning on and no delete role granted to any execution-plane identity. The manifest is the dataset's identity; examples are addressed from it.

| Option | Verdict | Reasoning |
|---|---|---|
| Immutable versions, corrections publish anew | Chosen | The only option compatible with ADR-01. Costs storage, which is the cheapest thing in this architecture. |
| Mutable datasets with an audit log of edits | Rejected | The log tells you an edit happened; it does not make the old scores meaningful again. It is forensics, not comparability. |
| Copy-on-write at run time — snapshot whatever the dataset is when a run starts | Rejected | Produces a version per run, which is correct and makes the corpus unmanageable and cross-run comparison nearly impossible. |

**What it buys**

- A three-year-old score still names a corpus that can be read, which is what makes a long quality series defensible.
- Dataset review becomes meaningful: a version is a reviewable artefact rather than a moving target.

**What it costs**

- Storage grows monotonically, and dataset sprawl becomes a real curation burden as versions accumulate.
- It collides head-on with the erasure obligation, which ADR-17 resolves imperfectly and deliberately.

**Choose differently when.** Nothing short of abandoning ADR-01 flips this. If the erasure obligation were to tighten such that tombstoning is insufficient, the corpus would have to move to synthesised or licensed material, which changes what is stored rather than whether it is immutable.

**Why it holds up over time.** Immutability is a property of the data model, not of the storage technology. Object stores have supported versioning and retention for two decades and will continue to; the decision outlives any particular one.

> **Lesson.** Immutability is not tidiness. It is the mechanism by which a measurement stays interpretable after the thing that produced it has been replaced.

#### ADR-04 · Scores are append-only facts; a correction supersedes rather than updates

**Status:** Accepted  ·  **Shown on views:** 10, 11

*When a score turns out to be wrong, is the row updated or is a new row written?*

**Context.** An updatable score row means a past verdict can change after the fact without anyone noticing. Release decisions are audited years later, and a verdict whose supporting evidence has been silently edited is not evidence. There is also a subtler problem: an update loses the information that the platform once believed something different, which is exactly the information needed to understand how a regression escaped.

**Decision.** Scores are append-only. A correction writes a new score row and sets `superseded_by` on the old one. Verdicts are immutable release records retained for the life of the product. No component has update or delete permission on the score table.

**How it is realised on Google Cloud.** BigQuery tables partitioned by run and clustered by slice, written through an append-only service account. Corrections are a distinct write path requiring a reason and an author. Reporting views filter on `superseded_by IS NULL` by default but can reconstruct any historical view.

| Option | Verdict | Reasoning |
|---|---|---|
| Append-only with supersession | Chosen | A past verdict stays reconstructible exactly as it was decided. Costs storage and query complexity. |
| Mutable scores with row-level history | Rejected | Equivalent in principle and worse in practice: the default read is the mutated one, so the correct behaviour requires everyone to opt in. |
| Immutable scores, no correction path at all | Rejected | Purity that forces a full re-run for a single bad score, and pushes people to fix things outside the system. |

**What it buys**

- Recovery of any component can never alter a past verdict, because there is no code path that writes to an existing row.
- The history of what the platform believed is itself data, which is what makes a post-incident review of the gate possible.

**What it costs**

- Every reporting query carries supersession logic, which is a small permanent tax on readability.
- Storage grows with corrections as well as with runs.

**Choose differently when.** If audit obligations disappeared entirely and nobody ever asked why a release was approved, the history would be dead weight. That is not a plausible direction of travel for AI product decisions.

**Why it holds up over time.** Append-only is a design stance rather than a feature of BigQuery; it transfers to any store. As regulatory interest in AI release decisions grows, the reconstructibility this buys becomes more valuable, not less.

> **Lesson.** If a decision will have to be defended later, the evidence behind it must be incapable of changing — not merely unlikely to.

### What is measured

*The subject of an evaluation, and the corpus it is evaluated against.*

#### ADR-05 · The evaluation subject is the full pinned configuration, not the model

**Status:** Accepted  ·  **Shown on views:** 01, 02, 11, 12

*What exactly is being scored when the platform says a candidate improved?*

**Context.** An assistant's behaviour is produced by a model, a system prompt, a set of tool definitions, a retrieval index, decoding parameters and a safety filter. Teams routinely attribute a quality change to 'the new model' when the prompt changed in the same week. If the subject of a score is underspecified, the score is attributed to nothing, and two runs that look like a comparison are measuring different systems.

**Decision.** The unit of evaluation is a release candidate: model identifier and weights digest, system prompt version, tool definitions version, retrieval index version, decoding parameters and safety filter version. A candidate whose configuration is not fully pinned is refused. Candidate identity is the digest of that configuration, so resubmitting the same configuration is recognised rather than re-evaluated.

**How it is realised on Google Cloud.** `candidate.config_digest` is a unique column in Cloud SQL, computed over the canonicalised configuration. The incumbent is a candidate row with `is_incumbent` set, maintained by the deployment pipeline, so the platform always knows what production is actually running.

| Option | Verdict | Reasoning |
|---|---|---|
| Full configuration digest as the subject | Chosen | Attribution is exact and deduplication is free. Costs discipline at submission time. |
| Model version as the subject, other components as tags | Rejected | Reintroduces the confound it was meant to remove, because a tag does not participate in identity. |
| Subject defined per-experiment by the submitter | Rejected | Makes comparability a per-team convention, which is how two teams end up with incomparable numbers and an argument. |

**What it buys**

- A quality change is attributable to a specific change, which is the difference between a measurement and an anecdote.
- Resubmitting an unchanged configuration is a cache hit rather than a repeat bill.

**What it costs**

- Engineers must pin everything before submitting, which is friction at exactly the moment they feel closest to done.
- A configuration surface that grows — a new retrieval component, a new filter — changes the digest for everyone and invalidates deduplication.

**Choose differently when.** If the assistant's architecture collapsed to a single model call with no prompt, tools or retrieval, the configuration and the model would be the same thing and this would be redundant. The direction of travel is the opposite.

**Why it holds up over time.** The principle — score the system, not the component — transfers to any composition of models, agents and tools, and becomes more important as the number of components rises.

> **Lesson.** Measure the thing the user meets. Attributing a system's behaviour to its most interesting component is the most common way a quality programme misleads itself.

#### ADR-06 · Generation and scoring are separate stages joined by a stored trace

**Status:** Accepted  ·  **Shown on views:** 02, 09, 13

*Does a scorer run inline with the model call, or against stored outputs?*

**Context.** Inline scoring is simpler and one fewer store. It also means that fixing a scorer bug, clarifying a rubric or adding a new metric requires re-running the model against the entire corpus — which is the expensive half. Given that scorer and rubric changes are frequent and generation is the dominant cost, this is a choice about how expensive it is to improve the measurement.

**Decision.** The harness generates and stores a full trace — prompt as sent, retrieval context, tool calls, raw output — as an addressable artefact. Scoring is a separate stage that reads traces. A scorer-only frame change re-scores stored outputs rather than re-running the candidate.

**How it is realised on Google Cloud.** Traces are written to Cloud Storage keyed by (run_id, example_id, generation_idx) and referenced from `score.trace_ref`. Scoring workers are a distinct GKE deployment that consumes a separate Pub/Sub subscription, so scoring capacity scales independently of generation capacity.

| Option | Verdict | Reasoning |
|---|---|---|
| Separate stages joined by stored traces | Chosen | Makes rubric iteration cheap and diagnosis possible. Costs ~2.2 TB/year and a second pipeline stage. |
| Inline scoring during generation | Rejected | Cheaper to build, and makes every scorer improvement cost a full generation sweep — which means scorers stop improving. |
| Store outputs only, not full traces | Rejected | Halves storage and removes the ability to tell why a candidate failed, which is what the engineer actually needs from a block. |

**What it buys**

- A rubric clarification costs a scoring pass, not a generation sweep — roughly an order of magnitude less at assumed prices.
- A blocked engineer can read the actual prompt, tool calls and output for the examples that moved, which is what makes a block arguable rather than infuriating.

**What it costs**

- Trace storage at ~2.2 TB/year, and a retention decision that must balance evidence value against cost and privacy exposure.
- Two pipeline stages to operate, monitor and keep in version agreement.

**Choose differently when.** If generation became nearly free relative to storage — a large drop in inference cost with no drop in storage cost — inline scoring with discarded traces would become defensible for the highest-volume suites. The diagnostic argument would still favour traces for gating suites.

**Why it holds up over time.** Separating expensive irreversible work from cheap repeatable work is a general pipeline principle. As models get cheaper this decision weakens economically and strengthens diagnostically, and the diagnostic case is the one that matters to users of the platform.

> **Lesson.** Put the boundary where the cost is. If improving the measurement requires redoing the expensive part, the measurement stops improving.

#### ADR-07 · Three dataset classes: a frozen golden set, a rolling set, and targeted regression sets

**Status:** Accepted  ·  **Shown on views:** 03, 09, 10, 18

*Should the gating corpus be a stable benchmark or a live sample of what users are actually asking?*

**Context.** A frozen golden set is comparable across time, contamination-detectable and stable, and drifts away from real traffic within months — at which point it measures a distribution nobody has. A rolling sample always represents the live distribution and destroys cross-time comparison, because every refresh is a frame change under ADR-01. Neither alone is adequate, and the conflict is genuine rather than a failure to choose.

**Decision.** Three classes with declared roles. The frozen golden set is the gate-bearing corpus and the long comparability axis; changing it requires an explicit reviewed decision. The rolling set is refreshed from recent production traffic on a cadence, is advisory rather than blocking, and exists to detect when the golden set has stopped representing users. Targeted regression sets are grown from real incidents and are blocking for the class of failure they encode.

**How it is realised on Google Cloud.** Each class is a separate dataset lineage in `mes-corpus` with its own owner, cadence and gate strength. A rolling-set refresh publishes a new version; comparisons spanning a refresh are re-scored rather than stitched. Staleness is measured continuously as divergence between the rolling set's distribution and current production traffic.

| Option | Verdict | Reasoning |
|---|---|---|
| Three classes with declared, different gate strengths | Chosen | Keeps the comparability axis stable while retaining a live signal. Costs roughly double the corpus to maintain. |
| Frozen golden set only | Rejected | Comparable and progressively irrelevant. The gate keeps passing candidates that are worse for real users. |
| Rolling set only | Rejected | Always relevant and never comparable; every week is its own frame and no trend can be read. |
| Frozen and rolling with equal blocking authority | Deferred | Leaves open what happens when they disagree, which is the question that actually matters. Deferred until disagreement frequency is measured. |

**What it buys**

- The golden set's drift becomes measurable rather than assumed, because the rolling set is the instrument that detects it.
- An incident can be converted into a permanent blocking test without touching the comparability axis.

**What it costs**

- Roughly twice the corpus curation effort, and three owners rather than one.
- A candidate the golden set likes and the rolling set does not is an unresolved situation the platform reports but does not decide.

**Choose differently when.** If the rolling set's verdicts were shown over a year to track online outcomes substantially better than the golden set's, blocking authority should move to it and the golden set should be demoted to a long-series instrument only.

**Why it holds up over time.** The tension between a stable benchmark and a live distribution is permanent in any measurement programme for a system facing changing demand. The three-class answer will outlive any particular corpus.

> **Lesson.** When two requirements genuinely conflict, name both instruments and declare which one is allowed to stop the line. Choosing one and pretending the other does not matter is how benchmarks become theatre.

### The judge as an instrument

*Using a model to score a model, without pretending the ruler is fixed.*

#### ADR-08 · The judge is pinned to an explicit version and monitored for silent drift

**Status:** Accepted  ·  **Shown on views:** 14, 18, 15

*Which model scores the outputs, and what happens when the provider changes it underneath the pin?*

**Context.** Using a model to judge a model is the only way to score open-ended quality at 39,000 judgements per gate run. But a judge is a measuring instrument, and an instrument whose calibration changes without notice invalidates every measurement taken with it. Provider-side updates to an endpoint that a customer believes is pinned are a documented reality, not a hypothetical. Under ADR-01 such a change is a frame change nobody recorded, which is the worst case the whole design exists to prevent.

**Decision.** The judge is pinned to an explicit model version and decoding configuration and is an axis of the frame. The platform does not assume the pin holds: it continuously re-scores a fixed calibration set with the pinned judge and alarms when the judge's own output distribution or its agreement with human labels shifts beyond a threshold. A detected shift retires the frame and quarantines runs scored inside the drift window.

**How it is realised on Google Cloud.** A scheduled job re-scores the calibration set on a fixed cadence and writes the result to BigQuery as a first-class monitored signal, surfaced on the operations view alongside availability and cost. Judge rationales are stored with every score so a disputed verdict can be inspected rather than merely re-run.

| Option | Verdict | Reasoning |
|---|---|---|
| Pin, and verify the pin continuously | Chosen | Treats the judge as an instrument under calibration control. Costs a standing re-score bill and an alerting path. |
| Pin and trust the provider | Rejected | Makes the platform's central guarantee depend on a third party's change management, undetectably. |
| Self-host the judge so the weights cannot change | Deferred | Removes drift entirely and is the strongest answer. Deferred on cost and on the capability gap against frontier judges; revisit as open-weight judges close it. |

**What it buys**

- Silent instrument drift becomes a detected incident with a defined response, rather than a slow corruption of the series.
- Frame retirement is automatic on detection, so affected scores are quarantined rather than quietly trusted.

**What it costs**

- A continuous re-scoring bill that buys no verdicts, only confidence that the verdicts are comparable.
- False drift alarms interrupt gating, and the threshold that separates real drift from sampling noise is itself a tuning problem.

**Choose differently when.** Self-hosting an open-weight judge of adequate quality removes the drift class entirely and makes the monitor a much cheaper regression check. That is the most likely change to this record within a few years.

**Why it holds up over time.** Calibration control over a measuring instrument is a principle from metrology, not from machine learning, and it long predates and will outlive any specific judge model.

> **Lesson.** If you measure with something you do not control, monitor the something. A pin you cannot verify is a hope with a version number.

#### ADR-09 · A judge may block a release only above a published agreement threshold

**Status:** Accepted  ·  **Shown on views:** 14, 18, 21

*How good does a model judge have to be before it is allowed to stop a release?*

**Context.** A judge with poor agreement against human preference is not measuring quality; it is measuring its own idiosyncrasies at scale and with great confidence. Yet judges are indispensable at these volumes, and a rule that admits only perfect judges admits none. The question is therefore not whether to use one but what authority to grant it, and at what evidentiary standard.

**Decision.** Every judge is calibrated against human labels on a named calibration set before it may gate. Agreement is published as a named statistic, re-measured for every judge version. A judge at or above 0.75 Cohen's κ may block; between 0.65 and 0.75 it may block only with human adjudication; below 0.65 it is inadmissible for gating and may produce observational scores only. The threshold is a property of the scorer row, read by the gate, not a convention people remember.

**How it is realised on Google Cloud.** `scorer.kappa_vs_human` and `scorer.gate_admissible` are columns the gate engine reads at decision time. The calibration set holds at least 1,200 human-labelled examples per rubric and is refreshed at least every six months. Raters see blinded pairs and never learn which output came from the candidate.

| Option | Verdict | Reasoning |
|---|---|---|
| Published κ threshold, admissibility as data | Chosen | Makes judge authority auditable and revocable. Costs a standing human labelling programme. |
| Use the judge, report agreement, let teams decide | Rejected | Authority without a standard. In practice the judge blocks and nobody checks the statistic. |
| Human-only gating | Rejected | Correct and unaffordable: 39,000 judgements per gate run against 40 reviewer-hours per week. |
| Ensemble of judges with disagreement routed to humans | Deferred | Raises reliability and multiplies the dominant cost. Phase 3, once single-judge agreement is characterised. |

**What it buys**

- The judge's authority is earned, measured and revocable, which is what makes an automated block defensible to the engineer it stops.
- A degraded judge loses blocking rights automatically rather than after someone notices.

**What it costs**

- A permanent human labelling programme is required whether or not anything is going wrong.
- κ is an imperfect summary of agreement and can mask systematic disagreement on the slices that matter most.

**Choose differently when.** If judges reached agreement levels comparable to inter-rater agreement among humans themselves, the threshold would become a formality and the calibration programme could shrink to periodic spot checks. Conversely, evidence that κ hides slice-level disagreement would push toward per-slice admissibility.

**Why it holds up over time.** Requiring an automated decision-maker to demonstrate agreement with human judgment before it is granted authority is a governance pattern that will apply to far more than evaluation judges.

> **Lesson.** Grant authority in proportion to demonstrated accuracy, and store the demonstration where the enforcement code can read it.

#### ADR-10 · Known judge pathologies are countered structurally, not by prompt wording

**Status:** Accepted  ·  **Shown on views:** 13, 14

*How are position bias, verbosity bias and self-preference handled?*

**Context.** Model judges exhibit well-documented systematic biases: preferring the first option presented, preferring longer answers, and preferring outputs from their own model family. These are not noise that averages out — they are directional errors that survive large samples, and a rubric instructing the judge to ignore length does not reliably make it do so. If a candidate happens to be more verbose than the incumbent, an uncorrected judge will report an improvement that does not exist.

**Decision.** Position bias is countered by evaluating both orderings of every pairwise comparison and combining. Verbosity bias is measured and reported as a standing statistic rather than assumed away. Self-preference is treated as disqualifying: a judge may not gate a candidate from its own model family. Human raters always see blinded pairs.

**How it is realised on Google Cloud.** The scoring pipeline issues two judge calls per pairwise comparison with the order swapped — which is why the gate run costs ~39,000 judge calls for 6,500 examples. Family affinity between judge and candidate is checked at plan time and blocks admissibility rather than warning.

| Option | Verdict | Reasoning |
|---|---|---|
| Structural controls: both orderings, family exclusion, blinding | Chosen | Doubles judge cost, which is the dominant cost. Bought because the bias is directional and would otherwise be indistinguishable from a real gain. |
| Instruct the judge to be unbiased in the rubric | Rejected | Cheap, popular and not reliably effective. It documents an intention rather than enforcing a control. |
| Correct statistically for measured bias | Rejected | Requires a stable bias estimate, and the bias moves with the judge version — which is the thing ADR-08 says not to assume. |

**What it buys**

- A verbosity-driven apparent improvement is caught rather than shipped, which is one of the most common false positives in LLM evaluation.
- A judge from the same family as a candidate cannot quietly favour it.

**What it costs**

- Judge cost doubles for every pairwise comparison, and judge inference is assumed to be ≤ 55% of run cost.
- Family exclusion reduces the pool of admissible judges, which can be limiting when the best available judge shares a family with the primary model.

**Choose differently when.** If a judge were demonstrated to be order- and length-invariant across versions, single-ordering evaluation would halve the dominant cost. That demonstration would have to be repeated per version, which is what makes it unattractive today.

**Why it holds up over time.** The specific pathologies may change as judges improve, but the principle — counter a known systematic bias structurally rather than by asking nicely — applies to any automated evaluator.

> **Lesson.** A directional bias does not average out with more samples. Design the measurement so the bias cancels, or measure and report it; never instruct it away.

### The decision

*How evidence becomes a verdict, and what happens when there is not enough of it.*

#### ADR-11 · The verdict is three-valued: PASS, BLOCK and INSUFFICIENT_EVIDENCE

**Status:** Accepted  ·  **Shown on views:** 08, 12, 17

*What does the platform return when a run did not finish, or finished with too many harness errors to trust?*

**Context.** A two-valued gate forces every incomplete run into one of two wrong answers. Returning PASS ships a candidate nobody measured. Returning BLOCK is safer but indistinguishable from a real regression, which trains engineers to retry until green — and a gate people retry past is not a gate. The distinction between 'this is worse' and 'we do not know' is the one the deployment pipeline most needs.

**Decision.** Three values. INSUFFICIENT_EVIDENCE is returned when a run is incomplete, when the harness error rate exceeds its declared budget, when a required judge was unavailable, or when a gate-bearing slice went unevaluated. The pipeline treats it as a stop, not as a retryable failure, and the verdict names what is missing.

**How it is realised on Google Cloud.** The gate engine evaluates completeness before it evaluates deltas. `run.harness_error_rate` is persisted by the harness and read as an admissibility precondition. Every verdict ships with the evidence it rests on: suites run, examples scored, slices evaluated, scorer and judge versions, and a confidence interval on every reported delta.

| Option | Verdict | Reasoning |
|---|---|---|
| Three-valued verdict with named gaps | Chosen | Makes 'we do not know' a first-class answer. Costs a third code path in every consumer. |
| Two-valued, incomplete runs block | Rejected | Conflates a regression with an outage, and teaches engineers that blocks are often spurious. |
| Two-valued, incomplete runs pass with a warning | Rejected | A warning in a pipeline log is a pass. This is how an unevaluated candidate reaches users. |

**What it buys**

- An infrastructure problem is diagnosed as one instead of being argued about as a quality regression.
- The block rate stays meaningful as a metric, because it is not inflated by harness flakiness.

**What it costs**

- Every consumer of the verdict must handle a third case, and the temptation to map it onto one of the other two is permanent.
- A platform having a bad day stops releases rather than degrading, which is deliberate and occasionally very unpopular.

**Choose differently when.** Nothing reasonable flips this. If harness reliability became high enough that incomplete runs effectively never occurred, the third value would be unused rather than wrong — and would still be the correct thing to return on the day it recurs.

**Why it holds up over time.** Distinguishing a negative result from an absent one is a principle of measurement rather than of software, and it will apply to any automated gate in any domain.

> **Lesson.** Absence of evidence must be representable, or it will be silently encoded as whichever answer is cheaper that day.

#### ADR-12 · Gates are evaluated per slice against a declared gate-bearing set

**Status:** Accepted  ·  **Shown on views:** 11, 12, 18

*How does the platform gate on cohorts without blocking most neutral releases by chance?*

**Context.** A headline average hides a collapse in one language behind gains everywhere else, so per-slice gating is necessary. But 34 gate-bearing slices tested independently at α = 0.05 will produce at least one false block in roughly four out of five fully-neutral releases. Correcting for multiple comparisons restores honesty and destroys sensitivity on small slices. Growing slices to restore power multiplies the cost of every run. Every available answer is a trade, and the design encodes a preference between a false block and a missed regression whether or not anyone states it.

**Decision.** The gate-bearing slice set is declared in advance and kept small — 34 slices against 18 suites — and each is chosen because it carries a product commitment rather than because it is measurable. Deltas are evaluated as paired comparisons against the incumbent on the same examples, which recovers substantial power. No gate blocks on a difference it cannot distinguish from noise at the declared confidence level. The preference between false blocks and missed regressions is stated explicitly: false-block rate ≤ 5%.

**How it is realised on Google Cloud.** `gate` rows carry slice, strength and alpha, so the trade is data rather than code. Suite size is derived from a power requirement — detect a 2 percentage-point regression at ≥ 90% power, α = 0.05 — which is what sets the 6,500-example gating suite rather than convenience.

| Option | Verdict | Reasoning |
|---|---|---|
| Small declared slice set, paired design, stated error preference | Chosen | Honest about the trade and bounded in cost. A regression in an ungated slice still ships, which is the accepted risk. |
| Gate every slice with multiple-comparison correction | Rejected | Statistically clean and practically blind: after correction, small slices detect nothing worth detecting. |
| Gate on the aggregate only | Rejected | Cheap, sensitive and exactly the failure the platform exists to prevent. |
| Grow every slice until all can be gated at full power | Deferred | Correct and multiplies run cost several-fold. Revisit if inference cost falls substantially. |

**What it buys**

- The multiple-comparisons cost is bounded and known rather than accumulating invisibly as teams add slices.
- Suite size is derived from a stated detection requirement, so 'is the suite big enough' has an answer.

**What it costs**

- A regression confined to an ungated slice ships. That is a real, accepted exposure and it should be reviewed as the slice set changes.
- The paired design constrains how datasets may be refreshed, since pairing requires the same examples on both sides.

**Choose differently when.** If the false-block rate proved much lower than modelled, or if run cost fell enough to grow every slice, the gate-bearing set could widen substantially. Evidence that regressions routinely land in ungated slices would force the same change at higher cost.

**Why it holds up over time.** Multiple comparisons is a permanent property of testing many hypotheses at once. The specific slice set will churn; the discipline of declaring it in advance will not.

> **Lesson.** Any system that gates on many metrics has already chosen between false alarms and missed faults. Make the choice explicit and numeric, or it is made by accident.

#### ADR-13 · Guardrail gates block regardless of quality gain, and safety guardrails are never waivable

**Status:** Accepted  ·  **Shown on views:** 12, 18, 21

*Can a large enough improvement in helpfulness justify a small regression in safety, latency or cost?*

**Context.** Any scalar quality score will eventually be improved by a candidate that is also less safe, slower or more expensive, and a single composite metric will happily trade one for the other. A candidate that is wittier and less safe is not a better candidate. Separately, waivers are necessary — a genuine emergency, a known-false positive — and a waiver mechanism that applies to everything is an off switch for the gate.

**Decision.** Guardrail gates — safety violation rate, benign refusal rate, p95 response latency, cost per request, tool-call error rate — block independently of the quality result and are never traded against it. Every other blocking gate may be waived by a named approver with a stated reason and an expiry. Safety guardrail gates have no waiver path at all.

**How it is realised on Google Cloud.** Guardrails are evaluated as a separate pass with their own thresholds before quality deltas are considered. The waiver API refuses a waiver against a safety gate as a matter of code rather than of permission, so there is no role that can grant one. Active waivers appear on a standing register, and waiver rate is a monitored metric.

| Option | Verdict | Reasoning |
|---|---|---|
| Independent guardrails, safety non-waivable | Chosen | Removes the possibility of trading safety for helpfulness. Costs genuine flexibility in a real emergency. |
| Composite score with weighted safety term | Rejected | A sufficiently large helpfulness gain outvotes safety, which is the exact outcome to prevent. |
| Safety waivable by a senior approver | Rejected | Creates a role whose existence is a standing target, and a precedent that is used a second time more easily than a first. |

**What it buys**

- There is no code path by which a safety regression reaches users because something else improved a lot.
- The waiver register makes quality debt visible and expiring rather than permanent.

**What it costs**

- A false positive on a safety gate stops a release with no override, which will at some point be the wrong outcome for a real incident.
- That pressure moves onto the safety scorer's accuracy, which becomes a critical component in its own right.

**Choose differently when.** None on the safety rule. The non-safety guardrails could reasonably become tunable per release class — a latency regression may be acceptable for a capability release and not for a routine one — which is a refinement rather than a reversal.

**Why it holds up over time.** Separating vetoes from trade-offs is a governance pattern older than software. As AI safety expectations become regulatory rather than voluntary, a non-waivable path becomes an asset rather than an inconvenience.

> **Lesson.** If a constraint must never be traded away, it cannot participate in a scalar score, and it cannot have an override — because a mechanism that exists will be used.

#### ADR-14 · Fail closed is a property of the pipeline's contract, not of the platform's availability

**Status:** Accepted  ·  **Shown on views:** 08, 17, 21

*What happens to releases when the evaluation platform is unavailable during a release window?*

**Context.** The obvious design makes the platform highly available and treats an outage as a degraded mode. But availability engineering cannot reach 100%, and the behaviour on the worst day is what the whole gate is worth. A gate with a bypass is a gate that will be bypassed on the day it matters most — typically during the incident that makes someone want to ship a fix quickly. Shipping an unevaluated candidate to 40 million users is a worse outcome than a delayed release.

**Decision.** The deployment pipeline will not proceed without an affirmative PASS. A pipeline that cannot reach the gate does not release. There is no bypass, no timeout-to-proceed and no break-glass release path in the platform. The gate is also protected as a production control: author and approver of a gate change must be different people, and no long-lived credential exists anywhere in the policy path.

**How it is realised on Google Cloud.** Fail-closed is implemented in the pipeline's contract rather than in the evaluation service, so no outage of the service can change it. Policy lives in Git with enforced two-person review; Cloud Build authenticates through Workload Identity Federation for short-lived tokens only; every gate change and waiver is written immutably.

| Option | Verdict | Reasoning |
|---|---|---|
| Fail closed in the consumer's contract | Chosen | The property survives any failure of the platform, including a total one. Costs release availability. |
| Fail closed implemented inside the evaluation service | Rejected | A service that is down cannot enforce anything. The guarantee has to live where it still holds when the platform does not. |
| Timeout-to-proceed after a bounded wait | Rejected | Converts an availability incident into an unevaluated release, quietly and on a schedule. |
| Break-glass release with post-hoc evaluation | Rejected | The mechanism will be used during exactly the incidents where judgement is worst. |

**What it buys**

- The worst-case behaviour is a delayed release rather than an unmeasured one, and that property does not depend on the platform being up.
- Gate integrity is enforced by review and short-lived identity rather than by trust in operators.

**What it costs**

- Evaluation platform availability becomes release availability, which raises the operational stakes of a system that is otherwise off the user path.
- A genuine emergency fix waits for the gate, and there is no sanctioned way around that.

**Choose differently when.** A strong argument could be made for a narrow exception covering rollbacks to a previously-passed incumbent, which is a release to a candidate that already has a verdict. That is a refinement worth considering and is deliberately not in the MVP.

**Why it holds up over time.** Placing a safety property in the consumer's contract rather than the provider's implementation is a general availability principle: a guarantee that depends on a component being up is not a guarantee.

> **Lesson.** Decide what happens on the worst day, and implement it somewhere the worst day cannot reach.

### Offline and online

*The proxy, the truth, and how each earns its authority.*

#### ADR-15 · Offline gates entry to exposure; online guardrails gate rollout and trigger rollback

**Status:** Accepted  ·  **Shown on views:** 05, 17, 19

*Does online evidence stop a release, or only observe one?*

**Context.** Offline scores are fast, safe and measure a proxy whose predictive validity is itself uncertain. Online behaviour is what actually matters and is slow, confounded by concurrent changes, and requires exposing real users to the candidate before it is cleared — which is the thing the gate exists to avoid. Choosing either alone gives up something the platform was built for.

**Decision.** Staged exposure. Offline gates entry to shadow and canary. Online guardrail metrics are monitored continuously during exposure and trigger automatic rollback on a declared threshold breach, without waiting for a scheduled analysis. Online evidence gates progression to full rollout. Metrics and thresholds are declared before exposure begins, never chosen after the data arrives.

**How it is realised on Google Cloud.** Guardrail evaluation runs against serving telemetry in BigQuery and emits a rollback trigger to the deployment pipeline; breach to rollback initiation is targeted at ≤ 5 minutes p99. User feedback signals — explicit ratings, regeneration rate, abandonment, copy-and-edit behaviour — are treated as biased signals to be interpreted, never as labels.

| Option | Verdict | Reasoning |
|---|---|---|
| Offline gates entry, online gates rollout, auto-rollback armed | Chosen | Bounds the exposure of a bad candidate while keeping the decision anchored in real behaviour. |
| Offline gating only | Rejected | Ships on a proxy and discovers the truth from users, which is the journey in view 05. |
| Online gating only | Rejected | Every candidate reaches users before it is cleared, which makes the gate a formality. |
| Thresholds chosen after inspecting canary data | Rejected | Guarantees a favourable reading, because the threshold is fitted to the result. |

**What it buys**

- A bad candidate's exposure is bounded by canary size and rollback latency rather than by how quickly someone notices.
- Offline suites acquire an external validity check, which ADR-16 turns into a retirement criterion.

**What it costs**

- Real users meet candidates that offline evaluation cleared but online evidence will reject — a deliberate, bounded exposure.
- Canary size must be large enough to detect the regressions that matter, which is a statistical requirement in tension with limiting exposure.

**Choose differently when.** If measured offline-online correlation became strong enough across all gate-bearing slices, the canary could shrink materially. Persistently weak correlation would push blocking authority toward online evidence and reduce offline gating to a smoke test.

**Why it holds up over time.** Proxy-then-truth staging is how every measurement programme with an expensive ground truth works, from drug trials to search ranking. It will outlive any particular metric set.

> **Lesson.** Declare the decision rule before seeing the data. A threshold chosen afterwards is a description of the result, not a test of it.

#### ADR-16 · Suites justify themselves by predicting online outcomes, and cached scores must not assert false determinism

**Status:** Accepted  ·  **Shown on views:** 13, 18, 19

*How does the platform know a suite is worth running, and when may a score be reused rather than recomputed?*

**Context.** Offline suites accumulate. Each was added for a good reason, each costs money on every run, and nothing ever removes one — so cost grows monotonically while the marginal information from the eighteenth suite is unknown. Separately, caching by (candidate, example, frame) is the single largest cost lever available and would make re-baselining nearly free. But a cached score for a candidate sampling at non-zero temperature asserts a determinism the candidate does not have, and a cached judge score assumes the pinned judge has not silently moved.

**Decision.** Offline-online correlation is measured per suite — the degree to which a suite's offline delta predicted the online outcome — and reported against that suite's cost. A suite that has never predicted anything is a visible candidate for retirement. Score caching is permitted only for candidates running at temperature zero, and only within a frame whose judge-drift monitor is current. Every other cache read is a miss.

**How it is realised on Google Cloud.** Correlation is computed in BigQuery over the verdict and online-outcome history and stored on `suite.online_correlation`. The cache probe in the scoring pipeline checks the candidate's decoding configuration before consulting Memorystore, so the determinism condition is enforced in the pipeline rather than trusted.

| Option | Verdict | Reasoning |
|---|---|---|
| Correlation-driven retirement, determinism-gated caching | Chosen | Keeps cost tied to demonstrated value and keeps the cache honest. Costs a slow feedback loop and forgoes caching on stochastic suites. |
| Cache everything by content hash | Rejected | Large saving bought by asserting a determinism that does not exist, which silently narrows measured variance to zero. |
| No cache at all | Rejected | Safe and forfeits the only lever large enough to make frequent re-baselining affordable. |
| Retire suites on a fixed schedule | Rejected | Removes suites by age rather than by value, and the oldest suite may be the one that works. |

**What it buys**

- The corpus can shrink as well as grow, which is the only defence against unbounded evaluation cost.
- Re-baselining a scorer-only change becomes substantially cheaper wherever suites can honestly run at temperature zero.

**What it costs**

- Correlation needs many releases before it is meaningful, so retirement decisions lag suite addition by a long way.
- Running gating suites at temperature zero may not represent production behaviour, which is an open question the cache depends on.

**Choose differently when.** If gating at temperature zero were shown to mis-measure production behaviour, caching would be confined to deterministic scorers over stored traces — still useful, much smaller. If variance across generations proved negligible, caching could widen considerably.

**Why it holds up over time.** Requiring a measurement to demonstrate predictive value, and requiring a cache to state what it assumes, are both durable disciplines independent of the technology underneath.

> **Lesson.** A cache is a claim that nothing relevant changed. Write down what 'relevant' means, and enforce it, or the cache is quietly fabricating data.

### Data, cost and safety

*User data under an immutable corpus, and running generated code at volume without an unbounded bill.*

#### ADR-17 · Production examples are redacted at admission, and erasure is a tombstone with bounded reproducibility

**Status:** Accepted  ·  **Shown on views:** 09, 10, 11, 20

*How can dataset versions be immutable while a user retains the right to have their data erased?*

**Context.** ADR-03 requires dataset versions never to change; privacy obligations require a user's data to be erasable wherever it exists. These are in direct conflict. Deleting the example mutates a version and silently changes every score computed from it. Keeping it honours comparability and breaches the obligation. Tombstoning preserves aggregate comparability but makes the verdict no longer fully reproducible from artefacts, which contradicts the reproducibility claim in the requirement.

**Decision.** Production-derived content is redacted or pseudonymised at admission, before the example is durable, and the redaction version is recorded. Every example carries its lawful basis, consent state and retention limit. An erasure request tombstones the example: content is removed; identity, slice labels and derived scores are retained. The consequence is stated openly rather than hidden — aggregate comparability survives, full reproducibility from artefacts does not, and this remains an open question rather than a solved one.

**How it is realised on Google Cloud.** Sensitive Data Protection applies versioned inspection templates at the admission stage of the data flow. `example.tombstoned_at` makes the state first-class and auditable. The corpus sits in its own project inside a VPC Service Controls perimeter, with every raw trace access logged and attributable.

| Option | Verdict | Reasoning |
|---|---|---|
| Redact at admission, tombstone on erasure, state the limit | Chosen | Honours the obligation and bounds the damage to the comparability claim. The residual gap is real and named. |
| Hard delete the example and its scores | Rejected | Mutates a dataset version and silently changes historical aggregates, which is worse than a bounded gap. |
| Refuse production-derived content entirely; synthesise or license everything | Deferred | Resolves the conflict completely and gives up the rolling set's representativeness. Genuinely worth revisiting if obligations tighten. |
| Keep the example and rely on the lawful basis | Rejected | Bets the platform's legal position on an interpretation, for a comparability benefit. |

**What it buys**

- There is no path by which raw production content becomes a durable dataset example, which shrinks the exposure to what redaction misses.
- Erasure is a first-class auditable state rather than a deletion nobody can account for.

**What it costs**

- A verdict resting on a tombstoned example cannot be fully reconstructed from artefacts, weakening the reproducibility claim in ADR-04.
- Redaction is imperfect, and an over-redacted example may no longer be a valid test of the behaviour it was collected for.

**Choose differently when.** If synthesised or licensed material were shown to carry the gating load without losing representativeness, refusing production-derived content entirely becomes the better architecture and removes this conflict rather than managing it.

**Why it holds up over time.** The tension between immutable evidence and a right to erasure is structural and will intensify. Naming the residual gap is what lets a future team make a different trade knowingly.

> **Lesson.** When two requirements genuinely cannot both be satisfied, implement the lesser compromise and document precisely what was given up. An unstated compromise is discovered later, by someone who assumed it did not exist.

#### ADR-18 · The harness is an untrusted workload, and every run carries a hard cost ceiling

**Status:** Accepted  ·  **Shown on views:** 07, 13, 16, 20

*How should a system that executes model-generated tool calls thousands of times an hour be contained, and what stops a misconfigured run spending without limit?*

**Context.** The evaluation harness does something unusual: it takes text a model wrote and executes it against tools, at volume, unsupervised, including inputs specifically designed to probe edge cases. Treating that as an internal batch job is a category error — it is closer to running untrusted user-submitted code. Separately, a fan-out pipeline with a bug in its work planner can generate an unbounded bill faster than any alerting loop can respond, and an alarm after the money is spent is not a control.

**Decision.** Model output and its tool calls are untrusted content. Tool execution happens in gVisor-isolated pods with egress denied and side-effect-free stubs, on a network path with no route to production systems. All outbound inference leaves through an allowlisting egress proxy. Separately, every run carries a cost ceiling enforced by the harness itself: the run halts at the ceiling and reports, rather than alarming afterwards.

**How it is realised on Google Cloud.** Sandbox pods run under gVisor with egress deny by default; the execution plane holds workload identity that can read the corpus and write traces, and cannot read a human label, write a verdict or delete a dataset version. Per-run cost is metered in-process against the ceiling set by the run planner; per-team monthly budgets are enforced at admission.

| Option | Verdict | Reasoning |
|---|---|---|
| Kernel-isolated sandbox, egress allowlist, in-harness cost ceiling | Chosen | Proportionate to running generated code at volume. Costs some performance and a more complex execution environment. |
| Plain containers with network policy | Rejected | Adequate for trusted workloads. This one executes text an adversary could have influenced through a prompt. |
| Cost alerting rather than a ceiling | Rejected | An alert fires after the spend. The ceiling is the only control that acts before it. |
| Real tools instead of stubs, for fidelity | Rejected | Higher fidelity bought by giving model-generated calls real side effects. Not a trade worth making. |

**What it buys**

- A prompt-injected tool call cannot reach an arbitrary host or produce a real side effect.
- A runaway run is halted rather than discovered on an invoice, which makes experimentation safe to encourage.

**What it costs**

- gVisor adds syscall overhead to a workload that is already the platform's largest compute consumer.
- Stubbed tools measure tool-call correctness rather than end-to-end tool behaviour, which is a genuine fidelity gap.

**Choose differently when.** If a class of tools were provably side-effect-free and network-isolated by their own design, running them directly would recover fidelity at no risk. The isolation requirement would remain for everything else.

**Why it holds up over time.** Treating model output as untrusted input is becoming the default stance across the industry, and will apply anywhere an agent's output reaches an execution surface. This record will read as obvious sooner than most.

> **Lesson.** Classify a workload by what it executes, not by who owns it. Code generated by a model and executed unsupervised is untrusted code, whoever deployed the pipeline.

## Every package used, in one table

The terms this package uses in a specific way, and what each one is doing in the architecture.

| Package | What it is | What it does here | Considered instead |
|---|---|---|---|
| Comparison frame | The tuple (dataset version × scorer version × judge version × harness version) under which a score was produced. | Part of a score's identity. Two scores from different frames may not be compared, which is the architecture's defining rule. | Treating these as metadata, which makes the rule advisory and therefore absent. |
| Release candidate | A fully pinned configuration: model, prompt, tools, retrieval index, decoding parameters, safety filter. | The subject of every score, so that a quality change is attributable to a specific change. | Attributing scores to a model version, which confounds a prompt edit with a model upgrade. |
| Incumbent | The candidate currently serving production traffic. | The baseline every gate compares against, maintained by the deployment pipeline and re-scored on every frame change. | An absolute threshold, which answers 'is this good enough' rather than 'is this better'. |
| Re-baseline | Re-scoring the incumbent in a newly registered frame. | The budgeted, recurring cost of changing any measurement axis. Gating is suspended until it completes. | Carrying the old score forward, which is the forbidden cross-frame comparison by another route. |
| Gate-bearing slice | A declared subset of the corpus — language, task type, cohort, safety category — on which a regression blocks. | Declared in advance so the multiple-comparisons cost is bounded and known rather than discovered. | Gating every slice, which produces a false block on most neutral releases. |
| Guardrail gate | A gate that blocks regardless of any quality improvement: safety violation rate, benign refusal rate, latency, cost, tool-call errors. | Prevents a candidate that is wittier and less safe from being scored as better. Safety guardrails have no waiver path. | Folding safety into the aggregate quality score, where a large enough gain can outvote it. |
| INSUFFICIENT_EVIDENCE | A verdict distinct from BLOCK, returned when a run did not complete, a judge was unavailable, or the harness error rate exceeded budget. | Keeps absence of evidence from being read as a pass. The pipeline treats it as a stop. | A two-valued verdict, under which an incomplete run either blocks wrongly or passes dangerously. |
| Harness error | A failure of the evaluation machinery — sandbox timeout, provider 5xx, expired credential — as opposed to a wrong answer from the model. | Classified separately and excluded from scoring, because scoring infrastructure failures as model failures corrupts every number in the run. | Counting any non-success as a model failure, which makes a flaky dependency look like a regression. |
| Contamination canary | A subset of examples never published in any form. | A published-versus-canary score gap reveals that a candidate has seen the golden set, which otherwise looks like a large genuine improvement. | Trusting that a training corpus excluded the benchmark, which is unverifiable from outside. |
| Judge admissibility | Whether a judge's measured agreement with human labels is high enough for it to block a release. | Stored on the scorer row and read by the gate, so authority is data rather than convention. | Using a judge because it is available, which grants unearned authority to an uncalibrated instrument. |
| Confound register | A record of changes — UI, traffic mix, infrastructure — that overlapped a candidate's exposure window. | Lets the platform refuse to attribute an online movement to a candidate, rather than guessing under pressure. | Attributing every online change to the most recent model release, which is how the wrong thing gets rolled back. |
| Tombstone | Removal of an example's content while retaining its identity, slice labels and derived scores. | The erasure path under an immutable corpus. It preserves aggregate comparability and bounds full reproducibility. | Deleting the example, which silently changes every score computed from it. |