Architecture Views

Is a score a property of a candidate, or a property of a candidate measured a particular way?

Context

The natural schema stores `candidate-7 scored 0.83`. It makes the dashboard trivial and it is wrong in a way that takes about a year to surface. Datasets get corrected. Rubrics get clarified. A judge is upgraded because the old one could no longer separate two good answers. The harness changes how it truncates context. Each change is individually reasonable and individually small, and each silently redefines what the stored number means. A year later the quality chart shows a smooth upward line assembled from measurements never taken the same way, and nobody can say which improvements were real. This is not a data-hygiene issue: it is the failure mode that makes an evaluation platform actively harmful, because it produces confident numbers that mean nothing and produces them indefinitely without ever erroring.

Decision

A comparison frame is the tuple (dataset version × scorer version × judge version × harness version). Every score carries its frame_id as part of its identity, not as metadata beside it. The gate engine and the reporting layer refuse — explicitly, with an error — any request that would place two scores from different frames on the same axis.

How it is realised on Google Cloud

`frame` is a first-class table in Cloud SQL with the four axes as columns and a `retired_at`. `frame_id` is a required column on both `run` and `score` in BigQuery, and every comparison query joins through it. The frame resolver runs at plan time, before any inference is purchased, so a run that cannot be compared is never paid for.

Options weighed

• ChosenFrame as part of score identity, cross-frame comparison refused: Comparability by construction. Costs a re-baseline on every frame change and occasionally forces the platform to tell someone their chart cannot be drawn.
• RejectedFrame as metadata, comparison discouraged by convention: Every team intends to check the metadata. None does, at 2 a.m., in a quarterly review deck. A convention that is not enforced is a defect with a documentation page.
• RejectedNormalise old scores into the new frame with a correction factor: Requires knowing the mapping between frames, which is exactly what has not been measured. It converts an honest 'cannot compare' into a plausible wrong number.

Consequences

What it buys

• A number produced today can be honestly placed next to one from eighteen months ago, or the platform says why it cannot.
• Every expensive property of this architecture — immutable datasets, append-only scores, a pinned judge — follows from one rule rather than from a list of best practices.
• A scorer defect becomes visible, because fixing it is a frame change that forces a re-baseline rather than a silent shift in the series.

What it costs

• Every frame change costs a full re-scoring of the incumbent, which is a standing budget line rather than an exception.
• The platform will sometimes refuse to draw a chart someone wants, which is an unpopular property that has to be defended repeatedly.

Choose differently when

If the organisation genuinely never changes a dataset, rubric, judge or harness — which no organisation does past its first year — the frame collapses to a constant and this machinery is overhead. More realistically: if a cheap, validated mapping between frames could be established empirically, normalisation would become defensible and the refusal could soften into a warning.

Why it holds up over time

The rule names no technology. It survives replacing BigQuery, Google Cloud, the judge and the harness, because it is a statement about what a measurement means rather than about where it is stored. It is also the property a future team would most easily throw away without noticing, which is why it is enforced by a rejected query rather than by a paragraph in a README.

When the dataset, rubric, judge or harness changes, what happens to the baseline the next candidate will be compared against?

Context

ADR-01 makes cross-frame comparison illegal, which leaves an operational question: a rubric is clarified on Tuesday, and on Wednesday an engineer submits a candidate. The incumbent has no score in the new frame. The tempting answers are to carry its old number forward — which is precisely the comparison ADR-01 forbids, performed by a different route — or to compare the candidate against an absolute threshold, which answers a different question than 'is this better than what users have'.

Decision

The incumbent is re-scored in the new frame before gating resumes. Gating is suspended between frame registration and the completion of that re-score, and the suspension is announced rather than discovered. Re-baselining capacity is budgeted as a recurring cost sized to complete across all gating suites within a declared window.

How it is realised on Google Cloud

The frame resolver checks for an incumbent score in the resolved frame at plan time. If it is absent, the run planner enqueues the re-baseline ahead of the candidate's own run on the same execution pool, at higher queue priority, and reports both costs against the change that caused them. Target: all gating suites re-scored within 12 hours.

Options weighed

• ChosenRe-score the incumbent, suspend gating until done: Honest and expensive. Makes the price of a frame change visible at the moment it is approved.
• RejectedCarry the incumbent's old score forward: It is the forbidden cross-frame comparison wearing a different name, and it is undetectable once done.
• RejectedGate against an absolute threshold while re-baselining: Answers 'is this good enough' rather than 'is this better', and thresholds drift out of date faster than baselines do.
• DeferredLazy re-baseline: score the incumbent only on slices the candidate touched: Attractive for cost, and correct only if slices are independent. Worth revisiting once per-slice variance is measured.

Consequences

What it buys

• A frame change is a priced decision. The quality lead approving a rubric edit sees the compute bill and the gating pause before approving it.
• The baseline is always real rather than inferred, so a gate never blocks on a difference that is partly an artefact of changed measurement.

What it costs

• Gating stops for up to a working day after a frame change, which is genuinely disruptive on a week with several.
• Re-baselining consumes capacity that would otherwise evaluate candidates, so frame changes compete with releases.

Choose differently when

If score caching proves safe at scale (ADR-16), a re-baseline for a scorer-only change becomes a re-score of stored outputs rather than a re-run of the model, which cuts the cost by roughly the generation share and could remove the gating pause entirely.

Why it holds up over time

The rule follows from ADR-01 rather than from any implementation, so it holds as long as that does. What will change is the price: cheaper inference and better caching make re-baselining less painful over time, which strengthens the decision rather than weakening it.

When an example is found to be wrong — a bad reference answer, a mislabelled slice — is it fixed in place or superseded?

Context

Fixing in place is obviously convenient and quietly catastrophic under ADR-01: a score references a dataset version, and if that version's content can change, the frame no longer identifies anything. Every score computed before the fix silently becomes a measurement of a corpus that no longer exists, and there is no way to detect that from the data.

Decision

A dataset version is immutable once published. Corrections, additions and removals produce a new version with a new identifier. Old versions remain readable indefinitely because scores reference them. Every example carries provenance — source, date, consent state, approver — and slice labels, and a gating dataset must have a named, current owner.

How it is realised on Google Cloud

Dataset versions are content-addressed objects in Cloud Storage in a separate `mes-corpus` project, with object versioning on and no delete role granted to any execution-plane identity. The manifest is the dataset's identity; examples are addressed from it.

Options weighed

• ChosenImmutable versions, corrections publish anew: The only option compatible with ADR-01. Costs storage, which is the cheapest thing in this architecture.
• RejectedMutable datasets with an audit log of edits: The log tells you an edit happened; it does not make the old scores meaningful again. It is forensics, not comparability.
• RejectedCopy-on-write at run time — snapshot whatever the dataset is when a run starts: Produces a version per run, which is correct and makes the corpus unmanageable and cross-run comparison nearly impossible.

Consequences

What it buys

• A three-year-old score still names a corpus that can be read, which is what makes a long quality series defensible.
• Dataset review becomes meaningful: a version is a reviewable artefact rather than a moving target.

What it costs

• Storage grows monotonically, and dataset sprawl becomes a real curation burden as versions accumulate.
• It collides head-on with the erasure obligation, which ADR-17 resolves imperfectly and deliberately.

Choose differently when

Nothing short of abandoning ADR-01 flips this. If the erasure obligation were to tighten such that tombstoning is insufficient, the corpus would have to move to synthesised or licensed material, which changes what is stored rather than whether it is immutable.

Why it holds up over time

Immutability is a property of the data model, not of the storage technology. Object stores have supported versioning and retention for two decades and will continue to; the decision outlives any particular one.

When a score turns out to be wrong, is the row updated or is a new row written?

Context

An updatable score row means a past verdict can change after the fact without anyone noticing. Release decisions are audited years later, and a verdict whose supporting evidence has been silently edited is not evidence. There is also a subtler problem: an update loses the information that the platform once believed something different, which is exactly the information needed to understand how a regression escaped.

Decision

Scores are append-only. A correction writes a new score row and sets `superseded_by` on the old one. Verdicts are immutable release records retained for the life of the product. No component has update or delete permission on the score table.

How it is realised on Google Cloud

BigQuery tables partitioned by run and clustered by slice, written through an append-only service account. Corrections are a distinct write path requiring a reason and an author. Reporting views filter on `superseded_by IS NULL` by default but can reconstruct any historical view.

Options weighed

• ChosenAppend-only with supersession: A past verdict stays reconstructible exactly as it was decided. Costs storage and query complexity.
• RejectedMutable scores with row-level history: Equivalent in principle and worse in practice: the default read is the mutated one, so the correct behaviour requires everyone to opt in.
• RejectedImmutable scores, no correction path at all: Purity that forces a full re-run for a single bad score, and pushes people to fix things outside the system.

Consequences

What it buys

• Recovery of any component can never alter a past verdict, because there is no code path that writes to an existing row.
• The history of what the platform believed is itself data, which is what makes a post-incident review of the gate possible.

What it costs

• Every reporting query carries supersession logic, which is a small permanent tax on readability.
• Storage grows with corrections as well as with runs.

Choose differently when

If audit obligations disappeared entirely and nobody ever asked why a release was approved, the history would be dead weight. That is not a plausible direction of travel for AI product decisions.

Why it holds up over time

Append-only is a design stance rather than a feature of BigQuery; it transfers to any store. As regulatory interest in AI release decisions grows, the reconstructibility this buys becomes more valuable, not less.

What exactly is being scored when the platform says a candidate improved?

Context

An assistant's behaviour is produced by a model, a system prompt, a set of tool definitions, a retrieval index, decoding parameters and a safety filter. Teams routinely attribute a quality change to 'the new model' when the prompt changed in the same week. If the subject of a score is underspecified, the score is attributed to nothing, and two runs that look like a comparison are measuring different systems.

Decision

The unit of evaluation is a release candidate: model identifier and weights digest, system prompt version, tool definitions version, retrieval index version, decoding parameters and safety filter version. A candidate whose configuration is not fully pinned is refused. Candidate identity is the digest of that configuration, so resubmitting the same configuration is recognised rather than re-evaluated.

How it is realised on Google Cloud

`candidate.config_digest` is a unique column in Cloud SQL, computed over the canonicalised configuration. The incumbent is a candidate row with `is_incumbent` set, maintained by the deployment pipeline, so the platform always knows what production is actually running.

Options weighed

• ChosenFull configuration digest as the subject: Attribution is exact and deduplication is free. Costs discipline at submission time.
• RejectedModel version as the subject, other components as tags: Reintroduces the confound it was meant to remove, because a tag does not participate in identity.
• RejectedSubject defined per-experiment by the submitter: Makes comparability a per-team convention, which is how two teams end up with incomparable numbers and an argument.

Consequences

What it buys

• A quality change is attributable to a specific change, which is the difference between a measurement and an anecdote.
• Resubmitting an unchanged configuration is a cache hit rather than a repeat bill.

What it costs

• Engineers must pin everything before submitting, which is friction at exactly the moment they feel closest to done.
• A configuration surface that grows — a new retrieval component, a new filter — changes the digest for everyone and invalidates deduplication.

Choose differently when

If the assistant's architecture collapsed to a single model call with no prompt, tools or retrieval, the configuration and the model would be the same thing and this would be redundant. The direction of travel is the opposite.

Why it holds up over time

The principle — score the system, not the component — transfers to any composition of models, agents and tools, and becomes more important as the number of components rises.

Does a scorer run inline with the model call, or against stored outputs?

Context

Inline scoring is simpler and one fewer store. It also means that fixing a scorer bug, clarifying a rubric or adding a new metric requires re-running the model against the entire corpus — which is the expensive half. Given that scorer and rubric changes are frequent and generation is the dominant cost, this is a choice about how expensive it is to improve the measurement.

Decision

The harness generates and stores a full trace — prompt as sent, retrieval context, tool calls, raw output — as an addressable artefact. Scoring is a separate stage that reads traces. A scorer-only frame change re-scores stored outputs rather than re-running the candidate.

How it is realised on Google Cloud

Traces are written to Cloud Storage keyed by (run_id, example_id, generation_idx) and referenced from `score.trace_ref`. Scoring workers are a distinct GKE deployment that consumes a separate Pub/Sub subscription, so scoring capacity scales independently of generation capacity.

Options weighed

• ChosenSeparate stages joined by stored traces: Makes rubric iteration cheap and diagnosis possible. Costs ~2.2 TB/year and a second pipeline stage.
• RejectedInline scoring during generation: Cheaper to build, and makes every scorer improvement cost a full generation sweep — which means scorers stop improving.
• RejectedStore outputs only, not full traces: Halves storage and removes the ability to tell why a candidate failed, which is what the engineer actually needs from a block.

Consequences

What it buys

• A rubric clarification costs a scoring pass, not a generation sweep — roughly an order of magnitude less at assumed prices.
• A blocked engineer can read the actual prompt, tool calls and output for the examples that moved, which is what makes a block arguable rather than infuriating.

What it costs

• Trace storage at ~2.2 TB/year, and a retention decision that must balance evidence value against cost and privacy exposure.
• Two pipeline stages to operate, monitor and keep in version agreement.

Choose differently when

If generation became nearly free relative to storage — a large drop in inference cost with no drop in storage cost — inline scoring with discarded traces would become defensible for the highest-volume suites. The diagnostic argument would still favour traces for gating suites.

Why it holds up over time

Separating expensive irreversible work from cheap repeatable work is a general pipeline principle. As models get cheaper this decision weakens economically and strengthens diagnostically, and the diagnostic case is the one that matters to users of the platform.

Should the gating corpus be a stable benchmark or a live sample of what users are actually asking?

Context

A frozen golden set is comparable across time, contamination-detectable and stable, and drifts away from real traffic within months — at which point it measures a distribution nobody has. A rolling sample always represents the live distribution and destroys cross-time comparison, because every refresh is a frame change under ADR-01. Neither alone is adequate, and the conflict is genuine rather than a failure to choose.

Decision

Three classes with declared roles. The frozen golden set is the gate-bearing corpus and the long comparability axis; changing it requires an explicit reviewed decision. The rolling set is refreshed from recent production traffic on a cadence, is advisory rather than blocking, and exists to detect when the golden set has stopped representing users. Targeted regression sets are grown from real incidents and are blocking for the class of failure they encode.

How it is realised on Google Cloud

Each class is a separate dataset lineage in `mes-corpus` with its own owner, cadence and gate strength. A rolling-set refresh publishes a new version; comparisons spanning a refresh are re-scored rather than stitched. Staleness is measured continuously as divergence between the rolling set's distribution and current production traffic.

Options weighed

• ChosenThree classes with declared, different gate strengths: Keeps the comparability axis stable while retaining a live signal. Costs roughly double the corpus to maintain.
• RejectedFrozen golden set only: Comparable and progressively irrelevant. The gate keeps passing candidates that are worse for real users.
• RejectedRolling set only: Always relevant and never comparable; every week is its own frame and no trend can be read.
• DeferredFrozen and rolling with equal blocking authority: Leaves open what happens when they disagree, which is the question that actually matters. Deferred until disagreement frequency is measured.

Consequences

What it buys

• The golden set's drift becomes measurable rather than assumed, because the rolling set is the instrument that detects it.
• An incident can be converted into a permanent blocking test without touching the comparability axis.

What it costs

• Roughly twice the corpus curation effort, and three owners rather than one.
• A candidate the golden set likes and the rolling set does not is an unresolved situation the platform reports but does not decide.

Choose differently when

If the rolling set's verdicts were shown over a year to track online outcomes substantially better than the golden set's, blocking authority should move to it and the golden set should be demoted to a long-series instrument only.

Why it holds up over time

The tension between a stable benchmark and a live distribution is permanent in any measurement programme for a system facing changing demand. The three-class answer will outlive any particular corpus.

Which model scores the outputs, and what happens when the provider changes it underneath the pin?

Context

Using a model to judge a model is the only way to score open-ended quality at 39,000 judgements per gate run. But a judge is a measuring instrument, and an instrument whose calibration changes without notice invalidates every measurement taken with it. Provider-side updates to an endpoint that a customer believes is pinned are a documented reality, not a hypothetical. Under ADR-01 such a change is a frame change nobody recorded, which is the worst case the whole design exists to prevent.

Decision

The judge is pinned to an explicit model version and decoding configuration and is an axis of the frame. The platform does not assume the pin holds: it continuously re-scores a fixed calibration set with the pinned judge and alarms when the judge's own output distribution or its agreement with human labels shifts beyond a threshold. A detected shift retires the frame and quarantines runs scored inside the drift window.

How it is realised on Google Cloud

A scheduled job re-scores the calibration set on a fixed cadence and writes the result to BigQuery as a first-class monitored signal, surfaced on the operations view alongside availability and cost. Judge rationales are stored with every score so a disputed verdict can be inspected rather than merely re-run.

Options weighed

• ChosenPin, and verify the pin continuously: Treats the judge as an instrument under calibration control. Costs a standing re-score bill and an alerting path.
• RejectedPin and trust the provider: Makes the platform's central guarantee depend on a third party's change management, undetectably.
• DeferredSelf-host the judge so the weights cannot change: Removes drift entirely and is the strongest answer. Deferred on cost and on the capability gap against frontier judges; revisit as open-weight judges close it.

Consequences

What it buys

• Silent instrument drift becomes a detected incident with a defined response, rather than a slow corruption of the series.
• Frame retirement is automatic on detection, so affected scores are quarantined rather than quietly trusted.

What it costs

• A continuous re-scoring bill that buys no verdicts, only confidence that the verdicts are comparable.
• False drift alarms interrupt gating, and the threshold that separates real drift from sampling noise is itself a tuning problem.

Choose differently when

Self-hosting an open-weight judge of adequate quality removes the drift class entirely and makes the monitor a much cheaper regression check. That is the most likely change to this record within a few years.

Why it holds up over time

Calibration control over a measuring instrument is a principle from metrology, not from machine learning, and it long predates and will outlive any specific judge model.

How good does a model judge have to be before it is allowed to stop a release?

Context

A judge with poor agreement against human preference is not measuring quality; it is measuring its own idiosyncrasies at scale and with great confidence. Yet judges are indispensable at these volumes, and a rule that admits only perfect judges admits none. The question is therefore not whether to use one but what authority to grant it, and at what evidentiary standard.

Decision

Every judge is calibrated against human labels on a named calibration set before it may gate. Agreement is published as a named statistic, re-measured for every judge version. A judge at or above 0.75 Cohen's κ may block; between 0.65 and 0.75 it may block only with human adjudication; below 0.65 it is inadmissible for gating and may produce observational scores only. The threshold is a property of the scorer row, read by the gate, not a convention people remember.

How it is realised on Google Cloud

`scorer.kappa_vs_human` and `scorer.gate_admissible` are columns the gate engine reads at decision time. The calibration set holds at least 1,200 human-labelled examples per rubric and is refreshed at least every six months. Raters see blinded pairs and never learn which output came from the candidate.

Options weighed

• ChosenPublished κ threshold, admissibility as data: Makes judge authority auditable and revocable. Costs a standing human labelling programme.
• RejectedUse the judge, report agreement, let teams decide: Authority without a standard. In practice the judge blocks and nobody checks the statistic.
• RejectedHuman-only gating: Correct and unaffordable: 39,000 judgements per gate run against 40 reviewer-hours per week.
• DeferredEnsemble of judges with disagreement routed to humans: Raises reliability and multiplies the dominant cost. Phase 3, once single-judge agreement is characterised.

Consequences

What it buys

• The judge's authority is earned, measured and revocable, which is what makes an automated block defensible to the engineer it stops.
• A degraded judge loses blocking rights automatically rather than after someone notices.

What it costs

• A permanent human labelling programme is required whether or not anything is going wrong.
• κ is an imperfect summary of agreement and can mask systematic disagreement on the slices that matter most.

Choose differently when

If judges reached agreement levels comparable to inter-rater agreement among humans themselves, the threshold would become a formality and the calibration programme could shrink to periodic spot checks. Conversely, evidence that κ hides slice-level disagreement would push toward per-slice admissibility.

Why it holds up over time

Requiring an automated decision-maker to demonstrate agreement with human judgment before it is granted authority is a governance pattern that will apply to far more than evaluation judges.

How are position bias, verbosity bias and self-preference handled?

Context

Model judges exhibit well-documented systematic biases: preferring the first option presented, preferring longer answers, and preferring outputs from their own model family. These are not noise that averages out — they are directional errors that survive large samples, and a rubric instructing the judge to ignore length does not reliably make it do so. If a candidate happens to be more verbose than the incumbent, an uncorrected judge will report an improvement that does not exist.

Decision

Position bias is countered by evaluating both orderings of every pairwise comparison and combining. Verbosity bias is measured and reported as a standing statistic rather than assumed away. Self-preference is treated as disqualifying: a judge may not gate a candidate from its own model family. Human raters always see blinded pairs.

How it is realised on Google Cloud

The scoring pipeline issues two judge calls per pairwise comparison with the order swapped — which is why the gate run costs ~39,000 judge calls for 6,500 examples. Family affinity between judge and candidate is checked at plan time and blocks admissibility rather than warning.

Options weighed

• ChosenStructural controls: both orderings, family exclusion, blinding: Doubles judge cost, which is the dominant cost. Bought because the bias is directional and would otherwise be indistinguishable from a real gain.
• RejectedInstruct the judge to be unbiased in the rubric: Cheap, popular and not reliably effective. It documents an intention rather than enforcing a control.
• RejectedCorrect statistically for measured bias: Requires a stable bias estimate, and the bias moves with the judge version — which is the thing ADR-08 says not to assume.

Consequences

What it buys

• A verbosity-driven apparent improvement is caught rather than shipped, which is one of the most common false positives in LLM evaluation.
• A judge from the same family as a candidate cannot quietly favour it.

What it costs

• Judge cost doubles for every pairwise comparison, and judge inference is assumed to be ≤ 55% of run cost.
• Family exclusion reduces the pool of admissible judges, which can be limiting when the best available judge shares a family with the primary model.

Choose differently when

If a judge were demonstrated to be order- and length-invariant across versions, single-ordering evaluation would halve the dominant cost. That demonstration would have to be repeated per version, which is what makes it unattractive today.

Why it holds up over time

The specific pathologies may change as judges improve, but the principle — counter a known systematic bias structurally rather than by asking nicely — applies to any automated evaluator.

What does the platform return when a run did not finish, or finished with too many harness errors to trust?

Context

A two-valued gate forces every incomplete run into one of two wrong answers. Returning PASS ships a candidate nobody measured. Returning BLOCK is safer but indistinguishable from a real regression, which trains engineers to retry until green — and a gate people retry past is not a gate. The distinction between 'this is worse' and 'we do not know' is the one the deployment pipeline most needs.

Decision

Three values. INSUFFICIENT_EVIDENCE is returned when a run is incomplete, when the harness error rate exceeds its declared budget, when a required judge was unavailable, or when a gate-bearing slice went unevaluated. The pipeline treats it as a stop, not as a retryable failure, and the verdict names what is missing.

How it is realised on Google Cloud

The gate engine evaluates completeness before it evaluates deltas. `run.harness_error_rate` is persisted by the harness and read as an admissibility precondition. Every verdict ships with the evidence it rests on: suites run, examples scored, slices evaluated, scorer and judge versions, and a confidence interval on every reported delta.

Options weighed

• ChosenThree-valued verdict with named gaps: Makes 'we do not know' a first-class answer. Costs a third code path in every consumer.
• RejectedTwo-valued, incomplete runs block: Conflates a regression with an outage, and teaches engineers that blocks are often spurious.
• RejectedTwo-valued, incomplete runs pass with a warning: A warning in a pipeline log is a pass. This is how an unevaluated candidate reaches users.

Consequences

What it buys

• An infrastructure problem is diagnosed as one instead of being argued about as a quality regression.
• The block rate stays meaningful as a metric, because it is not inflated by harness flakiness.

What it costs

• Every consumer of the verdict must handle a third case, and the temptation to map it onto one of the other two is permanent.
• A platform having a bad day stops releases rather than degrading, which is deliberate and occasionally very unpopular.

Choose differently when

Nothing reasonable flips this. If harness reliability became high enough that incomplete runs effectively never occurred, the third value would be unused rather than wrong — and would still be the correct thing to return on the day it recurs.

Why it holds up over time

Distinguishing a negative result from an absent one is a principle of measurement rather than of software, and it will apply to any automated gate in any domain.

How does the platform gate on cohorts without blocking most neutral releases by chance?

Context

A headline average hides a collapse in one language behind gains everywhere else, so per-slice gating is necessary. But 34 gate-bearing slices tested independently at α = 0.05 will produce at least one false block in roughly four out of five fully-neutral releases. Correcting for multiple comparisons restores honesty and destroys sensitivity on small slices. Growing slices to restore power multiplies the cost of every run. Every available answer is a trade, and the design encodes a preference between a false block and a missed regression whether or not anyone states it.

Decision

The gate-bearing slice set is declared in advance and kept small — 34 slices against 18 suites — and each is chosen because it carries a product commitment rather than because it is measurable. Deltas are evaluated as paired comparisons against the incumbent on the same examples, which recovers substantial power. No gate blocks on a difference it cannot distinguish from noise at the declared confidence level. The preference between false blocks and missed regressions is stated explicitly: false-block rate ≤ 5%.

How it is realised on Google Cloud

`gate` rows carry slice, strength and alpha, so the trade is data rather than code. Suite size is derived from a power requirement — detect a 2 percentage-point regression at ≥ 90% power, α = 0.05 — which is what sets the 6,500-example gating suite rather than convenience.

Options weighed

• ChosenSmall declared slice set, paired design, stated error preference: Honest about the trade and bounded in cost. A regression in an ungated slice still ships, which is the accepted risk.
• RejectedGate every slice with multiple-comparison correction: Statistically clean and practically blind: after correction, small slices detect nothing worth detecting.
• RejectedGate on the aggregate only: Cheap, sensitive and exactly the failure the platform exists to prevent.
• DeferredGrow every slice until all can be gated at full power: Correct and multiplies run cost several-fold. Revisit if inference cost falls substantially.

Consequences

What it buys

• The multiple-comparisons cost is bounded and known rather than accumulating invisibly as teams add slices.
• Suite size is derived from a stated detection requirement, so 'is the suite big enough' has an answer.

What it costs

• A regression confined to an ungated slice ships. That is a real, accepted exposure and it should be reviewed as the slice set changes.
• The paired design constrains how datasets may be refreshed, since pairing requires the same examples on both sides.

Choose differently when

If the false-block rate proved much lower than modelled, or if run cost fell enough to grow every slice, the gate-bearing set could widen substantially. Evidence that regressions routinely land in ungated slices would force the same change at higher cost.

Why it holds up over time

Multiple comparisons is a permanent property of testing many hypotheses at once. The specific slice set will churn; the discipline of declaring it in advance will not.

Can a large enough improvement in helpfulness justify a small regression in safety, latency or cost?

Context

Any scalar quality score will eventually be improved by a candidate that is also less safe, slower or more expensive, and a single composite metric will happily trade one for the other. A candidate that is wittier and less safe is not a better candidate. Separately, waivers are necessary — a genuine emergency, a known-false positive — and a waiver mechanism that applies to everything is an off switch for the gate.

Decision

Guardrail gates — safety violation rate, benign refusal rate, p95 response latency, cost per request, tool-call error rate — block independently of the quality result and are never traded against it. Every other blocking gate may be waived by a named approver with a stated reason and an expiry. Safety guardrail gates have no waiver path at all.

How it is realised on Google Cloud

Guardrails are evaluated as a separate pass with their own thresholds before quality deltas are considered. The waiver API refuses a waiver against a safety gate as a matter of code rather than of permission, so there is no role that can grant one. Active waivers appear on a standing register, and waiver rate is a monitored metric.

Options weighed

• ChosenIndependent guardrails, safety non-waivable: Removes the possibility of trading safety for helpfulness. Costs genuine flexibility in a real emergency.
• RejectedComposite score with weighted safety term: A sufficiently large helpfulness gain outvotes safety, which is the exact outcome to prevent.
• RejectedSafety waivable by a senior approver: Creates a role whose existence is a standing target, and a precedent that is used a second time more easily than a first.

Consequences

What it buys

• There is no code path by which a safety regression reaches users because something else improved a lot.
• The waiver register makes quality debt visible and expiring rather than permanent.

What it costs

• A false positive on a safety gate stops a release with no override, which will at some point be the wrong outcome for a real incident.
• That pressure moves onto the safety scorer's accuracy, which becomes a critical component in its own right.

Choose differently when

None on the safety rule. The non-safety guardrails could reasonably become tunable per release class — a latency regression may be acceptable for a capability release and not for a routine one — which is a refinement rather than a reversal.

Why it holds up over time

Separating vetoes from trade-offs is a governance pattern older than software. As AI safety expectations become regulatory rather than voluntary, a non-waivable path becomes an asset rather than an inconvenience.

What happens to releases when the evaluation platform is unavailable during a release window?

Context

The obvious design makes the platform highly available and treats an outage as a degraded mode. But availability engineering cannot reach 100%, and the behaviour on the worst day is what the whole gate is worth. A gate with a bypass is a gate that will be bypassed on the day it matters most — typically during the incident that makes someone want to ship a fix quickly. Shipping an unevaluated candidate to 40 million users is a worse outcome than a delayed release.

Decision

The deployment pipeline will not proceed without an affirmative PASS. A pipeline that cannot reach the gate does not release. There is no bypass, no timeout-to-proceed and no break-glass release path in the platform. The gate is also protected as a production control: author and approver of a gate change must be different people, and no long-lived credential exists anywhere in the policy path.

How it is realised on Google Cloud

Fail-closed is implemented in the pipeline's contract rather than in the evaluation service, so no outage of the service can change it. Policy lives in Git with enforced two-person review; Cloud Build authenticates through Workload Identity Federation for short-lived tokens only; every gate change and waiver is written immutably.

Options weighed

• ChosenFail closed in the consumer's contract: The property survives any failure of the platform, including a total one. Costs release availability.
• RejectedFail closed implemented inside the evaluation service: A service that is down cannot enforce anything. The guarantee has to live where it still holds when the platform does not.
• RejectedTimeout-to-proceed after a bounded wait: Converts an availability incident into an unevaluated release, quietly and on a schedule.
• RejectedBreak-glass release with post-hoc evaluation: The mechanism will be used during exactly the incidents where judgement is worst.

Consequences

What it buys

• The worst-case behaviour is a delayed release rather than an unmeasured one, and that property does not depend on the platform being up.
• Gate integrity is enforced by review and short-lived identity rather than by trust in operators.

What it costs

• Evaluation platform availability becomes release availability, which raises the operational stakes of a system that is otherwise off the user path.
• A genuine emergency fix waits for the gate, and there is no sanctioned way around that.

Choose differently when

A strong argument could be made for a narrow exception covering rollbacks to a previously-passed incumbent, which is a release to a candidate that already has a verdict. That is a refinement worth considering and is deliberately not in the MVP.

Why it holds up over time

Placing a safety property in the consumer's contract rather than the provider's implementation is a general availability principle: a guarantee that depends on a component being up is not a guarantee.

Does online evidence stop a release, or only observe one?

Context

Offline scores are fast, safe and measure a proxy whose predictive validity is itself uncertain. Online behaviour is what actually matters and is slow, confounded by concurrent changes, and requires exposing real users to the candidate before it is cleared — which is the thing the gate exists to avoid. Choosing either alone gives up something the platform was built for.

Decision

Staged exposure. Offline gates entry to shadow and canary. Online guardrail metrics are monitored continuously during exposure and trigger automatic rollback on a declared threshold breach, without waiting for a scheduled analysis. Online evidence gates progression to full rollout. Metrics and thresholds are declared before exposure begins, never chosen after the data arrives.

How it is realised on Google Cloud

Guardrail evaluation runs against serving telemetry in BigQuery and emits a rollback trigger to the deployment pipeline; breach to rollback initiation is targeted at ≤ 5 minutes p99. User feedback signals — explicit ratings, regeneration rate, abandonment, copy-and-edit behaviour — are treated as biased signals to be interpreted, never as labels.

Options weighed

• ChosenOffline gates entry, online gates rollout, auto-rollback armed: Bounds the exposure of a bad candidate while keeping the decision anchored in real behaviour.
• RejectedOffline gating only: Ships on a proxy and discovers the truth from users, which is the journey in view 05.
• RejectedOnline gating only: Every candidate reaches users before it is cleared, which makes the gate a formality.
• RejectedThresholds chosen after inspecting canary data: Guarantees a favourable reading, because the threshold is fitted to the result.

Consequences

What it buys

• A bad candidate's exposure is bounded by canary size and rollback latency rather than by how quickly someone notices.
• Offline suites acquire an external validity check, which ADR-16 turns into a retirement criterion.

What it costs

• Real users meet candidates that offline evaluation cleared but online evidence will reject — a deliberate, bounded exposure.
• Canary size must be large enough to detect the regressions that matter, which is a statistical requirement in tension with limiting exposure.

Choose differently when

If measured offline-online correlation became strong enough across all gate-bearing slices, the canary could shrink materially. Persistently weak correlation would push blocking authority toward online evidence and reduce offline gating to a smoke test.

Why it holds up over time

Proxy-then-truth staging is how every measurement programme with an expensive ground truth works, from drug trials to search ranking. It will outlive any particular metric set.

How does the platform know a suite is worth running, and when may a score be reused rather than recomputed?

Context

Offline suites accumulate. Each was added for a good reason, each costs money on every run, and nothing ever removes one — so cost grows monotonically while the marginal information from the eighteenth suite is unknown. Separately, caching by (candidate, example, frame) is the single largest cost lever available and would make re-baselining nearly free. But a cached score for a candidate sampling at non-zero temperature asserts a determinism the candidate does not have, and a cached judge score assumes the pinned judge has not silently moved.

Decision

Offline-online correlation is measured per suite — the degree to which a suite's offline delta predicted the online outcome — and reported against that suite's cost. A suite that has never predicted anything is a visible candidate for retirement. Score caching is permitted only for candidates running at temperature zero, and only within a frame whose judge-drift monitor is current. Every other cache read is a miss.

How it is realised on Google Cloud

Correlation is computed in BigQuery over the verdict and online-outcome history and stored on `suite.online_correlation`. The cache probe in the scoring pipeline checks the candidate's decoding configuration before consulting Memorystore, so the determinism condition is enforced in the pipeline rather than trusted.

Options weighed

• ChosenCorrelation-driven retirement, determinism-gated caching: Keeps cost tied to demonstrated value and keeps the cache honest. Costs a slow feedback loop and forgoes caching on stochastic suites.
• RejectedCache everything by content hash: Large saving bought by asserting a determinism that does not exist, which silently narrows measured variance to zero.
• RejectedNo cache at all: Safe and forfeits the only lever large enough to make frequent re-baselining affordable.
• RejectedRetire suites on a fixed schedule: Removes suites by age rather than by value, and the oldest suite may be the one that works.

Consequences

What it buys

• The corpus can shrink as well as grow, which is the only defence against unbounded evaluation cost.
• Re-baselining a scorer-only change becomes substantially cheaper wherever suites can honestly run at temperature zero.

What it costs

• Correlation needs many releases before it is meaningful, so retirement decisions lag suite addition by a long way.
• Running gating suites at temperature zero may not represent production behaviour, which is an open question the cache depends on.

Choose differently when

If gating at temperature zero were shown to mis-measure production behaviour, caching would be confined to deterministic scorers over stored traces — still useful, much smaller. If variance across generations proved negligible, caching could widen considerably.

Why it holds up over time

Requiring a measurement to demonstrate predictive value, and requiring a cache to state what it assumes, are both durable disciplines independent of the technology underneath.

How can dataset versions be immutable while a user retains the right to have their data erased?

Context

ADR-03 requires dataset versions never to change; privacy obligations require a user's data to be erasable wherever it exists. These are in direct conflict. Deleting the example mutates a version and silently changes every score computed from it. Keeping it honours comparability and breaches the obligation. Tombstoning preserves aggregate comparability but makes the verdict no longer fully reproducible from artefacts, which contradicts the reproducibility claim in the requirement.

Decision

Production-derived content is redacted or pseudonymised at admission, before the example is durable, and the redaction version is recorded. Every example carries its lawful basis, consent state and retention limit. An erasure request tombstones the example: content is removed; identity, slice labels and derived scores are retained. The consequence is stated openly rather than hidden — aggregate comparability survives, full reproducibility from artefacts does not, and this remains an open question rather than a solved one.

How it is realised on Google Cloud

Sensitive Data Protection applies versioned inspection templates at the admission stage of the data flow. `example.tombstoned_at` makes the state first-class and auditable. The corpus sits in its own project inside a VPC Service Controls perimeter, with every raw trace access logged and attributable.

Options weighed

• ChosenRedact at admission, tombstone on erasure, state the limit: Honours the obligation and bounds the damage to the comparability claim. The residual gap is real and named.
• RejectedHard delete the example and its scores: Mutates a dataset version and silently changes historical aggregates, which is worse than a bounded gap.
• DeferredRefuse production-derived content entirely; synthesise or license everything: Resolves the conflict completely and gives up the rolling set's representativeness. Genuinely worth revisiting if obligations tighten.
• RejectedKeep the example and rely on the lawful basis: Bets the platform's legal position on an interpretation, for a comparability benefit.

Consequences

What it buys

• There is no path by which raw production content becomes a durable dataset example, which shrinks the exposure to what redaction misses.
• Erasure is a first-class auditable state rather than a deletion nobody can account for.

What it costs

• A verdict resting on a tombstoned example cannot be fully reconstructed from artefacts, weakening the reproducibility claim in ADR-04.
• Redaction is imperfect, and an over-redacted example may no longer be a valid test of the behaviour it was collected for.

Choose differently when

If synthesised or licensed material were shown to carry the gating load without losing representativeness, refusing production-derived content entirely becomes the better architecture and removes this conflict rather than managing it.

Why it holds up over time

The tension between immutable evidence and a right to erasure is structural and will intensify. Naming the residual gap is what lets a future team make a different trade knowingly.

How should a system that executes model-generated tool calls thousands of times an hour be contained, and what stops a misconfigured run spending without limit?

Context

The evaluation harness does something unusual: it takes text a model wrote and executes it against tools, at volume, unsupervised, including inputs specifically designed to probe edge cases. Treating that as an internal batch job is a category error — it is closer to running untrusted user-submitted code. Separately, a fan-out pipeline with a bug in its work planner can generate an unbounded bill faster than any alerting loop can respond, and an alarm after the money is spent is not a control.

Decision

Model output and its tool calls are untrusted content. Tool execution happens in gVisor-isolated pods with egress denied and side-effect-free stubs, on a network path with no route to production systems. All outbound inference leaves through an allowlisting egress proxy. Separately, every run carries a cost ceiling enforced by the harness itself: the run halts at the ceiling and reports, rather than alarming afterwards.

How it is realised on Google Cloud

Sandbox pods run under gVisor with egress deny by default; the execution plane holds workload identity that can read the corpus and write traces, and cannot read a human label, write a verdict or delete a dataset version. Per-run cost is metered in-process against the ceiling set by the run planner; per-team monthly budgets are enforced at admission.

Options weighed

• ChosenKernel-isolated sandbox, egress allowlist, in-harness cost ceiling: Proportionate to running generated code at volume. Costs some performance and a more complex execution environment.
• RejectedPlain containers with network policy: Adequate for trusted workloads. This one executes text an adversary could have influenced through a prompt.
• RejectedCost alerting rather than a ceiling: An alert fires after the spend. The ceiling is the only control that acts before it.
• RejectedReal tools instead of stubs, for fidelity: Higher fidelity bought by giving model-generated calls real side effects. Not a trade worth making.

Consequences

What it buys

• A prompt-injected tool call cannot reach an arbitrary host or produce a real side effect.
• A runaway run is halted rather than discovered on an invoice, which makes experimentation safe to encourage.

What it costs

• gVisor adds syscall overhead to a workload that is already the platform's largest compute consumer.
• Stubbed tools measure tool-call correctness rather than end-to-end tool behaviour, which is a genuine fidelity gap.

Choose differently when

If a class of tools were provably side-effect-free and network-isolated by their own design, running them directly would recover fidelity at no risk. The isolation requirement would remain for everything else.

Why it holds up over time

Treating model output as untrusted input is becoming the default stance across the industry, and will apply anywhere an agent's output reaches an execution surface. This record will read as obvious sooner than most.