Model Evaluation Service  ·  View 04 of 21  ·  People and journeys

Journey — Taking a Candidate Through the Gate

The engineer's day: a prompt change that looked good on a dozen examples, and what the gate does with it.

Editable source SVG draw.io All views
ML Engineer ships 2-3 candidates a week Goal — Get a defensible yes or no on a prompt change before the end of the day Trigger — A rewritten system prompt that looks better on a dozen hand-picked examples Done when — A PASS with evidence, or a BLOCK naming the slice and the examples that moved 1 · Try local, hand-picked 2 · Smoke 400 examples 3 · Submit config pinned 4 · Wait ≤ 90 min ◆ moment of truth 5 · Read the verdict ◆ moment of truth 6 · Act What they do Eyeballs a dozen outputs Runs the smoke tier Pins the full config Submits the candidate Works on something else Opens the slice report Fixes, waives or ships Where Notebook CLI CLI · register Chat notification Verdict page Pull request What the platform does Nothing — not its job 400 examples · 3 gens Digest · frame resolve Cost ceiling check 6,500 × 3 gens 39k judge calls Per-slice deltas + CI Records the decision Feeling Confident Encouraged Fiddly Blocked and idle Blocked — on one slice Knows what to fix User Journey — ML Engineer Takes a Candidate Through the Gate The trough is the 90-minute wait followed by a block on a slice the engineer never looked at. The smoke tier exists to move that discovery earlier; the example-level diff exists to make the block arguable rather than final. v 1.0 · owner Data & AI Global Practice · date 2026-09

The trough

  • Ninety minutes of waiting followed by a block on a slice the engineer never looked at. That is the worst moment, and it is structural rather than cosmetic.
  • The smoke tier exists to move that discovery from 90 minutes to 8. The example-level diff exists to make the block arguable instead of final.

Decisions it forces

  • A smoke tier is a product requirement, not an optimisation — without it the feedback loop is too slow to be used.
  • A BLOCK must ship with the examples that moved, or engineers will route around the gate rather than argue with it.

Targets

  • Smoke ≤ 8 min p95 · gate suite ≤ 90 min p95 (stated assumptions).
  • False-block rate ≤ 5%, counting anything human adjudication overturns.
◀  Actors and Their Core Journeys All views Journey — Chasing a Regression Users Already Met  ▶
Solution Architecture v1.0 · Google Cloud · Data & AI Global Practice · 2026-09 · 21 views · 18 architecture decision records04-journey-ship-a-candidate