advanced 3 min answer

Your model inventory is a spreadsheet with 140 rows, maintained by the risk function. Serving decides nothing from it - a team puts a model into production by merging a config change. You have been asked to make the inventory enforceable, so that no model serves production traffic unless its registry record carries a current validation, without pausing any team's releases. Give the sequence.

model riskmodel inventoryadmission controlmigrationgovernance
Show the full answer Hide the answer

The sequence

  1. Make serving report before it enforces anything. Every model-loading process registers at startup: model id, artefact digest, config version, serving endpoint. One day of work per runtime, no behaviour change, instantly reversible.
  2. Reconcile the observed set against the spreadsheet. Expect a 20% to 40% mismatch: models serving that nobody registered, rows describing artefacts nobody serves, two rows for the same model under different names. The reconciliation report, not the gate, is the first deliverable, because it is also the first piece of evidence that the inventory is complete.
  3. Move the system of record. The publishing pipeline writes the registry entry - digest, owner, tier, validation run id, the conditions validation covered, expiry - so the record is a by-product of publishing rather than a form someone fills in. The spreadsheet becomes a view.
  4. Warn mode in the admission path. Serving still loads an unmatched model and emits a violation naming the owner. Move on only when unmatched loads stay below 2% for two consecutive weeks; that threshold is what stops the gate going live against a 30% mismatch and being switched off in its first week.
  5. Block at the deploy gate first, then at load time. The deploy gate is cheap and reversible. Load-time refusal is the real control and goes on for tier-one models only, with a documented break-glass that admits an unregistered model, opens a ticket automatically and expires in 72 hours. Teams build a worse bypass if you do not give them one.
  6. Enforce expiry last. It is the step that creates a cliff: validations granted in one quarter share an expiry date, and sixty models can fall due on a Tuesday. Stagger expiry at grant time and alert owners at 30 and 7 days.

Where data diverges, and how you would know

The divergence that matters is the registry describing a model nobody serves while an undescribed one takes traffic. The startup registration from step 1 is the detector, compared continuously rather than in an annual attestation. Second source of drift: a model loaded from a mutable path, so the digest changes under a stable name. Pin by digest and treat a digest change with no publish event as a violation on its own.

The point of no return

Switching load-time refusal on for tier one. Teams delete their old deployment paths within a sprint of that, and going back means re-creating them. Everything before it is a flag.

The rollback at each stage

Steps 1 to 4 are a configuration flag per cluster. Step 5 needs the break-glass path tested before it is needed, and the test is part of the rollout, not a follow-up ticket. Step 6 rolls back by shifting expiry dates, which is why expiries must be data in the registry rather than logic in the gate.

How long it really takes

Six to nine months for 140 models, and the engineering is weeks of it. The rest is backfilling validation evidence for models that have been serving for years, which needs model owners, not platform engineers. Plan the programme around that queue and publish its burn-down, or the gate will be ready months before it can be turned on.

When this is the wrong answer

With five models and one team, the spreadsheet and a review meeting are cheaper and work. Admission control earns its cost when the number of models exceeds what one person can hold in their head - a few dozen - or when a regulator will sample the population and the only defensible answer is a complete inventory. Below that, the gate is ceremony with an on-call rotation attached.