A Model Version Is No Longer a File: Registries for Compound AI Systems

On 15 June 2026 every request to claude-sonnet-4-20250514 on Anthropic's API began to fail. Anthropic had announced the retirement 62 days earlier and named a replacement (Anthropic, Model deprecations). For any team whose last known-good release pinned that snapshot, the rollback target in their runbook stopped existing that morning. Their registry still listed the version and its evaluation report; it could no longer be served.

The model registry, as MLflow, SageMaker and Vertex AI built it between 2019 and 2022, assumes a model version is an artefact you trained and store, restorable for as long as you keep the bytes. An LLM application's behaviour comes instead from a hosted model you do not own, prompts you edit weekly, a live index and tools whose semantics drift. It is a configuration of dependencies, some of which expire.

Why this matters: The version number is the only thing an incident responder, an auditor or an evaluator can hold on to. If it names a file, it misses the prompt edit that caused the regression. If it names an alias, it names nothing. If it names a retired snapshot, the rollback it promises is fiction.

TL;DR

• The release unit is a manifest of pinned components (model snapshot, prompts, index snapshot, tools, decoding, guardrails) identified by a content hash that every eval, trace and incident attaches to.
• Pinned names drift. Behind one API name, GPT-4's accuracy on a balanced prime-versus-composite task fell from 84.0% to 51.1% between March and June 2023, while its multi-hop exact match rose from 1.2% to 37.8% (Chen, Zaharia & Zou, 2024).
• Rollback has an expiry date: the earliest retirement among a manifest's components. Stated notice floors run from about 2 weeks (previews) to 60 days (Anthropic) to 6 months (OpenAI GA models).
• A behavioural PATCH means equivalent within a stated margin. Certifying ±2 points when 9% of cases flip between versions needs about 1,900 paired cases.
• Nondeterminism sets a floor: 1,000 temperature-0 completions of one prompt on Qwen3-235B gave 80 distinct outputs until batch-invariant kernels were used (He & Thinking Machines Lab, 2025).
• Changes interact. In the worked example a model swap alone adds 1 refund violation per 300 cases, a prompt edit alone adds 2, together they add 11, and a +1.0 point aggregate gate passes a 4-point slice regression.

At a Glance

flowchart LR
    subgraph Parts["What produces behaviour"]
        M["Model snapshot"]
        P["Prompt templates"]
        R["Embedder plus index snapshot"]
        T["Tool versions"]
        G["Guardrails and decoding"]
    end
    J["Pinned evaluator"]
    H["Manifest hash v"]
    E["Eval evidence per slice"]
    A["Alias prod"]
    D["Provider retirement dates"]
    W["Restorable until min expiry"]

    M --> H
    P --> H
    R --> H
    T --> H
    G --> H
    H --> E
    J --> E
    E -->|"gate passes"| A
    D --> W
    H --> W

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    class M,P,R,T,G,J blue
    class H,E purple
    class A teal
    class D,W amber

From Trained Artefacts to Moving Dependencies

In 2015 Sculley and colleagues at Google named the property that makes ML versioning hard: "We refer to this here as the CACE principle: Changing Anything Changes Everything" (Sculley et al., 2015, NeurIPS). The list of things that change has since grown, and several are changed by someone else.

timeline
    title From model files to release manifests
    2015 : Sculley et al. name the CACE principle
    2016 : ModelDB manages models and metadata
    2019 : MLflow Model Registry launches with stages
    2020 : SageMaker registry gates deployment on approval
    2022 : Vertex AI Model Registry GA with aliases
    2023 : Chen, Zaharia and Zou measure drift behind API names
         : MLflow 2.9 deprecates stages
    2024 : BAIR essay names compound AI systems
    2025 : MLflow 3 LoggedModel links traces, prompts and evals
    2026 : Claude Sonnet 4 snapshot retired on 15 June

After ModelDB's model catalogue (Vartak et al., 2016), the MLflow Model Registry (17 October 2019) added governance: numbered versions moving through stages such as Staging and Production (Databricks, 2019). In SageMaker, from December 2020, a "Pending manual approval" package would "block deployment until the model has been reviewed and approved manually" (Simon, 2020). Vertex AI went GA in October 2022 with aliases instead of stages (Nardini & Dugan, 2022), and by 2.9.0 MLflow had deprecated stages over "the inflexibility of model stages for expressing MLOps workflows" (MLflow docs).

The second lesson came from outside. In July 2023 Chen, Zaharia and Zou compared March and June versions of GPT-3.5 and GPT-4 behind the same API names: "the behavior of the 'same' LLM service can change substantially in a relatively short amount of time" (arXiv:2307.09009). Narayanan and Kapoor rebutted the preprint: its primality test contained only primes, and with 500 composites added "much of the performance degradation the authors found comes down to this choice of evaluation data"; the code metric penalised explanatory text around correct code (Narayanan & Kapoor, 2023). The published version used 500 primes and 500 composites and still reported GPT-4 falling from 84.0% to 51.1% (Chen et al., 2024, HDSR). For production the capability-versus-behaviour argument cuts one way: your parser cares whether the model emits bare code, not whether it could. And a badly built suite will mistake a bias flip for a regression.

In February 2024 Zaharia and co-authors named the architecture: a compound AI system "tackles AI tasks using multiple interacting components, including multiple calls to models, retrievers, or external tools" (BAIR, 2024). MLflow 3 (June 2025) promised "comprehensive lineage between models, runs, traces, prompts, and evaluation metrics" (MLflow 3). The registry's unit had shifted from a trained file to an application configuration.

[IMAGE: Two panels. Left, "2019 registry": a model file with a SHA badge moving through Staging and Production. Right, "2026 release unit": a manifest card of hashed leaves, with a red calendar on the model row reading "retires 15 Jun". Caption: "The unit of versioning moved from a file you own to a manifest that expires."]

How Versioning a Compound System Actually Works

The manifest is a lockfile

Package managers separate what you ask for (^4.2) from what you got (4.2.7 plus a hash). An LLM application needs the same split. Write its output for input \(x\) as

where each \(c_i\) is a component: model snapshot \(m\), prompt set \(p\), retrieval state \(r\) (embedder, chunking, index snapshot), tool versions \(T\), decoding parameters \(\theta\), guardrails \(g\). A manifest is releasable only if every \(c_i\) is an immutable reference: a dated model ID, a git SHA, a snapshot ID, a digest. The resolver's job is to reject aliases. Gemini's latest alias "will get hot-swapped with every new release of a specific model variation", with two weeks' email notice only for breaking changes (Google, Gemini models). An alias is a pointer someone else moves.

The release identity is a Merkle root over component hashes:

An unchanged configuration always yields the same \(v\), and "what changed between 3.4.0 and 3.5.0" becomes a comparison of \(k\) leaf hashes rather than archaeology in deploy logs. The prompt leaf is itself a tree, hashing each template and few-shot example, because formatting alone moved LLaMA-2-13B few-shot accuracy by up to 76 points (Sclar et al., 2024, ICLR).

The evaluator is the forgotten leaf. A judge's snapshot and rubric produce no user-facing behaviour, so they stay out of \(v\), but their hash belongs in the evidence record, and tooling should refuse comparisons across evaluator hashes.

Three sources of change

Authored changes (prompts, tools, guardrails) are yours; the only question is whether they were evaluated.

Dependency changes follow suppliers' schedules, and you choose when to adopt them only until retirement. As of September 2026, OpenAI promises at least six months' notice for generally available models, three for specialised variants and as little as two weeks for previews, unless safety or compliance require faster action (OpenAI, Deprecations). Anthropic promises at least 60 days for public models, and notes that Amazon Bedrock and Google Cloud "set their own retirement schedules", so one model name can expire on different days in different clouds (Anthropic).

Environmental changes need no release: an index ingests documents, a tool returns new data, the serving stack batches your request differently. The last is measured: 1,000 temperature-0 completions of one prompt from Qwen3-235B gave 80 unique outputs, first diverging at token 103, because common kernels are not batch-invariant; with batch-invariant kernels all 1,000 matched (He & Thinking Machines Lab, 2025). A manifest's behaviour is therefore a distribution, not a function.

Change decomposes, and does not add up

If two manifests differ in model and prompt, a metric changes by

where \(\Delta_m\) is the effect of changing only the model, \(\Delta_p\) only the prompt, and \(\Delta_{mp}\) the interaction. In a \(2 \times 2\) design with \(\mu_{ab}\) the metric for model \(a\) and prompt \(b\):

CACE says \(\Delta_{mp}\) is not negligible, and for LLM systems it often dominates: a prompt written against one model's habits encodes assumptions the next model does not share. That is the case for versioning the manifest as one unit. The cost is \(2^k\) runs for a full factorial over \(k\) changed leaves, and cells vanish once an old component retires.

Semantic versioning for behaviour

SemVer has a precondition people skip: "Software using Semantic Versioning MUST declare a public API" (Preston-Werner, SemVer 2.0.0). An LLM application's public API is the behaviour users rely on, which in practice means an evaluation contract: named slices (refunds, account closures, non-English queries), a metric on each, a direction of harm, and a tolerance \(\delta_s\).

A version bump becomes a statistical test on same-case paired differences \(d_i = s_i^{\text{new}} - s_i^{\text{old}}\). Evan Miller recommends inference "on the question-level paired differences, rather than the population-level summary statistics", with

and shows that when two models' scores correlate at 0.5, pairing cuts estimator variance by a third (Miller, 2024, arXiv:2411.00640). For binary outcomes, \(\operatorname{Var}(d) = \pi_{10} + \pi_{01} - (\pi_{01} - \pi_{10})^2\), where \(\pi_{10}\) is the fraction of cases only the old manifest got right and \(\pi_{01}\) only the new one. Cases where both agree contribute nothing: variance is driven entirely by disagreement.

The classification reads:

• PATCH if every protected slice is equivalent: the 90% interval of \(\bar d\) lies inside \([-\delta_s, +\delta_s]\) (two one-sided tests at \(\alpha = 0.05\)).
• MINOR if every slice is non-inferior (the one-sided bound in the harmful direction is within \(\delta_s\)) and some slice improves.
• MAJOR if any slice fails non-inferiority, the schema breaks, or the evidence cannot show either.

The last clause is the one teams resist, and the one that makes the number honest.

How much evidence is enough? For an equivalence test with true difference zero, the required paired cases are approximately

At \(\alpha = 0.05\) and 80% power, \(z_{0.95} + z_{0.90} = 1.645 + 1.282 = 2.927\). If 9% of cases flip, \(\sigma_d \approx \sqrt{0.09} = 0.30\), and a ±2-point margin needs \(n \approx (2.927 \times 0.30 / 0.02)^2 \approx 1{,}930\). A ±1-point margin quadruples that to about 7,700; at 4% discordance ±2 points needs about 860. Nondeterminism adds discordance even between two runs of the same manifest, so A/A disagreement sets the minimum suite size for any PATCH claim.

[IMAGE: Line chart of required paired cases (log scale) against discordance rate 2% to 16%, one curve per margin of 1, 2 and 3 points, with a band marking "typical golden set, 200 to 500 cases". Caption: "Suite size for a PATCH claim is set by how often cases flip."]

Rollback is a predicate with an expiry date

With self-trained weights, rollback redeploys bytes you kept. With a manifest, rollback to \(v\) works only while every component is servable. Define the restorable horizon

where expiry is a model's retirement date, an index snapshot's retention deadline, a tool API's end of support, and effectively infinity for a prompt in git. After \(R(v)\), "roll back to \(v\)" is not an operation, so the registry should show \(R\) beside every release. Anthropic's commitment to preserve weights of public models "for, at minimum, the lifetime of Anthropic as a company" (Anthropic, 2025) does not change \(R(v)\): preserved weights are not an endpoint.

Rollback also stops being binary: when the previous manifest is gone, the moves are partial (new model, old prompt), each a manifest that has never been evaluated, which the registry must build and evaluate in hours.

Seeing It in Motion

The implementation is a few services joined on the manifest hash.

flowchart TB
    subgraph Sources["Authored sources"]
        GIT["Prompts and tool code in git"]
        IDX["Index snapshot store"]
    end
    subgraph External["Supplier dependencies"]
        PROV["Provider model catalogue"]
        DEP["Deprecation feed"]
    end
    RES["Resolver rejects aliases"]
    MAN["Manifest store keyed by hash"]
    EVAL["Eval runner with pinned judge"]
    LIN["Lineage and evidence graph"]
    GATE["Semver classifier and gate"]
    ALIAS["Alias router prod and canary"]
    TR["Trace store tagged with hash"]
    HOR["Restorable horizon monitor"]

    GIT --> RES
    IDX --> RES
    PROV --> RES
    RES --> MAN
    MAN --> EVAL
    EVAL --> LIN
    LIN --> GATE
    GATE --> ALIAS
    ALIAS --> TR
    TR --> LIN
    DEP --> HOR
    MAN --> HOR
    HOR -.->|"flags non-restorable"| LIN

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    class GIT,IDX,PROV blue
    class RES,MAN,EVAL,LIN,GATE purple
    class ALIAS,TR teal
    class DEP,HOR amber

Traces carry the serving manifest's hash, so a complaint joins to the exact configuration and its evidence. The horizon monitor is the new piece. Two lifecycle transitions are triggered by suppliers, which the 2019 stage model could not express.

stateDiagram-v2
    [*] --> Draft
    Draft --> Resolved: all refs immutable
    Resolved --> Evaluated: suite run on pinned judge
    Evaluated --> Rejected: slice fails gate
    Evaluated --> Candidate: gate passes
    Candidate --> Live: canary healthy
    Candidate --> Rejected: canary regresses
    Live --> Superseded: newer release live
    Superseded --> Live: rollback
    Superseded --> NonRestorable: component retired
    Live --> Forced: component deprecated
    Forced --> Superseded: migration shipped
    NonRestorable --> Archived
    Rejected --> Archived
    Archived --> [*]

Forced begins the day a provider announces a retirement; the path from Superseded back to Live closes on the retirement date. In an incident the decision is a short exchange:

sequenceDiagram
    participant O as On-call engineer
    participant Reg as Registry
    participant Ev as Eval runner
    participant Rt as Alias router
    O->>Reg: Diff live release against previous
    Reg-->>O: Model, prompt and index leaves changed
    O->>Reg: Is previous release restorable
    Reg-->>O: No, model retired 15 June
    O->>Reg: Build hybrid with new model and old prompt
    Reg->>Ev: Evaluate hybrid on protected slices
    Ev-->>Reg: Paired deltas with intervals
    Reg-->>O: Hybrid passes refund slice gate
    O->>Rt: Point prod alias at hybrid hash
    Note over O,Rt: The hybrid is a new release with its own evidence

[IMAGE: Mock registry page for "support-bot 3.5.1": component leaves with green "unchanged" or amber "changed" badges, a "Restorable until" date, and per-slice paired deltas with interval bars. Caption: "A release page leads with diffs, slice evidence and the restorable horizon."]

By the Numbers

Sources: Chen, Zaharia & Zou (HDSR, 2024); Sclar et al. (arXiv:2310.11324); He & Thinking Machines Lab (2025); OpenAI (Deprecations), Anthropic (Model deprecations) and Google (Gemini models) pages, September 2026; notice computed from published dates. The timing row is a separate Qwen3-8B experiment. OpenAI does not say which tier the 20-day gpt-5.4-cyber notice falls under.

[IMAGE: Gantt-style chart, one bar per retirement above from announcement to shutdown, coloured by provider, with a vertical line at 60 days. Caption: "A two-month window leaves room for about one full migration attempt."]

A Concrete Example

The bot and every behavioural number here are illustrative; model IDs are placeholders, and dates mirror the real April to June 2026 window.

A support bot handles 150,000 conversations a week, a quarter of them refunds. A policy violation (an unallowed refund promise) costs about $30; an escalation costs about $4.

Step 1: the forced migration. Release 3.4.0 runs on vendor-large-2025-05-14. On 14 April the provider announces its retirement for 15 June. The team builds 3.5.0:

version: 3.5.0            # labelled MINOR by the team
model:    vendor-large-4.6         # was vendor-large-2025-05-14
decoding: {max_tokens: 1024}       # temperature removed
prompts:  git:prompts@a71be03      # v42, refund rules shortened
index:    kb-snapshot-2026-05-01   # was 2026-04-01
evidence: golden-v7 (1,200 cases; refund slice 300), judge pinned

The removed temperature is realistic: on Anthropic's Claude 4.7 and later models a non-default temperature, top_p or top_k returns a 400 error (Anthropic).

Step 2: the gate that passed. On all 1,200 cases, resolution rises from 83.5% to 85.5% and violations from 1.75% (21) to 2.75% (33), exactly on a "+1.0 point at most" gate. 3.5.0 ships on 25 May; the old model retires on 15 June.

Step 3: paired analysis. On 22 June support reports out-of-policy refunds. The engineer pulls stored per-case results, since 3.4.0 cannot run. Of 300 refund cases, 276 are clean under both, 6 violate under both, 3 only under 3.4.0, and 15 only under 3.5.0:

A 4.0-point regression, lower bound above a 1-point tolerance; McNemar's \((15-3)^2/(15+3) = 8.0\) gives \(p \approx 0.005\). The aggregate diluted it across 900 unaffected cases. Resolution genuinely improved (42 cases right only under 3.4.0, 66 only under 3.5.0: \(\bar d = +0.020\), 95% CI \([+0.003, +0.037]\)). By the behavioural rule, 3.5.0 was an unevidenced MAJOR change.

Step 4: attribution. In May the team had run single-change ablations, so all four cells exist for the refund slice:

The new index added one more (20 to 21). The regression is mostly interaction: v42 shortened the rules, the old model filled the gap from habit, the new one read them literally. The old-model cells exist only because they were run before 15 June.

Step 5: the decision. Rollback is impossible.

Against 3.4.0's stored results, 3.5.1 has 4 refund cases violating only under 3.4.0 and 5 only under 3.5.1: +0.3 points, 90% CI \([-1.3, +2.0]\). That does not fit inside ±1 point, so the suite cannot certify equivalence; 300 cases are too few. It is clearly better than 3.5.0.

Cost settles it. Of \(37{,}500\) weekly refund conversations, 3.5.1 avoids \((21-10)/300 \times 37{,}500 \approx 1{,}375\) violations (about $41,000) and adds 1,500 escalations ($6,000): net, about $35,000 a week for the hybrid while v43 is written.

Step 6: bookkeeping. 3.5.1 ships as its own release with its own hash, evidence and horizon, not as a rollback. The refund slice becomes protected, gated on the upper confidence bound, and grows toward the 1,900 cases a ±2-point claim needs.

[IMAGE: Panel strip: manifest diff with amber leaves; aggregate gate with a green tick; the 2x2 grid with the interaction cell in red; an interval bar right of the tolerance line; a new 3.5.1 release card. Caption: "After the old model retires, rollback becomes construction of an evaluated hybrid."]

Where It Breaks

Aliases leak back in

Teams pin the chat model and forget the rest: the embedder uses a family name, the judge uses latest. The fix is a resolver that fails the build on any identifier without a date suffix, digest or SHA, including in eval configs, where reviewers look least.

Aggregates hide slices

A regression in 25% of traffic appears at a quarter of its size in the aggregate. Per-slice confidence bounds fix the arithmetic, not coverage; slices found in incidents should become protected, so the contract grows like a test suite.

Nondeterminism floors every claim

If two runs of one manifest disagree on 3% of cases, no practical suite certifies a 0.5-point margin and every release becomes MAJOR. Publish the margin the suite supports; re-measure A/A discordance on every provider change.

Judges, indexes and tools drift unreleased

A judge whose snapshot changes between runs mixes grader change into every paired difference; re-grading an old manifest needs stored raw outputs, not scores. A live index has no stable identity, so evaluate on a snapshot and re-evaluate once newer documents pass a size threshold; a new embedding model means a full re-index, since vectors are not comparable across models. A refunds API can keep its schema and change what "eligible" means, so store tool versions and responses in traces.

Hosted release infrastructure retires

On 3 June 2026 OpenAI announced the shutdown of reusable prompt objects, its Evals platform and Agent Builder by 30 November 2026, advising teams to "move reusable prompt content into your application code" (OpenAI). History kept in a supplier's product has the snapshot's expiry problem one layer up.

[IMAGE: Grid of mini-diagrams: an alias arrow jumping boxes; a slice bar hidden in an aggregate bar; one manifest producing two output histograms; a judge icon changing under an eval chart; a vendor prompt store with a shutdown date. Caption: "Each failure is an unversioned or unrunnable leaf."]

Alternative Designs

Manifest versus matrix is a live disagreement. The matrix camp is right that re-evaluating everything for a one-word prompt fix is wasteful; the manifest camp is right that customer-visible regressions are disproportionately interactions. A defensible middle: version as a manifest, but scale suite depth to which leaves changed.

How It Is Used in Practice

MLflow 3's LoggedModel and prompt registry (MLflow 3) are the closest open-source analogue to this design, though restorable horizons are left to the user. SageMaker approval status and Vertex AI aliases supply the sign-off and the mutable production pointer.

DSPy changes what a prompt leaf is: it compiles modules into optimised prompts, reporting gains over standard few-shot prompting of over 25% for GPT-3.5 and 65% for llama2-13b-chat (Khattab et al., 2023). The prompt becomes a build output, so source and compiled prompt both need hashes.

Teams that absorb retirements calmly audit usage by model ID continuously, evaluate the likely successor before the notice, and store per-case outputs of the live release. Canaries still help, and Google's SRE workbook warns that before/after evaluation is risky (Google SRE Workbook), but with violations at a few per thousand, canaries catch gross failures and offline paired evaluation catches slices.

Insights Worth Remembering

1. A version number for an LLM application is a statistical claim. MAJOR, MINOR and PATCH mean something only with paired, slice-level evidence and stated margins. Without it, any model change is MAJOR.

2. Rollback has a horizon you can compute. The earliest retirement among a release's components is the last day it can be restored. A registry that hides that date promises what it cannot deliver.

3. Variance comes from disagreement, not accuracy. Only discordant cases contribute to the paired standard error, so A/A discordance under nondeterminism is the first calibration for any gate.

4. The expensive regressions are interactions. Component-wise testing, however thorough, is blind to \(\Delta_{mp}\).

5. Evidence not stored before a retirement is evidence you will never have. Ablations of the outgoing model are cheap in May and impossible in June.

6. The evaluator is a dependency of every number. Pin the judge and rubric; refuse comparisons across evaluator hashes.

Open Questions

Will long-lived snapshot access become a product? Known: stated floors are 60 days at Anthropic and six months for OpenAI GA models, and Anthropic has committed to preserving weights, not endpoints. OpenAI says some developers "may be able to provision dedicated capacity for continued access after a model's shutdown date" via sales. Whether that becomes a priced, published tier is unknown.

Will batch-invariant inference become a hosted default? Measured: it removed all variation in one 1,000-sample experiment; in a separate Qwen3-8B benchmark it ran in 42 seconds against vLLM's 26 (He et al., 2025). Unknown: whether providers will offer a deterministic tier, which would shrink PATCH suites.

How should contracts be specified for open-ended output? For judge-scored generation there is no agreed equivalence margin, and whether judge scores are stable across judge versions is open.

Can attribution scale past small factorials? \(2^k\) works for three leaves, not eight. Shapley-style attribution is plausible, but the author knows of no published evidence that it recovers interaction effects in LLM pipelines at acceptable cost.

Sources and Further Reading

1. Zaharia, M., et al. (2024, February 18). "The Shift from Models to Compound AI Systems." BAIR Blog. bair.berkeley.edu
2. Chen, L., Zaharia, M., & Zou, J. (2024). "How Is ChatGPT's Behavior Changing Over Time?" Harvard Data Science Review, 6(2). doi:10.1162/99608f92.5317da47. Preprint arXiv:2307.09009.
3. Narayanan, A., & Kapoor, S. (2023, July 19). "Is GPT-4 getting worse over time?" AI Snake Oil. normaltech.ai
4. Sculley, D., et al. (2015). "Hidden Technical Debt in Machine Learning Systems." NeurIPS 28. papers.nips.cc
5. Vartak, M., et al. (2016). "ModelDB: A System for Machine Learning Model Management." HILDA@SIGMOD. doi:10.1145/2939502.2939516
6. Mewald, C., Zaharia, M., & Simeone, C. (2019, October 17). "Introducing the MLflow Model Registry." Databricks Blog. databricks.com
7. MLflow. "MLflow Model Registry," 2.10 documentation mlflow.org; "MLflow 3" release notes (2025, June 11) mlflow.org/releases/3
8. Simon, J. (2020, December 8). "Amazon SageMaker Pipelines Brings DevOps Capabilities to your Machine Learning Projects." AWS News Blog. aws.amazon.com
9. Nardini, I., & Dugan, S. (2022, October 8). "Vertex AI Model Registry" (GA announcement). Google Cloud Blog. cloud.google.com
10. Sclar, M., et al. (2024). "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design." ICLR 2024. arXiv:2310.11324
11. Khattab, O., et al. (2023). "DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines." arXiv:2310.03714
12. Miller, E. (2024). "Adding Error Bars to Evals." arXiv:2411.00640
13. He, H., & Thinking Machines Lab. (2025, September 10). "Defeating Nondeterminism in LLM Inference." thinkingmachines.ai
14. OpenAI. "Deprecations." API documentation, accessed September 2026. developers.openai.com
15. Anthropic. "Model deprecations," accessed September 2026 (platform.claude.com); "Commitments on model deprecation and preservation" (2025, November 4) (anthropic.com).
16. Google. "Gemini models." Gemini API documentation, accessed September 2026. ai.google.dev
17. Preston-Werner, T. "Semantic Versioning 2.0.0." semver.org; Google. "Canarying Releases." The Site Reliability Workbook. sre.google

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.