Versioning Compound LLM Systems

Between March and June 2023, GPT-4's accuracy at identifying whether a number is prime fell from 84 percent to 51 percent, measured through the same API name with the same prompts (Chen, Zaharia & Zou, 2023, How is ChatGPT's behavior changing over time?, arXiv:2307.09009). Any team whose application depended on that behaviour had shipped nothing and still had a new release in production. That is the versioning problem for LLM systems in one number: behaviour is determined by several components, some of which change without you.

What a model version contains lists the prompt, tools and retrieval configuration as part of an LLM bundle. This concept takes that paragraph and makes it the whole design problem, because in a compound system the weights are usually the one component you do not control.

The unit of behaviour

A compound AI system reaches its result through multiple interacting components: model calls, retrievers and external tools (Zaharia et al., 2024, The Shift from Models to Compound AI Systems, BAIR blog). Its output for an input \(x\) is better written as

where \(m\) is the model snapshot, \(p\) the prompt templates including few-shot examples, \(r\) the retrieval state (embedding model, chunking rules and index snapshot), \(T\) the tool schemas and the versions of the services behind them, \(\theta\) the decoding parameters, and \(g\) the guardrail and post-processing logic. Changing any argument changes \(f\).

The practical identifier is a content hash over a manifest, \(v = H(h_m \,\|\, h_p \,\|\, h_r \,\|\, h_T \,\|\, h_\theta \,\|\, h_g)\), where each \(h\) is itself a hash or an immutable reference: a dated model identifier rather than an alias, a git SHA for prompts, an index snapshot ID. This is a lockfile for behaviour. Evaluation results, traces and incidents attach to \(v\), so "which system produced this output" has one answer.

Prompts deserve the same rigour as weights because small edits are large changes. Meaning-preserving formatting variations in few-shot prompts moved LLaMA-2-13B accuracy by up to 76 points (Sclar et al., 2024, Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design, ICLR, arXiv:2310.11324). A whitespace change in a template is a release.

The retriever is the awkward component

Retrieval state is large, slow to rebuild and has a freshness requirement that pulls against reproducibility. Two rules keep it versionable. Vectors produced by different embedding models live in unrelated spaces, so an embedding-model change is a full re-index and a new \(h_r\), never an in-place update. And an index that ingests documents continuously has no stable identity, so reproducible evaluation needs periodic immutable snapshots, with the live index versioned as "snapshot plus a bounded delta".

Teams disagree about how far to push this. One camp versions the whole system monolithically and re-evaluates on any change. The other versions components independently with a compatibility matrix, arguing that a monolithic version forces a full evaluation for a typo fix. The monolith is safer and slower; the matrix scales and misses the interactions between components, which are exactly the failures that are hard to find.

Deprecation turns rollback into migration

With self-hosted weights, rollback means redeploying an older artefact. With a hosted model, the older artefact can cease to exist. As of September 2026, OpenAI's policy promises at least six months' notice for generally available models and as little as two weeks for previews; gpt-4.5-preview was announced for shutdown on 14 April 2025 and removed on 14 July 2025 (OpenAI, Deprecations). Anthropic commits to at least 60 days' notice for publicly released models; claude-sonnet-4-20250514 was deprecated on 14 April 2026 and retired on 15 June 2026 (Anthropic, Model deprecations).

Deprecation reaches beyond model names. The same Anthropic page records that setting temperature, top_p or top_k to non-default values returns an error on Claude Opus 4.7 and later, so a pinned \(\theta\) can become invalid on the replacement model. A migration therefore changes \(m\) and possibly \(\theta\) and \(p\) at once, and needs the full evaluation any major version gets.

The planning consequence is a clock. If notice is 60 days and a full evaluation and prompt re-tuning cycle takes three weeks, a team has roughly one attempt with a margin for failure. Keeping a migration-ready evaluation suite and testing the likely successor before the deprecation notice arrives is what makes the clock survivable.

When it breaks

Aliases defeat pinning. An alias such as "latest" is a moving pointer. A manifest that records it has recorded nothing, and the Chen et al. drift is exactly what an alias delivers.

Tools change behind their schemas. A tool's JSON schema can stay constant while the API behind it changes its results. \(h_T\) should include the service version where one exists, and tool responses belong in traces so that a behaviour change can be attributed.

Rollback targets expire. A version whose model has been retired cannot be restored. The registry should mark such versions as non-restorable, so an incident runbook does not name a rollback target that returns an error.

Re-indexing has a real cost. Re-embedding 50 million chunks of 500 tokens is 25 billion tokens. At an illustrative $0.02 per million embedding tokens that is only $500, but the throughput limits and the dual-index period, when both old and new indexes must serve during cut-over, usually dominate the cost and the calendar.