GitOps for Model Deployment

At 02:40 an on-call engineer stops a paging incident by editing the serving deployment directly: replica count up, and the model URI pointed at last week's version. The fix works, the page clears, nobody writes it down. Six weeks later a routine deployment reverts both changes, because the pipeline deployed what the repository said and the repository never knew. The failure is not the hand edit, which was correct. The failure is that no process ever compared the cluster with the intended state.

The principle that does the work

OpenGitOps v1.0 names four: the desired state is declarative, it is stored versioned and immutable, software agents pull it automatically, and those agents continuously reconcile actual state toward it (OpenGitOps, 2021, OpenGitOps v1.0). The first three are good hygiene that a push pipeline can approximate. The fourth is the one that changes the properties of the system, because it turns an out-of-band change from a new baseline into a transient that gets closed.

That behaviour is usually opt-in. In Argo CD, automated sync is off by default, and when you enable it both prune and selfHeal still default to false, so a resource deleted from Git stays in the cluster and a hand-edited resource sits marked OutOfSync with nothing correcting it (Argo CD, Automated Sync Policy). Reconciliation you have not switched on is a diagram, not a control. The useful intermediate position is to enable detection everywhere and self-heal only where you are confident no other controller owns the field.

What the declaration has to name

For a conventional service the manifest names an image digest and some configuration. For a model service it has to name more, and the serving runtimes reflect that: a KServe InferenceService is a single custom resource carrying the model's storage URI, its format, the serving runtime and the autoscaling bounds, from which the controller derives deployment, routing and health checks. One object, one commit, one diff.

The discipline that matters is that every one of those references is immutable. A manifest that pins storageUri: s3://models/ranker/latest/ has declared nothing: the prefix can change under it without a commit, which is the exact property GitOps exists to remove. The same applies to a hosted model named by a floating alias rather than a dated snapshot. Pin the digest, the version prefix, or the snapshot identifier, and the diff becomes meaningful.

What a repository cannot hold

Weights are gigabytes, the retrieval index is tens of gigabytes, the training corpus is larger still, and none of them belong in Git. So the repository holds references, and the immutability guarantee moves to the artefact store, where it is nobody's default. A versioned object path with lifecycle rules that silently expire old versions is not immutable in the sense the second principle requires.

When it breaks

Self-heal fights other controllers. Replica count under a horizontal autoscaler, or the scale annotations a Knative autoscaler owns, will be rewritten by that controller and rewritten back by the reconciler. Those fields belong in ignoreDifferences, and finding out which they are is usually done by watching two loops oscillate in staging.

Rollback is a revert, and a revert takes a commit. Reverting, pushing, waiting for the reconcile loop and waiting for pods to roll is slower than editing the cluster, which is why engineers edit the cluster during incidents. That is the right call under pressure. The discipline is the follow-up commit within the hour, before the next sync undoes it.

The reconciler reports healthy while behaviour changes. A provider updates the model behind an alias, a nightly job rebuilds the retrieval index, an upstream feature pipeline shifts its encoding. The manifest is unchanged, the application is Synced, and the output distribution has moved. Sync status measures agreement with Git, not correctness, which is why the promotion still needs a smoke test against the live artefact.

Secrets reintroduce unversioned inputs. A manifest referring to a secret the repository does not version has an input nobody can diff. Sealed or externally referenced secrets keep the reference in Git without the value, which preserves the property at the cost of another system to operate.

Declarative delivery does not make the change safe. DORA's 2025 report, drawing on nearly 5,000 respondents, found AI adoption associated with higher throughput and higher instability at the same time (Google Cloud, 2025, Announcing the 2025 DORA report). A reconciler ships whatever the commit says, faster and more reliably. The gates in front of the commit are what decide whether that is an improvement.