Deployed-Artefact Assertion
also called Fleet Version Assertion, Release State Check
A check run after every deployment that compares the build identifier reported by each running host against the intended release, so a partially applied deployment fails loudly instead of running two behaviours at once.
A test suite verifies the code in the repository. An incident is caused by the code that is running. Nothing makes those the same thing except a check, and most release processes do not have one: the deploy tool reports success when its own steps completed, not when every host is serving the intended artefact.
The gap is invisible on ordinary days, because a host running an older but still correct build serves older but still correct responses. It becomes dangerous the moment a release reinterprets existing state — a configuration flag given new meaning, a message format changed, a column whose semantics moved, dead code that some new value makes reachable. Then the fleet is not running old behaviour, it is running different behaviour against the same inputs, and which one a request gets is a matter of routing.
Why it matters
On 1 August 2012 Knight Capital lost roughly $460M in about 45 minutes. The SEC's 2013 order records that new order-routing code had been deployed to eight servers over successive days, that a technician did not copy it to one of them, and that a flag the new code reused activated functionality on the old server that had been dormant for years. Every test had passed, because the untested object was the deployment, not the code.
The general form is more common than the extreme case suggests. Every rolling deployment passes through a window in which two versions are live against the same data, and the length of that window is a design choice — a slow rollout across hundreds of nodes holds it open for tens of minutes, which is the point of going slowly. The assertion is what tells you the window closed.
Implementation patterns
- Every process reports its build identifier on a health or info endpoint, and the deploy step queries all of them and fails if the set is not uniform. This is a few lines of code and it closes the exact gap: one host out of eight.
- Reconcile continuously, not only at deploy time. A host that restarts from a stale image hours later reintroduces the skew. A periodic job comparing running versions against the intended release catches it.
- Alert on version cardinality. More than two distinct versions live at once, or two live for longer than the expected rollout window, is the signal. It is a single number and it is nearly free to collect.
- Treat immutable artefacts as the enabler. If the deployed unit is a content-addressed image rather than a directory someone copies files into, "which version is this" has an exact answer.
- Never reuse a flag name for new semantics. Retire the old key, introduce a new one. The cost is one configuration entry; the alternative is meaning that depends on which binary reads it.
Industry example
Knight Capital is the canonical case because the sequence is fully documented in a regulatory order rather than reconstructed. The details that generalise are mundane: a manual copy step with no second pair of eyes, a reused flag, dead code left behind a condition nobody expected to be true, and no comparison of the eight servers afterwards. The firm's staff spent the window diagnosing and then removed the new code from the seven correct servers, which made the situation worse — rollback without a diagnosis is another change under pressure.
Failure scenarios
- Partial deployment reported as success, because the orchestrator counted the hosts it reached rather than the hosts that exist.
- Autoscaling reintroduces an old image after a clean deploy, so the fleet drifts back into a mixed state with no deployment event to notice.
- A canary that is not in the version set you are asserting, so the check is permanently red and gets disabled.
- Configuration and code versioned separately, so the artefacts match and the behaviour still differs because one host has stale configuration.
- Sidecars and agents left out of the inventory, which is how a security agent or a service mesh proxy ends up three versions behind on a subset of hosts.
Trade-offs
| Choose | Gains | Pays |
|---|---|---|
| Assert after every deploy | Partial deployment surfaces in seconds rather than during an incident | A new failure mode in the deploy pipeline, which will sometimes be the assertion's own bug |
| Assert continuously | Catches post-deploy drift from restarts and scaling | A background job, and alert tuning during legitimate rollouts |
| No assertion | Nothing to build or maintain | Version skew is discovered by its consequences |
The genuine cost is not compute, it is that the check will block deployments, including some it should not. Budget for the first few weeks of false positives from hosts that are legitimately draining.
When not to use it
For a stateless service where an older build serves correct-but-older responses, and releases are strictly additive, this is ceremony. It earns its place when a mixed fleet is semantically mixed rather than merely skewed, and when the cost per minute of wrong behaviour is high. The sharper test: can you name a piece of state whose meaning differs between the two versions? If not, the skew is benign. If you cannot answer at all, that uncertainty is itself the argument for the check.
Note also that the assertion bounds nothing on its own. Knight's loss was bounded by nobody; a pre-trade notional limit would have capped it regardless of why the orders were wrong. The version check tells you the fleet is wrong; a consequence limit decides how much that costs.
Interview question
Q: Your deployment tool reports success. Describe how you would establish, in under a minute and without logging into anything, that every process serving production traffic is running the build you intended — and say what you would do differently for configuration than for code.
What a strong answer covers: a build identifier exposed by every process and aggregated as a metric with version as a label, so the question is a query rather than an investigation; alerting on version cardinality and on the age of the oldest live version; the distinction that configuration changes propagate without a deployment event, so they need their own revision identifier reported the same way; the autoscaling case where a stale image returns after a successful deploy; and the observation that the check is worth building only where a mixed fleet is semantically mixed, with the flag-reuse rule as the cheaper preventive measure.
Quick check
Quiz: Seven of eight hosts run the new build. Under what circumstances is that merely untidy, and under what circumstances is it an incident waiting to happen? — Untidy when the old build serves correct-but-older behaviour; an incident when the release reinterprets existing state, so a flag, message or column means different things depending on which host reads it.
Flashcard: What does a deployed-artefact assertion compare? — The build identifier reported by every running process against the intended release, failing the deploy if the set is not uniform; it is the check that turns "the tool said success" into evidence.