Hermetic Release Set

A service is rolled back from v9 to v8 during an incident. The artefact is a known-good one that ran for a month. Within 2 minutes the error rate is worse than before the rollback, because the dynamic configuration moved forward last Tuesday into a shape only v9 parses. v8 with today's config is a combination that has never existed anywhere, and it was created by the one operation nobody reviews and everybody performs under pressure.

The usual pipeline cannot see this: it versions one input and treats configuration as ambient state. Each input was individually valid; the pairing was not.

Why it matters

Rollback is the mitigation of last resort, so a rollback path that can produce an untested state converts a recoverable incident into a compound one. It is also the operation with the least ceremony - no review, no staging run, often a single button, pressed by a tired on-call engineer.

What this pattern buys is categorical rather than procedural. An untested combination is not rejected by a reviewer or caught by a checklist; it is not expressible. That is the difference between a gate and a guardrail, and it is why the control survives organisational entropy.

Implementation patterns

• Name the inputs explicitly: application artefact, static configuration bundled with it, dynamic configuration read at runtime, and often the infrastructure definition. Anything outside the set is ambient state you will be surprised by.
• Register the tuple on success. When a pipeline completes in staging, record the exact combination as an artefact; that registry is what production consults.
• Make deploy a selection, not an assembly. The request names a registered combination rather than assembling one from current values at run time.
• Treat a config change as a release. A dynamic config edit creates a candidate combination that must earn its own staging run before production can select it.
• Split operational switches out. Flags and traffic weights that must move in seconds cannot live in the hermetic set; scope them so a wrong flip degrades a feature rather than breaking the rollback path.
• Expire combinations, or the registry grows combinatorially and a quiet service finds nothing recent it may deploy.

Industry example

Grab's engineering blog describes Conveyor, the continuous-delivery system it built from around 2018 for a transport and deliveries backend of roughly 270 mostly-microservice components. The 2021 instalment introduces hermeticity, borrowed from build systems: deployment parameters are registered as artefacts after a pipeline completes successfully, and production deployments use only combinations already tested in staging. Grab reports this largely removed a class of incidents in which a rollback was incompatible with the dynamic configuration then in force. The same posts note that production pipelines for large rolling clusters can run for hours, with per-cluster locks preventing concurrent deployments.

Failure scenarios

• Partial sets. The infrastructure definition is left out, so a rollback into a node pool that has since changed instance type behaves differently and the hermetic claim is false in the dimension that mattered.
• Registry as a single point of failure. The store of valid combinations is unavailable, so nothing deploys - including the rollback the incident needs. It must be at least as available as the deploy path it gates.
• Staleness. A service untouched for 8 months has no recent registered combination, so an urgent security patch waits for a staging pipeline nobody has run in 8 months.
• Escape hatch erosion. An emergency override is added, then used routinely, and within a quarter the guarantee is decorative.

Trade-offs

You trade speed of configuration change for certainty about what a rollback produces, and the trade is good exactly when rollback is a real part of your recovery strategy.

When not to use it

At 20 services with one configuration source and a 4-minute deploy, this is machinery for a problem you do not have - redeploying the known-good version costs less than operating a registry. It fits badly where configuration is genuinely operational, such as a kill switch flipped mid-incident. And it is premature before deploys are reproducible: if the same pipeline run twice produces different artefacts, registering combinations records noise.

Interview question

Q: "Your team rolled back an application version during an incident and made things worse. Walk me through the mechanism, then design the control that makes it impossible rather than unlikely, and tell me which configuration you would deliberately leave outside that control."

What a strong answer covers: that the inputs move independently so a rollback recreates an untested pairing; that a checklist is a gate applied at the worst possible moment while registration happens when nobody is under pressure; the cost of turning config edits into pipeline runs; and the deliberate carve-out for operational switches.

Quick check

Quiz: Why is rollback the riskiest deploy of the week when a pipeline versions only the application artefact? Because configuration moved on independently, so the old artefact runs against a configuration it has never been tested with - a combination created for the first time during an incident.

Flashcard: What does registering the full deployment tuple after a successful staging run buy? — Production can only select combinations that have actually run, so no rollback creates a novel state; the cost is a pipeline run per configuration change.