Grab's engineering blog describes Conveyor, the continuous-delivery system it built for a transport and deliveries backend that was around 270 services at the end of 2018. The 2021 follow-up introduces hermeticity - a production deployment may only use a combination of application code with static config and dynamic config that has already completed a staging pipeline and been registered as an artefact. What class of incident does that remove, and what does the rule cost?

The situation they were in

A deployment is usually treated as one versioned thing: the application artefact. Everything else - static configuration baked beside it, dynamic configuration read at runtime - moves on its own schedule and is invisible to the deploy record. That is fine until something has to be undone.

Grab's posts describe building Conveyor over roughly two years from 2018, starting with staging automation for a backend of about 270 mostly-microservice components, and then extending it to production.

What they chose

Conveyor treats the deployment as a set of versioned inputs rather than one artefact, and registers the whole combination after a pipeline completes successfully. Production may then only run a combination that has already run. Grab's write-up states this largely removed a class of incidents where a rollback was incompatible with the dynamic configuration then in force.

Why that class of incident exists at all

Roll an application back from v9 to v8 while the dynamic config has moved forward to a shape only v9 understands, and v8 is now running a combination that has never existed anywhere. The rollback - the move everyone reaches for under pressure, with the least review and the most urgency - becomes the riskiest deploy of the day. Nothing in a conventional pipeline notices, because each input was individually valid.

Hermeticity removes it by construction rather than by review: the unsafe combination is not rejected by a human, it is not expressible.

What it costs

• Config changes become deployments. A dynamic config edit that previously took effect in seconds now has to earn a staging run before production can use it, which is slower and is the point.
• The artefact registry becomes a deploy dependency. If the record of valid combinations is unavailable, nothing ships, including the rollback.
• Combinations grow combinatorially, so the registry needs expiry rules, and a long-untouched service can find that nothing it is allowed to deploy is recent.
• Grab notes production pipelines for large rolling clusters can run for hours, with locks acquired per cluster to prevent concurrent deployments, and auto-generated release notes before any change is applied so an engineer can abort.

Where copying it would be a mistake

At 20 services with one config source and a deploy that takes four minutes, this is machinery for a problem you do not have - a redeploy of the known-good combination costs less than building a registry. It also fits badly where configuration is genuinely operational: a feature flag flipped during an incident, or a traffic weight, cannot wait for a staging pipeline. The workable line is to split configuration into deployment parameters, which join the hermetic set, and operational switches, which stay fast and are scoped so that the worst a wrong flip can do is degrade rather than break the rollback path.

Common weak answers

• "Just version the config in the same repository." That makes the inputs change together, which helps, and it does not establish that the combination was ever tested together.
• "Add a pre-rollback checklist." A checklist is a human gate at the worst moment in the week. The lesson here is that the valid set was computed in advance, when nobody was under pressure.