advanced 4 min answer

One Terraform state file holds 4,000 resources across networking, databases, Kubernetes and DNS for production. A plan takes 14 minutes, every change risks unrelated resources, and four teams contend for the lock. You must split it into per-domain states without destroying anything. What is the sequence?

terraformstatemigrationblast-radiusinfrastructure-as-code
Show the full answer Hide the answer

The sequence

  1. Freeze structural change, not all change. Announce a window in which no new resources are added to the monolithic state. Ongoing attribute changes are fine; what you cannot absorb is the resource set moving while you are moving it.
  2. Draw the boundaries by lifecycle, not by technology. The right seam is how often something changes and who owns it: the network and the DNS zones change rarely and are shared; the Kubernetes workloads change daily and are per-team; the databases sit in between. A state per team-and-lifecycle is right; a state per tool is a filing decision that leaves the coupling intact.
  3. Define the interfaces between states before moving anything. Each new state publishes outputs (VPC id, subnet ids, cluster endpoint) and consumes others through a data source or a remote state reference. This is the step that determines whether you have four states or one state in four files. Prefer looking values up by tag or name over remote state references, because a reference creates a hard dependency between states that will bite during an incident.
  4. Move one leaf domain first, and choose the least critical. DNS or observability, not the network. Create the new state, then transfer resources with terraform state mv between states, or import into the new state and remove from the old — the modern path is moved blocks and terraform state rm with an import block, which is reviewable in a plan rather than performed by hand.
  5. Verify with an empty plan, twice. After the move, both states must plan clean: no changes, no destroys. An empty plan on both sides is the only acceptable evidence, and a plan proposing to create something in the new state means the import did not match, while a plan proposing to destroy in the old state means you are one apply away from deleting production.
  6. Repeat per domain, network last, and keep the old state until every domain has been migrated and has survived a normal change cycle.

Where things can diverge, and how you would know

  • A resource in both states. Two states each believing they own a resource produces alternating changes as each applies its own view. Detect it with a script that lists resource ids across all state files and asserts uniqueness, run in CI. This is the check that makes the whole migration safe and almost nobody writes it.
  • A resource in neither state, silently unmanaged, which is worse because nothing notices until the resource is changed by hand or deleted.
  • Drift introduced by the move itself, where the new state's configuration differs subtly from the original — a default that the provider computes, a tag the old module added. The empty plan catches this if you insist on it.

The point of no return

terraform state rm on the old state, once the new state holds the resource. Before that you can abandon the new state and continue. After it, the old state no longer knows about the resource, and the rollback is a re-import rather than a revert. So the order is always: import into the new state, verify an empty plan there, then remove from the old — and back up every state file before each step, because the file is the only record of the mapping.

The rollback at each stage

Re-import into the old state from the backup. This is why the backup and a written note of which resources moved in which step are not optional: without the mapping, recovery means reconstructing it from the provider's API under pressure.

How long it really takes

For 4,000 resources across four domains, with a team that has done it before: 2 to 3 weeks of elapsed time, mostly waiting for verification cycles. First-timers should assume 6 weeks, because the interface design in step 3 takes two attempts and the first domain takes as long as the other three.

When this is the wrong answer

If the plan takes 14 minutes and nobody is blocked, do not do this. The justification is lock contention and blast radius, so if one team owns everything and changes are weekly, a single state is simpler and the split adds four pipelines, four sets of credentials and a cross-state dependency graph. And if the real problem is only plan duration, targeted plans and provider caching are a day of work against a six-week migration. Split state when independent teams need to apply independently, and not before.