An estate has database passwords in environment variables across 200 services. Design the migration to a secrets manager.
Show the full answer Hide the answer
Sequence it by risk, not by convenience
Phase 0 — stop the bleeding. Secret scanning in CI and on the existing repositories, blocking new commits containing credentials. Otherwise the migration is a race against ongoing creation, and the scan usually finds secrets in history that need rotating regardless.
Phase 1 — inventory and triage. Which secrets exist, what they access, who uses them, and which are shared between services. Shared credentials are the priority: a leak is unattributable and rotation requires coordinating everything using them.
Phase 2 — solve secret zero first. This determines the whole design. If workloads must hold a token to reach the secrets manager, you have moved the problem rather than solved it. Use platform attestation — instance identity, Kubernetes service account tokens, managed identities — so no secret exists at the bootstrap point. Getting this wrong makes everything after it weaker.
Phase 3 — migrate reads, highest-value first. Applications fetch at start-up rather than reading an environment variable. This is a small code change per service and it is the bulk of the work by count.
Phase 4 — rotate everything. Every secret that has lived in an environment variable, a repository or a deployment manifest must be considered exposed. Migration without rotation preserves the compromise.
Phase 5 — dynamic secrets for the highest-value stores. Per-workload credentials with short leases. Rotation stops being an event, attribution becomes possible, and there is nothing at rest to steal.
What to design deliberately
Caching and renewal, so the secrets manager is not on every request path and its brief unavailability does not fail the service — cache the lease, renew before expiry.
Graceful credential expiry handling, because with dynamic secrets connections will eventually be rejected and the application must reconnect rather than fail.
Access policy per workload, not one policy for everything. A migration that gives every service read access to every secret has centralised the secrets and removed none of the blast radius.
The honest constraint
Phase 4 is the one that gets skipped, because rotation requires coordination and the migration already feels complete. It should be the acceptance criterion: the migration is done when every pre-existing credential has been replaced, not when every service reads from the manager.
What a strong answer adds
Proposing a deadline with the old path disabled, because a migration where both mechanisms work indefinitely never finishes. Remove environment-variable support from the shared library, and let the build fail for anyone who has not moved.