Restore Granularity

Backup design follows the failure that is expected. A platform that fears losing a database, a shard or a region takes backups it can restore at those boundaries, and restoring the whole thing to a point in time is fast and well practised.

An incident that affects a subset of tenants is a different shape of problem entirely. Rolling the store back would erase everyone else's work since that point, so each affected tenant must be extracted from a backup and reinserted into a live system that has moved on, with identifiers, integrations and cross-service references intact. That path usually exists in principle and has never been run at scale.

Why it matters

Recovery time is a property of the unit, not of the backup. A system with hourly backups and whole-store restore has an excellent RTO for losing a store and an unmeasured one for losing 500 customers, which is the incident that actually happens, because administrative tooling and bad scripts act on lists of tenants rather than on infrastructure.

Three multipliers make the fine-grained path slow. Automation does not exist for the rare path, so it is built during the incident. Verification is per unit, because returning the wrong data to the wrong tenant is worse than a slow restore. And the work is serialised by responders rather than by capacity, so it scales with people.

The consequence is counter-intuitive and worth stating plainly: restoring 775 customers is not 775 times the cost of restoring one. It is worse, because coordination and verification do not parallelise cleanly.

Implementation patterns

• Soft delete with a sweep window. Mark as deleted, remove after 30 days. This converts the entire class of incident into an undo and is by far the cheapest control available.
• Per-tenant export and import as a supported, tested operation, with a measured time per tenant and a known concurrency limit.
• A rehearsal at scale, not at unit size. Restoring one tenant proves the mechanism; restoring 200 proves the throughput, and only the second number is a recovery time.
• Destructive administrative tooling that validates identifier types, shows a dry run and requires a second person for anything above a threshold count.
• Scoped credentials for administrative tools, so the blast radius of a mistake is bounded by permission rather than by care.
• Recovery time published per granularity: whole store, one shard, one tenant, and a batch of tenants. Four numbers, three of which most organisations have never measured.

Industry example

Atlassian's April 2022 incident is the reference case. A maintenance script intended to remove a deprecated application was given identifiers that referred to whole sites, and deleted sites belonging to about 775 customers. Backups existed and were current; restoration nevertheless took up to about two weeks for some customers, because the recovery path had to return individual tenants into shared stores that were serving everyone else. The data was never lost. It was unreachable at the granularity in which it had to be returned.

Failure scenarios

• A subset incident with only whole-store restore available, so recovery means choosing between the affected customers and everybody else.
• Per-tenant restore that works for one and collapses at a hundred, because it was never load tested.
• Cross-service references broken after reinsertion: the tenant's data is back but their integrations, webhooks and identifiers point at things that no longer exist.
• Hard deletes in administrative tooling, removing the option to undo.
• A restore that succeeds into the wrong tenant, which is a data breach created by the recovery.
• Recovery serialised on two people who know the procedure, which caps throughput regardless of infrastructure.

Trade-offs

Half-built granular restore is worse than none, because it creates a belief that fails under load. If it will not be exercised, do not claim it.

When not to use it

For a single-tenant system, the store and the customer are the same unit and this distinction does not exist. For a platform where tenants are already isolated in their own databases, granularity comes free from the topology, which is one of the underrated benefits of per-tenant stores. Where neither holds, prefer soft delete and scoped tooling over building granular restore, because prevention here costs a sprint and the capability costs a quarter.

Interview question

Q: An administrative script has just hard-deleted 300 of your 20000 tenants. Walk me through recovery, then tell me what you would change so that the next occurrence is an inconvenience.

What a strong answer covers: why whole-store rollback is unavailable when others share the store · per-tenant extraction and reinsertion, with cross-service references as the hard part · that verification is per tenant and does not parallelise, so 300 is worse than 300 times one · soft delete with a sweep window as the highest-value prevention · dry runs, identifier validation and scoped credentials for administrative tooling · and publishing recovery time per granularity so the gap is visible before it is discovered.

Quick check

Quiz: Why can backups be healthy while recovery takes two weeks? Because they restore at a granularity larger than the incident, so returning a subset means extraction and reinsertion into a live system rather than a rollback.

Flashcard: What is the cheapest control against a bulk-delete incident? Soft delete with a sweep window, which turns the incident into an undo.