Shadow Planning

Before a large structural change — sharding a database, replacing a router, introducing a new query planner — someone has to answer the question "what does the new design fail to handle". The usual method is to read the code, which is reliable only for codebases nobody has worked in.

In a real system the awkward cases are not where anyone looks. They are in ORM call sites that build queries at runtime, in admin tools written for one incident, in background jobs nobody owns, and in a report generated by a scheduler four years ago. Reading cannot enumerate them, and a staging environment cannot either, because staging has neither the data distribution nor the caller population.

Shadow planning inverts the method. The new layer runs on live traffic and produces a plan or a route for every real request, while the old path continues to serve. The plans are logged and analysed offline. What comes back is not an opinion about readiness; it is a ranked list of the requests the new design cannot serve, with frequencies attached.

Why it matters

It converts an unbounded risk into a finite worklist. "We think we have found all the cross-shard queries" is not a statement anyone can act on. "There are 47 query shapes with no shard key, 3 of which account for 90% of the occurrences" is a plan, and it makes the remaining work estimable for the first time.

It also separates the routing decision from the data movement, which is the deeper value. Routing defects are wrong answers and are fixable forward; data-movement defects are lost rows and are fixable only by restore. Proving routing correct while everything still lives in one place means the dangerous step is taken with one variable rather than two.

Implementation patterns

• Compute, log, discard. The shadow path must not affect the response. Any coupling — a shared connection pool, a lock, an error that propagates — turns a diagnostic into an incident.
• Log the plan, not just the failure. The plans that succeed are how you verify that routing chose the right shard, not merely that it chose one.
• Send the log somewhere analytical. Figma logged plans to a warehouse for offline analysis; the volume is large and the questions are ad hoc, which is a warehouse workload rather than a log-search one.
• Rank by frequency and by tail. The top few shapes are usually a day's work; the long tail is where the decision about what to leave unsupported is made.
• Bound the overhead explicitly. Shadow work is real CPU on the serving path, so sample it if necessary and measure the latency effect before enabling it fleet-wide.
• Keep it on after cutover for a period, comparing old and new results, so a regression is a logged difference rather than a customer report.

Industry example

Figma's published account of sharding its Postgres stack (2024) describes a shadow planning framework: candidate sharding schemes are declared for tables, the logical planning phase runs against live production traffic, and the resulting queries and plans are logged for offline analysis. It sits inside a larger sequence — vertical partitioning first, then logical sharding through views with routing in a DBProxy layer, then physical data movement — in which each step is separately reversible and the riskiest one goes last.

Failure scenarios

• The shadow path affects production through a shared resource, and a diagnostic becomes the outage it was meant to prevent.
• Plans are logged but never analysed, so the team acquires the cost and not the information.
• Sampling hides the tail: a 1% sample of traffic finds the common shapes and misses the once-a-day report that cannot be routed at all.
• Success is measured as "no shadow errors" rather than "every plan routed to the right place", so silent mis-routing ships.
• The shadow is switched off at cutover, removing the comparison exactly when it would be most useful.

Trade-offs

When not to use it

When the change is small enough to reason about completely. A service with 30 query shapes, all written by hand, all in one repository, does not need a shadow framework — reading them is faster than building one. The same is true when the new path's failures are loud and cheap: if an unroutable request simply errors and the caller retries against the old path, you can learn the same thing from a canary at a fraction of the cost. Shadowing earns its keep when failures would be silent, the caller population is unknown, or the cutover is irreversible.

Interview question

Q: You are introducing a routing layer in front of a database that 200 services query, many through an ORM. How do you find out what it cannot handle before you move any data?

What a strong answer covers: that code reading and staging both fail for different reasons; running the new planner over live traffic and logging every plan without serving it; ranking unsupported shapes by frequency to size the work; keeping the comparison running past cutover; the isolation requirement so the shadow cannot affect the response; and the sequencing insight — routing goes live before data moves, so routing bugs are never data loss.

Quick check

Quiz: Why does shadow planning find cases that code review cannot? Because queries are built at runtime by ORMs, admin tools and scheduled jobs, so the population of real query shapes exists only in production traffic.

Flashcard: What does a shadow plan log tell you that a staging test does not? The ranked list of real request shapes the new layer cannot route, with frequencies, drawn from the actual caller population rather than from an environment with different data and different callers.