intermediate 2 min answer

A change passes every test in staging and fails immediately in production. Staging is a faithful copy of the topology. What are the most likely causes, and what would you change?

environmentsparitytesting
Show the full answer Hide the answer

What the interviewer is testing

Whether you know which dimensions of environment parity actually catch defects, rather than treating parity as a single property that is either present or absent.

The likely causes, in order of frequency

Data. Staging has 10,000 rows; production has 100 million with heavy skew. The query planner chooses a different plan, an index that was never needed becomes essential, and a full scan that took 40 ms takes 40 seconds. This single factor accounts for more of these failures than everything else combined.

Concurrency. Staging is exercised by one tester at a time. Production has 800 concurrent requests, so lock contention, connection pool exhaustion and race conditions appear for the first time.

Dependencies. Staging calls mocks that return instantly and never fail. Production calls a partner API with a 400 ms p99, a rate limit and occasional 503s.

Configuration. Different feature flag state, different timeout values, different TLS termination, a secret that resolves differently.

Traffic shape. Production has bots, retries, cached clients, an unusual locale, and users doing things nobody scripted.

Note that "faithful copy of the topology" — the same number of instances and services — is the dimension that catches the fewest defects, and it is the one most expensive to maintain.

What I would change

Reallocate the parity spend. Give the environment production-scale data with realistic distribution, generated or masked. Replace instant mocks with service virtualisation that reproduces latency and error behaviour. Run a load profile that matches the production mix rather than a smoke test.

Then accept that the remaining gap is irreducible and cover it downstream: canary release with automated analysis, feature flags to limit exposure, and fast rollback. Beyond a point, money spent on parity buys less than money spent on making production failure safe and brief.

What a strong answer adds

Naming the specific test that would have caught it — a load test at production cardinality is usually the answer — and observing that the environment's value should be judged by which incidents it has actually prevented.

Common weak answers

"Make staging identical to production", which is unattainable and misallocates the budget. Blaming the test suite without identifying which class of defect escaped.