An organisation has development, integration, staging, pre-production and production environments, and bugs still reach production. What should change?

Why the environments are not catching the bugs

They differ from production in the ways that matter. Data volume, data shape, cache state, traffic patterns, third-party behaviour and configuration all differ — and load-dependent and data-dependent failures live precisely in those differences.

A staging environment with a thousand rows and a synthetic traffic generator tests a different system.

What should change

• Reduce the number of environments and increase the fidelity of what remains. Five low-fidelity environments cost more and catch less than two high-fidelity ones.
• Make the pre-production environment genuinely production-like in data volume and shape — which usually means production data, masked, with the masking automated rather than a policy.
• Move verification into production through progressive delivery: deploy to a small percentage, observe real behaviour with real data, and halt automatically. This is the only environment with production's data shape, and it is where the remaining bugs actually surface.
• Use ephemeral per-branch environments for functional verification, created and destroyed automatically, rather than long-lived shared ones that drift and become contended.
• Test the things environments genuinely catch: functional regressions, integration contracts, migration mechanics. Accept that they do not catch load-dependent, data-dependent or emergent failures, and stop expecting them to.

The specific failure of long-lived shared environments

They drift — configuration diverges from production, someone's experiment is left running, data accumulates in an unrealistic state — and they become contended, so a team is blocked waiting for another team's testing.

The drift makes them unrepresentative and the contention makes them slow, which is the worst combination and is the normal state after a year.

The trade-off to state

Production data in a non-production environment is a recurring source of breaches, and the fix is automated masking rather than a policy that someone must remember to follow.

Masked data is less representative than real data, which weakens the fidelity argument — and that residual gap is exactly why progressive delivery in production is the necessary complement rather than an alternative.