What does testing in production genuinely mean, and what must exist before it is responsible?

What it means

Verifying properties that can only be verified with real traffic, real data and real conditions — not skipping testing.

Some properties are simply not available beforehand: real access distributions with their skew and correlated bursts, real network conditions across devices and geographies, third-party behaviour under real load, interaction with data that should not exist and does, true capacity limits, and user-perceived quality measured at the client.

The techniques

• Feature flags, so code is deployed inactive and enabled for a controlled population.
• Canary releases with automatic promotion and rollback, gated on a comparison against a control group rather than against yesterday.
• Shadow traffic, sending real requests to a new implementation and comparing responses without serving them — which finds behavioural differences no test suite will.
• Synthetic monitoring from diverse vantage points, exercising critical journeys continuously.
• Continuous low-level fault injection, so fallback paths are exercised routinely.

What must exist first

1. Observability sufficient to detect impact quickly, segmented — because an aggregate cannot distinguish "everyone slightly degraded" from "a segment completely broken", and the second is what a bad rollout produces.

2. A tested kill switch, operable without a deployment.

3. Blast-radius control, so an experiment affects a bounded population.

4. Business metrics alongside technical ones, since the failures that matter here are frequently the ones where the system works correctly and produces the wrong result.

5. Data safety. An experiment must not corrupt data or produce irreversible external effects — which is the constraint that determines what can be tested this way at all.

For a collaborative product specifically

Divergence detection is the enabling capability. Periodic client-side state checksums compared across clients turn "two users see different content" from an unreproducible user report into a detected event with data attached.

For any system with replicated mutable state, the ability to detect that replicas disagree is what makes production verification possible — without it, silent divergence is indistinguishable from correct operation.