A team wants to test in production. What does that legitimately mean, and what does it require first?

What it legitimately means

Not skipping pre-production testing. It means acknowledging that some properties can only be verified with production's data shape, traffic pattern, cache state and dependency behaviour — which is precisely where load-dependent and data-dependent failures live.

The legitimate practices:

• Progressive delivery: deploy to a small percentage, observe real behaviour, halt automatically on defined conditions.
• Synthetic monitoring: continuously exercise critical journeys in production, which catches failures affecting real users before they report them.
• Traffic shadowing: send production traffic to a new implementation without serving its results, and diff.
• Feature flags with graduated exposure, so a change is enabled for internal users, then a small cohort, then everyone.
• Controlled fault injection with a small blast radius, a fast abort and a stated hypothesis.

What it requires first

• Observability sufficient to detect a problem quickly, segmented enough to see it in a small cohort. A canary is only as good as the ability to tell it is failing, and an aggregate metric will not show a problem affecting 1% of traffic.
• A rollback measured in seconds, which usually means deployment separated from release so the reversal is a flag change.
• Automated halt conditions, since a human watching a dashboard will not react in time and will rationalise a slow degradation.
• A defined blast radius for anything deliberate, and a way to exclude high-value or high-risk customers from experiments.
• Data safety: synthetic transactions must be identifiable and excluded from business metrics, financial records and downstream processing — and getting that wrong pollutes the reporting that the business runs on.

The specific caution for a stateful product

A change that writes data cannot be rolled back by reverting the code. For a product where clients hold state and sync, an experiment that produces a bad record propagates to devices and persists after the rollback.

So the safe subset is smaller: read-path changes and additive writes are safe to experiment with; changes to the sync protocol or the conflict-resolution policy are not, and they need the slower path.

The framing

Testing in production is not a substitute for testing; it is the only way to test the properties that only exist there — and a team that adopts it without the observability and the rollback has adopted the risk without the mechanism.