Booking.com's Experimentation Platform
Booking.com runs over a thousand concurrent experiments and treats the ability to test any change safely as a platform capability rather than a product feature.
The practice
Booking.com has published extensively on running large-scale online controlled experiments — with more than a thousand running concurrently, and a culture in which effectively every change ships behind an experiment.
Their published findings are more interesting than the volume. A large proportion of ideas that experienced practitioners expected to help either did nothing or actively hurt. That result is consistent across organisations that measure honestly, and it is the entire justification for the infrastructure.
What it requires architecturally
Experimentation at this scale is a platform, not a library:
Assignment and consistency. A user must receive a stable variant across sessions and devices, which requires an identity and assignment service on the request path with very low latency.
Isolation between concurrent experiments. With a thousand running, most overlap on the same users. The platform must handle interactions, either by orthogonal assignment or explicit exclusivity, or results become meaningless.
Metric pipelines with guardrails. Beyond the target metric, every experiment is monitored against guardrail metrics — latency, error rate, conversion — so an experiment that improves one number while degrading the business is stopped automatically.
Automatic termination. Experiments that harm users are ended without human intervention, which is what makes it safe to let everyone experiment.
Statistical discipline in the tooling. Peeking at results, ignoring multiple comparisons and stopping early are the standard errors; the platform must make the correct behaviour the default, because most users will not be statisticians.
The architectural lesson
A/B testing capability is an architectural requirement with real consequences. It demands that multiple code paths coexist in production, that assignment be fast and consistent, that every metric be attributable to a variant, and that changes be reversible instantly.
Systems designed without it cannot acquire it cheaply — the retrofit touches routing, identity, logging and the whole metric pipeline.
And the deeper lesson is epistemic: if most confident predictions about user behaviour are wrong, then an architecture's most valuable property is the ability to try things cheaply and undo them quickly. That is an argument for feature flags, small deployments and decoupling on business grounds, which is a considerably stronger argument than engineering aesthetics.