Booking.com has described running more than 1,000 concurrent experiments in a 2017 paper on democratising experimentation. Before you let a thousand teams define their own metrics at that concurrency, what must be true of the semantic layer and the pipelines underneath?

What the scale actually demands

At one experiment, a metric definition is a local matter. At a thousand concurrent experiments, every experiment is reading metrics that other experiments' changes can move, and a definition that differs between two teams produces a decision conflict nobody can adjudicate.

Booking.com's published account stresses infrastructure and culture together: a shared library that keeps experimentation decoupled from business logic, close monitoring of the data-gathering pipelines so that people trust the numbers, and safeguards that let anyone own an experiment end to end.

Three requirements follow, and all three are architectural.

The three requirements

1. One computed definition per metric, resolved at query time. Conversion must be computed from one expression over one grain. If two teams each write their own SQL, the platform is producing two answers and a disagreement, and at a thousand experiments those disagreements arrive faster than anyone can resolve them.
2. Guardrail metrics that are not optional. Every experiment reports the same small set — latency, error rate, cancellation, a business-wide conversion — regardless of what it was testing. This is what catches an experiment that wins on its own metric and loses money overall, which is the characteristic failure of decentralised experimentation.
3. Pipeline trust, monitored as a product. If the data feeding the metrics is late or partial, experiments read a partial denominator and conclude something false. Freshness, completeness and assignment-balance checks belong in the platform, because no individual team will build them.

What it costs

A central metric layer is a dependency: changing a definition now requires a process, and the process is friction on the exact people the platform is trying to empower. The honest exchange is that decentralised execution requires centralised definition, and teams feel the second before they appreciate the first.

There is also a statistical bill nobody escapes. At 1,000 concurrent tests, a 5% significance threshold produces roughly 50 false positives by construction, so the platform needs a shared convention on thresholds and on when a result may be called.

Where copying this is a mistake

A company running five experiments a quarter should not build this. Its constraint is traffic, not coordination: at low volume the correct answer is fewer, larger experiments with manual review. Copying the platform without the traffic produces sophisticated infrastructure around results that are underpowered anyway.

Common weak answers

• "Use a feature flag service." That is delivery, not measurement, and it answers none of the three requirements above.
• "Certify dashboards." Certification is a label. What is needed is one computation, because labels do not stop two SQL statements from disagreeing.