A travel platform runs thousands of experiments simultaneously and needs its warehouse to support reliable analysis. What warehouse design decisions prevent cross-experiment contamination and infrastructure artefacts from being read as results?

Why this is a data architecture problem

With thousands of concurrent experiments, the dominant risk stops being statistical and becomes structural: assignment recorded inconsistently, exposure logged differently by different surfaces, metric definitions drifting between teams, and pipeline failures that look exactly like effects.

An experimentation platform is only as trustworthy as the warehouse underneath it, and the failures are quiet — a wrong result ships a wrong decision and nobody is paged.

The design decisions

1. Assignment is a logged fact, not a recomputation. The assignment event — unit, experiment, variant, timestamp, assignment version — is written when it happens and never recalculated. Recomputing assignment later from a hashing function is the single most common source of irreproducible analysis, because the function, the salt or the eligibility rule changed in the meantime.

2. Separate assignment from exposure. A unit assigned to a variant that never reached the surface under test must not be analysed as exposed. Conflating them dilutes real effects and manufactures fake ones, and the distinction can only be made if both are logged separately.

3. One metric definition, in a semantic layer. "Conversion" defined independently by six teams produces six numbers and an unresolvable argument. Metrics are defined once, versioned, and referenced by experiments — so a metric change is a visible, dated event rather than a silent shift.

4. Immutable, partitioned facts with late-arriving data handled explicitly. Events arrive late, especially from mobile clients. The warehouse needs a stated policy — a results window that waits for a defined completeness threshold — rather than analysing whatever happened to arrive.

5. Interaction detection between concurrent experiments. With thousands running, some share surfaces and interact. The platform needs to know which experiments overlap on which units, and either constrain overlap for sensitive surfaces or detect and flag interactions in analysis.

6. Pipeline health as an analysis input. A failed ingestion job that dropped a fraction of events from one platform looks identical to a treatment effect. Results must carry data-quality metadata — completeness by source, by platform, by region — and analysis must refuse to produce a verdict when completeness is below threshold. This is the control that turns an infrastructure artefact from an invisible wrong answer into a visible refusal.

7. Sample ratio mismatch as a first-class alarm. If a 50/50 experiment delivers 52/48, something is structurally wrong — assignment, logging or filtering — and no amount of statistical sophistication makes the result valid. This check catches more real defects than any other single control.

Why reproducibility drives the schema

Every result must be reproducible months later, which means the warehouse must retain the inputs to each analysis — assignment logs, exposure logs, metric definition versions, eligibility rules — not merely the outputs. A warehouse storing only computed results cannot answer the question that matters after a bad decision: was the analysis wrong, or was the data?