A marketplace wants to A/B test a change to its matching algorithm. Why is this harder than testing a button colour?

What the interviewer is testing

Whether you know that marketplace experiments violate the independence assumption that standard A/B testing relies on.

The problem

Standard A/B testing assumes the treatment applied to one user does not affect another — the stable unit treatment value assumption. In a marketplace this is false.

If the treatment group's matching algorithm assigns drivers more aggressively, those drivers are no longer available to the control group. The control group's experience degrades because of the treatment, so the measured difference overstates the effect — possibly enormously, and possibly with the wrong sign.

This is interference, and it means the experiment is measuring the treatment plus the harm it caused to the control, on a shared pool of supply.

The approaches

Switchback testing. Apply the treatment to the entire market for alternating time periods, and compare periods. Everyone experiences the same algorithm at any moment, so there is no cross-group interference. Costs: temporal confounding — time of day, weather, events — needs many switches and careful randomisation of period boundaries.

Geographic or market-level randomisation. Whole cities in treatment or control. Clean isolation, and the sample size becomes the number of markets rather than users, so statistical power is low and markets differ substantially from each other.

Cluster randomisation at a level where interference is contained — a region within a city, if supply does not move between regions much. A compromise, and the assumption needs validating.

Simulation against historical data before live testing, which is cheap and only as good as the model of participant behaviour.

What a strong answer adds

Recognising that the metric must also be market-level. Per-user conversion in a marketplace experiment is contaminated; the meaningful measures are market-level: total completed transactions, average wait time, supply utilisation, and the balance between the two sides.

And the platform consequence: switchback and geographic experimentation are different infrastructure from user-level A/B testing — different assignment, different analysis, different power calculations. A platform that only supports user-level splits cannot run these, which is the practical blocker in most organisations.

Common weak answers

Standard user-level A/B testing with a larger sample, which does not address interference. Testing in one city without a control.