A recommendation platform's ML systems span data pipelines, training, evaluation and online serving. Where should the boundaries be, and what causes the most costly class of bug?

The most costly bug class

Training/serving skew — the features used to train the model differ from those computed at serving time. The model performs well offline and worse in production, and the gap is invisible because nothing errors.

It arises from specific, avoidable causes:

• A feature computed one way in a training pipeline and another way in a serving path, because they are separate implementations of the same definition.
• Time leakage — training on data that would not have been available at prediction time, which produces excellent offline metrics and a model that cannot work.
• Different data distributions between the training snapshot and live traffic.
• Preprocessing differences — normalisation, encoding, handling of missing values — implemented twice.

The boundary that prevents it

A feature store as the seam. A feature is defined once and executed by both the batch path that produces training data and the online path that serves requests. When the definition exists once, skew from divergent implementations is impossible by construction.

The corresponding principle for data: one ingestion path, two consumption paths. Events are written once to a durable log; the streaming path builds low-latency features and the batch path builds training data and long-window features. Duplicating ingestion produces two pipelines that disagree, which is the origin of the problem.

The other boundaries

• Offline and online compute separated, with real resource isolation. A backfill replaying months of history will starve the real-time path if they share capacity, and at this scale that is not a tuning matter.
• The serving path does only lookup, retrieval and ranking. Everything derived from historical events is precomputed. The request budget is tens of milliseconds, which forbids computing anything over history.
• Model artefacts versioned with their feature definitions and training data snapshot, so a model can be reproduced and a regression attributed.
• Evaluation infrastructure separate from training, with held-out sets that reflect production distribution.

The experimentation requirement

At scale, thousands of concurrent experiments make the platform's correctness a structural rather than a statistical problem: assignment logged as a fact rather than recomputed (recomputation from a hash later is the top source of irreproducible analysis), assignment separated from exposure, one versioned metric definition, interaction detection between overlapping experiments, and data-quality metadata as an analysis input so a partial-ingestion bug is refused rather than read as a treatment effect.

A sample ratio mismatch alarm catches more real defects than any statistical sophistication.