A pricing model scores 3000 requests per second. Its features come from an online store written by a stream job; the training set is built nightly by a batch job that recomputes the same features from the warehouse. Offline accuracy is stable and production conversion has drifted down about 4% over two months with no model change and no errors anywhere. Design the fix.

Requirements that drive the structure

The model is fine. The vector it scores against in production is not the vector it was trained on, and the two were never required to be equal - they were only required to be computed from the same definition, by two different code paths, over two different inputs. A difference in how a null is handled, where a window boundary sits, what the late-data policy drops, or how a type is coerced becomes a systematic offset. Both implementations are internally consistent, so nothing errors and offline accuracy stays stable: the offline evaluation uses the offline features.

The design

Log the served vector. At inference, write the exact feature vector the model received, plus the model version, the request id and the serving timestamp. Training joins labels to those logged vectors on request id and never recomputes a feature. By construction the training distribution is the serving distribution.

The parts:

• The write, off the request path. 3000 requests per second × roughly 150 features × about 12 bytes is around 5 MB/s raw, perhaps 40 to 80 GB a day before compression. That is affordable; what is not affordable is a synchronous write in the latency budget. Push onto a bounded in-process queue and drain to a stream; when the queue is full, drop and increment a counter rather than blocking the response, and alert on the drop rate because a sampled log with an unknown sampling rate is worse than none.
• The label join. Labels arrive later than the prediction - a conversion may be hours later - so this is a join whose window is the label delay, or more simply a daily batch join on request id. The label delay sets the floor on retraining cadence, and writing it down is usually the first time anyone has.
• A skew monitor. Compare the distribution of each logged feature against the same feature as the training job computes it for the same day, per feature per day, and alert on a quantile delta or a population stability index. This is what turns a two-month conversion drift into a one-day feature-level alert.
• The old recomputation stays, demoted to a backfill path for periods before logging started, and reconciled against the logs so any residual offset is visible rather than assumed away.

What it costs

A new pipeline, its storage, and a cold start: the training set is only as deep as the logging history, so the first model trained purely on logged vectors has less data than its predecessor and may be worse before it is better. Run both training sets in parallel for one or two cycles and compare. Request ids must be stable and present end to end, which is often the real blocker in a system that generates them per service.

When this is the wrong answer

If every feature is a slowly-changing attribute read from one table by one implementation - a customer's country, their plan tier, a product's category - recomputation is identical to serving and logging is pure cost. The rule: log the vector when the serving path computes anything the training path cannot read verbatim - windowed counts, request-time context, anything derived from a stream or from the request itself. A model whose features are all joins against dimension tables does not need this and should not pay for it.