Inference-Time Feature Logging

A pricing model's conversion rate drifts down 4% over two months. The model has not changed. Offline accuracy is stable. No component logs an error, and no engineer can point at a release.

The online features are written by a stream job; the training table is built nightly by a batch job from the warehouse. Both implement the same definitions and neither implements them identically. A difference in null handling, in where a window boundary falls, in what the late-data policy drops, or in how a type is coerced becomes a systematic offset between the vector the model trained on and the vector it scores. Offline evaluation cannot see it, because offline evaluation uses the offline features.

Why it matters

This is the most expensive class of machine-learning defect precisely because it produces no error signal. The business metric moves slowly, the model artefact is innocent, and the usual response is to retrain - which bakes the offset in rather than removing it.

Logging the served vector removes the possibility by construction. If training reads the same bytes serving used, the two distributions cannot differ, and the entire category of bug stops existing rather than being monitored.

Implementation patterns

• Write the vector off the request path. At 3000 requests per second with roughly 150 features at about 12 bytes, raw volume is around 5 MB/s, or 40 to 80 GB a day before compression. Push onto a bounded in-process queue and drain asynchronously; when the queue is full, drop and increment a counter rather than blocking the response, and alert on the drop rate - a sample with an unknown rate is worse than no sample.
• Log the model version, the request id and the serving timestamp alongside the values, so the join key and the cohort boundary both exist.
• Join labels on request id. Labels arrive later than predictions, so the label delay sets the floor on retraining cadence. Writing that delay down is often the first time anyone has.
• Keep the batch recomputation as a backfill path for periods before logging started, and reconcile it against the logs so any residual offset is visible rather than assumed away.
• Monitor per-feature distribution between the logs and the training job daily, with a quantile delta or population stability index, which turns a two-month business drift into a one-day feature-level alert.

Industry example

The practice is standard in published feature-store designs, which separate an online store for serving from an offline store for training and then add a logging path precisely because two stores in production are the problem rather than the solution. The same reasoning drives the point-in-time correctness guarantees those systems advertise: both exist to make training read what serving saw.

Failure scenarios

• Unbounded logging on the request path, where the log write enters the latency budget and a slow log sink becomes a serving outage.
• Unknown sampling, where the queue drops silently under load, so the training set over-represents quiet periods and the model is worst exactly at peak.
• Unstable request ids generated per service rather than per request, so the label join loses rows in a pattern correlated with the thing being predicted.
• A cold start that looks like a regression: the first model trained purely on logged vectors has less history than its predecessor and can be worse before it is better.

Trade-offs

The cost is a pipeline, its storage and a training set that is only as deep as the logging history. The benefit is that one whole failure class disappears and that feature drift becomes observable per feature per day instead of inferable from revenue. Run both training sets in parallel for a cycle or two and compare, rather than switching on faith.

When not to use it

When every feature is a slowly changing attribute read from one table by one code path - a customer's country, a plan tier, a product category - recomputation is identical to serving and logging is pure cost with a new pipeline to operate. The rule: log the vector when the serving path computes anything the training path cannot read verbatim - windowed counts, request-time context, anything derived from a stream or from the request itself. A model whose features are all joins against dimension tables should not pay for this.

Interview question

Q: Production performance for a model has drifted down over two months. Offline metrics are stable, no model or code has shipped, and nothing errors. Where do you look, and what would you build so this cannot recur?

What a strong answer covers: naming training-serving skew from two implementations rather than data drift; why offline evaluation is blind to it; logging the served vector with request id and model version as the structural fix; the asynchronous write with a bounded queue and a drop counter; the label-delay join and what it implies for retraining cadence; the per-feature distribution monitor as the detector; and the cases where the whole practice is unnecessary.

Quick check

Quiz: Why can offline accuracy stay flat while production performance decays? Offline evaluation scores the offline feature implementation; production scores the online one and the two have diverged.

Flashcard: What single artefact makes a training set match serving by construction? The feature vector logged at inference, joined to labels on the request id.