Materialisation and the Online Store
How features get from an analytical table into a millisecond-latency lookup, the freshness-cost tradeoff each materialisation strategy makes, and why the online store's data model is nothing like the offline one.
A ranking service has 15 milliseconds to fetch features for 500 candidate items. Nothing in an analytical stack can answer that: a data lake query takes seconds, and a warehouse takes hundreds of milliseconds at best. The online store is a separate system with a separate data model whose only job is key-value lookup at low latency, and materialisation is the process of keeping it populated.
The two shapes
The offline store is columnar, append-heavy, keyed by entity and timestamp, and holds full history because training needs it. It is optimised for scanning many rows across few columns.
The online store holds only the current feature values per entity, keyed for point lookup, in a row-oriented or key-value engine such as Redis, DynamoDB, Cassandra or a similar store. It is optimised for reading many columns for one row, at high concurrency, which is the exact inverse of the offline access pattern. That is why one system cannot serve both, and why the feature store's central engineering problem is keeping two stores consistent.
Materialisation strategies
Batch materialisation runs a scheduled job that computes features and writes the latest values to the online store. Simple, cheap, and the feature is as stale as the schedule. Suitable for anything that changes slowly: account age, historical aggregates, embeddings of static content.
Streaming materialisation updates the online store as events arrive, giving seconds of freshness. It costs a streaming pipeline with all its state and correctness concerns, and it is what real-time aggregates such as "clicks in the last five minutes" require.
On-demand computation derives the feature at request time from the request payload and other fetched features. Nothing is stored, so there is no staleness and no materialisation cost. Suitable for transformations of request context, distance between a user's current location and an item, or ratios of two fetched features. The transformation must be identical offline, which brings back the parity problem it appeared to avoid.
Most production systems use all three, and the choice per feature is a freshness-versus-cost decision that should be made explicitly and recorded rather than defaulting to whatever the first feature used.
When it breaks
Write amplification during backfill. Materialising a new feature for 100 million entities means 100 million writes to a store sized for a steady-state update rate. This routinely saturates the online store and degrades live serving, so backfills need throttling and ideally a separate write path.
TTLs are a correctness mechanism, not a cleanup one. Without expiry, an entity that stops being updated keeps its last value indefinitely, and the model receives a stale feature with no indication of its age. Explicit TTLs turn silent staleness into an explicit miss the serving code must handle, which is strictly better than a wrong number.
Fan-out multiplies latency. Fetching 40 features for 500 candidates is 20,000 lookups. It only works with batched multi-get and careful key design that groups co-fetched features into one row. A schema requiring one lookup per feature per entity will not meet any realistic latency budget regardless of the store's per-lookup speed.
The online store is a cache without cache semantics. There is no origin to fall back to at request time, since the offline store is too slow. A missing key means the model runs on a default value, so miss rates directly affect prediction quality and belong on the same dashboard as model metrics rather than in an infrastructure dashboard nobody correlates with them.
12 flashcards for this concept
Click a card to reveal the answer.