A feed-ranking system needs embeddings, counters and aggregates pushed from offline jobs to an online store at very high write throughput. How should such a derived-data serving system handle full rebuilds, incremental updates, versioning and reads during a swap?

Why derived data needs its own system

Derived data has properties that make a general-purpose OLTP database a poor fit:

• It is written by machines in bulk and read by services one key at a time. Write throughput is enormous and bursty; read latency must be single-digit milliseconds.
• It is regenerable. Losing it means recomputing, not losing the truth — so durability requirements are weaker and availability requirements are stronger, the opposite of a system of record.
• It arrives in two modes: full daily rebuilds from batch jobs, and continuous incremental updates from streaming jobs, often for the same dataset.
• It is versioned by the pipeline that produced it. A model change means a new generation of the whole dataset.

LinkedIn's Venice is built for exactly this shape, sitting alongside Espresso (the online system of record) — a useful separation to internalise: one store for truth, another for things computed from truth.

The rebuild and swap

The critical mechanism is versioned datasets with an atomic pointer swap:

1. The batch job writes a new version of the store — a complete new generation, alongside the live one.
2. The write completes and is validated: row counts, key sampling, distribution checks against the previous version.
3. The serving pointer flips atomically from version N to version N+1.
4. Version N is retained, so a rollback is a second pointer flip rather than a re-run of a six-hour job.

Reads during the swap see either the old version or the new one, never a mixture — which is the entire reason for versioning rather than overwriting in place. Overwriting produces a window in which some keys are new and some are old, and for a ranking model that is a silently wrong model.

Combining rebuild with incremental updates

The hard case: a nightly full rebuild plus continuous streaming updates. The streaming writes that arrived during the rebuild must not be lost when the pointer flips.

The standard resolution: the streaming writes apply to the new version as it is being built as well, and the system tracks the stream offset the batch snapshot corresponds to, so the incremental writer resumes from that offset against the new version. Without offset tracking, every swap silently discards a day of incremental updates, and the symptom is a model that is subtly stale in a way nobody can reproduce.

What the validation gate must include

Row count within expected bounds (a job that produced 3% of yesterday's rows should never be promoted) · key-space overlap with the previous version · distribution comparison for numeric features · and a canary read set whose values are checked against expectations. The failure mode being defended against is a successfully-completed job that produced wrong data — which no infrastructure alarm detects.