Versioned Dataset Swap

Derived datasets — embeddings, feature tables, search indexes, precomputed recommendations, aggregate rollups — are regenerated wholesale by offline jobs. The naive publication method is to overwrite the live data in place, which creates a window during which some keys hold the new generation and some hold the old.

Versioned dataset swap eliminates that window. The job writes generation N+1 in full, alongside N. The result is validated. A single pointer moves. Readers before the flip see N entirely; readers after see N+1 entirely.

Why it matters

A partially-updated derived dataset is not stale — it is incoherent. Staleness is a known, bounded and usually acceptable condition. Incoherence means a ranking model reading features from two different generations, a search result merging two catalogue snapshots, or an aggregate computed over inconsistent inputs. These produce wrong answers that no monitoring detects, because every individual read succeeds.

The second reason is recovery. A bad generation published in place cannot be undone except by re-running the job, which for a large batch pipeline is hours — during which the wrong data is serving. With versions retained, rollback is the pointer moving back, measured in seconds.

Implementation patterns

• Write to an immutable, version-named location — a directory, a table with a generation suffix, a key prefix — never over the live one.
• Validate before promotion, as a hard gate: row count within expected bounds relative to the previous generation, key-space overlap, distribution comparison for numeric columns, and a canary set of keys whose values are asserted against expectations.
• Flip a single small piece of state — a pointer row, a metadata entry, an alias — so promotion is atomic by construction rather than by careful ordering.
• Retain the previous N generations with a documented retention count; retention is the rollback capability.
• Track the stream offset the batch snapshot corresponds to when incremental updates coexist with rebuilds, so the streaming writer resumes against the new generation from the right position.
• Apply streaming writes to the in-progress generation as well as the live one, so the new version is not born stale.
• Make the reader's generation choice sticky for the duration of a request, so a single request cannot span a flip.

Industry example

LinkedIn's Venice, which serves derived data to feed ranking and other online systems, is built around versioned stores with atomic swaps, precisely because its datasets are produced by both nightly batch jobs and continuous streaming updates, and both must land without exposing a mixed state. Airbnb's Mussel serves a similar role for bulk-loaded derived data with real-time updates and low-latency point reads.

The common shape across both: a store for truth (the online transactional system) and a separate store for things computed from truth, with the second designed around regeneration, versioning and availability rather than around durability and transactions.

Failure scenarios

• Overwriting in place, producing an incoherent window that no alarm detects.
• Promoting an under-produced generation — a job that silently processed 3% of the input, completed successfully, and was published because "the job succeeded."
• No retention, so rollback means re-running the pipeline.
• Losing incremental updates at every swap because the offset was not tracked — a subtle staleness nobody can reproduce.
• A flip that is not atomic — several pointers updated in sequence, so a reader catches half of them.
• Storage growth from retained generations with no lifecycle policy.
• Readers caching the resolved location for longer than the flip interval, so some readers stay on an old generation indefinitely.

Trade-offs

The pattern doubles or multiplies storage for the retention window, and it costs a full rewrite even when only a small fraction of rows changed — inefficient for datasets with small daily deltas, where an incremental update model with careful ordering may be cheaper.

It also adds a promotion step that can itself fail or be forgotten, and a validation gate that will occasionally block a legitimate generation and require human judgement at an inconvenient hour.

The trade is storage and pipeline complexity in exchange for coherent reads and second-scale rollback. For anything that feeds a model, a ranking, or a user-visible search result, that is a clearly good trade. For a dataset whose consumers genuinely tolerate a mixed view — an append-only log of events, for instance — the machinery is unnecessary.

Interview question

"Your nightly feature rebuild also receives streaming updates all day. Design the publication mechanism, then tell me exactly what happens to the streaming writes that arrive during the six hours the batch job is running."