Hybrid Store Rewind

A derived dataset is usually produced twice. A heavy offline job recomputes the whole thing on a schedule, and a nearline job nudges individual keys as events arrive. The textbook arrangement serves both and merges them in the application.

That merge is the expensive part. Every reader carries it, every reader can implement it differently, and the second team to write it will get an edge case wrong. The rewind inverts the responsibility: the store reconciles the two paths at write time and hands readers one answer.

Why it matters

A batch push does not update a dataset, it replaces it. The replacement was computed from an input snapshot taken hours earlier, so at the moment of the swap it is already behind every nearline update applied since that snapshot. Swapping it in without compensation silently regresses the dataset by the length of the push, and the regression raises nothing: the store is healthy and the values are simply old.

The rewind is the compensation, expressed as a duration rather than as a position, which is what makes it configurable per store rather than a bespoke piece of orchestration per dataset.

Implementation patterns

• Build the new version beside the live one, then rewind the real-time input by the configured interval and apply it to the new version, then swap. The live version keeps serving throughout.
• Set the interval longer than the batch job's own lag - the gap between its input cut-off and the push completing. A job that cuts at 02:00 and finishes at 05:30 needs more than three and a half hours of rewind, or the gap between the cut-off and the rewind start is lost.
• Require replayable writes. Re-applying four hours of updates must land in the same state as applying them once. Full-value overwrites and declarative partial updates are safe; a read-modify-write counter is not, and must be moved into the batch job or expressed as a mergeable type.
• Size for two copies. Roughly 2× the largest store exists during a push, which is a capacity-planning fact rather than a steady-state one.
• Expose the version and its push timestamp so a reader can tell which generation answered it.

Industry example

LinkedIn's Venice is the documented implementation. LinkedIn described the mechanism in a 2017 post titled "Venice Hybrid: Doing Lambda Better" and open-sourced the system in 2022, having run it in production since late 2016, behind more than 1800 datasets and over 300 applications including recommendation surfaces such as People You May Know. A Venice hybrid store takes a batch push and a stream write to the same keyspace, with the rewind time configured per store, and offers a thin client, a fast client, and the Da Vinci client, which keeps a local copy in the caller's process.

Failure scenarios

• Rewind shorter than the push duration: a window of nearline updates disappears at every push, so the dataset degrades on a schedule and only for keys touched in that window.
• Non-idempotent stream writes: counters double on every push, which looks like a data-quality problem rather than an architectural one.
• Rewind far too long: the push takes hours longer and the store spends its day re-applying, which shows up as push lag rather than as a configuration error.
• Capacity sized for one copy: the push fails at the swap, under peak, when the dataset is largest.

Trade-offs

The pattern buys one query path and one definition of correctness for every reader, and it pays for that with a store that is harder to operate, no read-your-writes guarantee because the write path is asynchronous, and the discipline of keeping every stream write replayable forever. The reconciliation cost does not vanish; it moves from N readers to one platform team, which is the right trade only when N is large.

When not to use it

For one derived table with a handful of readers, a materialised view refreshed on a schedule inside the database that already holds the data does the same job with no new system. Choose a hybrid store when the merge logic has already been written twice by different teams, not when the dataset is large. Volume is not the threshold; reader count is. If the batch path can be retired entirely because the stream is authoritative, do that instead and skip the whole mechanism.

Interview question

Q: You have a key-value store loaded nightly by a batch job and also written continuously by a stream. Users report that recommendations "go stale every morning for a few hours". What is happening, and what would you configure?

What a strong answer covers: that the push replaces the dataset with one computed from an older snapshot; that nearline updates applied between the snapshot and the swap are discarded; the rewind interval and why it must exceed the batch job's lag rather than some round number; the idempotency requirement on stream writes; and the storage cost of two live versions.

Quick check

Quiz: A batch push cuts its input at 02:00 and completes at 05:30. What is the minimum rewind time? More than three and a half hours - the gap between the input cut-off and the swap, plus margin.

Flashcard: Why must stream writes into a hybrid store be idempotent? Because the rewind re-applies hours of them on every push; anything read-modify-write double-counts.