A team needs a store for derived data that is bulk-loaded from offline jobs, updated in real time, and read at very low latency. Why does this workload get its own system rather than reusing the primary online database?

The workload's distinguishing properties

• Bulk-loaded: an offline job writes millions of rows at once, at a rate no OLTP database should absorb alongside serving traffic.
• Real-time updated: streaming jobs mutate the same dataset continuously.
• Read as simple key lookups at very low latency and very high volume.
• Regenerable: it can be recomputed from upstream sources.

That last property is the one that changes the engineering. Derived data does not require the durability guarantees of a system of record, because losing it costs a recomputation rather than the truth. It does require high availability, because serving traffic depends on it.

That is an inverted requirement profile relative to a transactional database, and it justifies a different system rather than a different table.

Why not just use the primary database

• Bulk loads compete with serving traffic for I/O, buffer pool and locks. A daily rebuild degrades user- facing latency for its duration, which is exactly when nobody wants it.
• The version-and-swap requirement is unnatural in a relational store. Derived data wants atomic replacement of an entire dataset; an OLTP database wants row-level updates in transactions.
• Storage growth and vacuum pressure from repeated bulk rewrites hurt the transactional workload.
• Coupled failure domains: a bug in an offline job should never be able to affect the ability to take an order. Separation buys blast-radius isolation on top of performance.

What the dedicated system provides

Bulk ingestion that does not disturb reads — typically by writing new immutable files and swapping, rather than by updating in place. Versioned datasets with atomic pointer swap, so a rebuild is visible all at once and rollback is a pointer flip. A read path optimised for point lookups with no query planner. Tunable replication reflecting that the data is regenerable, so replication factor and consistency can be relaxed for throughput. Multi-tenancy, since many teams have this same need and each building their own is the outcome being avoided.

When you should not do this

When the data is small enough to fit in memory in the serving process — then it is a file loaded at startup and refreshed periodically, which is dramatically simpler and should be the first thing tried. When there is one such dataset and a Redis instance with a versioned key prefix covers it.

The dedicated system earns its existence at the point where many teams have this need, because then the alternative is many bespoke implementations of the same tricky swap-and-rebuild logic, each with its own partial-visibility bug. That is a platform decision driven by organisation size, not by any individual workload.