Derived Store Discipline

Running several specialised datastores is often correct: a relational store for transactional invariants, an in-memory store for hot lookups, a time-series store for telemetry, a search engine for text. Each earns its place against a requirement the others demonstrably fail.

What makes the arrangement survivable is a single rule: one authoritative store per fact. Everything else is derived, rebuildable, and populated by a documented pipeline.

Why it matters

The moment two stores are both authoritative for the same fact, reconciliation becomes permanent work and every divergence is a bug class that does not exist in a single-store design. Teams rarely decide to have two authorities; they arrive at it by writing to both stores from the application, which seemed reasonable at the time.

The other cost is operational, and it is the one that dominates: each store adds a backup and restore procedure, a patching cycle, a failover to test, capacity planning, a monitoring surface, a security review, and expertise that must exist at 3am. The cost is not the licence, it is the on-call knowledge.

Implementation patterns

• Name the authority for every fact, in writing. If two teams disagree about which store is authoritative for a value, that disagreement is already a production incident waiting for its trigger.
• Derive rather than dual-write, via change data capture or a transactional outbox, so derived stores cannot silently diverge from the authority.
• Make every derived store rebuildable from the authority, and rehearse the rebuild. A projection that has never been rebuilt cannot be trusted to be rebuildable when it is wrong.
• Justify each new store against a measured failure of the existing ones, not against a judgement that a specialised tool would be better in principle. It almost always would be, and that is not sufficient.
• Prefer boring. A store the team already operates well beats a theoretically superior one nobody has run.

Industry example

A trading platform such as Zerodha has a genuinely polyglot case: relational for orders, accounts and the ledger, where transactional correctness across related entities is the requirement; an in-memory store for pre-trade risk limits, where a relational round trip is too slow and the data is reconstructible so durability is not needed; a time-series store for market data, which is append-heavy and queried by range; and a search engine for instrument lookup.

Each of those is a different requirement, not a preference — and the ledger remains the single authority for anything a regulator or a customer will ask about, with everything else explicitly downstream of it.

Failure scenarios

• Two authorities for one fact, producing permanent reconciliation work.
• Dual writes from the application, giving silent divergence with no detection.
• A derived store nobody can rebuild, so a projection bug becomes a data-loss event.
• A store added for a hypothetical requirement, then kept because migrating off is work.
• Expertise concentrated in one person, so the store is unoperable when they are unavailable.

Trade-offs

The counter-position deserves to be taken seriously: modern relational databases handle documents, full-text search and time-series data adequately, and for many products one well-operated Postgres is a better architecture than four specialised stores. Operational simplicity is frequently worth more than per-workload optimisation, particularly for a team under twenty engineers.

The threshold for adding a store should therefore be high, evidence-based, and reviewed — and the review should include who will be woken up for it.

Interview question

"You have four datastores. For each one, tell me which fact it is authoritative for, how the others get their copy, and what you would do if a projection was found to be wrong for the last three weeks."