advanced 2 min answer

A trading platform is considering a relational database, a key-value store, a time-series store and a search engine. What justifies each, and what is the cost of running four?

zerodhapolyglotdatastore-selectionoperationscomplexity
Show the full answer Hide the answer

What each earns its place with

  • Relational, for orders, accounts and the ledger. The requirement is transactional correctness across related entities with strong invariants. Nothing else does this as well, and this is the part where being wrong costs money and licences.
  • Key-value or in-memory, for the live position and limit cache on the hot path. The requirement is microsecond-scale lookups on the pre-trade risk check. A relational round trip is too slow, and the data is reconstructible from the authoritative store, so durability is not required.
  • Time-series, for market data and analytics. Append-heavy, time-ordered, queried by range and aggregate, compressed aggressively. A relational table with an index on timestamp handles this badly at volume and the gap widens with scale.
  • Search, for instrument lookup and text queries. Relevance ranking and fuzzy matching are not relational operations.

The cost of four

Each store adds: a backup and restore procedure, a patching and upgrade cycle, a failover mechanism to test, capacity planning, a monitoring surface, a security review, and a body of team expertise that must exist at 3am. The cost is not the licence; it is the on-call knowledge.

There is also a consistency cost. Data in four stores diverges, and every divergence is a class of bug that does not exist in a single-store design.

The discipline that makes it defensible

  • One authoritative store per fact. Every other copy is explicitly derived, populated by a documented pipeline, and rebuildable. The moment two stores are both authoritative for the same thing, reconciliation becomes permanent work.
  • Derive rather than dual-write, via CDC or an outbox, so the derived stores cannot silently diverge.
  • Justify each store against a requirement that the existing ones demonstrably fail, in writing, with the measurement that shows the failure.
  • Prefer boring. A second store that a team already runs well is usually a better answer than a theoretically superior one nobody has operated.

The counter-position worth taking seriously

Modern relational databases handle JSON documents, full-text search, and time-series data adequately. For many products, one well-operated Postgres is a better architecture than four specialised stores, because the operational simplicity is worth more than the per-workload optimisation.

The threshold for adding a store should be a measured failure of the existing one against a real requirement, not an assessment that a specialised tool would be better in principle. It almost always would be, and that is not sufficient.