A developer platform accumulates a relational database, an object store, a search index, a cache, a queue and a time-series store. When is this justified polyglot persistence, and when is it accidental sprawl?
Show the full answer Hide the answer
The test that distinguishes them
For each store, ask: what specific workload characteristic makes the primary database unsuitable, and has that been measured?
Justified answers name a property the relational database structurally lacks:
- Object storage for repository content and artefacts. Large immutable blobs, cheap durable storage, direct client access via signed URLs. A database storing gigabyte artefacts is a bad database.
- Search index for code and issue search. Relevance ranking, tokenisation and faceting are not relational operations; the query model is genuinely different.
- Cache for hot reads. Sub-millisecond access with explicit eviction.
- Queue for background work. Though this one deserves scrutiny — a database-backed queue is often better at moderate volume, precisely because enqueue joins the transaction that created the work.
- Time-series store for metrics. High-cardinality, append-heavy, with retention and downsampling as first-class features that relational schemas emulate poorly.
Unjustified answers sound like: "a team preferred it", "it seemed more scalable", "it was in a blog post", "we needed JSON" (modern relational engines handle JSON well), "we needed to scale" with no measurement.
The cost each store adds
The cost is not one system; it is a multiplier across everything:
- Backup and restore procedures, tested.
- Failover, replication and disaster recovery.
- Monitoring, alerting and dashboards.
- Version upgrades and security patching.
- On-call expertise at 3 a.m., in a technology used by one team.
- Local development environments that must run it.
- A new consistency boundary — data now spans stores with no transaction across them.
That last point is the underrated one. Every additional store is a new place data can disagree, and reconciliation between stores is permanent, ongoing work that appears on no roadmap.
When to consolidate
Signals that a store should be removed:
- It serves one feature that could be served acceptably by an existing store.
- Nobody on call is confident debugging it.
- Its data is fully derivable from another store and is only there for convenience.
- The workload it was chosen for has changed or gone away, and it remains from inertia.
- Its volume is a rounding error, so the primary database would absorb it unnoticed.
The judgement being tested
Polyglot persistence is a real and correct pattern — the failure is treating it as a default rather than as a decision with an owner and a justification. A good architecture can name, for each store, the workload property that requires it and the evidence that it was measured. An accidental one has stores whose original justification nobody can state.
The honest version of the consolidation question is not "can we remove this?" but "if we were starting today with what we now know, would we add it?" — and then costing the migration against the ongoing operational burden rather than assuming either answer.