Enterprise Metadata Management System  ·  View 08 of 22  ·  Data

Storage Architecture

Which store is authoritative, which stores are disposable, and what that split buys.

Editable source SVG draw.io All views
Metadata Repository
Metadata Repository
System of record — durable, versioned, backed up
System of record — durable, versioned, backed up
Aspect Store
PostgreSQL · partitioned
Aspect Store...
Version History
24 months online
Version History...
Audit Log
append-only · 7 y
Audit Log...
Payload Archive
S3 · object lock
Payload Archive...
Projections — derived, disposable, rebuilt from the change log
Projections — derived, disposable, rebuilt from the change log
Knowledge Graph
Neo4j · 50M edges
Knowledge Graph...
Search Index
OpenSearch · 5M docs
Search Index...
Read Cache
Redis · 15 min TTL
Read Cache...
Operational state
Operational state
Workflow State
Temporal · Postgres
Workflow State...
Watermarks & Jobs
per connector
Watermarks & Jobs...
Connector Secrets
KMS-backed vault
Connector Secrets...
Metadata Change Log
Kafka · 7-day retention
Metadata Change Log...
Backup Vault
PITR 35 days
Backup Vault...
Source systems
own their technical truth
Source systems...
emit on commit
emit on commit
project
project
continuous backup
continuous backup
replay on rebuild
replay on rebuild
harvest payload
harvest payload
Storage Architecture — System of Record and Projections
Storage Architecture — System of Record and Projections
Data store
Data store
Security / platform
Security / platform
Queue / topic
Queue / topic
External / third party
External / third party
event / async
event / async
batch
batch
Losing a projection costs a rebuild, not data. Losing the aspect store costs a restore — which is why only it is on the RPO 5 min path.
Losing a projection costs a rebuild, not data. Losing the aspect store costs a restore — which is why only it is on the RPO 5 min path.
v 1.0 · owner Data Architecture · date 2026-08
v 1.0 · owner Data Architecture · date 2026-08
Text is not SVG - cannot display

System of record

  • The aspect store in PostgreSQL is the only authoritative store. The graph and the search index are projections, rebuilt by replaying the change log.
  • This is the answer to graph-versus-relational: both, but only one of them is true. A graph database as system of record makes versioning, backup and point-in-time restore far harder than they need to be.
  • Source systems remain authoritative for their own technical metadata. The platform holds a described copy with provenance, never a fork.

Numbers

  • RPO 5 minutes and RTO 30 minutes apply to the aspect store alone.
  • Projection rebuild from the change log completes in under 4 hours at 5 million assets; the payload archive covers replay beyond the 7-day log retention.
  • Version history is online for 24 months, then compacted to first and current with the audit trail retained for 7 years.

Trade-off accepted

  • Projections are eventually consistent, typically under 60 seconds behind. Search may briefly miss a rename. The alternative — synchronous dual writes — trades a visible, measurable lag for invisible partial failures.