# Enterprise Metadata Management System **Solution Architecture v1.0 · Data & AI Global Practice · 2026-08 · 22 views** A metadata platform architected as a connected knowledge layer rather than a database of table and column descriptions. The value is in the relationships — between data assets, business concepts, people, policies, quality and lineage — so the architecture is organised around a canonical model with versioned aspects, one authoritative store with disposable projections, a precedence rule that protects curated metadata from being overwritten by a harvest, and an activation path that pushes classifications back out to the systems that enforce them. --- ## What is here | Path | Contents | |---|---| | `diagrams/index.html` | The view index — 22 views grouped into six acts, every format linked | | `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning it deliberately omits, with copy / PNG / PDF export | | `diagrams/svg/*.svg` | The same 22 views as SVG with the diagram XML embedded — re-opens in diagrams.net fully editable | | `diagrams/drawio/*.drawio` | draw.io native source | | `specs/views.json` | Diagram specifications — the source of truth for every view | | `specs/manifest.json` | Acts, page titles, subtitles and the decision / assumption / risk cards | | `scripts/pin-icons.mjs` | Icon corrections, with the reason for each pin | | `ask.txt` | The original requirement | Every component carries its own mark — 351 icons embedded in the files, so the deliverable renders identically anywhere with no external requests. Product marks are used only where the architecture actually names that product (Snowflake, Kafka, Neo4j, PostgreSQL, OpenSearch, Redis, Temporal, dbt, Power BI, Entra ID, Prometheus); our own components carry the semantic mark for their kind. View 16 is the one view genuinely *about* a cloud, so it carries the official AWS marks for EKS, Aurora, S3, MSK, Route 53 and AWS Backup. View 06 has no icons at all — its cells are coverage states, not components. --- ## The six acts | Act | What it lands | Views | |---|---|---| | 1 · Context and scope | The boundary, the four personas, and the three things the platform refuses to do | 01–02 | | 2 · Structure | Layering rule, deployable containers, integration surface, honest connector coverage | 03–06 | | 3 · Data | Canonical model, system of record versus projections, flow and precedence | 07–09 | | 4 · Runtime | Harvest, lineage construction, search, stewardship, drift, quality | 10–15 | | 5 · Operations | Deployment, delivery and extensibility, observability, the activation loop | 16–19 | | 6 · Assurance | Trust zones, authorisation and audit, failure modes and recovery | 20–22 | --- ## The decisions that are the architecture The requirement asks for ten design decisions to be settled before detailed design. They are settled as follows, each traceable to the view that carries it. | Decision | Position taken | View | |---|---|---| | Canonical metadata model | Typed entities addressed by URN, carrying versioned aspect documents. Classification, quality rules, ownership and documentation are all aspects, so a new metadata type needs no schema migration | 07 | | System of record | Source systems stay authoritative for their own technical metadata; the catalog is authoritative for curated metadata. Provenance is stored on every aspect version | 08, 09 | | Graph versus relational | Both, but only one is true. PostgreSQL is the versioned system of record; the property graph and the search index are projections rebuilt by replaying the change log | 08 | | Real-time versus batch | Hybrid: weekly full harvest, hourly incremental, event-driven within 5 minutes for sources that can push. One code path, differing only by watermark | 09, 10 | | Lineage granularity | Column level where an expression can be traced, table level everywhere else, with granularity and evidence recorded on each edge and published on the asset profile | 06, 11 | | Ownership model | Federated domains with central policy. Domains own assets, terms and certification; the centre owns the canonical model, the policy catalogue and the stewardship SLA | 01, 13 | | Extensibility | Pluggable connectors and registered aspect schemas, versioned independently of the platform. A new source or attribute ships without a core release | 05, 17 | | Source-of-truth precedence | Certified curation > curation > source-declared > inferred. Inference never overwrites a human value; two sources at the same rank raise a stewardship task rather than last-writer-wins | 09, 10 | | Retention and versioning | Aspect history online for 24 months then compacted; audit append-only for 7 years; raw payloads archived for replay | 08, 21 | | Active hub or catalog | Active hub. Classifications push out to policy enforcement, quality results come back in, impact opens change records. If the loop stops at curation the platform is shelfware | 05, 19 | --- ## Non-functional targets | Property | Target | |---|---| | Scale | 5 million assets, 50 million relationships, 200 source systems, ~40 connector types | | Users | 20,000 licensed, ~2,000 daily active | | Search latency | p95 under 300 ms; asset profile p95 under 500 ms; 3-hop lineage p95 under 1.5 s | | Availability | 99.9% read path; 99.5% ingest, which is allowed to lag | | Recovery | RTO 30 min, RPO 5 min on the aspect store; projection rebuild under 4 hours | | Freshness | Incremental harvest hourly; event-driven sources within 5 minutes | | Audit | Append-only, 7-year retention, separated credentials and object-lock | | Residency | Both regions in the EU; a second residency zone would be a separate deployment | --- ## Deliberate non-goals - **It does not run data-quality tests.** Results are ingested from Great Expectations, Soda, Monte Carlo, dbt and native platform checks. Rebuilding a quality engine would create the silo this platform exists to remove (view 15). - **It does not enforce access at query time.** It publishes labels that the platforms already enforcing access consume (view 05). - **It does not store data values.** Profile statistics are bounded, and no sample values are stored at all for confidential or restricted assets (view 20). --- ## Open items for the sponsor 1. **Stewardship resourcing.** The largest risk in the programme is not technical. If domains do not staff stewardship, the platform degrades to technical metadata only — a search tool, not governance (view 13). 2. **Audit immutability standard.** Object-lock plus separated credentials is assumed sufficient; confirm against the internal audit standard before build (view 21). 3. **Warehouse log retention.** Column-level lineage inference needs at least 90 days of access history. Confirm per platform (views 06, 11). 4. **Second residency zone.** Assumed not required at go-live. If it is, it is a separate deployment, not a configuration change (view 16). --- ## Rebuilding Requires Node and draw.io Desktop. ```bash bash scripts/build.sh ``` The pipeline is generate → validate (strict: zero errors and zero warnings) → editable SVG → plain SVG → white ground → draw.io copy → HTML pages and index → link check. Edit `specs/views.json` and `specs/manifest.json`; never edit the generated output.