Enterprise Metadata Management System

Architecture Views

22 views, in reading order. Every view ships three ways: an HTML page, an SVG that re-opens in diagrams.net fully editable, and draw.io source.

A metadata platform architected as a connected knowledge layer rather than a database of table descriptions. Read it in six acts: what sits inside the boundary, how the parts fit, what is stored and who owns it, what happens at runtime, how it is operated, and why it is safe. The load-bearing decisions are the canonical model in view 07, the system-of-record split in view 08, the precedence ladder in views 09 and 10, and the activation loop in view 19.

Context and scope

What the platform is accountable for, who it serves, and the three things it deliberately refuses to do.
01
Metadata sources — 200 registered systems
Metadata sources — 200 registered systems
Warehouses & Lakehouses
Snowflake · Databricks
Warehouses & Lakehouses...
BI & Reporting
Power BI · Tableau
BI & Reporting...
ETL / Orchestration
dbt · Airflow · Spark
ETL / Orchestration...
SaaS & Operational Apps
SAP · Salesforce
SaaS & Operational Apps...
People
People
Data Consumer
analyst · scientist
Data Consumer...
Data Steward
curates a domain
Data Steward...
Domain Owner
accountable
Domain Owner...
Governance & Security
CDO · CISO office
Governance & Security...
Data Engineer
platform team
Data Engineer...
Metadata consumers
Metadata consumers
Policy Enforcement
Unity Catalog · Ranger
Policy Enforcement...
Data Quality Platform
Soda · Great Expectations
Data Quality Platform...
ITSM & Incident
ServiceNow
ITSM & Incident...
Notebooks & IDEs
embedded lookup
Notebooks & IDEs...
Enterprise Metadata Platform
Active metadata hub
Enterprise Metadata Platform...
Enterprise platforms
Enterprise platforms
Identity Provider
Entra ID · SCIM
Identity Provider...
Notification
Teams · email
Notification...
Observability
Prometheus · Grafana
Observability...
schema · usage
schema · usage
reports · fields
reports · fields
run lineage
run lineage
object metadata
object metadata
find and trust data
find and trust data
curate · classify
curate · classify
certify · approve
certify · approve
policy · audit
policy · audit
integrate · automate
integrate · automate
labels · tags
labels · tags
scope · results
scope · results
impact tickets
impact tickets
GraphQL
GraphQL
OIDC · groups
OIDC · groups
tasks · alerts
tasks · alerts
metrics · traces
metrics · traces
Enterprise Metadata Platform — System Context
Enterprise Metadata Platform — System Context
External / third party
External / third party
Person or role
Person or role
Security / platform
Security / platform
batch
batch
event / async
event / async
synchronous
synchronous
two-way
two-way
Out of scope: running quality tests, enforcing access at query time, and storing data values.
Out of scope: running quality tests, enforcing access at query time, and storing data values.
v 1.0 · owner Data & AI Architecture · date 2026-08
v 1.0 · owner Data & AI Architecture · date 2026-08
Text is not SVG - cannot display
System Context Who depends on the platform, what it harvests from, and what it pushes back out. HTML page SVG draw.io
02
Data & tool estate
Data & tool estate
Warehouses & lakes
Warehouses & lakes
BI & reporting
BI & reporting
Pipelines & jobs
Pipelines & jobs
SaaS & apps
SaaS & apps
Harvest
Harvest
Connector Fleet
40 types
Connector Fleet...
Ingestion Agents
run in-VPC
Ingestion Agents...
Event Listeners
webhook · CDC
Event Listeners...
Harvest Orchestrator
watermarks
Harvest Orchestrator...
Process & enrich
Process & enrich
Canonical Mapper
Canonical Mapper
Classify & Profile
rules + ML
Classify & Profile...
Lineage Parser
column level
Lineage Parser...
Merge & Precedence
curated wins
Merge & Precedence...
Metadata repository
Metadata repository
Aspect Store
PostgreSQL
Aspect Store...
Knowledge Graph
Neo4j
Knowledge Graph...
Search Index
OpenSearch
Search Index...
Payload Archive
S3 · replay
Payload Archive...
Access
Access
GraphQL API
primary read
GraphQL API...
REST API
write · bulk
REST API...
Event Egress
Kafka · webhook
Event Egress...
Policy Sync
outbound tags
Policy Sync...
Experience
Experience
Catalog & Search
Catalog & Search
Glossary Workbench
Glossary Workbench
Lineage Explorer
Lineage Explorer
Governance Console
Governance Console
High-Level Architecture — Source to Consumption
High-Level Architecture — Source to Consumption
External / third party
External / third party
Interface / broker
Interface / broker
Queue / topic
Queue / topic
Application we own
Application we own
Data store
Data store
Identity, workflow, audit, notification and observability are cross-cutting; see views 03 and 04.
Identity, workflow, audit, notification and observability are cross-cutting; see views 03 and 04.
v 1.0 · owner Data & AI Architecture · date 2026-08
v 1.0 · owner Data & AI Architecture · date 2026-08
Text is not SVG - cannot display
High-Level Architecture The six stages a piece of metadata passes through, from a source system to a person who trusts it. HTML page SVG draw.io

Structure

The parts, the layering rule between them, and the integration surface in both directions.
03
Experience
Experience
Catalog & Search UI
Catalog & Search UI
Asset Profile
Asset Profile
Glossary Workbench
Glossary Workbench
Lineage Explorer
Lineage Explorer
Stewardship Inbox
Stewardship Inbox
Governance Console
Governance Console
Access
Access
GraphQL API
GraphQL API
REST API
REST API
Event Egress
Event Egress
Bulk Import / Export
Bulk Import / Export
SDK & CLI
SDK & CLI
Metadata services
Metadata services
Search Service
Search Service
Lineage Service
Lineage Service
Glossary Service
Glossary Service
Classification Service
Classification Service
Quality Metadata
Quality Metadata
Workflow Engine
Workflow Engine
Core platform
Core platform
Model & Schema Registry
Model & Schema Registry
Merge & Precedence
Merge & Precedence
Versioning & History
Versioning & History
Entitlement Filter
Entitlement Filter
Persistence
Persistence
Aspect Store
Aspect Store
Knowledge Graph
Knowledge Graph
Search Index
Search Index
Audit Log
Audit Log
Payload Archive
Payload Archive
Ingestion
Ingestion
Connector Framework
Connector Framework
Harvest Orchestrator
Harvest Orchestrator
Metadata Event Bus
Metadata Event Bus
Enrichment Workers
Enrichment Workers
Drift Detector
Drift Detector
Cross-cutting
Cross-cutting
Identity & Access
Identity & Access
Secrets & Keys
Secrets & Keys
Notification
Notification
Observability
Observability
Backup & DR
Backup & DR
Layered Architecture
Layered Architecture
A layer calls only the layer below it. Ingestion reaches persistence through the core platform, never directly.
A layer calls only the layer below it. Ingestion reaches persistence through the core platform, never directly.
v 1.0 · owner Data & AI Architecture · date 2026-08
v 1.0 · owner Data & AI Architecture · date 2026-08
Text is not SVG - cannot display
Layered Architecture The dependency rule: which layer is allowed to call which, and where the cross-cutting concerns sit. HTML page SVG draw.io
04
Enterprise Metadata Platform · Kubernetes
Enterprise Metadata Platform · Kubernetes
Experience
Experience
Catalog Web App
React SPA
Catalog Web App...
Governance Console
React SPA
Governance Console...
Access
Access
API Gateway
OIDC · rate limit
API Gateway...
GraphQL Service
graph-shaped read
GraphQL Service...
REST Service
write · bulk
REST Service...
Event Egress
Kafka · webhook
Event Egress...
Discovery services
Discovery services
Catalog Service
Catalog Service
Search Service
Search Service
Lineage Service
Lineage Service
Glossary Service
Glossary Service
Governance services
Governance services
Classification Service
Classification Service
Quality Metadata
Quality Metadata
Workflow Engine
Temporal
Workflow Engine...
Notification Service
Notification Service
Entitlement Filter
ABAC
Entitlement Filter...
Ingestion
Ingestion
Harvest Orchestrator
Temporal
Harvest Orchestrator...
Connector Runtime
plugin SDK
Connector Runtime...
Metadata Event Bus
Kafka · MCP/MCL
Metadata Event Bus...
Enrichment Workers
profile · classify
Enrichment Workers...
Merge & Precedence
Merge & Precedence
Persistence
Persistence
Aspect Store
PostgreSQL
Aspect Store...
Knowledge Graph
Neo4j
Knowledge Graph...
Search Index
OpenSearch
Search Index...
Payload Archive
S3
Payload Archive...
Audit Log
append-only
Audit Log...
Source systems
200 registered
Source systems...
Entra ID
OIDC · SCIM
Entra ID...
Policy enforcement
Policy enforcement
Quality engines
Quality engines
HTTPS
HTTPS
read
read
write
write
search
search
query
query
commit
commit
project
project
harvest
harvest
tags
tags
results
results
Container Architecture (C4 Level 2)
Container Architecture (C4 Level 2)
Application we own
Application we own
Interface / broker
Interface / broker
Queue / topic
Queue / topic
Security / platform
Security / platform
Data store
Data store
External / third party
External / third party
synchronous
synchronous
event / async
event / async
batch
batch
Eleven edges shown, others omitted for legibility. Identity is validated at the gateway on every call; the authoritative call graph is the service contract register.
Eleven edges shown, others omitted for legibility. Identity is validated at the gateway on every call; the authoritative call graph is the service contract register.
v 1.0 · owner Data & AI Architecture · date 2026-08
v 1.0 · owner Data & AI Architecture · date 2026-08
Text is not SVG - cannot display
Container Architecture The deployable units, grouped by responsibility, and the calls that cross between them. HTML page SVG draw.io
05
Metadata producers — inbound
Metadata producers — inbound
Snowflake / BigQuery
JDBC · ACCESS_HISTORY
Snowflake / BigQuery...
Databricks Unity Catalog
REST · system tables
Databricks Unity Catalog...
dbt / Airflow / Spark
OpenLineage
dbt / Airflow / Spark...
Power BI / Tableau
REST · usage API
Power BI / Tableau...
Kafka Schema Registry
subjects
Kafka Schema Registry...
SAP / Salesforce
metadata API
SAP / Salesforce...
Enterprise Metadata Platform
Enterprise Metadata Platform
Metadata Hub
canonical model · APIs
Metadata Hub...
Connector Framework
pluggable SDK
Connector Framework...
Metadata Event Bus
Kafka
Metadata Event Bus...
GraphQL · REST · Events
GraphQL · REST · Events
Metadata consumers — outbound
Metadata consumers — outbound
Policy Enforcement
masking · row filters
Policy Enforcement...
Quality Platform
Soda · GX
Quality Platform...
ITSM & Incident
ServiceNow
ITSM & Incident...
Notebooks & IDEs
inline lookup
Notebooks & IDEs...
Enterprise Search
M365 · Glean
Enterprise Search...
Metadata Exchange
OpenLineage · CSV
Metadata Exchange...
schema · query log
schema · query log
schema · lineage
schema · lineage
run events
run events
reports · usage
reports · usage
event schemas
event schemas
objects · fields
objects · fields
labels · tags
labels · tags
asset scope
asset scope
impact tickets
impact tickets
GraphQL
GraphQL
asset index
asset index
open export
open export
Integration Architecture — Producers, Hub, Consumers
Integration Architecture — Producers, Hub, Consumers
External / third party
External / third party
Application we own
Application we own
Interface / broker
Interface / broker
Queue / topic
Queue / topic
batch
batch
event / async
event / async
synchronous
synchronous
Every inbound connector is read-only. Outbound writes are limited to tags, labels and tickets — never to source data.
Every inbound connector is read-only. Outbound writes are limited to tags, labels and tickets — never to source data.
v 1.0 · owner Integration Architecture · date 2026-08
v 1.0 · owner Integration Architecture · date 2026-08
Text is not SVG - cannot display
Integration Architecture Every interface in and out, and the rule that keeps the platform from becoming another silo. HTML page SVG draw.io
06
Schema & structure
Schema & structure
Usage & popularity
Usage & popularity
Lineage granularity
Lineage granularity
Quality results
Quality results
Classification
Classification
Cloud warehouse
Cloud warehouse
Full
incl. constraints
Full...
Full
access history
Full...
Column level
SQL parsed
Column level...
Via DQ platform
Via DQ platform
Auto + curated
Auto + curated
Lakehouse
Lakehouse
Full
Unity Catalog
Full...
Full
Full
Column level
system tables
Column level...
Native + DQ
Native + DQ
Auto + curated
Auto + curated
Relational OLTP
Relational OLTP
Full
Full
Partial
sampled logs
Partial...
Table level
Table level
Via DQ platform
Via DQ platform
Auto + curated
Auto + curated
BI & reporting
BI & reporting
Full
datasets · fields
Full...
Full
views · viewers
Full...
Field to column
semantic model
Field to column...
Not available
Not available
Inherited
from upstream
Inherited...
Orchestration & ELT
Orchestration & ELT
Jobs & tasks
Jobs & tasks
Run history
Run history
Column level
OpenLineage
Column level...
Test results
dbt tests
Test results...
Not applicable
Not applicable
Streaming
Streaming
Full
Avro · Protobuf
Full...
Partial
consumer groups
Partial...
Topic level
Topic level
Not available
Not available
Auto + curated
Auto + curated
SaaS & files
SaaS & files
Objects & fields
Objects & fields
Not available
Not available
Manual only
declared
Manual only...
Not available
Not available
Curated
Curated
Connector Coverage by Source Class
Connector Coverage by Source Class
Application we own
Application we own
Interface / broker
Interface / broker
Risk / gap
Risk / gap
External / third party
External / third party
Red cells are declared gaps, not omissions. Lineage below column level is stated on each asset profile so trust is never assumed.
Red cells are declared gaps, not omissions. Lineage below column level is stated on each asset profile so trust is never assumed.
v 1.0 · owner Integration Architecture · date 2026-08
v 1.0 · owner Integration Architecture · date 2026-08
Text is not SVG - cannot display
Connector Coverage What each class of source can actually give up, and where the honest gaps are. HTML page SVG draw.io

Data

The canonical model, where authoritative metadata lives, and how a change reaches every projection.
07
business_term
term_id PK
name
definition
parent_term_id FK
status draft|approved
domain_id FK
business_term...
field
field_urn PK
asset_urn FK
name
data_type
ordinal
nullable
field...
lineage_edge
edge_id PK
upstream_urn FK
downstream_urn FK
granularity col|table
evidence sql|event
confidence
lineage_edge...
job_run
run_id PK
job_urn
platform
started_at
state
job_run...
domain
domain_id PK
name
parent_domain_id FK
owner_group
domain...
asset
asset_urn PK
type table|report|job
platform
domain_id FK
lifecycle draft|certified
deprecated_at
asset...
aspect
aspect_id PK
entity_urn FK
aspect_name
version
payload JSONB
provenance curated|source
created_at
aspect...
classification
class_id PK
aspect_id FK
label public..restricted
regulation GDPR|PCI
pii_flag
confidence
classification...
policy
policy_id PK
name
applies_to label|domain
enforcement_ref
status
policy...
ownership
own_id PK
entity_urn FK
principal
role owner|steward
valid_from
ownership...
quality_rule
rule_id PK
aspect_id FK
dimension
expression
engine soda|gx|dbt
quality_rule...
quality_result
result_id PK
rule_id FK
score
passed
run_at
quality_result...
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
N : M
N : M
N : 1
N : 1
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
N : M
N : M
N : M
N : M
Canonical Metadata Model
Canonical Metadata Model
Every entity is addressed by URN. Classification and quality rules are stored as aspects, so a new metadata type needs no schema change. workflow_task and audit_event are in views 13 and 21.
Every entity is addressed by URN. Classification and quality rules are stored as aspects, so a new metadata type needs no schema change. workflow_task and audit_event are in views 13 and 21.
v 1.0 · owner Data Architecture · date 2026-08
v 1.0 · owner Data Architecture · date 2026-08
Text is not SVG - cannot display
Canonical Metadata Model What counts as an asset, how it relates to people, terms, policy and quality, and how it is extended. HTML page SVG draw.io
08
Metadata Repository
Metadata Repository
System of record — durable, versioned, backed up
System of record — durable, versioned, backed up
Aspect Store
PostgreSQL · partitioned
Aspect Store...
Version History
24 months online
Version History...
Audit Log
append-only · 7 y
Audit Log...
Payload Archive
S3 · object lock
Payload Archive...
Projections — derived, disposable, rebuilt from the change log
Projections — derived, disposable, rebuilt from the change log
Knowledge Graph
Neo4j · 50M edges
Knowledge Graph...
Search Index
OpenSearch · 5M docs
Search Index...
Read Cache
Redis · 15 min TTL
Read Cache...
Operational state
Operational state
Workflow State
Temporal · Postgres
Workflow State...
Watermarks & Jobs
per connector
Watermarks & Jobs...
Connector Secrets
KMS-backed vault
Connector Secrets...
Metadata Change Log
Kafka · 7-day retention
Metadata Change Log...
Backup Vault
PITR 35 days
Backup Vault...
Source systems
own their technical truth
Source systems...
emit on commit
emit on commit
project
project
continuous backup
continuous backup
replay on rebuild
replay on rebuild
harvest payload
harvest payload
Storage Architecture — System of Record and Projections
Storage Architecture — System of Record and Projections
Data store
Data store
Security / platform
Security / platform
Queue / topic
Queue / topic
External / third party
External / third party
event / async
event / async
batch
batch
Losing a projection costs a rebuild, not data. Losing the aspect store costs a restore — which is why only it is on the RPO 5 min path.
Losing a projection costs a rebuild, not data. Losing the aspect store costs a restore — which is why only it is on the RPO 5 min path.
v 1.0 · owner Data Architecture · date 2026-08
v 1.0 · owner Data Architecture · date 2026-08
Text is not SVG - cannot display
Storage Architecture Which store is authoritative, which stores are disposable, and what that split buys. HTML page SVG draw.io
09
Capture
Capture
Scheduled harvest
hourly incremental
Scheduled harvest...
Event push
webhook · CDC
Event push...
Manual & bulk
UI · CSV · API
Manual & bulk...
Normalise
Normalise
Source adapter
Source adapter
Canonical mapper
to URN + aspects
Canonical mapper...
Schema valid?
aspect registry
Schema valid?...
Dead letter
quarantine
Dead letter...
Enrich
Enrich
Profiler
sampled stats
Profiler...
PII classifier
rules + ML
PII classifier...
Term suggester
advisory only
Term suggester...
Drift detector
diff vs last
Drift detector...
Resolve
Resolve
Precedence engine
curated wins
Precedence engine...
Conflict
two sources disagree
Conflict...
Stewardship task
routed by domain
Stewardship task...
Commit
Commit
Aspect Store
version + 1
Aspect Store...
Change log
Kafka MCL
Change log...
Audit Log
who · what · before
Audit Log...
Activate
Activate
Knowledge Graph
Knowledge Graph
Search Index
Search Index
Policy Sync
labels outbound
Policy Sync...
Notify owners
Teams · email
Notify owners...
reject
reject
unresolved
unresolved
raise
raise
Metadata Flow — Capture to Activation
Metadata Flow — Capture to Activation
Interface / broker
Interface / broker
Queue / topic
Queue / topic
Application we own
Application we own
Decision point
Decision point
Risk / gap
Risk / gap
Data store
Data store
Security / platform
Security / platform
failure / alternate
failure / alternate
synchronous
synchronous
Precedence: certified curation > curation > source-declared > inferred. Inference never overwrites a human value; it only proposes.
Precedence: certified curation > curation > source-declared > inferred. Inference never overwrites a human value; it only proposes.
v 1.0 · owner Data Architecture · date 2026-08
v 1.0 · owner Data Architecture · date 2026-08
Text is not SVG - cannot display
Metadata Flow and Precedence The path from capture to activation, and the rule applied when two systems disagree. HTML page SVG draw.io

Runtime

What actually happens when metadata is harvested, searched, curated and found to have drifted.
11
Evidence
Evidence
Query logs
Snowflake · BigQuery
Query logs...
dbt manifests
model graph
dbt manifests...
OpenLineage events
Airflow · Spark
OpenLineage events...
BI semantic models
Power BI · Tableau
BI semantic models...
Parse
Parse
SQL parser
sqlglot · 12 dialects
SQL parser...
Manifest reader
Manifest reader
Event consumer
Event consumer
Report binder
field to column
Report binder...
Resolve
Resolve
URN resolver
platform · db · schema
URN resolver...
Column mapper
expression trace
Column mapper...
Confidence scorer
parsed vs declared
Confidence scorer...
Emit
Emit
Column edges
high confidence
Column edges...
Table edges
fallback
Table edges...
Unresolved refs
manual review
Unresolved refs...
Serve
Serve
Knowledge Graph
Neo4j
Knowledge Graph...
Lineage API
GraphQL n-hop
Lineage API...
Lineage Explorer
collapsed by default
Lineage Explorer...
below threshold
below threshold
Lineage Construction — Evidence to Column Edges
Lineage Construction — Evidence to Column Edges
External / third party
External / third party
Queue / topic
Queue / topic
Application we own
Application we own
Interface / broker
Interface / broker
Risk / gap
Risk / gap
Data store
Data store
failure / alternate
failure / alternate
Column-level lineage is produced only where an expression can be traced. Everything else degrades to table level and says so on the asset profile.
Column-level lineage is produced only where an expression can be traced. Everything else degrades to table level and says so on the asset profile.
v 1.0 · owner Platform Engineering · date 2026-08
v 1.0 · owner Platform Engineering · date 2026-08
Text is not SVG - cannot display
Lineage Construction How four kinds of evidence become column-level edges, and what happens when they cannot. HTML page SVG draw.io
13
Trigger
Trigger
Assign
Assign
Curate
Curate
Review
Review
Publish
Publish
Monitor
Monitor
Data consumer
Data consumer
Requests a definition
Requests a definition
Suggests description
Suggests description
Sees certified badge
Sees certified badge
Reports an issue
Reports an issue
Data steward
Data steward
Stewardship inbox
Stewardship inbox
Claims task
Claims task
Writes term
definition · synonyms
Writes term...
Links term to columns
Links term to columns
Submits for approval
Submits for approval
Coverage scorecard
Coverage scorecard
Domain owner
Domain owner
Drift alert
Drift alert
Route by domain
Route by domain
Approve or reject
Approve or reject
Certifies asset
Certifies asset
Recertifies
annual
Recertifies...
Platform team
Platform team
Harvest gap
Harvest gap
Auto-assign rule
Auto-assign rule
Fixes connector
Fixes connector
Backfills metadata
Backfills metadata
SLA dashboard
SLA dashboard
Stewardship Workflows by Persona
Stewardship Workflows by Persona
Application we own
Application we own
Security / platform
Security / platform
Queue / topic
Queue / topic
Decision point
Decision point
Risk / gap
Risk / gap
Workflows are configurable per domain and asset type. Unclaimed tasks escalate to the domain owner after 5 working days.
Workflows are configurable per domain and asset type. Unclaimed tasks escalate to the domain owner after 5 working days.
v 1.0 · owner Data Governance · date 2026-08
v 1.0 · owner Data Governance · date 2026-08
Text is not SVG - cannot display
Stewardship Workflows How a gap becomes an owned, reviewed, published definition — through the four personas in turn. HTML page SVG draw.io
14
Detect
Detect
Schema diff
vs last harvest
Schema diff...
Deprecation set
by owner
Deprecation set...
Quality breach
from DQ platform
Quality breach...
Classify
Classify
Breaking change?
drop · retype · rename
Breaking change?...
Additive only
log and move on
Additive only...
Traverse
Traverse
Graph traversal
n-hop downstream
Graph traversal...
Blast radius
assets · reports · jobs
Blast radius...
Certified assets hit
Certified assets hit
Notify
Notify
Owner notification
Teams · email
Owner notification...
Change record
ServiceNow
Change record...
Catalog banner
on every consumer
Catalog banner...
Act
Act
CI change gate
block or warn
CI change gate...
Deprecation window
90 days
Deprecation window...
Retire asset
lifecycle state
Retire asset...
no
no
Schema Drift and Impact Analysis
Schema Drift and Impact Analysis
Application we own
Application we own
Queue / topic
Queue / topic
Decision point
Decision point
Risk / gap
Risk / gap
Interface / broker
Interface / broker
External / third party
External / third party
failure / alternate
failure / alternate
Impact is answered from the graph, not from a spreadsheet. A change gate is advisory for uncertified assets and blocking for certified ones.
Impact is answered from the graph, not from a spreadsheet. A change gate is advisory for uncertified assets and blocking for certified ones.
v 1.0 · owner Data Governance · date 2026-08
v 1.0 · owner Data Governance · date 2026-08
Text is not SVG - cannot display
Drift and Impact Analysis What happens between a column being dropped upstream and the report that breaks because of it. HTML page SVG draw.io
15
Quality engines — execution stays there
Quality engines — execution stays there
Great Expectations
suite results
Great Expectations...
Soda Core
scan results
Soda Core...
Monte Carlo
anomaly alerts
Monte Carlo...
dbt tests
run_results.json
dbt tests...
Native platform checks
constraints · freshness
Native platform checks...
Quality Metadata Service
Quality Metadata Service
Quality Metadata Service
dimensions · scores
Quality Metadata Service...
Rule Registry
stored as aspects
Rule Registry...
Score Aggregator
asset · domain roll-up
Score Aggregator...
Issue Linker
to asset and owner
Issue Linker...
Where quality becomes visible
Where quality becomes visible
Asset profile badge
score + last run
Asset profile badge...
Search ranking signal
Search ranking signal
Owner alert
breach on owned asset
Owner alert...
Certification gate
no cert while failing
Certification gate...
Domain SLA report
Domain SLA report
results API
results API
results API
results API
incidents
incidents
test outcomes
test outcomes
check state
check state
quality panel
quality panel
trust boost
trust boost
notify
notify
gate
gate
monthly
monthly
Data Quality Metadata — Ingest and Surface
Data Quality Metadata — Ingest and Surface
External / third party
External / third party
Application we own
Application we own
Interface / broker
Interface / broker
Decision point
Decision point
Security / platform
Security / platform
event / async
event / async
batch
batch
synchronous
synchronous
The platform stores quality dimensions, rules, scores and history. It deliberately runs no tests — rebuilding a quality engine would create the silo this system exists to remove.
The platform stores quality dimensions, rules, scores and history. It deliberately runs no tests — rebuilding a quality engine would create the silo this system exists to remove.
v 1.0 · owner Data Governance · date 2026-08
v 1.0 · owner Data Governance · date 2026-08
Text is not SVG - cannot display
Data Quality Metadata How quality results reach the catalog without the catalog becoming a quality engine. HTML page SVG draw.io

Operations

How the platform is deployed, extended, watched, and kept from decaying into a stale catalog.
16
AWS eu-west-1 · primary · active
AWS eu-west-1 · primary · active
AZ-a
AZ-a
EKS node group
services · workers
EKS node group...
Aurora writer
PostgreSQL 16
Aurora writer...
OpenSearch data
OpenSearch data
AZ-b
AZ-b
EKS node group
EKS node group
Aurora reader
failover target
Aurora reader...
OpenSearch data
OpenSearch data
AZ-c
AZ-c
EKS node group
EKS node group
Neo4j cluster
3 core members
Neo4j cluster...
MSK brokers
Kafka
MSK brokers...
AWS eu-central-1 · warm standby
AWS eu-central-1 · warm standby
Standby estate
Standby estate
EKS scaled to zero
IaC identical
EKS scaled to zero...
Aurora global replica
RPO under 1 min
Aurora global replica...
S3 cross-region copy
S3 cross-region copy
Network-isolated estates
Network-isolated estates
Restricted VPC or data centre
Restricted VPC or data centre
Ingestion agent
outbound 443 only
Ingestion agent...
Private sources
no inbound route
Private sources...
Route 53 + WAF
health-checked
Route 53 + WAF...
Entra ID
Entra ID
Backup vault
PITR 35 days
Backup vault...
HTTPS 443
HTTPS 443
global replication
global replication
mTLS outbound
mTLS outbound
read-only
read-only
continuous backup
continuous backup
Deployment and Infrastructure
Deployment and Infrastructure
Application we own
Application we own
Data store
Data store
Queue / topic
Queue / topic
Interface / broker
Interface / broker
External / third party
External / third party
Security / platform
Security / platform
synchronous
synchronous
event / async
event / async
batch
batch
Warm standby, not active-active: metadata reads tolerate a 30-minute RTO, and dual-region write consistency for a versioned store is not worth its cost here.
Warm standby, not active-active: metadata reads tolerate a 30-minute RTO, and dual-region write consistency for a versioned store is not worth its cost here.
v 1.0 · owner Platform Engineering · date 2026-08
v 1.0 · owner Platform Engineering · date 2026-08
Text is not SVG - cannot display
Deployment and Infrastructure Where it runs, how it survives an availability-zone loss, and how it reaches isolated networks. HTML page SVG draw.io
17
Source
Source
Platform services
monorepo
Platform services...
Connector plugins
versioned separately
Connector plugins...
Aspect schemas
the metadata model
Aspect schemas...
Build & test
Build & test
Unit + contract tests
Unit + contract tests
Connector conformance
golden payloads
Connector conformance...
Schema compatible?
backward only
Schema compatible?...
Gate
Gate
SAST · SCA · secrets
SAST · SCA · secrets
Migration dry run
on prod snapshot
Migration dry run...
Model change board
new asset types only
Model change board...
Blocked
breaking change
Blocked...
Deploy
Deploy
Dev
synthetic estate
Dev...
Staging
prod metadata subset
Staging...
Production
blue/green
Production...
Verify
Verify
Smoke harvest
one asset per class
Smoke harvest...
Projection lag check
under 60 s
Projection lag check...
Rollback
one release back
Rollback...
breaking
breaking
on failure
on failure
Delivery Pipeline and Extensibility
Delivery Pipeline and Extensibility
Application we own
Application we own
Decision point
Decision point
Security / platform
Security / platform
Risk / gap
Risk / gap
failure / alternate
failure / alternate
A new source type ships as a plugin and a new metadata attribute ships as an aspect schema. Neither requires a core platform release.
A new source type ships as a plugin and a new metadata attribute ships as an aspect schema. Neither requires a core platform release.
v 1.0 · owner Platform Engineering · date 2026-08
v 1.0 · owner Platform Engineering · date 2026-08
Text is not SVG - cannot display
Delivery and Extensibility How a new connector, a new asset type and a platform change each reach production. HTML page SVG draw.io
18
Emit
Emit
Collect
Collect
Store
Store
Consume
Consume
Act
Act
Service health
Service health
OpenTelemetry SDK
OpenTelemetry SDK
OTel Collector
OTel Collector
Prometheus
Prometheus
SLO dashboard
99.9% read path
SLO dashboard...
Page on burn rate
Page on burn rate
Harvest health
Harvest health
Connector metrics
rows · duration
Connector metrics...
OTel Collector
OTel Collector
Prometheus
Prometheus
Freshness by source
age of last run
Freshness by source...
Retry, then quarantine
Retry, then quarantine
Projection lag
Projection lag
Change log offsets
Change log offsets
Lag exporter
Lag exporter
Prometheus
Prometheus
Index vs store lag
Index vs store lag
Show staleness banner
Show staleness banner
Metadata coverage
Metadata coverage
Nightly coverage job
Nightly coverage job
Batch export
Batch export
Aspect Store
Aspect Store
Domain scorecard
owned · described
Domain scorecard...
Steward campaign
Steward campaign
Access & audit
Access & audit
Audit events
Audit events
Log shipper
Log shipper
OpenSearch
7-year archive
OpenSearch...
Access review
quarterly
Access review...
Revoke and investigate
Revoke and investigate
Observability and Operations
Observability and Operations
Application we own
Application we own
Interface / broker
Interface / broker
Data store
Data store
Security / platform
Security / platform
Queue / topic
Queue / topic
Coverage is treated as a production signal, not a report: a domain whose ownership drops below target raises work, not just a number.
Coverage is treated as a production signal, not a report: a domain whose ownership drops below target raises work, not just a number.
v 1.0 · owner SRE · date 2026-08
v 1.0 · owner SRE · date 2026-08
Text is not SVG - cannot display
Observability and Operations What is measured, and which signals are allowed to wake somebody up. HTML page SVG draw.io

Assurance

Where the blast radius is, how every governance action is proved, and what is known to be able to fail.
20
Untrusted · internet
Untrusted · internet
Catalog user
managed device
Catalog user...
Automation client
service principal
Automation client...
Metadata scraper
column-name harvest
Metadata scraper...
Perimeter · DMZ
Perimeter · DMZ
WAF + DDoS
WAF + DDoS
Load balancer
TLS 1.3
Load balancer...
API Gateway
OIDC · quota
API Gateway...
Application · private subnets
Application · private subnets
Metadata services
mTLS mesh
Metadata services...
Policy decision point
ABAC
Policy decision point...
Entra ID
MFA · SCIM
Entra ID...
Metadata · restricted
Metadata · restricted
Aspect Store
KMS at rest
Aspect Store...
Knowledge Graph
Knowledge Graph
Search Index
per-domain aliases
Search Index...
Audit Log
write-once · 7 y
Audit Log...
Source estate · least privilege
Source estate · least privilege
Ingestion agent
outbound 443 only
Ingestion agent...
Read-only role
catalog views only
Read-only role...
Source systems
Source systems
HTTPS 443
HTTPS 443
client credentials
client credentials
rate limited
rate limited
mTLS + token
mTLS + token
authorise
authorise
TLS · scoped role
TLS · scoped role
every action
every action
mTLS outbound
mTLS outbound
read-only metadata
read-only metadata
Security Architecture — Trust Zones
Security Architecture — Trust Zones
Person or role
Person or role
External / third party
External / third party
Risk / gap
Risk / gap
Security / platform
Security / platform
Interface / broker
Interface / broker
Application we own
Application we own
Data store
Data store
synchronous
synchronous
failure / alternate
failure / alternate
event / async
event / async
batch
batch
Metadata leaks too. No data values are persisted; profile statistics are bounded, and sample values are never stored for confidential or restricted assets.
Metadata leaks too. No data values are persisted; profile statistics are bounded, and sample values are never stored for confidential or restricted assets.
v 1.0 · owner Security Architecture · date 2026-08
v 1.0 · owner Security Architecture · date 2026-08
Text is not SVG - cannot display
Security Trust Zones Where the boundaries are, what authenticates at each crossing, and why metadata is treated as sensitive. HTML page SVG draw.io
22
Tier 1 · read path · 99.9% · RTO 30 min · must not fail
Tier 1 · read path · 99.9% · RTO 30 min · must not fail
Search & read API
3 AZ · autoscaled
Search & read API...
Search Index
3 replicas
Search Index...
Read Cache
serves on index loss
Read Cache...
Stale-read banner
shows lag age
Stale-read banner...
Tier 2 · write and ingest · may lag, must not lose
Tier 2 · write and ingest · may lag, must not lose
Metadata Event Bus
7-day retention
Metadata Event Bus...
Aspect Store
RPO 5 min
Aspect Store...
Connector failure
quarantine + alert
Connector failure...
Poison payload
dead letter, skip
Poison payload...
Tier 3 · projections · disposable
Tier 3 · projections · disposable
Knowledge Graph
rebuild under 4 h
Knowledge Graph...
Index rebuild
replay change log
Index rebuild...
Payload Archive
replay beyond 7 days
Payload Archive...
Standing risks · owned, not hidden
Standing risks · owned, not hidden
Graph hot spot
wide lineage fan-out
Graph hot spot...
Source rate limits
harvest falls behind
Source rate limits...
Stale curation
recertification overdue
Stale curation...
Shadow catalogs
the silo returns
Shadow catalogs...
Backup vault
PITR 35 days
Backup vault...
Warm standby region
RTO 30 min
Warm standby region...
fall back to cache
fall back to cache
replay
replay
deep replay
deep replay
continuous backup
continuous backup
global replication
global replication
Failure Modes, Tiers and Recovery
Failure Modes, Tiers and Recovery
Application we own
Application we own
Data store
Data store
Risk / gap
Risk / gap
Queue / topic
Queue / topic
External / third party
External / third party
failure / alternate
failure / alternate
batch
batch
event / async
event / async
Adoption, not infrastructure, is the largest risk: a catalog nobody trusts is replaced by spreadsheets, which is why coverage and certification are on the operations dashboard in view 18.
Adoption, not infrastructure, is the largest risk: a catalog nobody trusts is replaced by spreadsheets, which is why coverage and certification are on the operations dashboard in view 18.
v 1.0 · owner Data & AI Architecture · date 2026-08
v 1.0 · owner Data & AI Architecture · date 2026-08
Text is not SVG - cannot display
Failure Modes and Recovery What is allowed to fail, what is not, and the risks that are being accepted with their eyes open. HTML page SVG draw.io
Open svg/<view>.svg or drawio/<view>.drawio in draw.io Desktop or at app.diagrams.net to edit. The SVG carries the diagram inside it, so it is both the picture and the source. This folder is self-contained — copy it whole and every link still resolves.