Enterprise Metadata Management System  ·  View 11 of 22  ·  Runtime

Lineage Construction

How four kinds of evidence become column-level edges, and what happens when they cannot.

Editable source SVG draw.io All views
Evidence
Evidence
Query logs
Snowflake · BigQuery
Query logs...
dbt manifests
model graph
dbt manifests...
OpenLineage events
Airflow · Spark
OpenLineage events...
BI semantic models
Power BI · Tableau
BI semantic models...
Parse
Parse
SQL parser
sqlglot · 12 dialects
SQL parser...
Manifest reader
Manifest reader
Event consumer
Event consumer
Report binder
field to column
Report binder...
Resolve
Resolve
URN resolver
platform · db · schema
URN resolver...
Column mapper
expression trace
Column mapper...
Confidence scorer
parsed vs declared
Confidence scorer...
Emit
Emit
Column edges
high confidence
Column edges...
Table edges
fallback
Table edges...
Unresolved refs
manual review
Unresolved refs...
Serve
Serve
Knowledge Graph
Neo4j
Knowledge Graph...
Lineage API
GraphQL n-hop
Lineage API...
Lineage Explorer
collapsed by default
Lineage Explorer...
below threshold
below threshold
Lineage Construction — Evidence to Column Edges
Lineage Construction — Evidence to Column Edges
External / third party
External / third party
Queue / topic
Queue / topic
Application we own
Application we own
Interface / broker
Interface / broker
Risk / gap
Risk / gap
Data store
Data store
failure / alternate
failure / alternate
Column-level lineage is produced only where an expression can be traced. Everything else degrades to table level and says so on the asset profile.
Column-level lineage is produced only where an expression can be traced. Everything else degrades to table level and says so on the asset profile.
v 1.0 · owner Platform Engineering · date 2026-08
v 1.0 · owner Platform Engineering · date 2026-08
Text is not SVG - cannot display

Decisions

  • Column-level lineage where an expression can be traced, table-level everywhere else, and the granularity is recorded on the edge so the difference is visible.
  • Parsing is dialect-aware and version-pinned. An unparsed statement produces an unresolved reference for review rather than a confident wrong edge.
  • OpenLineage is preferred over log parsing wherever the platform emits it — a declared edge beats an inferred one.

Numbers

  • Target: column-level coverage on 80% of certified assets by end of year one.
  • Confidence threshold 0.85; below that the edge is table-level and flagged.

Risks

  • Dynamic SQL, stored procedures and hand-written extracts are effectively opaque. They surface as unresolved references and need declared lineage.
  • A single wide fact table can produce a fan-out that makes graph traversal expensive; depth is capped and paginated (view 22).