Data Lineage Graph

Which dataset was derived from which, through what job — the artifact you produce when someone asks where a number came from.

Type Data View Stage Run Topic Data Observability Area Data Governance & Semantics

flowchart LR
  s1[("core_banking.accounts<br/><i>source</i>")]
  s2[("payments.transactions<br/><i>source</i>")]
  s3[("crm.customer<br/><i>source</i>")]

  j1{{"job: cdc_ingest<br/><i>hourly</i>"}}
  j2{{"job: conform_party<br/><i>daily 02:00</i>"}}
  j3{{"job: build_balances<br/><i>daily 03:15</i>"}}
  j4{{"job: reg_extract<br/><i>monthly</i>"}}

  r1[("raw.accounts")]
  r2[("raw.transactions")]
  c1[("clean.party")]
  c2[("clean.balance_daily")]
  o1[("report.capital_adequacy")]
  o2["Dashboard:<br/>Treasury Daily"]

  s1 --> j1 --> r1
  s2 --> j1 --> r2
  s3 --> j2
  r1 --> j2 --> c1
  r1 --> j3
  r2 --> j3 --> c2
  c1 --> j3
  c2 --> j4 --> o1
  c2 --> o2

What it is

A directed graph whose nodes are datasets and whose edges are the jobs that produced them. Read forwards it answers "if this source is wrong, what else is wrong?". Read backwards it answers "where did this number come from?" — which is the question a regulator, an auditor or a furious head of finance actually asks.

Note that it is not a system diagram. Two tables in the same database are two nodes; one job that reads three tables is one node with three inbound edges.

When you produce it

For any figure that is externally reported or externally audited, and before any migration of a source system, because the graph is the impact assessment. In practice it should be generated from the orchestrator and the query logs rather than drawn — a hand-drawn lineage graph is out of date the week it is finished.

Who reads it

Regulators and auditors, who want provenance. Data engineers doing impact analysis before a schema change. Incident responders working out the blast radius of a bad load.

What good looks like

Generated, with a timestamp, from the systems that actually run the jobs.
Column-level where the obligation is column-level; table-level is often not enough for a privacy or reporting question.
Job schedules on the edges, so the reader can see where staleness enters.
Terminal consumers included — the dashboard and the report are nodes, because they are what somebody is looking at.

Common mistakes

Drawing it by hand. It will be wrong within a sprint.
Stopping at the warehouse boundary. The spreadsheet somebody exports and emails is part of the lineage whether it is drawn or not.
Table-level only. "This report uses that table" does not answer whether a particular field is personal data.