Column-Level Lineage

Someone asks whether a customer's email address, which they have requested be deleted, appears anywhere in the analytics platform. Table-level lineage tells you which tables derive from the source table, which after four hops is most of the warehouse. Column-level lineage tells you that the email column feeds exactly three downstream columns, one of which is a hashed identifier and two of which are direct copies. The second answer is actionable and the first is not.

The two granularities

Table-level lineage records that table B was produced from table A. It is easy to capture, since it can be inferred from job configuration, and it supports coarse questions: what breaks if this table changes, what must be rebuilt after a backfill.

Column-level lineage records that column B.revenue_usd derives from A.revenue_cents through a specific transformation. It supports the questions that actually get asked: where does this number come from, what depends on this specific field, which downstream columns contain personal data, and what is the blast radius of changing this one column's semantics.

The difference in usefulness is large because tables are wide and dependencies are sparse. A table with 200 columns feeding a table with 50 usually has each output column depending on one or two inputs, so table-level lineage over-reports the impact of a change by an order of magnitude, and an impact analysis that says "everything is affected" is ignored.

How it is captured

Static analysis parses SQL and pipeline code to derive the mapping. It works well for SQL, which is declarative and analysable, and it is where most tooling concentrates. It requires a parser per dialect, and it degrades on dynamic SQL, user-defined functions and anything constructed at runtime.

Runtime capture instruments the query engine to emit lineage as part of execution. It is accurate because it observes what actually ran rather than what the code appears to do, and it requires engine support, which limits it to the platforms that provide it.

Declared lineage has authors annotate dependencies. It is always available and always drifts, since nothing forces the annotation to match the code.

Production systems combine the first two and treat the third as a fallback for the paths neither covers, typically anything passing through application code.

When it breaks

Transformations in application code are opaque. A Python job reading a dataframe, transforming it in arbitrary code, and writing a result gives no column-level information to a SQL parser. This is where lineage graphs quietly become wrong rather than incomplete, because the table-level edge exists and implies column relationships nobody verified.

Wildcards destroy precision. SELECT * means every input column potentially feeds every output column, so the graph reverts to table granularity for that hop and stays there for everything downstream. Prohibiting wildcards in persisted transformations is one of the few style rules with a concrete, checkable payoff.

Lineage without semantics is only half an answer. Knowing that revenue_usd derives from revenue_cents does not say whether the conversion is correct or which currency it assumed. Lineage supports the audit; it does not perform it.

It rots without automation. A lineage graph regenerated on every deployment is trustworthy. One built during a project and maintained manually is wrong within a quarter, and a lineage graph that is wrong is worse than none, because decisions get made on it.