Lineage Coverage

An engineer asks the catalogue what breaks if a column changes. The answer is four consumers. They make the change, and a fifth breaks — a mart built by jobs the harvester stopped parsing after a version upgrade three months ago.

A catalogue renders the graph it has, and an asset whose lineage was never derived looks exactly like an asset with no dependencies. There is no error, no marker and nothing to query for. The gap is discovered by consequence.

Coverage is the metric that closes that hole: for each harvest run, the proportion of known assets for which lineage was actually produced. It is unglamorous and it is the difference between a catalogue that can carry a control and one that cannot.

Why it matters

Lineage is load-bearing for more processes than its owners usually realise: impact analysis, ownership routing, sensitivity-label propagation, retention and erasure scope, cost attribution by dataset. A single parser regression degrades all of them at once, in the same direction, and none of them alarms, because each is consuming a plausible answer.

It also matters for trust dynamics. Once a team has been burned twice by an incomplete impact analysis, they stop consulting the catalogue — which means the people most likely to spot the gap are now the people who never look. Adoption falls, the owners read it as a change-management problem, and the parser failure runs for another year.

Implementation patterns

• Report coverage per harvest run and track it over time. Alert on a relative drop rather than an absolute floor: a platform that normally derives 98% should page below 96%, while one sitting at 70% has a different problem and must raise the number before it can monitor it.
• Make harvesters fail per asset, not silently skip. A parser that cannot read a job definition must record an error against that asset. Skipping is the design defect; an upgrade merely triggers it.
• Cross-check against an independent source. Warehouse query logs observe edges that actually executed; an edge seen in execution but absent from the declared graph is a discrepancy worth reporting.
• Show coverage in the interface. A user looking at an unharvested asset should see "not harvested" rather than a confident empty graph.
• Break down coverage by source system, because the aggregate hides exactly the failure mode described above — one orchestrator at 0% inside a platform-wide 94%.

Industry example

Every mature catalogue product has converged on the same pair of features for the same reason: per-source harvest status, and lineage derived from query logs as a complement to parsed job definitions. The second exists precisely because the first is not enough — parsing is brittle against format changes, and execution logs are the only evidence that does not depend on a parser understanding a dialect. Organisations subject to statutory subject-access and erasure obligations since 2018 have driven this hardest, because a catalogue that under-reports silently produces a written statement to a regulator that is wrong.

Failure scenarios

• A format change after an orchestrator upgrade, silently skipped, so one whole source system disappears from the graph while overall coverage barely moves.
• A purge scoped from lineage that misses derived tables in the unparsed set, leaving residual copies no process will ever find.
• Label propagation stopping at the gap, so personal data flows into an unlabelled table and every downstream control treats it as non-sensitive.
• Coverage measured only as an aggregate, hiding a source at zero behind a healthy total.
• Coverage instrumented but never alerted on, which is the most common of all: the number exists on a dashboard nobody opens.

Trade-offs

Raising coverage is real engineering: parsers for every job format, a query-log ingestion path, and reconciliation between two sources that will disagree. The bill is ongoing, because every tool upgrade can break a parser.

The alternative is cheaper and honest: narrow what lineage is trusted for. A catalogue used as a searchable inventory — who owns this, what does this column mean — needs no coverage guarantee. The cost only becomes necessary when lineage is wired into a control.

When not to use it

Do not instrument coverage for a catalogue that is an inventory. If nothing automated consumes the graph, an incomplete graph is an inconvenience and the monitoring is overhead.

It becomes mandatory the moment lineage feeds a control — erasure scope, label propagation, a change-approval gate. Before wiring lineage into any of those, ask what the control does when lineage is missing. If the answer is "proceeds", the control is worse than nothing, because it converts a visible manual process into an invisible automated gap. The right default is that a control refuses to act on an asset whose lineage was not derived.

Interview question

Q: Your catalogue's impact analysis said four consumers and there were five. The catalogue reported no error. Tell me what metric you would add, how you would set its threshold, and what else in the platform is now suspect.

What a strong answer covers: that missing lineage and absent lineage are indistinguishable in the data model, which is the root defect · coverage per run, broken down by source, alerted on relative drop · harvesters recording per-asset errors instead of skipping · query logs as an independent cross-check · the list of other processes silently degraded — erasure scope, label propagation, ownership routing, cost attribution · and the rule that a control consuming lineage must refuse rather than proceed when coverage is absent.

Quick check

Quiz: Why is a silently skipped lineage parse worse than a failed harvest? Because a failed harvest is visible while a skip renders identically to an asset that genuinely has no dependencies, so every consumer of the graph is confidently wrong.

Flashcard: What threshold should a lineage coverage alert use? A relative drop from the platform's own normal — two points below a steady 98% is a parser regression — not a round absolute floor.