A catalogue harvests column-level lineage by parsing the orchestrator's job definitions. A version upgrade changes one job format and the parser silently skips those jobs. The catalogue reports no error. What happens over the next three months?

Week one to week four: nothing visible

Lineage that is absent looks exactly like lineage that is correct, because a catalogue renders the graph it has. There is no red marker for "this asset's upstreams were not parsed", so the missing edges present as a table that happens to have no dependencies. Nobody queries for the absence.

Week four to week eight: the first wrong answers

Impact analysis is now quietly incomplete. An engineer asks "what breaks if I change this column", gets a list of four consumers, changes it, and breaks a fifth that was in the unparsed jobs. The failure is attributed to the engineer, not to the catalogue, so the organisation learns "check manually as well" rather than "the catalogue is broken".

Erasure and retention work inherits the same gap: a purge scoped from lineage misses the derived tables in the unparsed set, and the residual copies are not discovered by any process that exists.

Week eight onwards: the trust collapse

Once a team has been burned twice, they stop consulting the catalogue, which means the people best placed to notice the gap now never look at it. Adoption falls, the catalogue's owners read that as a change-management problem and buy training, and the underlying parser failure runs for another year.

Where it amplifies

Downstream of the gap, everything derived from lineage is wrong in the same direction — impact analysis, ownership routing, sensitivity-label propagation, retention scope, cost attribution by dataset. One silent parser failure degrades five governance processes simultaneously, and none of them raises an alarm because each is consuming a plausible answer.

What stops it

• Coverage as a first-class metric: the share of known assets whose lineage was successfully derived this run, tracked over time, with an alert on a drop. A drop of even two percent is a parser regression. This is the single control that makes the class of failure visible.
• Harvesters that fail loudly. A parser that cannot read a job definition must record an error for that asset, not skip it. Skipping is the design defect; the upgrade merely triggered it.
• Cross-checks from a second, independent source — query logs from the warehouse, for instance — so that an edge observed in execution but missing from the declared graph is reported as a discrepancy rather than absorbed.
• Displaying coverage in the interface, so a user asking about an asset with no parsed lineage sees "not harvested" rather than an empty and confident graph.

What would have to be true for it to self-heal

Nothing in the current design self-heals, and that is the point: the failure is silent because the system cannot distinguish "no dependencies" from "not parsed". Until that distinction exists in the data model, no amount of monitoring above it helps.

When this is the wrong control to build

For a catalogue used as a searchable inventory — "who owns this table, what does this column mean" — lineage coverage is a convenience and this failure costs little. It becomes critical the moment lineage is load-bearing for a control: erasure scope, label propagation, or a change-approval gate. Before wiring lineage into a control, ask what the control does when the lineage is missing, and make sure the answer is "refuses" rather than "proceeds".

Choose the coverage alert threshold from the harvest's own variance, not from a round number: a run that normally derives lineage for 98% of assets should alert below 96%, while a platform sitting at 70% coverage has a different problem and needs the number raised before it can be monitored. It flips to urgent the moment a regulator asks which systems hold a subject's data, because since 2018 that answer has had a statutory response window measured in days and a catalogue that silently under-reports will produce a written statement that is wrong.