Metadata Harvesting
Populating a catalogue automatically from the systems that hold the data, because anything requiring manual entry will be incomplete within a quarter.
Catalogues fail for one dominant reason: their contents go stale. A catalogue listing datasets that no longer exist, omitting ones that do, and attributing ownership to people who left is worse than none, because people trust it briefly and then stop.
Harvesting is the only arrangement that survives. Technical metadata — schemas, table sizes, partition layouts, last-modified times — is read directly from the platforms. Lineage is derived from query logs and orchestrator definitions rather than drawn. Usage is derived from access logs, which also answers the most practical question a catalogue is asked: is anyone actually using this?
What cannot be harvested is the part that carries meaning: what the dataset is for, what its caveats are, why the number differs from the other number. That is human input, and the way to get it is to make the cost tiny and the moment natural — a description field in the transformation model, reviewed in the same pull request as the logic, rather than a separate portal someone is asked to visit.
The measure of a working catalogue is not how many datasets it lists but whether people search it before asking a colleague.