Catalogues, Metadata and Discovery

A data catalogue is an index of what exists in a platform, and the pattern of its adoption is consistent enough to be predictable: it is deployed, populated automatically with technical metadata, a documentation push adds business descriptions, the descriptions go stale, and within a year people ask colleagues instead. Understanding which parts of a catalogue maintain themselves is the difference between one that survives and one that does not.

Three layers of metadata

Technical metadata is schemas, types, partitions, sizes, file counts and physical locations. It is extracted automatically from the systems themselves, so it is always current and costs nothing to maintain.

Operational metadata is freshness, update frequency, run history, quality check results, and query patterns. It is also automatic, coming from orchestrators and query logs, and it is the layer that most influences whether a dataset is trustworthy. A table last updated three months ago, with a failing quality check and no queries in sixty days, is described accurately by its operational metadata regardless of what its description claims.

Business metadata is descriptions, ownership, classification, glossary terms and usage guidance. It is human-authored, it is what people actually need, and it is the only layer that decays.

What makes a catalogue used

Usage statistics are the best quality signal available. A table queried by forty people daily is more trustworthy than any description, and one queried by nobody in six months is a candidate for deletion regardless of its documentation. Ranking search results by usage rather than by name match is the single change that most improves a catalogue's perceived quality.

Popular queries beat schema documentation. Showing the five most common queries against a table teaches a newcomer how it is actually used, including the joins and filters that are always applied, faster than any prose description. This is free metadata sitting in the query log.

Ownership must resolve to a person or a rota that responds. A catalogue entry naming a team that no longer exists is worse than an empty field, since it sends people down a dead end. Validating owners against the directory and flagging unresolvable ones is a small automation with disproportionate benefit.

Documentation belongs next to the code. Descriptions maintained in a separate UI drift from the transformation that produces the column. Descriptions declared alongside the pipeline definition and published to the catalogue on deployment stay closer to accurate, because they are reviewed with the change that made them wrong.

When it breaks

Coverage targets produce empty descriptions. Mandating that every column be documented yields "customer_id: the customer id" at scale, which satisfies the metric and helps nobody. Better to document the tables and columns that are actually used, which usage statistics identify precisely, and leave the rest blank and honestly so.

Automated description generation is plausible and unverified. A language model can write a description from a column name and sample values, and it will produce a confident, fluent guess about semantics it cannot know. Generated descriptions belong marked as generated, or they contaminate the one layer whose value is being human-verified.

A catalogue is not governance. Recording that a column contains personal data is not the same as controlling access to it. Catalogues describe; policy engines enforce. Treating documentation as a control is a gap that becomes visible during an audit.

Search only works if people can express what they want. Users search for business concepts and find technical names, so "revenue" fails to match fct_txn_amt_usd. A glossary linking business terms to physical assets is the bridge, and it is the piece most often skipped because it is neither automatic nor technically interesting.