Catalogue Coupling

The argument for consolidating a warehouse and a lake onto open table formats is portability: the files are Parquet, the table metadata is a published specification, and any engine can read them. That part is true.

What the argument leaves out is how an engine finds the table. A query says analytics.orders_fact, and something has to turn that name into a pointer to the current metadata file. That something is the catalogue - a Hive Metastore, a vendor's managed catalogue, a REST catalogue service, or a cloud catalogue - and every read and every commit goes through it.

So the portable thing is the data and the binding thing is the catalogue. An engine that cannot speak your catalogue's protocol cannot see your tables at all, however open the files are, and the permissions, lineage and table-level governance that accumulate in the catalogue are not part of any file format.

Why it matters

It relocates the lock-in conversation to where the cost actually is. A team that chose an open format to keep its options open can still find that switching engines means re-registering every table, re-creating every grant and reconciling every view definition - work that scales with the number of tables and consumers rather than with the volume of data, which is why it surprises people who budgeted for a data migration.

It also decides correctness, not just convenience. The catalogue is where an atomic commit is serialised: a writer swaps the table's current metadata pointer through it, and that swap is what makes the commit atomic. Anything that bypasses or duplicates the catalogue breaks the guarantee the table format was adopted for.

Implementation patterns

• One catalogue per table, named explicitly. Multiple engines may read and write, through the same catalogue. Two catalogues over one table location is a correctness bug waiting for concurrent writes.
• Prefer a catalogue with a documented open protocol - the Iceberg REST catalogue specification exists for exactly this reason - so engine choice stays a client decision.
• Keep governance where you can export it. Grants and policies held only in a proprietary catalogue UI are the part of a migration nobody has a script for.
• Treat table registration as code. If every table is created by a job from a definition in version control, re-registering 4000 tables in a new catalogue is a run rather than a project.
• Separate the catalogue's availability from the data's. It is a small, highly available service on whose uptime every query depends, and it deserves the monitoring that implies.

Industry example

The long-running case is the Hive Metastore, which became the default catalogue in lake estates and has outlived most of the engines that used it. Teams in production regularly keep it running purely because the table definitions and permissions inside it have no exporter - a relational database holding thousands of table entries is the thing blocking the modernisation, not the petabytes of Parquet beside it.

The newer form of the same problem appears when a team adopts an open table format inside one vendor's platform and later wants a second engine on the same tables. The files are readable. The catalogue requires the vendor's own client, and the project becomes a catalogue migration with a re-grant exercise attached.

Failure scenarios

• Two catalogues over one table. Each believes it holds the current metadata pointer, so concurrent commits overwrite each other and committed data disappears with no error anywhere.
• Catalogue outage, total read outage. The data is intact in object storage and unreachable, because nothing can resolve a table name.
• Permissions that exist in one catalogue only, so a second engine reading the same files enforces nothing - an access-control hole created by adding an engine.
• Metadata drift after a bulk file operation, where a lifecycle rule or a manual copy changes files the catalogue still describes, producing queries that fail on missing files.
• A migration that is estimated from data volume and then takes months, because the work is thousands of registrations, views and grants.

Trade-offs

The summary: open file formats remove the data-rewrite cost of switching engines and leave the registration-and-governance cost in place, and the second is the one measured in months.

When not to use it

This is a dependency to manage, not an option to decline - any table needs a catalogue, and running without one means paths in queries, which gives up atomic commits and schema evolution entirely.

The useful "when not" is about effort. A small estate does not need a catalogue strategy. With 40 tables, one engine and one team, the vendor's integrated catalogue is the right answer and the re-registration cost in a hypothetical future migration is a week. The strategy earns its keep from roughly the point where tables run into the hundreds, more than one engine is in use, or a regulator asks who can read what - below that, catalogue independence is a cost paid for optionality nobody will exercise.

Interview question

Q: Your team moved 90 TB from a proprietary warehouse to Iceberg tables on object storage "so we are never locked in again". Eighteen months later you want to add a second query engine. What actually stands in the way, and what would you have done at the start to make it a configuration change?

What a strong answer covers: the files are portable and the catalogue is the dependency, since every read and commit resolves through it; the switching cost scales with tables, views and grants rather than terabytes; the correctness point that the catalogue serialises atomic commits, so two catalogues over one table lose data silently; and the preventions - a catalogue with an open protocol, table registration and grants held as code, and governance kept in an exportable form.

Quick check

Quiz: After adopting an open table format, what still ties you to one engine? - The catalogue: an engine must speak its protocol to resolve table names and commit, and the grants and view definitions inside it have no file-format equivalent.

Flashcard: Why is two catalogues over one table a correctness bug rather than a configuration mess? - Each one serialises commits by swapping the table's current metadata pointer, so concurrent writers through different catalogues overwrite each other's commits and the lost data is reported by nothing.