Metadata Read Amplification

A dashboard that normally returns in 4 seconds takes 40. Bytes scanned is unchanged, the data files are the same size, the cluster is at 8% CPU, and the storage service reports no errors — only that its read latency has roughly quadrupled. The scan is 4x slower as expected. The planning is 20x slower, and nobody has a graph of planning.

A table format stores the answer to "which files are in this table" as a chain of small objects: a catalogue pointer to a metadata file, which names a manifest list, which names manifests, each of which lists data files with their column statistics. Planning walks that chain, and the walk is sequential where the scan is parallel. A table with 4,000 manifests issues thousands of small requests, one dependent on the last, before the first byte of actual data is read.

That is the amplification: latency on the metadata path multiplies by the depth and count of the chain, while latency on the data path multiplies by one.

Why it matters

It decides how a lakehouse behaves under partial degradation, which is the common failure of object storage — slow, not down. Every capacity model based on bytes and CPU predicts a 4x slowdown and gets 20x, and the gap is where the incident lives.

It also decides the floor on interactive query latency. A table with a long, uncompacted history cannot answer a small question quickly, however selective the filter, because planning happens before pruning has any effect. Teams read this as "the lakehouse is slow for small queries" and reach for a separate serving store, when the actual cause is a maintenance job that has not run.

Implementation patterns

• Manifest rewriting on a schedule, which merges many small manifests into few large ones. This is a different job from data-file compaction and is the one more often missing. Target tens of manifests per table, not thousands.
• Snapshot expiry, so metadata does not accumulate one entry per commit forever. A streaming writer committing every minute produces about 525,000 commits a year, and each leaves a trace in the metadata chain.
• Metadata caching in the engine, keyed by table version, so repeated queries within a session do not re-walk the chain.
• Partitioning that keeps manifest counts bounded, since manifests are typically organised by partition; a table over-partitioned into 200,000 partitions has a metadata problem before it has a data problem.
• Measuring planning time as a distinct metric from scan time and queue time. Without this split, the amplification is invisible.

Industry example

Datadog's published design for Husky treats fragment count, not byte volume, as the scaling variable for its compactors, and states the goal of issuing about one read request per input fragment. The reasoning generalises beyond their bespoke format: when storage is remote, the number of objects touched is the cost driver, and compaction exists to keep that number down. Every open table format has since grown the same pair of maintenance operations for the same reason.

Failure scenarios

• A streaming writer with a one-minute commit interval and no manifest rewriting. Query latency degrades a few percent a week, so no single day looks wrong, and after eight months the dashboards are unusable.
• Storage latency rises by a multiple without erroring. Planning saturates the worker pool, clients retry into the same dependency, and write commits start losing optimistic-concurrency races — including the compaction job, so the metadata grows while the incident runs.
• Time-travel retention set to 90 days on a high-commit table, which prevents snapshot expiry from removing anything and defeats the rewrite job.
• A migration that generates manifests once and never schedules maintenance, so the new format performs worse than the directory layout it replaced within a quarter.

Trade-offs

When not to use it

Do not build metadata monitoring for a table written once a day and read by three dashboards. A daily batch writer produces about 365 commits a year, the chain stays short by itself, and the default maintenance settings are sufficient. The instrumentation is worth the effort at the point where either commits per day exceed a few hundred or planning time exceeds about a fifth of total query time — and that second number is the better trigger, because it is the one the user feels.

It flips towards urgency on any table with a streaming writer, a multi-tenant lakehouse where commit frequency is somebody else's decision, or a table whose readers include an interactive product surface rather than only dashboards.

Interview question

Q: A lakehouse query takes 40 seconds instead of 4. Bytes scanned is identical to last week, CPU is at 8%, and object storage reports no errors but higher latency. Where do you look, and what would you have needed in place to answer this in a minute rather than an afternoon?

What a strong answer covers: splitting query duration into planning and scan before anything else · why planning amplifies more than scan, given a sequential chain of small requests · manifest count and snapshot count per table as the standing metrics · manifest rewriting and snapshot expiry as the structural fix, distinct from data compaction · retry budgets and bounded metadata concurrency as the survival mechanism during degradation · and the recognition that low CPU during a slowdown points at a dependency, not at capacity.

Quick check

Quiz: Why does a 4x increase in object-storage latency produce a much larger increase in lakehouse query time? Because planning is a chain of sequential dependent metadata reads while the scan is parallel, so the multiplier compounds along the chain.

Flashcard: What keeps planning fast on a high-commit table? Manifest rewriting and snapshot expiry, targeting tens of manifests rather than thousands — not data-file compaction, which is a different job.