An Iceberg lakehouse on object storage is healthy. Then the storage service starts serving reads at about four times its usual latency with no errors at all. Nothing is down. Walk through what happens over the next ten minutes and what stops it.

Second by second

Minute 0–1. Query planning degrades first, and by far more than 4x. Planning walks a chain: catalogue pointer, then the table metadata file, then the manifest list, then the manifests themselves. These reads are sequential and there are many of them — a broad time-range query on a table with a few thousand manifests issues thousands of small reads before the first byte of actual data. A 4x latency increase on a chain of sequential round trips shows up as planning going from about 2 seconds to 30 or more.

Minute 1–3. Worker threads block on those reads. Thread pools are sized for a latency that no longer holds, so they saturate, and new queries queue behind planning rather than behind scanning. Dashboard users see spinners and press refresh, which submits a second query without cancelling the first.

Minute 3–6. Clients hit their timeouts and retry. Retries consume the same degraded capacity as first attempts, so the offered load rises exactly when the dependency is least able to serve it. This is the amplification step, and it is the one that turns a slow dependency into an outage.

Minute 3–10, on the write side. Table commits are optimistic: a writer reads the current metadata, prepares its change and swaps the pointer, retrying if someone else won. Slower reads lengthen the window in which a conflict can occur, so concurrent writers to the same table begin to lose commit races. Compaction, which is the least urgent writer and usually the one configured to back off, loses consistently, so file counts start growing — which makes planning slower still. That feedback loop is the reason this does not simply recover when latency returns to normal.

What the user sees

Dashboards that time out, a streaming ingest whose lag graph climbs, and an engine dashboard showing low CPU. Low CPU during a slowdown is the signature of a slow dependency rather than a capacity problem, and it is the signal most teams misread.

What stops it

• Timeouts shorter than the caller's remaining budget, so work is shed instead of queued. A query holding a worker for 120 seconds has already failed its user.
• A retry budget — retries capped as a fraction of first attempts, commonly around 10% — rather than exponential backoff alone. Backoff spaces retries out; it does not bound them.
• Bounded concurrency on metadata reads, so a single wide query cannot occupy the whole pool.
• Manifest rewriting on a schedule, which is the structural fix. A table whose history is compacted into tens of manifests plans through this degradation; one with thousands does not. The resilience was decided weeks earlier by a maintenance job nobody watches.

What would have to be true for it to self-heal

The queue must drain faster than it fills, which requires admission control, not more workers. Adding workers to a dependency-bound system increases the offered load on the dependency and makes it worse.

The alert that would have caught it: planning time reported separately from scan time, and manifest count per table with a threshold. Almost every platform measures only total query duration, which cannot distinguish "the data is large" from "the metadata is slow".

When not to engineer for this

A platform whose tables are compacted weekly and whose largest query plans over 40 manifests will ride out a 4x slowdown with nothing worse than slow dashboards, because the metadata chain is short enough that a multiplier on a small number stays small. The mitigations above cost real work: a maintenance job to own, concurrency limits that reduce peak throughput, and timeouts that will occasionally kill a legitimate long query and generate a complaint.

Prefer spending that effort on manifest hygiene rather than on admission control, in that order, because one removes the amplification and the other only survives it. The order flips for a platform that cannot control its own write patterns — a shared lakehouse where dozens of teams commit small files — since there the file count is somebody else's decision and defence is the only lever you hold. This is the general shape of every slow-dependency failure documented in postmortems since the 2010s: the system degrades where round trips are chained, not where bytes are large.