Which requirement most often forces an open table format over plain files in object storage?

Why deletion is usually the forcing requirement

A plain lake stores immutable files. Deleting one user's rows means finding every file containing them, rewriting each without those rows, and swapping atomically under concurrent readers — across petabytes. Without transactional metadata this is either impossible or a bespoke, error-prone project repeated per request.

An open table format supplies exactly the missing pieces: row-level deletes via delete vectors or merge-on-read, snapshot isolation so readers see a consistent view during rewrites, metadata mapping values to files so deletion touches only relevant files, and compaction applying accumulated deletes on a schedule rather than synchronously.

Volume alone would justify a plain lake. Targeted deletion is what eliminates the simple answer — and recognising which requirement is doing the eliminating is the actual architectural skill.

The other properties it brings

• Concurrent writes with conflict detection, so several pipelines can write without corrupting each other.
• Schema evolution without rewriting history, essential when event schemas change constantly and old data must remain readable.
• Time travel, which makes model training reproducible — a model can be retrained against the exact snapshot it originally saw.
• Engine independence, so the storage decision does not lock in the compute decision.
• Partition evolution in some formats, so a partitioning mistake is correctable without rewriting everything.

The costs to state

Compaction and metadata maintenance are continuous operational work. Neglected, small-file accumulation destroys query performance — and this is the most common way these deployments degrade.

Metadata itself becomes large enough to need tuning. Deletes are not free: merge-on-read shifts cost to query time, so heavy delete volume degrades reads until compaction catches up.

The decision framing

Plain files where data is append-only, read by one engine, and never needs targeted modification. A table format as soon as any of updates, deletes, concurrent writes or independent schema evolution is required — and for anything holding personal data, that is immediately.