An organisation is consolidating a warehouse and a lake onto open table formats. What guarantees do ACID-on-object-storage tables provide, where do small files and compaction bite, and how are streaming and batch writers coordinated?

What the table formats actually provide

Object storage gives durable, cheap, immutable objects and no transactions, no atomic multi-file updates and only eventual listing consistency in some implementations. A directory of Parquet files is not a table: a reader can observe a half-written update, and two writers can silently overwrite each other.

Open table formats — Delta Lake, Iceberg, Hudi — add a metadata layer that defines which files constitute the table at a given version.

That gives:

• Atomic commits. A write publishes a new metadata version referencing a new file set; readers see the old or the new version, never a mixture.
• Snapshot isolation, since a reader pins a version for the duration of its query.
• Optimistic concurrency between writers, with conflict detection at commit — the write is rejected and retried rather than silently lost.
• Time travel, because previous versions remain referenced until expired.
• Schema evolution with defined compatibility rules.
• Efficient pruning from per-file statistics in the metadata, which is where most query performance comes from.

Where small files bite

Streaming ingestion writes frequently, and frequent writes produce many small files. The costs compound:

• Per-file overhead on read. Opening a thousand small files is dramatically slower than opening ten large ones, and object storage per-request pricing makes it expensive as well as slow.
• Metadata growth, since every file is tracked.
• Poor compression and poor statistics, because both work better over larger row groups — which degrades pruning, which is the main performance mechanism.

Compaction is therefore not optional maintenance; it is a required background process, and a lakehouse without it degrades continuously and silently.

• Compact on a schedule or by trigger — file count or average size crossing a threshold.
• Compaction is itself a writer and must not conflict with ingestion, which is what the format's concurrency control is for.
• Expire old snapshots and remove orphaned files, or storage grows without bound and time travel retains everything forever.
• Sort or cluster during compaction on the columns queries filter by, so compaction improves pruning rather than merely reducing file count.

Coordinating streaming and batch writers

• Optimistic concurrency handles the common case: two writers touching different partitions commit independently; overlapping writers conflict and one retries.
• Partition by writer where possible, so streaming and batch touch disjoint partitions and never conflict.
• Serialise the operations that rewrite existing files — compaction, backfills, corrections — since these conflict with everything and retrying a large rewrite is expensive.
• Idempotent streaming writes, keyed by a batch identifier, so a retried micro-batch does not duplicate.
• Backfills into a separate version or partition, published atomically, rather than rewriting in place.

The honest limitations

• Not a transactional database. Commit latency is seconds, not milliseconds; this is an analytical store with transactional integrity, not an OLTP replacement, and using it for high-frequency single-row updates produces a file-per-update disaster.
• Concurrent writers to the same partition contend, and under heavy contention optimistic retries can starve.
• Metadata itself becomes a scaling concern on very large tables.
• Maintenance is a real operational commitment. Compaction, snapshot expiry and orphan cleanup must be scheduled, monitored and paid for — and the platform degrades gradually when they are not, which is the most common lakehouse failure and the least dramatic.