Object Storage Semantics for Analytics

Every table format, commit protocol and file-sizing heuristic in the modern data stack is a workaround for the same substrate. Object storage is not a filesystem with a different API. It is a flat key-value store with HTTP semantics, and four of its properties determine almost every physical design decision made above it.

There is no rename, and that is why table formats exist

A POSIX filesystem gives you an atomic rename within a directory, and a generation of query engines used exactly that to publish results: write to a temp path, rename into place, and readers either see the old directory or the new one. Object stores have no rename. A "rename" is a server-side copy followed by a delete, which is neither atomic nor cheap, and for a multi-gigabyte object it is a full data rewrite billed as such.

Remove atomic rename and a directory of Parquet files can no longer represent a table, because a reader listing that directory mid-write sees a partial set. The metadata layer that Iceberg, Delta Lake and Hudi add exists to put the atomic operation somewhere else: a single pointer swap in a catalog, or a conditional write of one small metadata object. The data files become immutable and append-only; the only thing that ever changes is which files the current snapshot names.

Consistency arrived late, and the workarounds outlived it

S3 was eventually consistent for overwrites and deletes until 1 December 2020, when AWS made read-after-write consistency automatic for all applications, including list operations, at no extra cost and with no performance change (AWS, 2020). Before that, a writer could commit a manifest that a reader would then fail to see, and systems such as S3Guard and Databricks' transactional log existed partly to paper over it.

Strong consistency removed one class of bug but not the design. Immutable data files, snapshot isolation and optimistic concurrency with a conditional commit are still the right architecture, because the remaining three properties have not changed.

Requests, not bytes, are the scaling unit

S3 sustains at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on the number of prefixes in a bucket (AWS S3 performance guidelines). Ten prefixes read in parallel gives you 55,000 GET/s. One prefix does not, and a workload that concentrates on a single prefix gets HTTP 503 SlowDown responses while the service repartitions, which takes time and is not instantaneous.

Listing has its own ceiling. ListObjectsV2 returns at most 1,000 keys per call with a continuation token, so enumerating a partition with 200,000 files costs 200 sequential round trips before a single byte of data is read. This is the mechanical reason a table format keeps a manifest: the manifest is a file list you can read in one GET instead of discovering by pagination.

Pricing follows the same shape. S3 Standard charges roughly $0.005 per 1,000 PUT, COPY, POST or LIST requests and roughly $0.0004 per 1,000 GET requests. A job that reads ten million small objects pays about $4 in request fees regardless of how few bytes those objects hold, and a pipeline that lists aggressively can spend more on LIST than on storage.

Latency is high, bandwidth is elastic, and the two must not be confused

First-byte latency to S3 Standard is tens of milliseconds and does not improve with effort. Aggregate bandwidth is effectively unbounded if you are willing to have enough requests in flight. The consequence is that every efficient reader on object storage is concurrent: it issues byte-range GETs for the column chunks it needs, in parallel, with a prefetch depth chosen so that concurrency = throughput × latency is satisfied. A single-threaded reader on object storage is slow no matter how fast the storage is.

S3 Express One Zone changes the latency constant rather than the model, offering consistent single-digit millisecond access and up to 80% lower request costs, at the price of single-Availability-Zone durability and a different bucket type (AWS S3 Express One Zone).

When it breaks

Partition layouts designed for HDFS fail here. Hive-style partitioning by day and hour was cheap when a directory listing was a namenode lookup. On object storage it is pagination, and a table partitioned to hour granularity over three years needs 26,000 prefix listings just to plan a full scan.

Deletes are not free and not immediate. Removing a million expired objects is a million DELETE requests, billed as PUT-class. Lifecycle rules do it asynchronously and cheaply, but they operate on prefixes and tags, which constrains how you lay data out in the first place.

Read-after-write consistency is per object, not per table. Strong consistency tells you that a GET after a PUT returns the new bytes. It says nothing about two writers committing conflicting snapshots, which is why the commit path still needs a conditional put or a catalog with a compare-and-swap.

The 503 is a capacity signal, not an error. Treating SlowDown as a failure rather than backpressure turns a slow job into a failed one. Exponential backoff with jitter is mandatory, and sustained 503s mean the key layout, not the retry policy, is wrong.