Feeding the Accelerator: Why the Training Loop Is a Storage Problem

Eight H100s, a 100 Gb/s NIC, an object store advertising unlimited throughput, and the accelerators are at 11% utilisation. The network graph is flat at 450 MB/s. Nothing is saturated. Nothing is broken. The job will take nine days instead of one.

This is the most expensive failure mode in machine learning infrastructure and the least discussed, because every component involved reports itself healthy. Jayashree Mohan and colleagues measured it directly across a range of hardware configurations and found DNN training spent between 10% and 70% of epoch time blocked on I/O despite prefetching and pipelining; on one spinning-disk configuration, ResNet-50 on ImageNet was stalled for 75% of every epoch (Mohan et al., 2021, Analyzing and Mitigating Data Stalls in DNN Training, PVLDB 14(5), arXiv:2007.06775). Google's own fleet analysis put 20% of training jobs above a third of total compute time spent ingesting data (Murray et al., 2021, tf.data: A Machine Learning Data Processing Framework, PVLDB 14(12), arXiv:2101.12127).

The reason the graphs look healthy is that the constraint is not the one the graphs measure. A training data path is bound by request rate and latency, not bandwidth, and almost every instinct carried over from analytics engineering points the wrong way.

Why this matters: Accelerator hours are the dominant line item in a training budget, and the data path decides how many of them are useful. Getting the storage layout right is usually a larger and cheaper win than any kernel optimisation, and getting it wrong is invisible in every dashboard that measures bytes.

TL;DR

• The binding constraint on a training data path is requests per second, not bytes per second. Feeding eight accelerators at 4,000 samples/s from one-object-per-sample storage means 4,000 GET/s, against an S3 ceiling of 5,500 GET/s per partitioned prefix. The same workload moves only 450 MB/s.
• Latency sets the required concurrency. By Little's law, 4,000 samples/s at 20 ms first-byte latency needs 80 requests in flight permanently. Eight synchronous dataloader workers deliver 400 samples/s, and the storage system never notices.
• Sharding samples into 100 MB to 1 GB sequential archives cuts request rate by roughly three orders of magnitude and converts random reads into streams. NVIDIA measured 3x to 10x throughput gains from the switch.
• The price is shuffle quality. A read-time shuffle buffer mixes only about buffer / shard_size shards at once, so shard composition at write time, not buffer size at read time, determines whether batches are actually random.
• A lakehouse table is the wrong shape for training reads. Predicate pushdown, column projection and clustered statistics all optimise reading less; training reads everything, every epoch.
• Storage has become a benchmarked, first-class part of the training stack. MLPerf Storage v2.0 (August 2025) validates a result only if simulated accelerators stay above 90% utilisation on Unet3D and 85% on RetinaNet, and added checkpointing as a measured workload.
• Failures make the data path stateful. Meta recorded 419 unexpected interruptions in a 54-day Llama 3 pre-training window on 16,384 H100s. A loader that cannot resume mid-epoch deterministically turns each of those into lost work.

At a Glance

flowchart LR
  A[Object store<br/>shards or samples] --> B[Fetch<br/>N requests in flight]
  B --> C["Local NVMe /<br/>page cache"]
  C --> D[Decode and augment<br/>host CPU or GPU]
  D --> E[Shuffle buffer<br/>k samples]
  E --> F[Collate to batch<br/>pinned memory]
  F --> G[Accelerator HBM]
  G -. step barrier .-> B

  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
  class A,B blue
  class C,D,E purple
  class F slate
  class G teal

Six stages, and any one of them can be the bottleneck. The diagnostic discipline of this whole article is that you must know which one before you change anything, because the fixes for stage two and stage four are opposites.

Before the Data Path Had a Name

For most of the deep learning era the input pipeline was an afterthought, because it could be. AlexNet-scale datasets fit on a local disk, and a single GPU consumed samples slowly enough that a Python loop over image files kept up. The pipeline became a system only when two things happened at once: datasets outgrew any single machine, and accelerators got fast enough that a millisecond of stall per sample mattered.

timeline
    title How the training data path became its own system
    2019 : NVIDIA publishes AIStore and WebDataset, arguing the format itself is the bottleneck
         : Plain tar shards make sequential reads the default access pattern
    2020 : S3 delivers strong read-after-write consistency on 1 December, removing a class of workaround
         : Mohan et al. release DS-Analyzer and quantify data stalls at 10 to 70 percent of epoch time
    2021 : tf.data paper lands at VLDB with fleet evidence that input pipelines consume real compute
         : CoorDL shows coordinated caching beating DALI by up to 5x on a single server
    2023 : MosaicML StreamingDataset makes elastically deterministic mid-epoch resumption a shipped feature
         : AWS launches S3 Express One Zone on 28 November with single-digit millisecond access
    2025 : DeepSeek open-sources 3FS, reporting 6.6 TiB/s aggregate read across 180 storage nodes
         : MLPerf Storage v2.0 results published in August, with checkpointing added as a measured workload

Two things in that sequence are worth dwelling on. The 2019 WebDataset argument was not about a faster filesystem; it was that the format determines the access pattern, and that a format nobody would choose for analytics (uncompressed tar, no index, no random access) is the right one for training (Aizman, Maltby & Breuel, 2019, High Performance I/O For Large Scale Deep Learning, IEEE Big Data, arXiv:2001.01858). And the 2025 MLPerf Storage round marks the point at which "can your storage keep GPUs busy" became a number vendors compete on rather than an anecdote.

[IMAGE: Stacked area chart of a single training step over time, 0 to 400 ms, with bands for fetch wait, decode, augment, collate, H2D copy and compute. Two panels: "one object per sample" where fetch wait occupies 78% of the step, and "sharded sequential" where fetch wait shrinks to 6% and decode becomes the widest band. Caption: "Fixing the layout does not remove the bottleneck, it moves it to preprocessing."]

How the Data Path Actually Works

The four stages, and the one that usually fails

Every training data path does the same four things: get bytes off storage, turn bytes into tensors, randomise the order, and land the result in accelerator memory. They pipeline, so the slowest stage sets throughput and the others show up as idle capacity.

Fetch is the stage that fails most often on cloud storage, and it fails in a way that looks like nothing. Decode failures peg host CPUs. Collation failures show up as memory pressure. Fetch failures show up as an idle accelerator, an idle CPU, an idle NIC, and a storage system reporting a comfortable load. Everything is waiting, so nothing looks busy.

Request rate, not bandwidth, is the ceiling

Object storage is a key-value store reached over HTTP, and its scaling unit is the request. Amazon S3 sustains at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on the number of prefixes in a bucket (AWS S3 performance guidelines). Exceeding it produces HTTP 503 SlowDown while the service repartitions, which is gradual rather than instantaneous.

Now put a dataset of individual JPEGs behind that. A node with eight accelerators consuming 4,000 samples per second issues 4,000 GET requests per second. One node nearly saturates one prefix. Four nodes cannot run on that prefix at all. The bytes involved, at a typical 110 KB per image, total about 450 MB/s, which is 3.6 Gb/s on a 100 Gb/s link. The pipeline is request-starved while being nowhere near bandwidth-starved, and every capacity dashboard in the stack reports green.

[IMAGE: Dual-axis line chart over node count 1 to 8. Left axis, blue line: aggregate GET requests per second, rising linearly and crossing a dashed red ceiling at 5,500 after the first node. Right axis, teal line: aggregate MB/s, rising to 3.6 GB/s against a dashed 12.5 GB/s NIC ceiling it never approaches. Caption: "Two ceilings, one workload: the request ceiling binds at one node while the bandwidth ceiling stays 28x away."]

Latency sets the required concurrency

The second half of the fetch problem is Little's law. To sustain a throughput \(\lambda\) against a per-request latency \(W\), the number of requests in flight must be at least

First-byte latency to S3 Standard is tens of milliseconds and does not improve with effort, so at \(\lambda = 4000\) samples per second and \(W = 0.020\) seconds you need \(L = 80\) outstanding requests, continuously, forever. A PyTorch DataLoader with num_workers=8 doing synchronous reads has \(L = 8\), which by the same equation caps it at

Ten times too slow, from a parameter most people treat as a CPU-count heuristic. The correct mental model is that num_workers on cloud storage is not a parallelism knob for decode, it is a queue-depth knob for I/O, and those two want very different values.

This is also why S3 Express One Zone exists. Announced generally available on 28 November 2023, it targets consistent single-digit millisecond access from directory buckets supporting up to two million reads per second (AWS S3 Express One Zone). Cutting \(W\) from 20 ms to 5 ms cuts the concurrency requirement fourfold for the same throughput. It changes the constant in Little's law; it does not change the law.

Sharding is a bargain, not a free win

The standard fix packs many samples into one object. A shard is a contiguous archive of roughly 100 MB to 1 GB holding a thousand or more samples, read start to finish. WebDataset uses plain POSIX tar, deliberately: tar is sequentially readable, needs no index, and every tool already parses it. TFRecord, Mosaic Streaming's MDS and Ray Data's formats make the same structural choice with different framing.

The arithmetic changes by three orders of magnitude. One thousand samples per shard turns 4,000 GET/s into 4 shard opens per second, each streaming at whatever the connection sustains rather than paying a round trip per sample. NVIDIA's measurements put sequential shard reading at 3x to 10x the throughput of random per-sample access.

What you give up is granularity in both directions. Samples inside a shard are immutable, so deleting one record for a takedown request means rewriting a shard or maintaining a deny-list the loader consults on every sample. And the read order inside a shard is the write order, which brings us to the part everyone underestimates.

Shuffling becomes an approximation

A uniformly random permutation over a billion samples requires random access to a billion samples, which is exactly what sharding abolished. What sharded loaders do instead is a two-level approximation:

1. Permute the shard list each epoch and deal shards out to workers.
2. Maintain a buffer of \(k\) samples; emit a uniformly random element from the buffer and refill from the sequential stream.

The sample order this produces is not uniform. Two samples from the same shard are far more likely to co-occur in a batch than two samples chosen independently. A useful approximation of how much mixing you get is the number of shards simultaneously represented in the buffer:

With \(k = 10{,}000\) and 1,000 samples per shard, roughly ten shards are in play at any moment. If those shards were written from data grouped by class, crawl date or source domain, ten shards is not enough mixing to rescue batch composition, and the model sees systematic structure the loss curve will not obviously reveal.

The rule that follows is unglamorous and load-bearing: shuffle thoroughly once, at write time, and treat read-time shuffling as a top-up. Writing shards from a globally shuffled sample order costs one expensive pass over the dataset, once, and every epoch of every future run benefits.

[IMAGE: Three 64x64 heatmaps of batch composition, rows are batches and columns are source-class identity, for (a) a global permutation, uniform noise; (b) shards written from shuffled data with a 10,000-sample read buffer, nearly uniform with faint banding; © shards written in crawl order with the same buffer, strong diagonal blocks. Caption: "The read-time buffer is identical in (b) and ©. Only the write-time order differs."]

Getting bytes into accelerator memory

The last hop is usually invisible and occasionally dominant. A conventional read lands in host memory, is copied into a pinned staging buffer, and is DMA'd across PCIe. NVIDIA GPUDirect Storage removes the CPU bounce buffer by having a DMA engine near the NIC or drive write directly into GPU memory, which NVIDIA reports as typically a 3x latency reduction (most visible on small transfers) and, on DGX-2 class hardware, the difference between roughly 50 GB/s of CPU-memory-mediated bandwidth and close to 200 GB/s aggregated from local drives and NICs (NVIDIA GPUDirect Storage overview).

For image and video pipelines the bigger win is usually moving decode itself onto the accelerator. Once fetch is fixed, JPEG decode and augmentation become the widest band in the step, and a host with eight accelerators and too few cores starves on any storage layout.

Seeing It in Motion

The request-level view makes the concurrency argument concrete. Here is one dataloader worker's steady state under each layout.

sequenceDiagram
    participant W as Loader worker
    participant S as Object store
    participant C as Local cache
    participant G as Accelerator
    Note over W,S: Layout A: one object per sample
    W->>S: GET sample_00417.jpg
    S-->>W: 110 KB after 20 ms
    W->>W: decode plus augment, 4 ms
    W->>G: one sample
    Note over W,G: 24 ms per sample per worker, 8 workers, 333 samples/s<br/>worse than the 400 that fetch latency alone implies
    Note over W,S: Layout B: 1000-sample shard
    W->>S: GET shard_0142.tar
    S-->>C: stream 110 MB, one round trip
    loop 1000 samples
        C->>W: next record, no network
        W->>G: one sample
    end
    Note over W,G: 20 ms amortised over 1000 samples, decode now dominates

The same contrast, drawn as the two pipelines rather than one worker's timeline:

flowchart TB
  subgraph A["Layout A: object per sample"]
    A1["4000 GET/s"] --> A2[80 in-flight needed]
    A2 --> A3[8 workers supply 8]
    A3 --> A4["400 samples/s"]
    A4 --> A5[accelerator at 10 percent]
  end
  subgraph B["Layout B: sharded sequential"]
    B1["4 shard opens/s"] --> B2[3 streams in flight]
    B2 --> B3[8 workers ample]
    B3 --> B4["4000 samples/s"]
    B4 --> B5[decode becomes the limit]
  end

  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  class A1,A2,A3,A4,A5 rose
  class B1,B2,B3,B4 emerald
  class B5 amber

And the lifecycle of a single shard, which is where caching and resumption logic actually live:

stateDiagram-v2
  [*] --> Assigned: epoch permutation deals shard to worker
  Assigned --> Fetching: prefetch depth allows
  Fetching --> Cached: bytes land on local NVMe
  Cached --> Streaming: records emitted into shuffle buffer
  Streaming --> Exhausted: last record emitted
  Exhausted --> Evicted: cache pressure
  Exhausted --> Cached: retained for next epoch
  Streaming --> Assigned: preemption, resume from record offset
  Evicted --> [*]

[IMAGE: Two side-by-side scatter plots of achieved samples/s (y) against requests in flight (x), for 5 ms, 20 ms and 80 ms first-byte latency. Straight lines through the origin with slope 1/W, and a horizontal ceiling where the prefix request limit bites. Caption: "Little's law drawn twice: concurrency buys throughput linearly until the per-prefix request ceiling caps it."]

Watch It Run

By the Numbers

Sources: request rates, pagination and pricing from AWS documentation as of 2026 and subject to change; data-stall figures from Mohan et al., 2021; fleet figures from Murray et al., 2021; sequential-read speedups from Aizman et al., 2019; MLPerf Storage v2.0 thresholds from the MLCommons submission rules; 3FS and GPUDirect Storage figures are vendor-reported and have not been independently reproduced here; Llama 3 interruption counts from Grattafiori et al., 2024, arXiv:2407.21783. The S3 Express One Zone launch announcement quoted request costs up to 50% below S3 Standard while the storage-class page now quotes up to 80%; treat vendor cost multiples as current-pricing claims, not constants.

[IMAGE: Horizontal bar chart of cost per 10-day training run, broken into storage-at-rest, GET request fees and egress, for three layouts: one object per sample, 1,000-sample shards, and 10,000-sample shards. The request-fee bar dominates the first and nearly vanishes in the others. Caption: "Request fees are the line item that scales with file count rather than data volume."]

A Concrete Example

Take a concrete job and work the numbers all the way through. The setup: one node, eight H100s, a vision model at global batch 1,024 and a 250 ms step, training on 1.28 million images averaging 110 KB, stored in an S3 bucket under a single prefix.

Step 1: what the loop demands. A 1,024-image batch every 250 ms is

Step 2: what that is in bytes. \(4096 \times 110\,\text{KB} \approx 450\ \text{MB/s}\), which is 3.6 Gb/s. The NIC is 100 Gb/s. Bandwidth headroom is 28x. Cross bandwidth off the list.

Step 3: what it is in requests. One object per sample means 4,096 GET/s against a prefix ceiling of 5,500 GET/s. That is 74% of a single prefix consumed by one node. Add a second node and the job starts collecting 503 SlowDown responses.

Step 4: what concurrency it needs. At 20 ms first-byte latency, \(L = 4096 \times 0.020 = 82\) requests in flight. With num_workers=8 doing synchronous reads, \(L = 8\), so achieved throughput is \(8 / 0.020 = 400\) samples/s. The step now takes \(1024 / 400 = 2.56\) seconds instead of 0.25. Accelerator utilisation: 9.8%.

Step 5: what that costs. Ten days at 400 samples/s is 345.6 million GETs, about $138 in request fees, and it buys you 10 days of 90% idle H100s. Run it properly at 4,096 samples/s and you would issue 3.54 billion GETs in the same ten days, roughly $1,416 in request charges, for bytes that cost a small fraction of that to store.

Step 6: reshape into shards. Pack 1,000 images per tar shard, about 110 MB each, giving 1,280 shards for the dataset. Now the loop needs \(4096 / 1000 = 4.1\) shard opens per second. At 200 MB/s per connection a shard takes 0.55 s to stream, so required concurrency is \(4.1 \times 0.55 = 2.3\) streams. Eight workers is comfortable. Request fees for the ten-day run drop from roughly $1,416 to roughly $1.42.

Step 7: check the new bottleneck. With fetch no longer binding, the step needs 4,096 JPEG decodes per second. At roughly 3 ms of CPU per decode-and-augment that is 12.3 core-seconds per second, so the node needs at least 13 dedicated cores for decode alone and realistically 24 to hold the pipeline steady. If the instance has 16 vCPUs available to the loader, you are now decode-bound at about 5,300 samples/s and only just clearing the requirement. That is the moment to move decode onto the accelerator rather than to buy faster storage.

Step 8: check the shuffle you just bought. With 1,000 samples per shard, a 10,000-sample buffer mixes about 10 shards. If the 1,280 shards were written in crawl order, every batch draws from a 10-shard window of highly correlated data. The fix costs one shuffled write pass over 140 GB, and it is not optional.

Steps 4 and 8 are the two that get skipped, and they are the two that decide whether the run is fast and whether it is correct.

[IMAGE: Waterfall chart tracking achieved samples/s through the eight steps of the worked example: demand 4,096, capped to 400 by concurrency, restored to 4,096 by sharding, then capped again at 5,300 by decode with 16 vCPUs. Annotate each bar with the binding constraint. Caption: "The bottleneck does not disappear, it relocates, and the worked example is a record of where it went."]

Where It Breaks

The last epoch lies to you

Once the working set fits in page cache or local NVMe, fetch stops touching the network and throughput jumps. A benchmark that measures epoch five on a dataset that fits in cache is measuring a different system from the one that will run on the full dataset. This cuts both ways: teams over-provision storage after benchmarking epoch one, and under-provision after benchmarking epoch five. Always state which epoch a throughput number came from, and whether the dataset exceeds aggregate cache.

Shard count and worker count interact badly

With 900 shards and 128 workers, four workers get an eighth shard while the rest get seven. At the epoch barrier everybody waits for the stragglers, so the epoch costs eight shards of time with seven shards of average work. The remainder is a pure tax that grows as shards get larger. Choosing shard count means choosing against a maximum cluster size you may not know at write time, which argues for more, smaller shards inside the 100 MB to 1 GB band.

Resumption is a correctness problem, not a convenience

Meta's Llama 3 405B pre-training recorded 466 job interruptions in a 54-day window, 419 of them unexpected, on a 16,384-GPU cluster, with roughly 78% attributable to confirmed or suspected hardware faults (Grattafiori et al., 2024, The Llama 3 Herd of Models, arXiv:2407.21783). A loader that can only resume at epoch boundaries throws away up to an epoch of work per interruption. Worse, a loader that resumes by re-seeding without tracking position silently re-shows samples already seen in that epoch, which is a data-ordering bug that no test catches and no metric reveals.

Mosaic Streaming addresses this by making sample order a function of a fixed number of canonical buckets, num_canonical_nodes, rather than of the physical worker count, so a run can stop on 24 GPUs, resume on 16 and finish on 48 with the same loss curve (Mosaic Streaming, elastic determinism). Changing that value changes the global order, so it is a config value to pin, not to tune.

Checkpoint writes collide with sample reads

The same storage system serves both, and their profiles are opposites. Sample reads are many small-to-medium sequential streams; a checkpoint write is a sudden burst of very large objects from every rank at once. On a shared prefix the checkpoint burst can push the bucket into 503 territory and stall the reads that were fine a second earlier. MLPerf Storage v2.0 added checkpointing as a measured workload in the August 2025 round precisely because this interaction had become the operational problem at scale. Separate the prefixes, and think of checkpoint frequency as a storage-contention parameter as well as a recovery parameter.

Small files punish the metadata layer, not just the data layer

If the dataset is being served through a table format rather than raw objects, file count hits planning before it hits reading. The engine reads a manifest entry and a Parquet footer per file, each a round trip on object storage, before it can decide anything. Millions of tiny files turn dataset enumeration into a multi-minute preamble on every job start, which is invisible in per-epoch throughput and very visible in time-to-first-batch on a fleet of short jobs.

Determinism and performance pull against each other

Asynchronous prefetch with a completion queue is fast and returns samples in completion order, which varies run to run. A deterministic global order means either buffering to restore order (memory) or forcing in-order completion (latency). Most frameworks default to the fast, non-deterministic option, and most teams discover this while trying to reproduce a run.

The data path is where privacy obligations get hard

Deleting a user's records from a lakehouse table is a DELETE statement. Deleting them from 4,000 immutable tar shards replicated across local NVMe on 512 nodes is a project. The deny-list workaround keeps the sample out of future batches without removing the bytes, which satisfies a training-exclusion requirement but not a deletion requirement. If the dataset carries deletion obligations, the shard rewrite schedule is a compliance control and should be designed as one.

Alternative Designs

No row here is a default. The honest selection rule is: measure which stage stalls, compute the request rate and concurrency your throughput target implies, and pick the cheapest row that clears both.

[IMAGE: Decision flowchart. Root: "accelerator under 85% utilisation?" Branches through "run with working set fully cached, did throughput move?" to separate storage-bound from compute-bound, then storage-bound splits on "requests/s above 2000 per node?" into sharding versus latency reduction, and compute-bound splits into accelerator-side decode versus more host cores. Caption: "The differential test comes first: cache the working set and see whether anything changes."]

How It Is Used in Practice

Frontier labs converged on roughly the same shape. Datasets are written once as shuffled shards in a self-describing container, with a manifest listing shard names, sample counts and byte offsets so the loader can plan without listing the bucket. Shards live in object storage as the durable origin and are cached on node-local NVMe. Sample order is derived from a seed and a step counter so that resumption is exact. Preprocessing is split: anything deterministic and expensive (tokenisation, resizing to a canonical resolution) is done once at shard-write time, while anything random (crops, masking, augmentation) stays at read time because baking it in would freeze the randomness.

The storage tier underneath varies by whether the compute is owned or rented. DeepSeek built 3FS, an RDMA-first parallel filesystem over NVMe, reporting 6.6 TiB/s aggregate read throughput on a 180-node cluster with two 200 Gb/s InfiniBand NICs per node. The design deliberately optimises random reads and largely forgoes read caching, which is coherent when the access pattern is a fresh permutation every epoch. That is a vendor-reported figure on their own hardware, and the design rationale is the interesting part, not the headline number.

On rented compute the equivalent is layered: object store as origin, an aggressive local NVMe cache, and shard placement spread across prefixes to multiply the request ceiling. Teams running many short jobs over a shared dataset get more from a cooperative cache than from faster storage, which is the CoorDL result restated: coordinating what gets cached across concurrent jobs beat improving any single job's pipeline, by up to 5x on one server.

Two practices separate teams that have been burned from teams that have not. They instrument the stage, not the job, emitting per-stage wait times so a stall names its own cause. And they keep the shard-writing pipeline as production code with tests, because a subtly mis-shuffled shard set is a defect that outlives several model generations and never announces itself.

[IMAGE: Architecture diagram of a production training data path: object-store origin on the left holding shards plus a manifest, a per-node NVMe cache tier, eight training processes with prefetch queues, and a separate checkpoint path writing to a different prefix. Annotate the three places prefixes are split and the one place backpressure is applied. Caption: "The checkpoint path and the sample path share a bucket and should not share a prefix."]

Insights Worth Remembering

1. The training data path is bound by requests and latency, not bandwidth. A pipeline can be catastrophically starved while the NIC sits at 4% and the storage service reports a light load. Any diagnosis that starts with throughput graphs will miss it.

2. num_workers is a queue-depth parameter, not a CPU-count parameter. On cloud storage its job is to satisfy \(L = \lambda W\). Sizing it to core count is a habit inherited from local-disk training, and it silently caps throughput an order of magnitude below the hardware.

3. The format determines the access pattern, and the access pattern determines everything else. This was the WebDataset argument in 2019 and it has survived every subsequent hardware generation. Choosing tar over a sophisticated indexed format was the point, not a compromise.

4. Shuffle quality is set at write time. A read-time buffer mixes roughly buffer / shard_size shards. No buffer size rescues shards written in crawl order, and the resulting correlation does not show up as a clear signal in the loss curve.

5. Fixing fetch moves the bottleneck to decode, and that is success. The step should end up CPU-bound or accelerator-bound. Once fetch is not the constraint, the remaining optimisations are the ones worth doing.

6. Resumption granularity is a throughput feature. At 419 unexpected interruptions in 54 days, the gap between epoch-granular and step-granular resumption is measured in accelerator-weeks, and the incorrect middle option (re-seeding without tracking position) is a silent data bug.

7. Storage is now benchmarked as part of the training stack. MLPerf Storage validating results only above 90% accelerator utilisation on Unet3D formalises what was folklore, and the v2.0 addition of checkpointing acknowledges that reads and writes contend for one system.

8. Governance and training layout pull in opposite directions. A lakehouse table gives you deletion, lineage and time travel; a shard set gives you throughput. Most organisations end up with both and a pipeline between them, and the pipeline is where deletion obligations get lost.

Open Questions

Can a table format serve training reads directly without giving up throughput? Iceberg and Delta now have the pieces (manifests that avoid listing, deletion vectors for cheap row-level deletes, sorted compaction), and several vendors are pushing the idea. What is measured is that per-row random access over Parquet remains far slower than sequential shard reads. What is not established is whether a training-aware scan (large contiguous row-group reads plus a shuffle buffer over row-group order) closes enough of the gap to be worth the governance benefit at pre-training scale.

What is the actual cost of approximate shuffling? The mechanism by which shard-level shuffling could hurt (correlated batches biasing gradient estimates) is clear, but published controlled measurements of the effect on final model quality at scale are thin. Confidence in current practice rests on the empirical absence of disaster rather than a measured bound.

Does a single copy of the data serve both analytics and training? Most organisations maintain a governed table and a derived shard set, paying for two copies and an ETL between them. Whether that is a permanent split or a transitional state depends largely on the previous two questions.

How should checkpoint and sample traffic be scheduled against each other? Both are known to contend, and MLPerf Storage v2.0 measures them separately. Whether the answer is physical separation (different prefixes, tiers or systems), temporal scheduling into a known pipeline slack window, or admission control at the storage layer has no consensus answer.

Do disaggregated, RDMA-first stores make the local cache tier unnecessary? 3FS is built on the premise that random reads can be served fast enough that caching is not worth the complexity, and its reported numbers are consistent with that. Whether the premise holds on rented infrastructure, where the network between compute and storage is not yours to design, is unresolved.

Sources and Further Reading

1. Mohan, J., Phanishayee, A., Raniwala, A., & Chidambaram, V. (2021). "Analyzing and Mitigating Data Stalls in DNN Training." Proceedings of the VLDB Endowment, 14(5), 771-784. arXiv:2007.06775
2. Murray, D. G., Šimša, J., Klimovic, A., & Indyk, I. (2021). "tf.data: A Machine Learning Data Processing Framework." Proceedings of the VLDB Endowment, 14(12), 2945-2958. arXiv:2101.12127
3. Aizman, A., Maltby, G., & Breuel, T. (2019). "High Performance I/O For Large Scale Deep Learning." 2019 IEEE International Conference on Big Data. arXiv:2001.01858
4. Grattafiori, A., et al. (2024). "The Llama 3 Herd of Models." arXiv:2407.21783
5. An, W., et al. (2024). "Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning." arXiv:2408.14158
6. Amazon Web Services. "Best practices design patterns: optimizing Amazon S3 performance." Amazon S3 User Guide. docs.aws.amazon.com
7. Amazon Web Services. "Amazon S3 now delivers strong read-after-write consistency automatically for all applications." 1 December 2020. aws.amazon.com
8. Amazon Web Services. "Amazon S3 Express One Zone storage class." aws.amazon.com
9. MLCommons. "New MLPerf Storage v2.0 Benchmark Results Demonstrate the Critical Role of Storage Performance in AI Training Systems." August 2025. mlcommons.org
10. MLCommons. "MLPerf Storage submission guidelines." github.com/mlcommons/storage
11. NVIDIA. "GPUDirect Storage Overview Guide." NVIDIA Magnum IO documentation. docs.nvidia.com
12. DeepSeek. "3FS: Fire-Flyer File System." github.com/deepseek-ai/3FS
13. WebDataset project. "webdataset: A high-performance Python-based I/O system for large deep learning problems." github.com/webdataset/webdataset
14. Databricks / MosaicML. "Elastic Determinism." Streaming documentation. docs.mosaicml.com
15. Apache Software Foundation. "Apache Parquet configurations." parquet.apache.org

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.