Arrow and Zero-Copy Columnar Interchange

Before Arrow, moving a result set from a query engine to a dataframe library meant serialising it into some interchange encoding and parsing it back out, row by row, allocating a Python or JVM object per value. For a wide result of a few million rows, that conversion routinely cost more than the query that produced the data. Every pair of systems needed its own converter, so \(n\) systems needed \(O(n^2)\) of them.

Arrow's contribution is a specification for how columnar data is laid out in memory, precise enough that two processes can agree on the bytes and skip the conversion entirely.

A memory format, not a file format

The distinction people most often collapse is Arrow versus Parquet, and they solve opposite problems.

Parquet is optimised for bytes at rest: dictionary encoding, run-length encoding, bit-packing and block compression, all of which make the data smaller and make random access require decoding. Arrow is optimised for bytes being computed on: fixed-width values in contiguous buffers, a separate validity bitmap for nulls, offsets arrays for variable-length types, and no compression by default. An Arrow int64 column is literally an array of 8-byte integers that a SIMD loop can walk, and the \(i\)-th element is at a known offset with no decoding.

That is why the pairing is so common. Parquet on disk, Arrow in memory, and a reader whose job is to turn one into the other once.

Record batches and the IPC format

The unit of transfer is a record batch: a schema plus one contiguous set of buffers per column, covering some number of rows. Arrow's IPC format frames these batches with a small Flatbuffers metadata header describing buffer offsets and lengths, followed by the buffers themselves.

The payoff is asymmetric and worth stating precisely. The receiver can point its column objects directly at the received buffers, so reading a record batch out of a mapped file or a network buffer is genuinely zero-copy and effectively free. The sender is not free: producing a contiguous IPC payload usually means copying column buffers into one laid-out region. Measurements on Arrow Flight RPC put roughly 30% of RPC duration in serialising the record batch against roughly 0.0004% in deserialising it (Thallus: An RDMA-based Columnar Data Transport Protocol, arXiv:2412.02192). "Zero-copy" describes the read side.

Where the copies actually are

Three places, in practice.

Crossing a process boundary without shared memory: the bytes must traverse a socket, which is a copy on each side regardless of format. Arrow Flight and shared-memory plasma-style stores exist to cut this down, and RDMA transports cut it further.

Type coercion: a zero-copy read requires the consumer to accept Arrow's layout exactly. Converting an Arrow string column into Python str objects, or into a NumPy array of a different dtype, materialises new memory. Pandas with Arrow-backed dtypes avoids this; Pandas with NumPy-backed dtypes does not.

Compression: Arrow IPC supports optional LZ4 and Zstd buffer compression, which trades the zero-copy read for a smaller payload. That is often the right trade over a slow network and it is not the default for a reason.

When it breaks

Arrow is not a storage format and behaves badly as one. Uncompressed fixed-width layout means an Arrow IPC file of the same data is typically several times larger than the Parquet equivalent. For anything that lives on disk longer than a cache entry, that is the wrong trade.

Very wide schemas cost metadata. Every column contributes buffers and Flatbuffers metadata to every record batch. With tens of thousands of columns, the per-batch overhead becomes significant, and small batches over a wide schema are pathological.

Batch size is a real tuning parameter. Batches too small and per-batch overhead dominates; too large and memory spikes and pipelining stalls. Tens of thousands of rows is the usual operating range, and the correct value depends on row width, not row count.

Zero-copy is a property of the pair, not of Arrow. If either side converts on the boundary, the guarantee is gone. The common disappointment is an "Arrow-native" path that ends in a library which immediately materialises its own representation, at which point you have paid for the Arrow layout and received none of its benefit.