Tensor Memory Layout and Contiguity

Two tensors with identical shapes can have completely different performance, and the reason is invisible in the shape. Memory is one-dimensional; a tensor is a flat buffer plus metadata saying how to index it. That metadata is the stride tuple: how many elements to step in the buffer to advance one position along each dimension. A row-major tensor of shape (4, 3) has strides (3, 1) — moving one row skips three elements, moving one column skips one.

Transposing that tensor gives shape (3, 4) and strides (1, 3). No data moved. The transpose is a metadata edit, which is why it appears free in a profile and why the cost shows up somewhere else entirely.

Contiguity, and what breaks without it

A tensor is contiguous when its elements sit in memory in exactly the order a row-major walk of its shape would visit them. The transposed tensor above is not: walking its rows jumps by 3 elements each step. Nothing is wrong with it, and most elementwise operations handle it fine, but two things follow.

First, reshape needs contiguity or a copy. view in PyTorch demands a compatible layout and raises otherwise; reshape silently calls contiguous() when it must, which allocates a fresh buffer and copies every element. That is a full memory-bandwidth pass, and on a large activation tensor it is not cheap. A reshape that fails to be free is one of the more common invisible costs in a training step.

Second, kernels care deeply. A matmul kernel is written to load tiles into shared memory with coalesced reads, where consecutive threads read consecutive addresses (see memory coalescing). A non-contiguous input either forces the kernel to materialise a contiguous copy first, or forces a strided access pattern that wastes most of each memory transaction. Two mathematically identical implementations can differ by a factor of several purely on layout.

The multi-head reshape everyone writes

The canonical example lives in attention. A projection produces shape (B, S, H·D), and attention needs (B, H, S, D), so every implementation writes something like x.view(B, S, H, D).transpose(1, 2). The view is free; the transpose is free; the result is non-contiguous, and the next operation that needs a contiguous buffer pays for the copy. Where that copy lands, how many times it happens per layer, and whether the attention kernel can consume the strided form directly are exactly the details that separate a fast implementation from a slow one with the same maths.

When it breaks

• reshape hides an allocation. It is the convenient function and it is the one that can quietly copy your largest tensor. Where performance matters, use view and let it raise, so you learn that a copy was needed rather than paying for it silently.
• Alignment matters for tensor cores. Tensor core paths generally require the contracted dimensions to be multiples of 8 or 16 depending on dtype; a hidden size of 4095 can fall off the fast path entirely while 4096 stays on it. Model dimensions are chosen with this in mind, not for aesthetics.
• Layout choices propagate. A permute inserted early to make one operation convenient forces copies at every downstream operation that wants the original order. Layout is a property of a pipeline, not of a line.
• Some libraries silently disagree. Passing a non-contiguous tensor into a custom or third-party kernel may produce a copy, an error, or wrong results, depending on whether it checks. Wrong results are the dangerous case and the reason many kernels call contiguous() defensively at their boundary.
• Reading strides is the debugging tool. When two implementations of the same maths differ in speed, print shapes and strides at each step. The extra bandwidth pass is almost always visible there and almost never visible in the shapes alone.