Views, Aliasing, and In-Place Operations

Nothing but metadata. A tensor is a pointer into a storage buffer, a shape, and a stride tuple; transpose returns a new tensor object over the same buffer with the strides permuted. A contiguous \((4, 8)\) tensor has strides \((8, 1)\); its transpose has shape \((8, 4)\) and strides \((1, 8)\). Slicing, permute, expand, and often reshape behave the same way.

view requires that the requested shape be expressible as a new stride tuple over the existing buffer, and raises when no valid one exists, which happens after a transpose or a strided slice. reshape attempts the same thing and silently falls back to allocating a contiguous copy. The failure is informative: it tells you the copy is happening, whereas reshape hides the cost.

The gradient of \(x^2\) is \(2x\), so it needs the original input, which the in-place square has overwritten. ReLU's backward needs only the sign of the input, and that is recoverable from the output, so overwriting is safe. Whether an in-place op is legal depends entirely on what its gradient formula reads, not on any general rule about mutation.

Every storage carries a version counter incremented by any in-place write. An op that saves a tensor for backward records the version it saw and compares at backward time, raising "one of the variables needed for gradient computation has been modified by an inplace operation" on mismatch. It is deliberately conservative: it flags the write even when the specific gradient formula would not have read the changed value, because a false positive is far cheaper than a silently wrong gradient.

.detach() returns a view over the same storage, cut from the autograd graph. It stops gradients, not aliasing: writing into it still mutates the original and still bumps the version counter. .clone() copies the data and stays in the graph, so gradients flow to the source. .detach().clone() gives a genuinely independent tensor. Reaching for .detach() when you meant "a separate copy" is a standard source of silent corruption.

Because parameter and gradient tensors are views into large shared buffers: DDP gradient buckets, FSDP flattened shards. An in-place edit of one parameter's .grad writes into a bucket that may be mid-reduction, so ranks end up with different values. The result is silent numerical divergence between processes rather than an exception, and it typically surfaces much later as a training instability with no obvious cause.

Nothing but metadata. A tensor is a pointer into a storage buffer, a shape, and a stride tuple; transpose returns a new tensor object over the same buffer with the strides permuted. A contiguous \((4, 8)\) tensor has strides \((8, 1)\); its transpose has shape \((8, 4)\) and strides \((1, 8)\). Slicing, permute, expand, and often reshape behave the same way.

view requires that the requested shape be expressible as a new stride tuple over the existing buffer, and raises when no valid one exists, which happens after a transpose or a strided slice. reshape attempts the same thing and silently falls back to allocating a contiguous copy. The failure is informative: it tells you the copy is happening, whereas reshape hides the cost.

The gradient of \(x^2\) is \(2x\), so it needs the original input, which the in-place square has overwritten. ReLU's backward needs only the sign of the input, and that is recoverable from the output, so overwriting is safe. Whether an in-place op is legal depends entirely on what its gradient formula reads, not on any general rule about mutation.

Every storage carries a version counter incremented by any in-place write. An op that saves a tensor for backward records the version it saw and compares at backward time, raising "one of the variables needed for gradient computation has been modified by an inplace operation" on mismatch. It is deliberately conservative: it flags the write even when the specific gradient formula would not have read the changed value, because a false positive is far cheaper than a silently wrong gradient.

.detach() returns a view over the same storage, cut from the autograd graph. It stops gradients, not aliasing: writing into it still mutates the original and still bumps the version counter. .clone() copies the data and stays in the graph, so gradients flow to the source. .detach().clone() gives a genuinely independent tensor. Reaching for .detach() when you meant "a separate copy" is a standard source of silent corruption.

Because parameter and gradient tensors are views into large shared buffers: DDP gradient buckets, FSDP flattened shards. An in-place edit of one parameter's .grad writes into a bucket that may be mid-reduction, so ranks end up with different values. The result is silent numerical divergence between processes rather than an exception, and it typically surfaces much later as a training instability with no obvious cause.