Multidimensional RoPE for Images and Video
A video is three coordinates, not one, and flattening it into a token index throws away the fact that two patches were vertically adjacent; M-RoPE and 2D RoPE split the head dimension into independent axes so a single mechanism can encode time, height, and width.
Flatten a 24 × 24 patch grid into a sequence and the patch directly above another one ends up 24 positions away, indistinguishable from a patch 24 columns to the right in the same row. One-dimensional RoPE will happily encode that distance, and it will be the wrong distance. For video the problem compounds: the same spatial patch in the next frame is 576 positions later, so a model asked to track an object across time has to learn a periodic structure that the position encoding actively obscures.
Splitting the head across axes
The fix follows directly from RoPE's construction. Rotations on disjoint coordinate pairs are independent, so a head's \(d/2\) pairs can be partitioned into groups, each group rotated by a different coordinate.
Qwen2-VL's M-RoPE partitions into three sections and rotates them by temporal, height, and width indices respectively (Wang et al., 2024, Qwen2-VL, arXiv:2409.12191). A text token gets identical values in all three, which makes M-RoPE numerically equal to ordinary 1D RoPE on text, so nothing about language modelling changes. An image token holds the temporal index constant and varies height and width with the patch's grid location. A video token varies all three.
The backwards-compatibility property is the reason the design won. A pretrained language model can be extended to vision without invalidating the positional behaviour its text weights already learned.
Vision transformers got there separately
RoPE-ViT applies the same idea to pure vision, comparing an axial variant that assigns half the pairs to \(x\) and half to \(y\) against a mixed variant that learns per-head frequencies over both axes (Heo et al., 2024, Rotary Position Embedding for Vision Transformer, ECCV 2024, arXiv:2403.13298). The interesting result is not the ImageNet delta, it is resolution extrapolation: a model trained at one resolution and evaluated at a higher one degrades far less with 2D RoPE than with an interpolated learned position table, for the same reason RoPE extrapolates better than absolute embeddings in text.
The position-index budget
Multidimensional schemes change how fast position indices are consumed, and that interacts with the base frequency. Under M-RoPE, a video segment advances the temporal index once per frame while the spatial indices cycle within a frame, so a long video consumes far fewer distinct positions than the token count suggests. Qwen2.5-VL pushes this further by aligning the temporal index to absolute time rather than frame count, so a 2 fps clip and a 30 fps clip of the same scene produce comparable temporal positions (Bai et al., 2025, Qwen2.5-VL Technical Report, arXiv:2502.13923).
When it breaks
The partition is a fixed budget. Giving the temporal axis a third of the pairs means the spatial axes get a third each, and the highest-frequency pairs available to each axis are correspondingly fewer, which is exactly the resource Barbero et al. identify as necessary for sharp local attention (Barbero et al., 2024, arXiv:2410.06205). Fine-grained OCR on a dense document and long-video tracking compete for the same 64 pairs.
Cross-modal distance becomes ill-defined. If an image occupies temporal index 7 and the next text token continues from 8, the model is implicitly told that "one patch to the right" and "one word later" are the same unit of displacement. They are not, and there is no principled conversion; the choice of how much to advance the index after an image is a tuning decision that shows up in interleaved image-text tasks.
Implementation bugs in this area are quiet. Position IDs for multimodal inputs are computed outside the attention kernel, usually in preprocessing, and a wrong stride produces a model that still generates fluent text while its spatial grounding is subtly scrambled. The symptom is a vision model that describes the right objects and gets their relative positions wrong.
10 flashcards for this concept
Click a card to reveal the answer.