Interleaving RoPE and NoPE Layers
Llama 4 and Cohere's Command A both build long-context models by using rotary embeddings on most layers and no positional encoding at all on the rest, turning a choice everyone treated as global into a per-layer one.
Positional encoding has always been specified once and applied everywhere. Two 2025 frontier models broke that assumption in the same way: most layers get rotary embeddings with a bounded attention window, a minority get no positional signal and full attention over the entire context. Meta calls the arrangement iRoPE and credits it for Llama 4 Scout's 10-million-token window (Meta, 2025, The Llama 4 herd); Cohere ships a 3:1 ratio of sliding-window RoPE layers to full-attention NoPE layers in Command A (Cohere, 2025, Command A Technical Report).
Why removing the signal helps
A decoder-only transformer with no positional encoding is not position-blind. Causal masking already breaks permutation symmetry: token \(m\) attends to \(m\) predecessors and token \(m+1\) attends to \(m+1\), so attention-weight magnitudes carry a counting signal that the network can decode. Kazemnejad et al. prove NoPE can represent both absolute and relative schemes, and measure that it generalises to longer sequences better than APE, ALiBi, or RoPE on downstream reasoning tasks (Kazemnejad et al., 2023, NeurIPS, arXiv:2305.19466).
The advantage is structural. A NoPE layer has nothing to extrapolate, because there is no position-dependent term to push out of distribution. Whatever it learned at 8k applies unchanged at 800k.
Division of labour
The hybrid exploits the fact that the two layer types fail in opposite directions.
| Layer type | Attention span | Position signal | Good at | Bad at |
|---|---|---|---|---|
| RoPE + sliding window | fixed, e.g. 4,096 or 8,192 | explicit rotation | local syntax, sharp adjacency | anything past the window |
| NoPE + full attention | entire context | implicit, from causal mask | long-range retrieval | fine-grained ordering |
Because the windowed layers dominate by count, the KV cache stays bounded on most of the network. In Command A's 3:1 arrangement, three of every four layers cache at most 4,096 positions regardless of prompt length; only the fourth grows with context. That is a roughly 4x reduction in long-context KV memory before any compression technique is applied, and it is the reason these designs make very long windows economically plausible rather than merely representable.
Cohere's accompanying study sweeps the design space and reports that the hybrid beats full-attention RoPE baselines on both long and short context tasks while cutting training and inference cost (Yang et al., 2025, Rope to Nope and Back Again, arXiv:2501.18795).
When it breaks
Retrieval quality now concentrates in a quarter of the network. Every fact that must travel more than one window depends on the NoPE layers, so their heads are load-bearing in a way no single layer is in a uniform architecture. Pruning, quantising, or head-dropping without knowing which layers are which is a good way to destroy long-context behaviour while short-prompt benchmarks stay flat.
NoPE's positional signal is implicit, which makes it weaker for tasks requiring exact ordering. Reversing a list, reporting the fifth item, or comparing "before" and "after" leans on precision the counting signal does not reliably supply, and the RoPE layers cannot help once the span exceeds their window.
Meta pairs iRoPE with inference-time temperature scaling of attention to hold length generalisation together, which is an admission that the architecture alone does not reach 10M tokens. Treat headline context numbers here as capacity, not measured accuracy; the RULER-style gap between advertised and effective length applies to hybrids as much as to anything else (Hsieh et al., 2024, arXiv:2404.06654).
Serving is harder than the memory saving suggests. Two attention shapes in one model means two kernel configurations, two cache layouts, and chunked masks that most inference stacks implemented late and inconsistently. The efficiency is real; the integration cost is charged separately.
10 flashcards for this concept
Click a card to reveal the answer.