Contextual Position Encoding

Its only ruler measures tokens. "Three words back" is a fixed index offset; "three sentences back" is 24 tokens in one passage and 120 in another, so no fixed offset expresses it. Every standard scheme, absolute, relative, rotary, ALiBi, differs in how it encodes an index but agrees that position means token count, and content-defined units like sentences, clauses, or nesting depth fall outside what any of them can address.

Position is the running sum of sigmoid gates between query and key, \(p_{ij} = \sum_{k=j}^{i} \sigma(\mathbf{q}_i \cdot \mathbf{k}_k)\). A sigmoid rarely returns exactly 0 or 1, so the sum lands between integers. CoPE therefore interpolates between the learned embeddings at the two adjacent integer positions, which keeps the path differentiable and lets gradients flow back into the gate.

It produces a distinct position for every query-key pair. Standard RoPE applies one rotation per token, \(O(n)\) work outside the score matrix, so a fused kernel can compute scores in tiles without ever materialising them. CoPE's gate is defined over all pairs, giving an \(n \times n\) tensor of positions, which is exactly the object FlashAttention exists to avoid writing to HBM. That cost, not the idea, is why no frontier model has shipped it.

Selective copy, counting, and Flip-Flop. All three require addressing by a content-defined unit rather than by token offset: copy every element matching a marker, count occurrences, track the most recent write instruction. A token-index ruler cannot express "the i-th thing of type X" because the token distance to it varies with content. Code benefits for the same reason, since nesting depth and argument index are content-defined.

As a strong diagnosis and weak deployment evidence. The gains come from small-scale language modelling and from algorithmic tasks designed to expose exactly the weakness CoPE targets, not from a frontier pretraining budget. It also reintroduces a bounded embedding table, the problem RoPE was adopted to escape. The durable contribution is the argument that token counting is the limiting assumption; the specific mechanism remains unproven at scale.

Its only ruler measures tokens. "Three words back" is a fixed index offset; "three sentences back" is 24 tokens in one passage and 120 in another, so no fixed offset expresses it. Every standard scheme, absolute, relative, rotary, ALiBi, differs in how it encodes an index but agrees that position means token count, and content-defined units like sentences, clauses, or nesting depth fall outside what any of them can address.

Position is the running sum of sigmoid gates between query and key, \(p_{ij} = \sum_{k=j}^{i} \sigma(\mathbf{q}_i \cdot \mathbf{k}_k)\). A sigmoid rarely returns exactly 0 or 1, so the sum lands between integers. CoPE therefore interpolates between the learned embeddings at the two adjacent integer positions, which keeps the path differentiable and lets gradients flow back into the gate.

It produces a distinct position for every query-key pair. Standard RoPE applies one rotation per token, \(O(n)\) work outside the score matrix, so a fused kernel can compute scores in tiles without ever materialising them. CoPE's gate is defined over all pairs, giving an \(n \times n\) tensor of positions, which is exactly the object FlashAttention exists to avoid writing to HBM. That cost, not the idea, is why no frontier model has shipped it.

Selective copy, counting, and Flip-Flop. All three require addressing by a content-defined unit rather than by token offset: copy every element matching a marker, count occurrences, track the most recent write instruction. A token-index ruler cannot express "the i-th thing of type X" because the token distance to it varies with content. Code benefits for the same reason, since nesting depth and argument index are content-defined.

As a strong diagnosis and weak deployment evidence. The gains come from small-scale language modelling and from algorithmic tasks designed to expose exactly the weakness CoPE targets, not from a frontier pretraining budget. It also reintroduces a bounded embedding table, the problem RoPE was adopted to escape. The durable contribution is the argument that token counting is the limiting assumption; the specific mechanism remains unproven at scale.