The fourth layer remembers

Search Moonshot AI's Kimi Linear implementation for a rotary embedding and you will not find one. The word appears once in the whole modeling file, inside a docstring explaining that the slice of the keys where rotation would normally be applied is left unrotated. There is no rotary module, no inverse-frequency table, nothing that tells any layer where in the sequence a token sits. This is a model built for contexts measured in the hundreds of thousands of tokens, and it has no positional encoding at all.

That is the second-strangest thing about it. The strangest is a number.

Kimi Linear was contributed to Hugging Face Transformers on 4 September and shipped in the v5.17.0 release. Read its configuration and you find that when a checkpoint does not specify the layout explicitly, the layer types are generated by a rule: every fourth layer is a full-attention block, and every other layer is Kimi Delta Attention, a linear-attention variant. Moonshot's own repository describes the released models as having a "3:1 KDA-to-global MLA ratio". Three cheap layers, then one expensive one.

Now open Qwen3-Next, contributed a year earlier. Forty-eight layers, and a configuration that pops a parameter called full_attention_interval with a default of 4: linear attention everywhere except every fourth layer. Then open GLM-5.3-Flash, contributed on 26 August, whose documentation says that for the first time in the GLM series the architecture combines sparse and linear attention. Forty-five layers, and the rule that assigns their types marks a layer linear unless its index modulo four is three.

Three labs, three different linear-attention formulations, and one number they agree on. The ratio is not a coincidence and it is not, I think, a compromise. The 3:1 hybrid is the field's standing price for exact recall: three layers may forget, and one may not.

What each kind of layer keeps

The distinction worth being precise about is not speed. It is what the layer stores between tokens.

A full-attention layer keeps a KV cache. For every token it has seen, it holds that token's key and value vectors, exactly as computed, and when it processes the next token it reads all of them. Nothing is compressed and nothing is approximated, which is why the memory grows in proportion to the sequence and the work of attending grows with it. The cache is a log. It appends.

A linear-attention layer keeps a matrix. One per head, of fixed shape: key dimension by value dimension. In the Kimi implementation the update for token \(t\) reads, in effect,

and the middle term is the part to slow down on. Before the layer writes anything, it asks what it would currently return for this key: that is \(S_{t-1}^{\top}k_t\). It subtracts that from the value it actually wants stored, and writes back only the difference, scaled by a learned rate \(\beta_t\). This is the delta rule, and it is error correction rather than accumulation. The layer is not keeping a record of what it has seen. It is maintaining an associative memory and repeatedly correcting it.

The \(g_t\) term is the forget gate, and it is where Kimi Delta Attention differs from the Gated DeltaNet it extends. In the earlier design the decay is one number per head per token: the whole state fades at one rate. KDA gives each key channel its own gate, so the state decays per channel. Some directions in key space can be held for a long time while others are flushed almost immediately, inside the same head, with the rates learned rather than set.

Read the recurrence once more and the trade becomes visible. \(S\) is the same size at token one and at token one million. Memory per sequence is constant, and so is the work per decoded token. What is also true, and unavoidable, is that a fixed matrix cannot hold an unbounded number of distinct key-value associations without them interfering with one another. The delta rule is a good way of managing that interference. It is not a way of abolishing it.

Which is what the fourth layer is for

Moonshot's repository reports up to a 75 per cent reduction in the KV cache required, up to six times faster time per output token against MLA at long context, and a score of 84.3 on RULER at 128k context, which it describes as Pareto-optimal against the speedup. These are the vendor's own figures on the vendor's own harness, and they should be read as evidence of what Moonshot wants believed. But notice the shape of the first one. Not a hundred per cent. Seventy-five. The cache does not go away, because one layer in four still keeps one, and that layer's cache is the entire remaining cost.

If the hybrid were simply linear attention with training wheels, the interesting engineering would all be in making the wheels smaller. The opposite is happening. GLM-5.3-Flash's configuration contains a line that rewrites any layer typed full_attention into deepseek_sparse_attention, unconditionally. In that model there is no full-attention layer at all: the one layer in four that keeps per-token state keeps it sparsely. The count of global layers did not move. Their kind did.

That is a refinement of the commitment rather than a retreat from it, and the convergence goes deeper than a ratio. The forget gate in Kimi Linear's transformers implementation carries the docstring "Same as Glm5NextTextForgetGate but with no gate_lower_bound and no A_log reshape". Two labs, two flagship models, one module, different only in a bound and a reshape.

The strongest objection is that I am reading a law into an act of caution. Pretraining runs of this size cost millions, and nobody bets one on an attention mechanism with a short track record; keeping a quarter of the layers familiar is cheap insurance. Kimi Linear's own abstract claims that KDA "for the first time, outperforms full attention under fair comparisons" across short-context, long-context and RL scaling regimes. If that is so, why keep any full-attention layers at all? On this reading 3:1 is risk management, it will drift to one layer in eight, and then to none.

It is a fair objection, and the evidence against it is the absence of drift. A hedge loosens as confidence accumulates. This number has not moved across a year and three independent groups, while a great deal else about these architectures has. And when a lab did spend engineering effort on the global layer, it spent it making that layer cheaper instead of deleting it. GLM's documentation gives the reason in the vendor's own framing: the hybrid sharply reduces long-context serving costs "while preserving precise long-context capabilities". Something has to do the preserving.

Here is what I cannot show you. I did not find, in what I read, an ablation that removes the global layers from one of these models and measures what happens to recall. The convergence is strong evidence that three groups arrived at the same design conclusion. It is not proof of the mechanism I am attributing to them, and a reader should hold my explanation of why one layer in four more loosely than the fact that there is one.

The rest of the evidence here is of two kinds, and they are worth separating. The layer counts, the generating rules, the delta-rule update and the absent rotation are read from implementation code and configuration in Hugging Face Transformers, which is a third party to all three labs and has to make its code agree with the released checkpoints. That is about as solid as a claim about architecture gets without training something. The performance numbers are of the other kind: every one of them is a lab reporting on itself, on a harness it chose, and I opened no independent evaluation of any of them. Take the 3:1 ratio as established and the six-fold speedup as a claim.

Back to the missing rotation

The positional encoding turns out to be the same story told from the other end. A recurrence is order-dependent by construction: \(S_t\) depends on the sequence in which the updates arrived, so a KDA layer knows about position without being told. With three-quarters of the stack recurrent, the remaining attention layers can be left with no positional signal, and Kimi's released checkpoints do exactly that by setting mla_use_nope=True. Position stopped being something added to the vector and became a property of the state.

For anyone learning this material, that is the belief most worth loosening. A transformer does not need positional encodings. It needs some source of order, and rotary embeddings were one answer, not the answer.

The practical takeaway is smaller and more useful: read layer_types. In a modern configuration it is the most informative line in the file, and it is not in any marketing copy. Two models advertising the same context window can differ by a large factor in memory per sequence depending on how many of their layers keep a cache, and that difference will not show up in an accuracy number at 4k, where Kimi reports parity with full attention at similar speed. It shows up in time per output token at long context, which is the measurement most people never run.

We spent a decade learning how to make networks remember. The interesting parameters now are the ones that govern what gets thrown away and how quickly, learned per channel, per head, per layer. And in every one of these designs there is one layer that is not permitted to throw away anything. Nobody has published an argument for why it should be one in four. It simply is, in three labs at once, which is usually what it looks like when a field has found something it does not yet know how to explain.