Model Architecture
18 min
Trainable Sparse Attention: When the Model Learns What to Skip
For years, sparse attention was a bolt-on applied after training to shrink the KV cache. NSA, MoBA, and DeepSeek's DSA flip that order, training the sparsity end to end and reporting up to 11x faster decoding without losing accuracy.