Tagged “long-context”

6 posts.

All Model Architecture13 Training & Alignment8 Inference & Serving11 Agents & Orchestration10 Reasoning & Evaluation7 Safety, Security & Governance3 Platforms & Practice11

Inference & Serving 21 min

Cache-Augmented Generation: When Preloaded KV-Caches Replace Retrieval Pipelines

Retrieval-augmented generation fetches documents at query time, scores them, and hopes the retriever got it right. Cache-Augmented Generation sidesteps the entire pipeline by preloading knowledge into the model's KV-cache before …

kv-cache cache-augmented-generation transformer-inference rag-alternative ∑ ◫

Model Architecture 23 min

Context Rot: Why Bigger Context Windows Don't Mean Better Retrieval

A million-token window promises perfect recall of everything you feed it. Controlled tests on 18 frontier models show recall degrading steadily, unevenly, and well before the window fills, a pattern researchers now call context rot.

llm long-context rag context-engineering ∑ ◫

Model Architecture 21 min

From 4K to a Million Tokens: How RoPE Scaling, YaRN, and Ring Attention Stretch the Context Window

A model trained on 4,096 tokens can be coaxed into reading a quarter-million without retraining from scratch. The trick is not bigger attention; it is lying to the model about position, and splitting the sequence across a ring of GPUs.

long-context rope yarn ring-attention ∑ ◫

Model Architecture 20 min

Half Mamba, Half Attention: Why Hybrid State-Space Models Took Over

Pure Mamba was supposed to replace attention. Instead the most efficient open models in 2025 are roughly seven-eighths Mamba and one-eighth attention. The reason is the KV cache, and what each layer can and cannot remember.

state-space-models mamba transformers long-context ∑ ◫

Model Architecture 26 min

Titans: The Sequence Architecture That Learns to Remember While It Runs

Most long-context models compress the past into a fixed-size vector and hope nothing important got squeezed out. Titans instead gives the model a small neural network as its memory, and lets that network keep training on the data…

architecture long-context memory test-time-training ∑ ◫

Model Architecture 18 min

Trainable Sparse Attention: When the Model Learns What to Skip

For years, sparse attention was a bolt-on applied after training to shrink the KV cache. NSA, MoBA, and DeepSeek's DSA flip that order, training the sparsity end to end and reporting up to 11x faster decoding without losing accuracy.

attention long-context sparse-attention transformers ∑ ◫