Tagged “deepseek”

3 posts.

All Model Architecture13 Training & Alignment8 Inference & Serving11 Agents & Orchestration10 Reasoning & Evaluation7 Safety, Security & Governance3 Platforms & Practice11

Model Architecture 25 min

Mixture of Experts: How Frontier LLMs Achieve Scale Without Proportional Cost

Every frontier LLM released since 2023 uses some form of Mixture of Experts. The idea is deceptively simple: build a model with hundreds of billions of parameters but activate only a fraction for each token. The result is dense-m…

moe architecture scaling inference ∑ ◫

Model Architecture 22 min

Multi-Head Latent Attention: How DeepSeek Compressed the KV Cache Without Losing Quality

Every token an LLM generates forces it to reload the keys and values of every token that came before. Multi-Head Latent Attention rewrites that trade by caching one compressed vector instead of dozens of separate heads, cutting m…

llm-architecture attention-mechanisms deepseek kv-cache ∑ ◫

Model Architecture 18 min

Trainable Sparse Attention: When the Model Learns What to Skip

For years, sparse attention was a bolt-on applied after training to shrink the KV cache. NSA, MoBA, and DeepSeek's DSA flip that order, training the sparsity end to end and reporting up to 11x faster decoding without losing accuracy.

attention long-context sparse-attention transformers ∑ ◫