Tagged “llm-inference”

2 posts.

All Model Architecture13 Training & Alignment8 Inference & Serving11 Agents & Orchestration10 Reasoning & Evaluation7 Safety, Security & Governance3 Platforms & Practice11

Model Architecture 21 min

From 4K to a Million Tokens: How RoPE Scaling, YaRN, and Ring Attention Stretch the Context Window

A model trained on 4,096 tokens can be coaxed into reading a quarter-million without retraining from scratch. The trick is not bigger attention; it is lying to the model about position, and splitting the sequence across a ring of GPUs.

long-context rope yarn ring-attention ∑ ◫

Inference & Serving 23 min

Speculative Decoding: How a Small Draft Model Makes Large Language Models Think Faster

Autoregressive decoding forces even the largest language models to emit one token at a time, bottlenecked by memory bandwidth rather than compute. Speculative decoding breaks this sequential constraint by letting a lightweight dr…

speculative-decoding llm-inference parallel-decoding draft-model ∑ ◫