Tagged “llm-inference”

3 posts.

All Model Architecture19 Training & Alignment22 Inference & Serving18 Agents & Orchestration11 Reasoning & Evaluation26 Safety, Security & Governance7 Platforms & Practice20

Model Architecture 21 min

From 4K to a Million Tokens: How RoPE Scaling, YaRN, and Ring Attention Stretch the Context Window

A model trained on 4,096 tokens can be coaxed into reading a quarter-million without retraining from scratch. The trick is not bigger attention; it is lying to the model about position, and splitting the sequence across a ring of GPUs.

long-context rope yarn ring-attention ∑ ◫

Inference & Serving 24 min

Running Language Models on a Phone: Memory Bandwidth, NPUs, and the Few-Billion-Parameter Ceiling

Phones ship NPUs rated in trillions of operations per second, yet the speed at which a reply appears is set by how fast LPDDR memory can hand a couple of gigabytes of weights to the processor, over and over. This post derives the…

on-device-and-edge-ai on-device-ai llm-inference memory-bandwidth ∑ ◫

Inference & Serving 23 min

Speculative Decoding: How a Small Draft Model Makes Large Language Models Think Faster

Autoregressive decoding forces even the largest language models to emit one token at a time, bottlenecked by memory bandwidth rather than compute. Speculative decoding breaks this sequential constraint by letting a lightweight dr…

speculative-decoding llm-inference parallel-decoding draft-model ∑ ◫