Tagged “efficiency”

4 posts.

All Model Architecture19 Training & Alignment22 Inference & Serving18 Agents & Orchestration11 Reasoning & Evaluation26 Safety, Security & Governance7 Platforms & Practice20

Model Architecture 20 min

Do Transformers Need Three Projections? Rethinking Q, K, and V

Every attention head learns three weight matrices for query, key, and value. A 2026 study trained models up to 1.2B parameters to ask which of them are actually load-bearing, and found that keys and values can share one projectio…

transformers attention kv-cache inference ∑ ◫

Inference & Serving 21 min

Four Bits Per Weight: How Low-Precision Quantization Stopped Hurting LLMs

A 70B model in FP16 needs 140 GB of memory it spends most of its time waiting to read. Dropping each weight to four bits cuts that to 35 GB, and for years that cut also broke the model. Here is what changed.

quantization inference gptq awq ∑ ◫

Reasoning & Evaluation 24 min

Inside Gradient-Boosted Trees: The Engineering That Made XGBoost, LightGBM and CatBoost Win Tabular ML

XGBoost, LightGBM and CatBoost minimise the same objective with the same kind of tree. What separates them is bookkeeping: XGBoost turned two sums of derivatives into a split score, LightGBM made those sums cheap, and CatBoost ma…

trees-and-ensembles gradient-boosting xgboost tabular ∑ ◫

Model Architecture 4 min

The Length of a Thought: Why Context Windows Became the New Battleground

A 512-token model could read a paragraph. A million-token model can read a novel. The journey between those two numbers is a story about quadratic cost, clever approximations, and what it means for a machine to "remember."

transformers attention context scaling