Tagged “optimisation”

4 posts.

All Model Architecture20 Training & Alignment22 Inference & Serving21 Agents & Orchestration11 Reasoning & Evaluation27 Safety, Security & Governance8 Platforms & Practice22

Reasoning & Evaluation 24 min

Inside Gradient-Boosted Trees: The Engineering That Made XGBoost, LightGBM and CatBoost Win Tabular ML

XGBoost, LightGBM and CatBoost minimise the same objective with the same kind of tree. What separates them is bookkeeping: XGBoost turned two sums of derivatives into a split score, LightGBM made those sums cheap, and CatBoost ma…

trees-and-ensembles gradient-boosting xgboost tabular ∑ ◫

Training & Alignment 24 min

The Bandwidth Wall: How Low-Communication Training Unbundled the Datacentre

Data-parallel training all-reduces the entire gradient after every step, which is why frontier pretraining happens inside one building with a purpose-built fabric. DiLoCo synchronises every five hundred steps instead of every one…

distributed-training scaling infrastructure optimisation ∑ ◫

Training & Alignment 26 min

The Bias-Variance Tradeoff Is a Special Case: Double Descent, Benign Overfitting, and Grokking

A network that fits ImageNet with randomly shuffled labels should not generalise on real ones. It does. That single experiment invalidated the textbook account of why machine learning works, and the three phenomena that replaced …

training-dynamics generalisation double-descent grokking ∑ ◫

Training & Alignment 25 min

The Four Faces of KL Divergence: Mode-Seeking, Mode-Covering, and Why Your Estimator Went Negative

You add a KL penalty to an RLHF objective, log it, and it prints minus 0.03. KL divergence is provably non-negative, and nothing is broken. One formula does four different jobs in modern machine learning, and almost every confusi…

math information-theory kl-divergence rlhf ∑ ◫