Tagged “scaling-laws”

2 posts.

All Model Architecture19 Training & Alignment22 Inference & Serving18 Agents & Orchestration11 Reasoning & Evaluation26 Safety, Security & Governance7 Platforms & Practice20

Training & Alignment 3 min

How Much Data Is Enough? The Chinchilla Correction

For years the field raced to build bigger models. Then a 2022 study showed most of those giants were undertrained, and quietly redrew the map of how to spend a compute budget.

scaling-laws training compute data

Training & Alignment 22 min

Language Modelling Is Compression: The Seventy-Year-Old Idea Underneath Every LLM

In 1951 Claude Shannon estimated the entropy of English by having people guess the next letter. In 2023 a 70-billion-parameter language model compressed a gigabyte of Wikipedia to 8.3% of its size, beating every compressor ever p…

information-theory compression entropy language-models ∑ ◫