Training & Alignment
22 min
Language Modelling Is Compression: The Seventy-Year-Old Idea Underneath Every LLM
In 1951 Claude Shannon estimated the entropy of English by having people guess the next letter. In 2023 a 70-billion-parameter language model compressed a gigabyte of Wikipedia to 8.3% of its size, beating every compressor ever p…