Tagged “compression”

3 posts.

All Model Architecture19 Training & Alignment22 Inference & Serving20 Agents & Orchestration11 Reasoning & Evaluation27 Safety, Security & Governance7 Platforms & Practice22

Inference & Serving 27 min

Everything Is Lossy Compression: A Rate-Distortion View of Quantisation, KV Caches, and Distillation

Weight quantisation, KV cache eviction, prompt compression and distillation are treated as four separate engineering disciplines with four separate literatures. They are one problem: choosing a point on a rate-distortion curve. S…

information-theory quantisation kv-cache distillation ∑ ◫

Training & Alignment 22 min

Language Modelling Is Compression: The Seventy-Year-Old Idea Underneath Every LLM

In 1951 Claude Shannon estimated the entropy of English by having people guess the next letter. In 2023 a 70-billion-parameter language model compressed a gigabyte of Wikipedia to 8.3% of its size, beating every compressor ever p…

information-theory compression entropy language-models ∑ ◫

Model Architecture 24 min

Why GANs Lost Image Generation, and Why Adversarial Losses Are Everywhere Anyway

In 2021 a diffusion model beat BigGAN-deep on ImageNet while nearly doubling its recall, and GANs stopped being the default way to generate images. Yet latent autoencoders, SDXL-Turbo, HiFi-GAN and the neural audio codecs behind …

adversarial-generative-models gans image-generation diffusion-models ∑ ◫