A 2-billion-parameter model that trades blows with one ten times its size is not an accident of architecture. It is the product of a teacher pouring its full probability distribution into a student, token by token.
Weight quantisation, KV cache eviction, prompt compression and distillation are treated as four separate engineering disciplines with four separate literatures. They are one problem: choosing a point on a rate-distortion curve. S…
You add a KL penalty to an RLHF objective, log it, and it prints minus 0.03. KL divergence is provably non-negative, and nothing is broken. One formula does four different jobs in modern machine learning, and almost every confusi…
In 2021 a diffusion model beat BigGAN-deep on ImageNet while nearly doubling its recall, and GANs stopped being the default way to generate images. Yet latent autoencoders, SDXL-Turbo, HiFi-GAN and the neural audio codecs behind …