4 posts.
Weight quantisation, KV cache eviction, prompt compression and distillation are treated as four separate engineering disciplines with four separate literatures. They are one problem: choosing a point on a rate-distortion curve. S…
In 1951 Claude Shannon estimated the entropy of English by having people guess the next letter. In 2023 a 70-billion-parameter language model compressed a gigabyte of Wikipedia to 8.3% of its size, beating every compressor ever p…
Two variational autoencoders can report the same ELBO to three decimal places while one has a latent code carrying five bits of information about the input and the other has a latent carrying exactly zero. The objective cannot se…
You add a KL penalty to an RLHF objective, log it, and it prints minus 0.03. KL divergence is provably non-negative, and nothing is broken. One formula does four different jobs in modern machine learning, and almost every confusi…