Model Architecture
32 min
The Wavelength Problem: How One Number Decides a Model's Context Length
Llama 2 and Llama 3 share an attention implementation almost line for line. One constant differs: the RoPE base went from 10,000 to 500,000. That number sets the entire wavelength spectrum of the model's position signal, and it h…