Context & In-Context Learning
Autoregressive generation, the prompt stack, context engineering, and long-context degradation.
10concepts
88flashcards
74minutes of reading
- 01 Context Rot The measured fact that model accuracy falls as the input grows, non-uniformly and in cliffs, so a bigger window is a bigger desk rather than a better memory.
- 02 Context Windows and Long-Context Models Why a model advertised at a million tokens can still lose the fact in the middle, and what actually sets the limit: memory, compute, position, and attention itself.
- 03 Long-Context Training Recipes How a model trained at 8k becomes a genuine 128k model, why the data mixture matters more than the token count, and what the Llama 3 and ProLong recipes agree on.