Context & In-Context Learning
Autoregressive generation, the prompt stack, context engineering, and long-context degradation.
10concepts
88flashcards
74minutes of reading
- 01 Autoregressive Generation How a language model turns next-token prediction into a paragraph, and why generation is a loop that feeds its own output back as input.
- 02 Context Compaction and Handoff When an agent's conversation approaches the window limit, compaction summarises the history and reinitialises a fresh window from the summary; what survives the compression determines whether the agent continues or silently restarts.
- 03 Context Engineering The discipline of curating exactly which tokens occupy the model's window during inference, and why it became the core skill for building agents that run longer than a single turn.
- 04 In-Context Learning How large models learn a task from examples in the prompt alone, with no weight updates, and why this emergent ability reframed how we use LLMs.
- 05 Many-Shot In-Context Learning What changes when you put hundreds or thousands of examples in the prompt instead of five, why the gains keep coming after few-shot plateaus, and how model-generated rationales substitute for scarce human data.
- 06 RAG vs Long Context The engineering decision the million-token window forced, what controlled comparisons actually found about quality and cost, and why routing between retrieval and full-context beats picking a side.
- 07 The Prompt Stack and Chat Roles What a chat prompt actually is under the hood: a single token sequence built from system, user, and assistant turns wrapped in special tokens, and why that structure is load-bearing.
- 08 Context Rot The measured fact that model accuracy falls as the input grows, non-uniformly and in cliffs, so a bigger window is a bigger desk rather than a better memory.
- 09 Context Windows and Long-Context Models Why a model advertised at a million tokens can still lose the fact in the middle, and what actually sets the limit: memory, compute, position, and attention itself.
- 10 Long-Context Training Recipes How a model trained at 8k becomes a genuine 128k model, why the data mixture matters more than the token count, and what the Llama 3 and ProLong recipes agree on.