Context & In-Context Learning
Autoregressive generation, the prompt stack, context engineering, and long-context degradation.
6concepts
28flashcards
46minutes of reading
- 01 Autoregressive Generation How a language model turns next-token prediction into a paragraph, and why generation is a loop that feeds its own output back as input.
- 02 Context Engineering The discipline of curating exactly which tokens occupy the model's window during inference, and why it became the core skill for building agents that run longer than a single turn.
- 03 In-Context Learning How large models learn a task from examples in the prompt alone, with no weight updates, and why this emergent ability reframed how we use LLMs.
- 04 The Prompt Stack and Chat Roles What a chat prompt actually is under the hood: a single token sequence built from system, user, and assistant turns wrapped in special tokens, and why that structure is load-bearing.
- 05 Context Rot The measured fact that model accuracy falls as the input grows, non-uniformly and in cliffs, so a bigger window is a bigger desk rather than a better memory.
- 06 Context Windows and Long-Context Models Why a model advertised at a million tokens can still lose the fact in the middle, and what actually sets the limit: memory, compute, position, and attention itself.