Context Window Budget
The finite token allowance per request, treated as an engineering resource to be allocated deliberately between system instructions, retrieved context, history and output.
Treating the context window as effectively unlimited is the most common design error in LLM applications, and it degrades three things at once: cost scales with tokens, latency scales with tokens, and retrieval accuracy degrades as irrelevant material accumulates.
The budget needs explicit allocation. System instructions and tool definitions are fixed overhead paid on every call — and tool definitions are larger than people assume, with a dozen tools easily consuming several thousand tokens before any work begins. Retrieved context is the variable most worth tuning. Conversation history grows without bound unless managed. Output tokens must be reserved or generation truncates mid-answer.
The finding that should govern the design: models do not attend uniformly across a long context. Material in the middle is used less reliably than material at the start or end, so stuffing fifty retrieved passages in hope produces worse answers than five well-chosen ones, at ten times the cost.
The practices that follow: rank and truncate retrieval rather than passing everything; summarise or window conversation history rather than replaying it verbatim; place the most important instructions and the most relevant context at the edges; and instrument token usage per component so the budget is a measured quantity rather than an assumption.