Where State Lives in an LLM Application
Why the context window is the worst place to keep state, the four stores a mature application actually uses, and the assembly step that decides what the model sees.
The tempting design keeps everything in the conversation: append each turn, pass the whole history, let the model remember. It works until the history exceeds the window, costs grow linearly in turn count, retrieval degrades in the middle of a long context, and there is no way to query or correct what the system believes. The context window is a working buffer, not a database, and treating it as storage is the most common architectural mistake in these systems.
The four stores
Conversation history is the raw turn sequence, kept in ordinary application storage and used to reconstruct context rather than being the context. Keeping it outside the prompt means it can be truncated, summarised or selectively included without losing the record.
Structured state is what the application actually knows: the entities under discussion, the user's stated constraints, the current step of a workflow, the values collected so far. This belongs in a database with a schema, because it is queryable, correctable, and can be validated. Extracting it from conversation into a schema, rather than leaving it implicit in the transcript, is what makes the application inspectable.
Long-term memory spans sessions: preferences, facts about the user, prior conclusions. It needs storage plus a retrieval mechanism plus, critically, a policy for what is written and what expires, since an unbounded append-only memory becomes noise and a privacy liability.
Retrieved documents are transient, fetched per request and not persisted in the conversation, so the same query later retrieves current content rather than a stale copy embedded in history.
The assembly step
Given these stores, each request assembles a context from them: the system prompt, the relevant structured state, retrieved documents, and a bounded slice of history. That assembly is a piece of application logic that deserves to be explicit, versioned and tested, because it determines everything the model sees and is where most quality problems in these systems actually originate.
The property this buys is that context length becomes a design parameter rather than a growing accumulation. A conversation of two hundred turns can assemble the same size of context as one of five, containing what matters rather than what happened.
When it breaks
Summarisation loses what was not anticipated. Compressing history into a summary discards detail irreversibly, and the detail that turns out to matter is not predictable at compression time. Keeping the full history in storage and summarising only for the prompt means the detail can be recovered when a later turn needs it.
Structured extraction fails silently. A model asked to maintain state from conversation can mis-extract, and the wrong value then persists and conditions everything after. Validating extracted state against a schema and surfacing it to the user, so it can be corrected, is what makes the failure recoverable.
Memory writes need a policy. Storing everything makes retrieval noisy and creates a growing store of personal data with no retention rationale. What is worth remembering, how long it lives, and how a user removes it are product decisions that have to be made rather than defaulted.
Multi-agent state is the hard case. When several agents share a task, whether state is passed in messages, held in a shared store or reconstructed per agent determines both correctness and cost, and the shared store is a trust boundary as well as a coordination mechanism.
12 flashcards for this concept
Click a card to reveal the answer.