Agent Transcript Compaction

An agent that browses and edits files runs 30 steps on a hard task. Step 1 sends 2,000 tokens. Step 30 sends 20,000, and the run as a whole has sent about 330,000 input tokens rather than the 20,000 a naive estimate predicts, because every step re-sends the entire transcript. The bill is one problem. The other is that by step 25 the agent has started re-running a tool it already ran and has stopped honouring a constraint stated in step 2.

Compaction is the mechanism that bounds both. At a threshold, the resolved portion of the transcript is replaced with a compact record of what was learned and what was done, while the goal, the constraints and the open questions are carried forward unchanged.

Why it matters

Input tokens in an agent loop grow as roughly base x n + delta x n(n-1)/2. That quadratic term is why agent costs surprise teams by an order of magnitude and why a step budget alone does not control spend.

The quality effect is the less obvious half. Instruction-following degrades as the transcript lengthens, so a system prompt's constraints compete with thousands of tokens of tool output that arrived more recently. Compaction is as much a correctness mechanism as a cost one.

Implementation patterns

• Trigger on token count, not step count. One step that fetched a 40 KB file matters more than ten small ones. A common threshold is 50 to 70% of the context window.
• Preserve three things verbatim: the original task, the hard constraints, and any identifiers the agent must not paraphrase (file paths, ticket ids, account numbers). Summarising an identifier corrupts it.
• Summarise into a structured record rather than prose: what was tried, what the result was, what remains open, what has been ruled out. Prose loses the negative results, and negative results are what stop the agent repeating itself.
• Keep the last two or three turns raw, because the immediate next action depends on their exact content.
• Write durable findings out of the transcript entirely — to a file or a scratchpad the agent can re-read by name — so the transcript holds pointers rather than content.
• Log the pre-compaction transcript, because a post-compaction failure is otherwise undebuggable.

Industry example

Long-running coding and research agents shipped by the major model providers from 2024 onwards all expose some form of this: a context threshold at which the session is summarised and continued, paired with prefix caching so the stable head of the transcript is charged at a reduced rate. The pairing matters — compaction invalidates the cached prefix at the moment it fires, so a system that compacts too often pays full price on every subsequent request and loses the saving it was trying to make.

Failure scenarios

• A constraint dropped in the summary, so the agent violates a rule it was given at step 2 and no log line records the loss.
• An identifier paraphrased — "the config file" instead of the exact path — after which every subsequent tool call targets the wrong thing.
• Negative results discarded, so the agent retries the approach that already failed, burns its step budget and reports that the task is impossible.
• Compaction thrashing near the threshold, firing every second step, destroying the prefix cache and multiplying cost.
• A summary produced by a weaker model to save money, which is where most of the above originate.

Trade-offs

Compaction buys bounded cost and better late-step instruction-following. It pays an extra model call per compaction, a cache invalidation, and an irreversible loss of detail: the agent can no longer quote something it saw at step 4. It also makes runs harder to reproduce, since the summary is model output and two runs of the same task diverge after the first compaction.

When not to use it

If typical runs finish in under about ten steps, do not build it. The quadratic term is small, prefix caching handles the cost, and a hard step budget plus trimmed tool output is sufficient. Build compaction when long runs are the normal case rather than the exception — and before that, prefer the cheaper levers: return less from each tool, prune the tool list, and ask whether the workflow needs a loop at all rather than a fixed sequence of calls with one model step at the end.

Interview question

Q: Your coding agent completes short tasks well and fails on long ones, and the failures look like it forgot its instructions. Spend on the long tasks is also far above forecast. Diagnose both, and tell me what you would build first and what you would measure.

What a strong answer covers: identifying quadratic input growth as the cost cause and transcript length as the instruction-following cause, so both share one mechanism; proposing token-triggered compaction that preserves goal, constraints and identifiers verbatim; noting the interaction with prefix caching and the risk of thrashing; measuring constraint-violation rate and repeated-tool-call rate before and after, not just tokens; and stating that reducing tool output size is the cheaper first move.

Quick check

Quiz: Why does a 30-step agent loop cost far more than 30 times a single call? — Because each step re-sends the whole transcript, so total input grows with the square of the step count rather than linearly.

Flashcard: What must never be summarised when an agent transcript is compacted? — The original goal, the hard constraints and any exact identifiers; paraphrasing those is how a compacted agent starts violating rules and targeting the wrong resources.