An agent's search tool begins returning about 20000 tokens per call instead of 200 after an upstream API change. The agent makes up to 12 tool calls per task and nothing in the agent code changed. What happens, turn by turn, and what mechanism stops it?

Turn by turn

Turn 1: Works. The response is longer and the answer is arguably better, so nobody notices.

Turn 2 to 3: The context is now carrying 40,000 tokens of tool output. Three things move together: prefill time rises, because it is linear in prompt length; input cost rises roughly a hundredfold on the tool-output portion; and prompt caching stops helping, because the cached prefix is no longer a prefix - the conversation diverges earlier on every turn.

Turn 4 to 6: The context window is exhausted. What happens next depends on a framework decision nobody made deliberately:

• The provider returns an error. The best outcome: loud, attributable, and the task fails cleanly.
• The framework truncates the tool result, frequently mid-structure. The model receives malformed JSON, cannot parse it, and calls the tool again - turning a size problem into a retry loop that multiplies both cost and latency.
• A compaction step summarises the history. The summary preserves the narrative and drops the specifics - identifiers, exact figures, the constraint from turn 1 - so the agent continues confidently with the task's parameters quietly altered. This is the most damaging variant because there is no error at all.

Steady state: quality degrades even where the context fits, because a relevant fact in the middle of a very long context is attended to less reliably than one near either end. The agent gets worse at tasks it was previously good at, and the only visible signal is spend.

What the user sees

Slower responses, then either a failure or an answer that has lost a detail from early in the task. The failure mode most often reported is "it forgot what I asked".

What stops it - mechanisms, not vigilance

1. A tool-result budget enforced at the tool boundary, not in the prompt. Every tool returns at most N tokens; beyond that it truncates at a structural boundary, states that it truncated, and offers a way to fetch more (an identifier, a page token, a narrower query). The agent then has an affordance instead of a mystery.
2. Never truncate mid-structure. Truncating serialised output must happen at a record boundary, with a machine-readable marker, or the retry loop above is inevitable.
3. A per-task token budget with a hard stop and a defined behaviour when it is reached - return what exists and say the budget was exhausted. An agent with no budget has an unbounded worst case, and the worst case is what arrives when an upstream changes.
4. Alert on tokens per task, not on total spend. Spend changes with traffic; tokens per task changes only when behaviour changes, so it is the signal that moves first.
5. Treat tool output size as part of the tool's contract, tested the way a schema is tested. This is the control that would have caught the upstream change on the day it shipped.

What would have to be true for it to self-heal

Nothing here self-heals: the agent has no feedback loop from cost or context pressure to its own behaviour. Give it one - a budget it can observe - or accept that an upstream change controls your cost and quality.

When this is the wrong thing to build

A single-turn tool call with a response bounded by construction - a currency conversion, a status lookup - needs no budget machinery, and adding it is complexity for nothing. That is the argument for keeping single-purpose tools narrow in the first place.

The trade-off a budget imposes is real: capping a search tool at 2000 tokens means some tasks genuinely need the data you truncated, so the agent must be given a way to ask for more rather than simply being starved. Prefer a budget with an affordance over a smaller budget, and set the per-task ceiling from measured token use rather than from a round number: if the 95^th percentile task uses 40000 tokens, a 60000-token ceiling bounds the worst case without touching normal work.