A team gives an assistant five tools. A task that needs four tool calls costs roughly six times a plain answer and takes about 9 seconds end to end, and the team expected the tools to be nearly free. What actually happens on each tool call?

What is being tested

Whether you understand that a model does not execute anything. It emits a request to call a function, your application runs it, and the result goes back into a fresh inference request. The cost and latency follow from that loop, not from the tools.

The mechanism

A tool call is not a callback inside one inference. The sequence for a four-call task is:

1. Request 1: system prompt plus tool schemas plus the user message. The model returns a tool-call request and stops.
2. Your code runs the tool and appends the result to the transcript.
3. Request 2: everything from request 1 plus the tool request plus its result.
4. Repeat, then a final request that produces the answer.

Four tool calls means five inference requests, and the input grows on every one. If the base prompt is 2,000 tokens and each tool result adds 800, the five requests carry roughly 2,000 + 2,800 + 3,600 + 4,400 + 5,200 input tokens: about 18,000 input tokens for a task whose single-shot version was 2,000. That is the factor of six, and it is arithmetic rather than a defect.

Latency behaves the same way. Each round trip pays time-to-first-token plus generation plus the tool's own execution, and they are strictly serial because the model cannot choose call n+1 until it has seen result n. Four calls at roughly 1.5 to 2 seconds each is the 9 seconds in the stem.

What reduces it

• Prefix caching. The growing transcript shares a long common prefix with the previous request. Providers that cache prefixes charge much less for the cached portion, which turns the quadratic input growth from a cost problem into mostly a latency problem. Keep the stable content at the front and never reorder it, or the cache misses.
• Fewer round trips, not smaller prompts. One tool that returns the whole customer record beats three that each return a field. The round trip dominates.
• Parallel tool calls where the model is allowed to request several at once and they are genuinely independent. Two independent lookups in one turn remove a whole round trip.
• Trim tool results before appending them. A 40 KB JSON blob costs on every subsequent turn, not just the one that fetched it.
• Prune the tool list. Every schema sits in the input of every request in the loop. Twenty tools is a standing tax on every call and it also makes selection worse.

When the loop is the wrong answer

If the sequence of calls is known in advance, write it as code and use the model once at the end. A workflow that always fetches the order and then the shipping status does not need a model to discover that. The loop earns its cost only when the next step genuinely depends on what the last step returned.

Common weak answers

• "Use a faster model." It shortens each hop and leaves the number of hops unchanged, so latency improves at most linearly while the token growth is untouched.
• "Stream the tokens." Streaming improves perceived latency of the final answer only. The intermediate turns produce no user-visible output.