The benchmark it never saw

Of the six performance changes an agent proposed for Google's Agent Development Kit, four were thrown away, and each of the four had a number attached that said it worked. One cut a multi-agent event cache's time by 17 percent on a synthetic benchmark, and was rejected because it held its entries through a weak reference that could return stale data across sessions. One read a private OpenTelemetry attribute that a minor upgrade could remove. One rewrote the formatting of conversation history, made it harder to read, and bought nothing on a held-out end-to-end benchmark. One sped up workflow event replay, but only in synthetic replay tests.

That is the detail worth keeping from Google's account of letting Gemini 4 Argon loose on the ADK repository, roughly 205,000 lines in about 800 files with nearly 17,000 unit tests. The headline is the two changes that shipped: filtering session events before copying them, and lazy-loading heavy dependencies. The team reports about 40 percent less time preparing in-memory sessions in long conversations, and 21 percent faster import google.adk. The instructive part is the four that did not, because they show where the quality of this kind of work is actually decided.

My argument is that an optimisation is a claim about a workload, and that an agent which can write six such claims in an evening has made the refusal of claims the scarce resource. The refusing instrument is not the test suite. In Google's account it was a benchmark the agent never saw, plus people who knew what the code was promising to its callers.

What the tests did not hold

Take the lazy-import change. All 16,892 open-source unit tests passed on the draft. Twenty-five internal Google test targets failed, because they patched attributes directly on modules, an entirely ordinary thing for a test to do and a thing no public document promises will keep working. The shipped version uses lazy accessors so those attributes stay patchable. The passing suite was a necessary condition and it was silent about a contract that existed anyway.

This is the general shape of the problem. A test suite is a sample of the behaviour a system's callers depend on, taken by people who were thinking about something else at the time. A human making a change carries a rough picture of what is outside the sample: a maintainer knows that a private attribute is private for a reason, that a cache which can serve another session's data is not a cache but a defect with good latency. An agent given a goal and a green suite carries the sample and nothing more. Google's team says the first, two-hour round, which used a single prompt and no repository rules, produced real speedups but broke maintainer conventions, and ran only 216 nearby unit tests. The second round added repository guidelines in .agents/skills files and ran the whole suite. The improvement came from handing the agent more of the contract. It still did not hand over all of it.

What caught the rest was a held-out benchmark the team wrote and the agent never saw, with each reported number a median of six to twelve runs, and model requests confirmed byte-identical before and after. That is a specification of the thing that matters, kept where the optimiser cannot fit itself to it. The 17 percent cache result came from a benchmark the agent could see, and it was a review, not a number, that found the stale-data risk behind it.

A constant nobody derived

The same week's other performance story has no agent in it, and it makes the point from the opposite direction. Cloudflare's post, published in September and summarised by InfoQ in October, describes Pingora Backend Router, which uses consistent hashing to send each cacheable URL to the same server so a data centre holds one copy of a file. Weighted servers, and a separate ring for each combination of features such as compliance and caching, meant dozens of rings, and memory use from the hashing structures that the post says reached 6 GB in some cases.

The cause was a number. Each server is placed on a ring at many points, 160 by default, a figure NGINX hardcodes and Pingora inherited. The post gives the arithmetic that justifies some number: with a hundred servers and a single point each, the load's coefficient of variation is about 99 percent, so some servers take twice their share and some almost none; at 160 points it falls to about 8 percent. More points, smoother load. The team's contribution was to ask how smooth it needed to be. The error shrinks slowly, the last 90,000 points in the example buy about 0.7 percent, and with 32-bit hashes collisions start to push error back up for large data centres. They cut the points per server by about 90 percent. They also packed each point from 8 bytes to 6, a 25 percent saving on that structure. Cloudflare reports roughly 100 TB of RAM saved across its fleet, read from a week-over-week memory chart. I would treat that as the company's measurement of itself, not an audited figure, and the post is itself candid that the 25 percent applies to the hashing structures rather than to all router memory.

Nothing in the optimisation is clever in the sense a profiler is clever. A default was treated as a decision and a decision was given a derivation. What interests me is what Cloudflare did next, because it is the thing Google's team did with a benchmark. A new ring changes which server owns a URL, so a global switch would have emptied caches and sent a surge to origins. The router held the old and new rings side by side, chose between them per request from the request's hash, and could roll back without a deploy. Rollout was staged through small validation locations, then larger groups of data centres, and traffic share and geography were controlled separately so any cache churn stayed local. Only at 100 percent was the old path deleted. The thing being refused or accepted was never the maths. It was a mechanism that let the system disagree with the maths cheaply.

The packed type is a decision too

One line of the Cloudflare post deserves more attention than it got. Shrinking the index from 32 bits to 16 saves nothing by itself, because alignment pads the struct back to its old size, so the 6-byte layout stores the pieces in a packed byte array behind getters. And a 16-bit index means the structure cannot address more than about 65,000 servers. The post says as much: the router is not expected to coordinate more than that.

That is sensible, and it is also a capacity limit moved out of a document and into a type, where it is enforceable and no longer discussed. This is what good optimisation usually does. It converts a flexible, expensive assumption into a rigid, cheap one. Each such conversion is correct on the day it is made and each one removes an option. An agent proposing lazy loading or a cheaper copier is making the same kind of trade, and the human reading the pull request is the only party in the loop who knows which options the organisation would have wanted to keep.

The counterargument

The strongest objection is that this is a counsel of perfection. Four rejected out of six sounds like a failure rate, but it is a triage rate, and triage at that price is a bargain: a team that could review six candidate optimisations of a large codebase in a morning, and discard four, is ahead of the one that never found them. The agent's audit even caught a bug in its own cache before the humans did. Verification by hand has always been the bottleneck, and one can say the honest reading is that rejection by review works, so the system is fine.

I agree with the arithmetic and disagree with the comfort. Google's team had a held-out benchmark, repository guidelines the agent loaded as needed, an agent run that logged its reasoning, and maintainers who could tell a stale-data cache from a fast one. Those are exactly the pieces a typical organisation lacks, and the whole result depends on them. The more instructive number is not 4 of 6 but how many of the four would have shipped under a team that trusted a green suite and a before-and-after table, which are the two artefacts the agent was most able to produce. The article itself is sober about scope: a single run by the vendor's own team, a benchmark with an instant model and instant tools so that it isolates framework overhead, a 40 percent gain that applies only to the in-memory session service because external stores deserialise from storage, and per-call savings no human would notice on a call that takes seconds. Whether this generalises is a question one run cannot answer.

What this changes

If generation is cheap, then the cost of a change moves wholly to deciding whether to take it, and the deciding needs three things the change cannot supply for itself: a measurement the proposer cannot see, a record of the contracts the tests leave unsaid, and a way to be wrong cheaply, which is what Cloudflare's two-ring router was. All three predate agents. What is new is the rate at which an organisation without them will be handed plausible candidates.

That has an uncomfortable consequence for how architects spend their time. The craft of an optimisation used to be mostly in finding it, and finding it scaled with the scarce person who understood the profiler output. Finding is now nearly free, so what is left is the part that was always the real skill and was always the less visible one: knowing which numbers to distrust, and what the system has quietly promised to people who never wrote it down.

Cloudflare's lesson is the gentler version. One constant, inherited and never derived, sat in the fleet until someone asked how much error they were actually buying with it. Google's is the sharper one. A tireless collaborator can now ask that question six times before breakfast, and it will answer with evidence each time. The test of the system around it is whether any of that evidence was gathered where the collaborator could not reach.