A change ships on 2 March. On 8 April finance asks why cloud spend is 24% above forecast. The cause is a client retry loop added on 2 March that doubled cross-region traffic. Why did detection take 37 days and what structural fix would have caught it?

The trigger

A client library gained a retry on a timeout path. The timeout fires on about 3% of calls to a service in another region, and each one now makes three attempts. Cross-region transfer for that path tripled on the affected 3%, which roughly doubled the total because that path carried most of the bytes. No error rate changed, no latency alarm fired, and the feature worked better than before - retries do improve success rates. Nothing in the service's own telemetry said anything was wrong.

Why it propagated

The retry was in a shared client library, so every consumer of that library inherited it on their next build. The cost grew as adoption spread over weeks, which also made the spend curve a gentle ramp rather than a step - the shape that anomaly detectors are worst at seeing.

Why detection lagged, in three parts

1. Billing data is late and coarse. Usage records typically land a day or more behind, and transfer charges are among the slowest to resolve. A 24-hour-lagged daily total is a poor detector for a 2% daily ramp.
2. The data is aggregated against the wrong axis. Cost is reported per account and service - "data transfer, this region" - while the question is "which code path". No allocation scheme in common use maps a bill line to a client library version.
3. Nobody owned the gradient. Monthly reviews compare month to month, so a ramp that starts on the 2^nd is a small variance at the first review and an emergency at the second. The team that could have recognised the cause never saw the signal, and the team that saw the signal could not recognise the cause.

The tempting local fix

Set a budget alert at forecast plus 10%. It would have fired around day 20 and told finance something they would soon know anyway, with no indication of what caused it. Useful as a backstop and not a fix.

The structural fix

• Measure the physical driver in the service's own telemetry, not on the bill. Bytes sent cross-region per request, and retry attempts per logical call, as first-class metrics with alerts on ratio rather than absolute volume. A retry rate that triples is visible within minutes in a system that measures it; it is invisible for a month in a system that waits for an invoice.
• Put a cost estimate in the change. A deploy-time check that flags changes to retry policy, replication, log volume, retention or instance shape, with an order-of-magnitude estimate attached. The point is a conversation at review time, not a precise number.
• Define a retry budget rather than a retry count - retries capped at a small percentage of successful traffic - which bounds both the cost and the overload risk by construction.
• Alert on unit cost, not total spend. Cost per thousand requests rises immediately when efficiency drops, and is immune to the growth that masks a regression in a total.

The general lesson

Cost is a lagging indicator of an engineering change, so cost data is the wrong place to detect engineering regressions. Every expensive resource has a physical proxy the system can measure in real time: bytes moved, gigabyte-months stored, rows scanned, tokens generated, instance-hours. Alert on the proxy with a ratio threshold, reconcile against the bill monthly, and the detection gap falls from weeks to minutes.

The same mechanism produces outages as well as invoices, which is why this is worth building even in a team with no cost mandate. An added retry multiplies offered load on a dependency that is already failing, and the published incident record is full of cases where that loop was the amplifier rather than the trigger: retry traffic sustains a failure after its cause has passed, which is why Google's SRE writing recommends a retry budget per client rather than a retry count per call. Prefer a retry budget expressed as a fraction of successful traffic over a per-call retry count, unless the call is genuinely idempotent and cheap, and the cost regression and the overload risk are bounded by one control.

When not to build the cost gate

A deploy-time cost check is machinery, and below roughly a few tens of thousands a month of spend it will cost more attention than it saves. Start with the two cheap things: the physical-driver metrics, which a team wants anyway for reliability, and a monthly review that looks at the largest line rather than the total. Add the gate when a single change can plausibly move the bill by more than a few per cent, which is the threshold at which a human review earns its place.