Anthropic reported in 2025 that a routing bug sent a share of Claude Sonnet 4 requests to servers configured for a different context length, peaking at 16% of those requests in the worst hour on 31 August, and that it took weeks to identify. Error rates never moved. Which design decision allowed the delay and what telemetry closes it?

The trigger

Anthropic's published postmortem describes a context-window routing error that began affecting a small fraction of Sonnet 4 requests on 5 August 2025, roughly 0.8%, and was then amplified by a routine load-balancing change on 29 August. At the worst hour on 31 August it reached 16% of Sonnet 4 requests. The routing was sticky, so a user who landed on a misconfigured server tended to stay there; Anthropic reported that around 30% of Claude Code users who made requests in that window had at least one message served by the wrong server type. The fix shipped on 4 September and finished rolling out across platforms by mid-September.

Why detection lagged

Every conventional signal was healthy. The requests returned 200, within normal latency, with no malformed responses. The degradation lived entirely in output quality, which no error budget measures.

Three properties made it worse, and all three are design decisions rather than accidents:

• The unit of impact was a user session, not a request. A 16% request-level rate spread evenly would be background noise. Sticky routing concentrated it, so a minority of users saw a badly broken product while the fleet average barely moved.
• The model often recovered. A capable model produces an acceptable answer despite a degraded path much of the time, so the defect appears as an elevated rate of mediocre answers rather than as a class of failures.
• Symptoms differed by platform and by model, which made the incoming reports look like several unrelated complaints instead of one cause.

Anthropic also noted that its own privacy controls limit engineers' access to user interactions that were not explicitly reported, which slowed reproduction.

The structural fix

• Emit the serving configuration on every response, not just the model name: the exact server pool, context configuration, and deployment build. A regression is only bisectable if the dimension that changed is a field you can group by.
• Alert on quality distributions per configuration, not per fleet. Run a small continuous evaluation set against every serving pool independently and compare pools against each other. A pool that drifts from its peers is detectable in hours, and it needs no ground truth.
• Segment by user, not only by request. Track the share of users who had any bad interaction in a window. That metric would have shown a 30% user-level figure while the request-level figure looked survivable.
• Cheap proxy signals that need no labels: refusal rate, retry rate, conversation abandonment, output length distribution, thumbs-down per user rather than per message.

Common weak answers

• "Roll back the load-balancing change." The tempting local fix. It masks the symptom and leaves the routing bug in place at its original 0.8%. The amplifier is not the cause, and treating a traffic change as the root cause is the most common error in this class of incident.
• "Add an LLM judge on a sample of production traffic." Useful, and not sufficient alone: a judge scoring a fleet-wide sample dilutes a 16% effect into noise unless the sample is stratified by serving configuration. The grouping dimension matters more than the scorer.
• "Trust user reports." They arrived and were not actionable, because they described different symptoms on different platforms. Reports tell you something is wrong; only a configuration dimension in the telemetry tells you what.

The general lesson

An AI system can be fully available and substantially broken at the same time. Availability and latency are necessary and insufficient, and the observability gap is that quality has no natural error signal. Where copying this would be a mistake: a team with one model on one pool does not need per-pool evaluation. The per-configuration comparison earns its cost once you have more than one serving path, which happens the moment you add a region or a second provider.