Serving Path Divergence

"The model" is never one thing in production. A request resolves to a tuple: model snapshot, server pool, accelerator type, context configuration, prompt version, region, provider. A fleet with three pools, two regions and two prompt versions has a dozen distinct paths, and each one is an independent opportunity for a configuration to drift away from its peers.

Divergence is when one of those paths starts producing different output from the rest while serving the same traffic mix. It is a different problem from a regression that ships in a deploy, and it needs a different detection method: peers, not history. A pool compared against its siblings on the same day needs no ground truth, no labels, and no memory of what good looked like last month.

Why it matters

Fleet-wide averages are structurally blind to it. A path carrying 8% of traffic can be badly broken while every aggregate metric moves by less than its normal daily variance, so the alert threshold that would catch it would also fire constantly.

The damage is also concentrated rather than spread, which is what makes it a product crisis rather than a statistical curiosity. Sticky routing, session affinity and per-tenant pinning all convert a small request-level percentage into a large user-level one. A defect on 16% of requests in a bad hour can mean roughly a third of active users hit it at least once, because the users who landed on the bad path kept landing on it.

Implementation patterns

• Emit the resolved tuple on every response: model snapshot (not the alias you sent), server pool, accelerator type, context configuration, deployment build, prompt version, region. You can only group by a dimension you recorded, so this list is the whole investigation.
• Run a small fixed probe set against every path on a schedule and compare paths against each other rather than against a stored baseline. Fifty prompts every ten minutes per path is cheap and detects drift in hours.
• Compare distributions, not scores. Output length, refusal rate, retry-within-a-minute rate, schema-parse failure rate and tool-call rate per path. A pool two standard deviations from its peers on any of these is worth paging on, and none of them needs a label.
• Segment by user as well as by request. "Share of users with at least one bad interaction this week" is the metric that matches the damage under sticky routing.
• Make the path a first-class canary dimension. A canary that routes 1% of traffic to a new pool proves nothing if the comparison is against the fleet average rather than against the pool it replaces.

Industry example

Anthropic's 2025 postmortem on three overlapping infrastructure bugs is the clearest published account of a path-level defect. One of them was a context-window routing error that sent Claude Sonnet 4 requests to servers configured for a different context length — a second serving path that should not have existed for that traffic: it began at roughly 0.8% of requests on 5 August 2025, was amplified by a routine load-balancing change on 29 August, and peaked at 16% of Sonnet 4 requests in the worst hour on 31 August. Because routing was sticky, around 30% of Claude Code users in that window had at least one message served by the wrong server type. The fix shipped on 4 September and finished rolling out across platforms in mid-September. Anthropic also reported that its own privacy controls, which restrict engineers' access to user interactions not submitted as feedback, slowed reproduction.

Failure scenarios

• A new pool added for capacity with a subtly different configuration, which then serves a slice of traffic worse than the rest for months.
• A canary compared against the fleet average rather than against the pool it is replacing, so a pool-specific defect is averaged away.
• A provider-side change on one region only, which no deploy of yours can explain and no bisect over your commits will find.
• A version field that logs an alias rather than the resolved snapshot, so the dimension that changed is absent from the data.
• Treating the amplifier as the cause: rolling back the load-balancing change that exposed a latent path defect leaves the defect in place at its original rate.

Trade-offs

Per-path probing costs inference on synthetic traffic multiplied by the number of paths, and it produces false alarms when a pool legitimately differs, since a smaller-model pool should score differently. Recording the resolved tuple on every response adds fields to every log line and a retention question with them. Rich per-interaction telemetry also collides with privacy commitments: the data that makes reproduction easy is often the data you promised not to keep. The workable position is content telemetry on synthetic and opt-in traffic only, and structural metadata on everything else.

When not to use it

One model, one pool, one region and one prompt version has no peers to compare against, and a fixed evaluation set run on each deploy is the right and sufficient tool. The point at which path-dimensioned telemetry starts paying is the second serving path — a second region, a second provider, a canary pool, a per-tenant configuration. Before that it is cost with no signal, and the effort belongs in the deploy-time evaluation instead.

Interview question

Q: Users report your assistant has got worse over the last three weeks. Error rate, latency and throughput are unchanged, your evaluation suite passes on every build, and the complaints describe different symptoms on different platforms. Where do you start, and what would you have built beforehand?

What a strong answer covers: treating "the model" as a tuple and enumerating every serving path; comparing paths against each other rather than against history, because no deploy may have occurred; recognising that sticky routing makes request-level and user-level rates diverge; using label-free distribution signals; and naming the pre-work, which is recording the resolved snapshot and pool on every response so the grouping is possible at all.

Quick check

Quiz: Why is comparing serving paths against each other stronger than comparing a metric against last week? — Because it needs no labels and no stable baseline, and it isolates the one variable that differs; a week-over-week comparison confounds a path defect with every change in traffic mix.

Flashcard: Your fleet runs two regions and three pools. Where does a quality defect hide? — In one path, averaged into invisibility by any fleet-wide metric; record the resolved snapshot and pool on every response and compare paths against their peers.