Segment-Isolated Serving

When p99 latency is investigated properly, the tail frequently turns out not to be random. It is a specific population being consistently slow: the account with a hundred thousand records, the customer with the largest catalogue, the workspace with the deepest hierarchy.

That is a different problem from statistical variance, and it has a different solution. Optimising the shared path to accommodate the heavy population makes the common case worse for everyone and usually still fails the tail.

Why it matters

The heavy population is usually the commercially important one. The largest customers generate the most revenue, exercise the most features, and experience the worst performance — a correlation that is invisible in aggregate metrics and obvious once the segmentation exists.

Implementation patterns

• Segment latency by every dimension that could correlate with work: account size, tenant, region, device, app version, feature usage, data volume. The segmentation is the diagnostic, and it usually resolves the question in one observation.
• Give the heavy segment a different data path: a precomputed or materialised view, a different index strategy, pagination where the shared path returns everything, or an asynchronous export instead of a synchronous query.
• Give it a different resource allocation — a dedicated pool or dedicated capacity — so its expensive work does not consume the shared allocation.
• Consider a different placement entirely. For a multi-tenant platform, the largest tenants moving to dedicated infrastructure is the same decision expressed at the deployment level.
• Set expectations differently where the work is genuinely large. A report over ten million records can legitimately be asynchronous with a notification, and that is a better product than a synchronous request that times out.
• Alert on the segment's SLI separately, since the aggregate will not show it.

Industry example

Multi-tenant platforms such as Freshworks and commerce platforms such as Myntra both encounter this as they grow: the distribution of account sizes widens until the largest accounts are orders of magnitude bigger than the median, and every query written for the median becomes a problem for them.

The transition point is predictable and rarely anticipated — the design that served a uniform customer base does not survive a heavy-tailed one, and the symptom is always tail latency before it is anything else.

Failure scenarios

• Treating the tail as variance, optimising broadly and improving nothing.
• Optimising the shared path for the heavy case, degrading the common one.
• No segmentation in metrics, so the pattern is invisible.
• Aggregate SLIs, which stay green while the largest customers are unhappy.
• Synchronous APIs for unbounded work, which cannot be made fast and should have been asynchronous.

Trade-offs

A separate path is a second implementation to maintain, with the risk that it diverges from the primary one. Dedicated capacity costs money and reduces the density that makes multi-tenancy economic.

The counter-argument is that the alternative is a shared path optimised for nobody, and that the heavy segment's cost is usually justified by its revenue. The discipline is to keep the divergence minimal — a different index or a precomputed view rather than a different codebase — and to move a tenant back to the shared path if their profile changes.

Interview question

"Your p99 is bad and your p50 is excellent. Before you optimise anything, what would you measure, and what would you do differently if the slow requests all came from your ten largest customers?"