Outlier Alerting

Aggregate metrics answer "how is the system doing on average". In a population with orders-of-magnitude variation — tenants, regions, device classes, customer tiers — that average is dominated by the many small members, and the few large ones can be completely broken without moving it.

Outlier alerting inverts the question: rather than "is the aggregate above threshold", it asks "is any segment below its expectation".

Why it matters

In a multi-tenant platform, tenants differ enormously in data volume, query complexity and usage. The same query against the same schema can be fast for 95% of tenants and pathological for one — because the query plan was chosen from statistics describing the aggregate of all tenants, and no individual tenant resembles the aggregate.

Business impact correlates with tenant size, while request count correlates inversely with it. So the metric most likely to be affected is the one least represented in the aggregate.

Implementation patterns

• Per-segment percentiles, with a leaderboard of the worst performers rather than a single number.
• Alert on "any segment exceeding N", which catches what an aggregate threshold never will. This is the central inversion.
• Normalise for size. A large tenant's queries being slower is expected; the signal is slower than comparable tenants or slower than that tenant's own baseline. Absolute thresholds generate noise for large segments and miss regressions in small ones.
• Attribute resource consumption per segment — database time, cache occupancy, connection pool share, queue time. "Who is affected" and "who is causing it" are frequently different tenants requiring different responses.
• Capture slow queries and query plans with the segment identifier, since a slow query log without tenant context cannot be acted on in a shared-table architecture.
• Cap the alert volume — with thousands of segments, an alert per segment is unmanageable, so alert on the count of segments exceeding threshold, or on the worst few.

Industry example

Enterprise multi-tenant platforms are where this is unavoidable. A shared-table architecture is efficient and operationally simple, and it makes the largest tenants pay a design cost imposed by the smallest — plans chosen for aggregate statistics, indexes tuned for the common case, buffer cache dominated by whoever is busiest.

The monitoring consequence is that the platform's own dashboards will report health while its most valuable customers are experiencing severe degradation, and the discrepancy will be discovered through an account manager rather than through an alert.

Consistent per-tenant outlier findings are also a tenancy signal, not merely a performance one. Beyond a certain size, sharing is a losing proposition regardless of tuning, and the architecture should include an explicit promotion path — dedicated schema, database or cell — with a measured threshold for crossing it. Discovering that need during an incident, with no path prepared, is the common and avoidable outcome.

Failure scenarios

• Aggregate-only alerting, so concentrated failures are invisible.
• Absolute per-segment thresholds, noisy for large segments and blind for small ones.
• No per-segment attribution of shared resources, so noisy-neighbour effects cannot be diagnosed.
• Alert storms when many segments cross a threshold simultaneously during a general degradation.
• Segments defined too finely, producing statistically meaningless per-segment metrics.

Trade-offs

Per-segment metrics multiply cardinality, which is the primary driver of observability cost — so this practice is in direct tension with telemetry budgets and must be applied to the dimensions that matter rather than to every dimension available.

It also increases alerting surface and requires a policy for handling many simultaneous segment alerts. The usual resolution is to alert on aggregate plus the count of outliers, keeping the detail available for investigation rather than paging on it.

Interview question

"Your p99 latency dashboard is flat and your three largest customers are complaining. Explain how both can be true, and design the alert that would have caught it."