advanced 2 min answer

In a multi-tenant platform, aggregate application performance metrics look healthy while specific tenants experience severe slowness. What must performance monitoring do differently?

apmmulti-tenancysegmentationoutlierssalesforcedebugging
Show the full answer Hide the answer

Why aggregates lie in multi-tenant systems

Tenants differ by orders of magnitude in data volume, query complexity and usage patterns. The aggregate is dominated by the many small tenants, whose queries are fast because their datasets are small. A large tenant whose every query is slow contributes a negligible fraction of the request count and a large fraction of the business impact.

The same query, against the same schema, can be fast for 95% of tenants and pathological for one — because the query plan was chosen from statistics describing the aggregate of all tenants, and no individual tenant resembles the aggregate.

What monitoring must do differently

1. Percentiles per tenant, not only per endpoint. The metric that matters is "the worst experience any tenant is having", which requires per-tenant aggregation and a leaderboard of the worst.

2. Alert on outliers, not on aggregates. "Any tenant whose p95 exceeds N" catches the case an aggregate threshold never will. This inverts the usual alerting design and is the single most important change.

3. Normalise for tenant size. A large tenant's queries being slower is expected; the signal is slower than comparable tenants of similar size, or slower than that tenant's own baseline. Absolute thresholds generate noise for large tenants and miss regressions for small ones.

4. Attribute resource consumption per tenant — database time, cache occupancy, queue time, connection usage. This answers both "who is affected" and "who is causing it", which are frequently different tenants and require different responses.

5. Capture query plans and slow queries with tenant context. In shared-table architectures the plan is the usual culprit, and a slow query log without the tenant identifier cannot be acted on.

6. Track saturation of shared resources per tenant — connection pool share, worker pool share, cache hit rate. Noisy-neighbour problems are invisible in aggregates by construction.

The architectural consequence

Consistent per-tenant outlier findings are a tenancy signal, not just a performance one. Beyond a certain size, sharing is a losing proposition regardless of tuning: the largest tenants pay a design cost imposed by the smallest.

The architecture should include an explicit promotion path — dedicated schema, database, or cell — with a measured threshold for when a tenant crosses it. Discovering that need during an incident, with no path prepared, is the common and avoidable outcome.