A multi-tenant SaaS platform cannot tell which customer caused a performance problem. What must be present in the telemetry, and what does that cost?
Show the full answer Hide the answer
What must be present
Tenant identity on every telemetry signal, propagated through the whole request path including asynchronous work. Without it, a platform experiencing degradation can see that it is degraded and not who is causing it or who is affected — which makes both mitigation and prevention guesswork.
Specifically:
- Tenant in the trace context, propagated across service boundaries and into queued work. Background jobs are where attribution is usually lost, and background jobs are frequently the cause.
- Per-tenant resource consumption: query time, connection usage, storage, job execution time, event volume. This is what identifies a noisy neighbour before it becomes an incident.
- Per-tenant SLI, so the platform can answer "is this customer having a good experience", which aggregate metrics cannot — a platform at 99.9% overall can be at 90% for one large customer and the average hides it.
What it costs
Tenant is a high-cardinality dimension, and putting it on every metric multiplies series count by the tenant count. At thousands of tenants that is exactly the cost blowout that cardinality budgets exist to prevent.
The resolution
- Tenant on traces and logs, where high cardinality is native and expected. Traces are sampled and logs are not indexed by default, so the cardinality is affordable there.
- Tenant on a deliberately small set of metrics only — the handful that drive quota enforcement and noisy neighbour detection — rather than on everything.
- Aggregate metrics for the top N tenants explicitly, plus an "other" bucket. This gives per-tenant visibility where it matters and bounds the series count, and it works because the distribution is heavily skewed.
- Exemplars: metrics that link to representative traces, so an aggregate metric can be drilled into without carrying the cardinality itself. This is the mechanism that resolves most of the tension.
The organisational payoff
Per-tenant attribution is what makes quotas enforceable and what turns a conversation with a customer about their usage from an argument into a data exercise. It is also what allows tiered isolation decisions — knowing which tenants should move to dedicated infrastructure requires knowing what they consume, and most platforms cannot answer that when the question first arises.