Seven published cross-tenant incidents, from Steam's Christmas cache to ChatGPT's Redis client, and not one of them defeated the authorization layer. This guide reconstructs where multi-tenant isolation actually breaks, how Salesforce, Shopify, GitLab, AWS and Cloudflare place their tenants, and which tests would have caught each published leak.
One system, many customers, and a promise that each customer behaves as if alone. Where that promise is made, where it is broken, and who has written down what happened.
Strip the technology names away and the problem is this: for economic reasons, many customers share one running system, and the operator promises each of them that nothing they store or do will ever reach another. The revenue side of that bargain is old and well understood; Aulbach, Grust, Jacobs, Kemper and Rittinger opened their SIGMOD 2008 paper with it, noting that "multiple tenants are often consolidated into the same database to reduce total cost of ownership" [24]. The risk side is what this guide is about, and it is best learned from the operators who have already paid it: Valve, Cloudflare, GitHub, OpenAI, Microsoft and Atlassian have each published an account of a day the promise failed.
Read those accounts together and a pattern appears that should reorganise how you spend
an isolation budget. In every published incident in this corpus, the layer whose job is
tenant scoping, the WHERE tenant_id = ? clause, the session check, the
org-scoped query layer, worked exactly as designed. The data crossed between customers
somewhere else: in an edge cache keyed only by URL, in a proxy buffer, in a reused
request object, in a pooled Redis connection, in a shared build VM, in an operator's
deletion script. The layers that leak are the ones that have no concept of a tenant at
all. That is the surprise of this dig, and sections 2 and 4 are built around it.
Scope. This guide covers data and performance isolation between customers of one product: where the tenant boundary lives, how identity travels with a request, and what broke in the published record. It deliberately does not cover network segmentation between corporate environments, Kubernetes cluster multi-tenancy for internal teams, or the adjacent-but-different problem of per-tenant encryption key management. Cell-based availability partitioning appears only where it doubles as a tenancy boundary; the 2026-08-29 guide in this collection covers it as a blast-radius mechanism.
One enforcement point, six layers that can undo it. The reference shape below is reconstructed from the systems that published both their designs and their failures.
Every multi-tenant system in this corpus has the same skeleton: requests arrive carrying an identity, the identity is resolved to a tenant early, and a scoping mechanism applies that tenant to every data access. The mechanism varies, and section 3 treats the choice, but the variation matters less than something the incident record makes vivid: the request crosses far more layers than the one that enforces scoping. An edge cache, a proxy with shared buffers, an application server that reuses request objects, a connection pool whose connections outlive requests, a shared cache, a job queue whose workers inherit ambient state. Each of those layers multiplexes tenants, and most of them address their contents with keys that say nothing about tenancy.
The divergence points are at the edges of that skeleton. At one extreme, Salesforce's core platform is the maximal pool: Craig Weissman's QCon SF presentation describes one shared database and one application stack for all tenants, every physical row carrying the org identifier, with a metadata-driven runtime materialising each tenant's virtual schema from shared tables [22]. The 2008 whitepaper put the count at fifty-five thousand organisations on that shared design [27]. At the other extreme, Shopify runs pods: "an isolated instance of Shopify consisting of an individual MySQL database shard, along with other datastores like Redis and Memcached", each holding a unique subset of shops [12]. GitLab's Cells design document is the most explicit recent statement of the silo position: "Organizations serve as logical boundaries and Cells serve as physical boundaries", with cross-organization operations forced through public APIs [11].
Both ends work in production at enormous scale, which is itself informative: the spectrum position is not what decides whether you leak. What the pool end buys is unit economics and uniform operations; what the silo end buys is bounded blast radius and a story you can tell a regulated customer. What neither end buys is safety in the tenant-blind middle layers, because those layers exist in both designs. Shopify's pods still share job workers, app servers and load balancers outside the pod boundary, with the constraint that shared resources talk to one pod at a time; Salesforce's pool still terminates every tenant's traffic in shared infrastructure.
Application-level scoping on every query, or a database policy. The Postgres
version: enable row-level security and a policy comparing
tenant_id to a per-request setting, bound with SET LOCAL
inside the transaction so it cannot survive connection reuse.
Runs this way at: Crunchy Data pattern, Supabase
Caches, pools, proxies and reused request objects. None of them re-checks identity; each is safe only if its addressing key carries the full identity context or its state is destroyed between tenants. Section 4 shows one published failure per layer.
Failed at: Valve, Cloudflare, GitHub, OpenAI
Provisioning, deletion, backup and restore tooling that operates across tenants by design. Its tenant-awareness must extend to recovery: a backup of intermingled tenants is not a per-tenant restore capability, as Atlassian's fourteen-day restore demonstrated.
Four forks, each with a published system on both sides and a condition that flips the answer.
SET for RLS context: survives connection reuse under a pooler.| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Data placement | Shared tables + tenant column | Schema- or database-per-tenant at scale | Catalog and connection overhead grow with tenant count | PlanetScale, 2025; pgsql list, 2014 |
| Filter enforcement | App scoping + RLS | Either alone | Independent failure modes; policy catches the forgotten filter | Crunchy Data, 2023 |
| Context propagation | Explicit, per unit of work | Ambient thread/session state | Pools, queues and cancellation let ambient state outlive its request | acts_as_tenant #141; GitHub, 2021 |
| Untrusted code | MicroVM (Lambda) or isolate + mitigations (Workers) | Shared-kernel containers; process-per-tenant | Containers judged too weak; processes too expensive at density | Firecracker, NSDI '20; Workers security model |
| Shared workers | Shuffle-sharded assignment | Dedicated workers per tenant; fully shared fleet | Random small subsets make full overlap between two tenants combinatorially rare | AWS Builders' Library, 2019 |
Each decision is really the same question at a different layer: does tenant identity travel with the data, or is it ambient? Ambient identity is cheaper everywhere, and every published failure in section 4 is a place where ambient identity met a component that outlived the request it was ambient in.
Seven published incidents in three classes. Note what is absent: not one is a defeated authorization check.
Class 1: the tenant-blind shared layer. Four incidents, eight years apart, in four different components, with one mechanism: a layer that multiplexes tenants addressed its contents with less than the full identity context. Valve's emergency cache rule cached authenticated responses by URL. Cloudflare's parser stepped past a buffer and returned whatever neighbouring tenant's traffic sat in memory. GitHub's Unicorn server reuses a single env hash per worker, and a thread-safety bug in exception logging let one request's session reach another's response. OpenAI's redis-py client, after an asyncio cancellation, left an unread reply on a pooled connection so that the next request on that connection received the previous user's data. The component names could not be more different; the design failure is identical.
Class 2: the feature that crossed the boundary. Wiz's ChaosDB and Orca's AutoWarp are researcher discoveries rather than exploited incidents, but they are production isolation failures all the same. Cosmos DB bundled a Jupyter notebook feature, auto-enabled from 2019, whose container could reach shared infrastructure and read other customers' primary keys; Wiz reported full read, write and delete access to other tenants' databases. Azure Automation ran different tenants' jobs on the same VM, and an internal endpoint on sequential local ports handed out other tenants' managed-identity tokens. In both cases a mature, well-isolated data plane acquired a new compute feature that silently inherited the weakest isolation in the stack. The transfer is direct: every feature that executes anything on shared infrastructure re-opens the isolation question, and the review has to happen per feature, not per platform.
Class 3: the control plane that saw no boundaries. Atlassian's April 2022 outage leaked nothing, and belongs here anyway. A cleanup script offering both "mark for deletion" and "permanently delete" modes was run with the wrong mode and the wrong ID list, deleting 883 sites across 775 customers in 23 minutes. The recovery is the architectural lesson: backups existed, restore drills had passed, but restoring a subset of tenants out of shared databases meant standing up staging clusters and surgically extracting rows, in batches Gergely Orosz reported at up to about sixty tenants taking four to five days each; full recovery took two weeks. The data plane was multi-tenant; the restore path was effectively single-tenant-at-a-time.
The plain missing-filter bug, an endpoint that forgets the tenant clause, is almost absent from this record: it lives in bug-bounty writeups and disclosure notes rather than operator postmortems. Read that absence carefully. It does not mean the class is rare; it means it is usually caught quietly, reported by a researcher, or never detected, because a correct-looking response to the wrong tenant fires no alert. Detection for this class has to be built (rung 1 of section 7), not waited for.
What the published record supports. Every figure carries its context and date; leak rates in particular are properties of specific bugs, not of architectures.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Users served another user's page | ~34,000 | Valve / Steam | Emergency cache rule live about 90 minutes | 2015-12 | Valve statement |
| Requests leaking memory, peak window | 1 in 3.3M | Cloudflare | 2017-02-13 to 2017-02-18; leak had run since 2016-09-22 | 2017-02 | postmortem |
| Authenticated sessions misrouted | <0.001% | GitHub | Under two weeks of cumulative exposure; all sessions revoked | 2021-03 | postmortem |
| Plus subscribers with billing details exposed | 1.2% | OpenAI | Active users in a nine-hour window, 2023-03-20 | 2023-03 | postmortem |
| Sites deleted by one script run | 883 | Atlassian | 775 customers, 23 minutes; restores ran 2022-04-08 to 2022-04-18 | 2022-04 | PIR |
| Per-tenant restore throughput | ~60 / 4-5 days | Atlassian | Reported batch size and elapsed time per batch during recovery | 2022-04 | Pragmatic Engineer |
| Migration across all tenant schemas | ~2 hours | pgsql list report | Schema-per-tenant Postgres, thousands of schemata, catalog queries repeated per tenant | 2014-11 | thread |
| Organisations on one shared pool design | 55,000+ | Salesforce | Vendor figure for the whole platform in the 2008 whitepaper; per-instance counts not published | 2008 | whitepaper |
| Guest memory ceiling per isolate | low MB | Cloudflare Workers | "Each guest cannot take more than a couple megabytes of memory"; thousands of guests per machine | 2025 | security model |
| CPU cost of process-per-tenant | ~10× | Cloudflare Workers | Vendor comparison of strict process isolation against shared-process isolates | 2025 | security model |
| Events served by one multi-tenant query engine | ~100T | Datadog | Husky event store; reader isolation against noisy neighbours is a stated design goal | 2025-10 | Husky posts |
| Distinct 2-worker shuffle shards from 8 workers | 28 | AWS | The combinatorial basis for noisy-neighbour containment without dedicated hardware | 2019 | Builders' Library |
The incident rates are measured by the operators involved; the Salesforce tenant count, the Workers overhead figures and the shuffle-shard arithmetic are vendor statements serving an argument. Two quantities nobody publishes: the cost delta of silo versus pool at a stated tenant count, and the base rate of missing-filter bugs. Estimates of either in vendor decks are unsourced; treat them as sales material.
Every source behind this page, graded. Filter by kind. The ledger with
per-claim quotes ships alongside as sources.md.
The redis-py cancellation bug as experienced by its most famous victim: chat titles across users, and billing details of 1.2% of active Plus subscribers in a nine-hour window.
The mechanism in reproducible form, three days before OpenAI's postmortem: cancel between send and receive and every later response on the connection is off by one.
The advisory records that the first fixed versions were "believed to be incomplete", with CVE-2023-28859 assigned to the remainder: even the patch for a cross-tenant bug needed a second pass.
The fullest public account of a shared-proxy memory leak: rates, window, the equality-check root cause, and the cleanup problem of leaked data cached by crawlers.
An unusually candid walk from symptom to root cause: Unicorn's single reused env hash, a thread-unsafe exception logger, and sessions misrouted at under 0.001%.
The definitive account of tenant blast radius in the control plane: wrong mode, wrong IDs, 883 sites gone, and a restore path that had never been exercised for a subset of tenants at once.
The independent account, sourced from affected customers and insiders while the incident ran; supplies the restore batch sizes and elapsed times the PIR leaves vague.
The operator statement, reproduced in full by the press: a DoS mitigation deployed a caching configuration that "incorrectly cached web traffic for authenticated users" for about 34,000 people.
Escape from an auto-enabled notebook container to shared Cosmos DB infrastructure and other customers' primary keys; the canonical feature-crosses-the-boundary case.
Different tenants' automation jobs on one VM, and managed-identity tokens served by port number rather than by requester identity. Reported, fixed in four days, disclosed three months later.
A rare public record of an operator re-partitioning a monolithic SaaS around a tenancy unit: Organizations as logical boundary, Cells as physical, cross-cell access only via public APIs.
The silo position argued from operations: full isolated instances per shop subset, chosen to bound database incidents and noisy neighbours to a fraction of merchants.
Tenant placement as a continuous process: binlog-streamed shop moves between shards, because a static assignment of tenants to silos decays into hot spots.
The reference RLS pattern: policy on tenant_id against a per-request
setting, with the transaction-scoped binding that survives connection pooling.
What the policy costs and how to pay less: index the policy columns, wrap volatile calls in scalar subqueries so they evaluate once, and repeat the filter in the query for the planner.
A practitioner running schema-per-tenant reports catalog operations collapsing at thousands of schemas, with one all-tenant migration taking close to two hours.
A current, vendor-adjacent but concrete comparison of shared-schema, schema-per-tenant and database-per-tenant, honest about the connection-model problem of the last.
The rejected PR this guide was required to find, and it earns its place: worker threads persisting the previous job's tenant, a maintainer acknowledging the fragility, and no merged fix.
Performance isolation as an architecture driver: compute separated from storage partly so the query path can be isolated per workload, against poison-pill queries and noisy neighbours, at around 100 trillion events.
The written-down reasoning for moving Lambda's tenant boundary to hardware virtualization: containers judged insufficient for untrusted code at multi-tenant density, so the VMM was shrunk until VMs were cheap.
The opposite bet, documented: isolates confining thousands of guests per machine, timers frozen and shared memory removed against Spectre, and suspicious workloads dynamically evicted into their own process.
The chief architect explaining the maximal pool from the inside: one schema for all tenants, org identifier on every row, and a metadata runtime in place of per-tenant DDL.
The density argument for isolates over containers and VMs, from the platform's architect, with the security trade-offs stated rather than waved away.
The paper that framed pooling as a cost decision and catalogued the schema-mapping price of it; still the clearest statement of why the middle of the spectrum is hard.
The silo / pool / bridge vocabulary the industry now speaks, with the honest framing that pooling "adds a level of complexity to the isolation story".
The combinatorial trick for noisy-neighbour containment on shared fleets: give each tenant a random small worker subset, and two tenants sharing all workers becomes vanishingly unlikely.
The written companion to Weissman's talk: metadata-driven virtual schemas over shared physical tables, and the platform-wide tenant count of the era.
Seven rungs. The first three take an evening each; the line from toy to production-shaped is crossed at rung four, where the bugs from section 4 become reproducible.
Build the smallest CRUD app with shared tables and a tenant_id column.
Then write the test that matters: for every endpoint, authenticate as tenant B and
request tenant A's object ids, expecting 404s. Delete one filter to prove the suite
fails the build.
Done when: a seeded missing-filter bug cannot pass CI. Teaches: cross-tenant denial is a tested property, not a code-review habit; this is the detection the "no postmortem" class never gets.
Enable RLS with a policy on current_setting('app.tenant_id'), bound via
SET LOCAL in a transaction per request, following the Crunchy Data
pattern. Re-seed the rung-1 bug and watch the policy catch what the application
missed.
Done when: the seeded bug returns zero rows instead of another tenant's rows. Teaches: two enforcement layers with independent failure modes, and the fail-closed default.
Add PgBouncer in transaction mode. Change the rung-2 binding from
SET LOCAL to session-level SET and run concurrent requests
for both tenants until one reads the other's rows; then change it back.
Done when: you can produce the leak on demand with SET and cannot with SET LOCAL. Teaches: the acts_as_tenant and GitHub failure shape: ambient identity dies at any boundary where execution outlives the request.
Cache rendered responses in Redis. Write a fuzzer that replays each tenant's authenticated URLs as the other tenant and diffs the bodies. Then reproduce Steam's incident deliberately: cache one authenticated route keyed by URL alone.
Done when: the fuzzer catches the deliberately broken key and passes on (tenant, URL) keys. Teaches: the cache key is part of the authorization model, and the fuzzer is cheap enough to run in CI forever.
Drive the app with a load generator that cancels a percentage of in-flight requests. Use an async Redis client and watch for cross-request responses (redis-py <4.5.3 reproduces the historical bug faithfully; current clients should survive). Add an application-level check that cached values embed the tenant id they belong to.
Done when: one hour at 5% cancellation yields zero wrong-tenant reads, verified by the embedded ids. Teaches: cancellation as an adversary, and reconciliation as the safety net the client library cannot provide.
Take scheduled backups of the shared database, then run Atlassian's drill in miniature: delete tenant A at a known time and restore A alone, to that point, while tenant B keeps writing with no downtime.
Done when: tenant A is back within an hour, tenant B's writes are untouched, and the runbook is written down. Teaches: the gap between having backups and having per-tenant restore, which is measured in days at production scale.
Give tenant B a pathological workload (unindexed scans, hot loops). Measure tenant A's p99 before and after per-tenant connection quotas, statement timeouts, and a shuffle-sharded worker assignment across four app processes.
Done when: B saturates its shard and A's p99 stays within 20% of baseline. Teaches: performance isolation as its own discipline, separate from data isolation, per Datadog's Husky design and the shuffle-sharding argument.
The queries that found this material, grouped by what they surface. The vocabulary is the value: "cross-tenant", "noisy neighbour", "silo pool bridge" and "shuffle sharding" each unlock a literature that "multi-tenancy best practices" never reaches.
"cross-tenant" vulnerability postmortem OR "incident report""saw another user's" OR "generated for other users" incident"incorrectly cached web traffic for authenticated users"site:wiz.io OR site:orca.security cross-tenant isolation researchrepo:redis/redis-py cancel connection unsafe state"wrong tenant" sidekiq OR celery OR worker site:github.comis:pr is:closed is:unmerged tenant leak thread"race condition" session "returned to the pool"gitlab cells design document organization isolation"tenant isolation" silo pool bridge whitepaper"pods architecture" OR "cell-based" shard blast radius engineering"shuffle sharding" noisy neighbor workload isolation"row level security" multi-tenant performance current_setting index"schema per tenant" migrations slow information_schema thousands"database per tenant" connection model postgres trade-offspgbouncer "SET LOCAL" tenant rls transaction poolingAll links were live on 2026-09-18. This session's research environment
could fetch github.com and gitlab.com directly; other pages were read through a search
layer, and the evidence ledger (sources.md) marks which quotes are
near-verbatim on that account.