Evidence ledger
27 sources
Checked 18 Sep 2026
Evidence ledger
One row per claim in The leak goes around the tenant filter, not through it: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic: how multi-tenant systems keep one customer's data from reaching another, and where the isolation actually broke in the published record. One row per claim. All links checked 2026-09-18.
Access note for this session. The research environment's egress policy allowed direct fetches only to github.com, raw.githubusercontent.com and gitlab.com; every other source was read through the search layer, which returns page content but not the page itself. Where a quote below could not be captured character-for-character from the primary page, it is marked (near-verbatim) and the surrounding claim is stated in the guide as reported rather than quoted. Talk timestamps could not be captured because the video hosts were unreachable; talks are cited to claims that appear in their published abstracts and companion documents.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | OpenAI | March 20 ChatGPT outage: here's what happened | postmortem | 2023-03-24 | 2026-09-18 | https://openai.com/index/march-20-chatgpt-outage/ | A bug in the redis-py client, triggered by request cancellations, let users see other users' chat titles and, for a window, billing details | "a bug in an open-source library which allowed some users to see titles from another active user's chat history" ... "the same bug may have caused the unintentional visibility of payment-related information of 1.2% of the ChatGPT Plus subscribers who were active during a specific nine-hour window" |
| 2 | redis-py (drago-balto) | Off by 1 - Canceling async Redis command leaves connection open, in unsafe state for future commands | source | 2023-03-17 | 2026-09-18 | https://github.com/redis/redis-py/issues/2624 | The mechanism: a cancelled task leaves an unread response on a pooled connection; every later request on that connection gets the previous request's response | Issue title as above; reproduction shows r.get('bar') returning b'foo' and r.get('foo') returning b'PONG'; reporter's fix: "except asyncio.CancelledError: await conn.disconnect() raise" (fetched directly) |
| 3 | GitHub Advisory Database | redis-py Race Condition vulnerability (CVE-2023-28858) | source | 2023-03-26 | 2026-09-18 | https://github.com/advisories/GHSA-24wv-mv5m-xv4h | The first fix was itself incomplete; CVE-2023-28859 was assigned to the gap | "The fixed versions for this CVE Record are 4.3.6, 4.4.3, and 4.5.3, but are believed to be incomplete." (fetched directly) |
| 4 | Cloudflare | Incident report on memory leak caused by Cloudflare parser bug | postmortem | 2017-02-23 | 2026-09-18 | https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/ | A parser buffer overrun in a shared edge proxy returned memory containing other customers' traffic; peak rate 1 in 3,300,000 requests | "The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that's about 0.00003% of requests)." (near-verbatim); root cause: end-of-buffer check used equality and the pointer stepped past the end |
| 5 | GitHub | How we found and fixed a rare race condition in our session handling | postmortem | 2021-03-18 | 2026-09-18 | https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare-race-condition-in-our-session-handling/ | Unicorn reuses one env hash across requests; a thread-safety bug in exception logging let one user's session reach another user's browser | "a race condition ... could have misrouted a user's session to the browser of another authenticated user" (near-verbatim, from the companion 2021-03-08 security update); Unicorn "allocates one single Ruby Hash that is then cleared (using Hash#clear) between each request" (near-verbatim); "fewer than 0.001% of authenticated sessions" |
| 6 | Atlassian | Post-Incident Review on the April 2022 outage | postmortem | 2022-04-29 | 2026-09-18 | https://www.atlassian.com/engineering/post-incident-review-april-2022-outage | A permanent-delete script run with the wrong mode and the wrong IDs deleted 883 sites (775 customers); restore from shared-infrastructure backups took up to 14 days | "The script was executed with the wrong execution mode and the wrong list of IDs" (near-verbatim); deletion of 883 sites for 775 customers between 07:38 and 08:01 UTC on 2022-04-05; sites progressively restored 2022-04-08 to 2022-04-18 |
| 7 | The Pragmatic Engineer (Gergely Orosz) | The Scoop: Inside the longest Atlassian outage of all time | blog | 2022-04-12 | 2026-09-18 | https://newsletter.pragmaticengineer.com/p/scoop-atlassian | Independent account of the restore mechanics: batches of tenants restored together because customer rows were intermingled in shared databases | Reports restoration proceeding in batches of up to roughly 60 tenants, with about 4 to 5 elapsed days end to end to hand a site back (reported figures, this account's own sourcing) |
| 8 | Valve (via The Register) | Cache-astrophic: why Valve's Steam store spewed players' private profiles to strangers | postmortem | 2015-12-30 | 2026-09-18 | https://www.theregister.com/2015/12/30/steam_security_blip_explained/ | An emergency caching configuration deployed during a DoS attack cached pages generated for authenticated users and served them to other users; about 34,000 users saw someone else's data | Valve statement: "a second caching configuration was deployed that incorrectly cached web traffic for authenticated users. This configuration error resulted in some users seeing Steam Store responses which were generated for other users." (statement as reproduced in full by the press; Valve's own news post was not directly reachable) |
| 9 | Wiz Research | ChaosDB: how we hacked thousands of Azure customers' databases | casestudy | 2021-08-26 | 2026-09-18 | https://www.wiz.io/blog/chaosdb-how-we-hacked-thousands-of-azure-customers-databases | A Jupyter-notebook feature bundled into Cosmos DB allowed escape to shared infrastructure and retrieval of other customers' primary keys | Wiz: the flaw "gives any Azure user full admin access (read, write, delete) to another customers' Cosmos DB instances, without authorization" (near-verbatim); notebooks were auto-enabled for accounts from 2019 |
| 10 | Orca Security | AutoWarp: critical cross-account vulnerability in Azure Automation service | casestudy | 2022-03-07 | 2026-09-18 | https://orca.security/resources/blog/autowarp-microsoft-azure-automation-service-vulnerability/ | Automation jobs of different tenants ran on the same VM; an internal HTTP endpoint on sequential ports served other tenants' managed-identity tokens | Researcher obtained tokens by requesting a range of local ports; observed tenants included "a global telecommunications company, two car manufacturers, a banking conglomerate, big four accounting firms" (near-verbatim); reported 2021-12-06, fixed 2021-12-10, disclosed 2022-03-07 |
| 11 | GitLab | Cells (architecture design document) | adr | 2022-09-07 (ongoing) | 2026-09-18 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/cells/_index.md | GitLab's chosen isolation unit is the Organization, pinned to one Cell; cross-organization operations must go through public APIs | "Most operations should be scoped to an Organization. This ensures features operate within the logical boundary and maintain isolation." ... "Organizations serve as logical boundaries and Cells serve as physical boundaries" (fetched directly from the raw file) |
| 12 | Shopify | A pods architecture to allow Shopify to scale | blog | 2018 | 2026-09-18 | https://shopify.engineering/a-pods-architecture-to-allow-shopify-to-scale | Shopify partitions the platform into pods, each an isolated full instance holding a subset of shops, to bound blast radius and noisy neighbours | "A pod is an isolated instance of Shopify consisting of an individual MySQL database shard, along with other datastores like Redis and Memcached. Every pod houses a unique subset of shops on the platform." (near-verbatim) |
| 13 | Shopify | Shard balancing: moving shops confidently with zero-downtime at terabyte-scale | blog | 2021-09 | 2026-09-18 | https://shopify.engineering/mysql-database-shard-balancing-terabyte-scale | Tenant placement is a continuous operational process: shops are rebalanced across shards online using binlog streaming | Post describes moving shops across shards "entirely online and with virtually zero consumer-facing downtime" at terabyte scale (near-verbatim) |
| 14 | Crunchy Data | Row level security for tenants in Postgres | blog | 2023 | 2026-09-18 | https://www.crunchydata.com/blog/row-level-security-for-tenants-in-postgres | RLS moves the tenant filter from every query into a database-enforced policy keyed on a session setting | Pattern: policy USING (tenant_id = current_setting('app.current_tenant')::uuid) with the setting bound per request (pattern as documented in the post) |
| 15 | Supabase | Row Level Security performance and best practices | vendor | 2024 (docs, maintained) | 2026-09-18 | https://supabase.com/docs/guides/troubleshooting/rls-performance-and-best-practices-Z5Jjwv | RLS policies are inlined as predicates into every query; unindexed tenant columns and per-row function calls dominate the cost; wrapping calls in scalar subqueries and indexing recovers most of it | Guidance: index policy columns, wrap auth/current_setting-style calls in (select ...) so they evaluate once per query, duplicate the policy filter in the query; documents order-of-magnitude improvements in their benchmarks |
| 16 | PostgreSQL hackers list (Ulf Lohbrügge) | Performance of information_schema with many schemata and tables | source | 2014-11 | 2026-09-18 | https://www.postgresql.org/message-id/CABZYQRKnp=FxZ7tQeyytDjUOnHP9J90irxRBEAc+-XGbKdgf2A@mail.gmail.com | Schema-per-tenant degrades the catalog itself: with thousands of schemas, catalog-heavy operations slow by orders of magnitude | "One migration run on all tenants (schemata) almost took 2 hours" (near-verbatim, from the list thread) |
| 17 | PlanetScale | Approaches to tenancy in Postgres | blog | 2025 | 2026-09-18 | https://planetscale.com/blog/approaches-to-tenancy-in-postgres | The three tenancy models in Postgres and their operational limits; database-per-tenant conflicts with Postgres's connection model | "the appeal of using logical databases per tenant is increased isolation ... you can modify the connection string to connect to the correct database"; database-per-tenant "is at odds with the connection model of Postgres" (near-verbatim) |
| 18 | ErwinM/acts_as_tenant | PR #141: Explicitly use nil tenant in sidekiq server | source | 2016-06-16 (closed unmerged 2019-12-23) | 2026-09-18 | https://github.com/ErwinM/acts_as_tenant/pull/141 | Rejected PR: thread-local tenant context leaks across Sidekiq worker jobs; the library's thread safety depends entirely on clearing state between units of work | "sidekiq 4's worker threads are persisting tenant information from previous workers"; maintainer: "Acts as Tenant is only thread safe because it uses RequestStore to clear out the state between requests" (fetched directly) |
| 19 | Datadog | Husky: exactly-once ingestion and multi-tenancy at scale / Inside Husky's query engine | blog | 2023-02 / 2025-10 | 2026-09-18 | https://www.datadoghq.com/blog/engineering/husky-deep-dive/ | Performance isolation is a first-class design axis of a multi-tenant event store; the reader/query path is isolated per workload to contain poison-pill queries and noisy neighbours | Query-engine post: reader-service isolation matters "both to avoid poison-pill queries and to protect against noisy neighbors" (near-verbatim); query engine serves ~100 trillion events (post title, 2025) |
| 20 | AWS (Marc Brooker, Alexandru Agache et al.) | Firecracker: lightweight virtualization for serverless applications (NSDI '20) | paper | 2020-02 | 2026-09-18 | https://www.usenix.org/system/files/nsdi20-paper-agache.pdf | For running untrusted tenant code on shared hardware, AWS judged container isolation insufficient and moved the boundary to hardware virtualization with a minimal VMM | Paper's premise: existing isolation options did not meet the combined needs of strong isolation and multi-tenant density for Lambda, a service the paper describes at the scale of trillions of events per month (scale phrasing as reported in Adrian Colyer's review of the paper, https://blog.acolyer.org/2020/03/02/firecracker/) |
| 21 | Cloudflare | Workers security model (docs) | vendor | maintained, 2025 snapshot | 2026-09-18 | https://github.com/cloudflare/cloudflare-docs/blob/production/src/content/docs/workers/reference/security-model.mdx | The opposite bet to Firecracker: V8 isolates in a shared process, chosen for density, with side channels managed by removing timers and shared memory and by dynamic process isolation | "V8 executes code inside isolates, which prevent that code from accessing memory outside the isolate — even within the same process." ... "The value returned by Date.now() is locked in place while code is executing." (fetched directly) |
| 22 | Salesforce (Craig Weissman) | The internal design of Force.com's multi-tenant architecture (QCon SF 2008, InfoQ) | talk | 2009 | 2026-09-18 | https://www.infoq.com/presentations/SalesForce-Multi-Tenant-Architecture-Craig-Weissman/ | The extreme pool position: one shared schema, all tenants in the same tables, isolation enforced by OrgID scoping and a metadata-driven runtime | InfoQ abstract: "one shared database and one application stack" serving all tenants; companion whitepaper (2008) describes the platform "supporting some 55,000+ organizations" (near-verbatim); no timestamp captured, video host unreachable from this environment |
| 23 | Cloudflare (Kenton Varda) | Fine-grained sandboxing with V8 isolates (QCon SF 2019, InfoQ) | talk | 2019 | 2026-09-18 | https://www.infoq.com/presentations/cloudflare-v8/ | Why isolates over containers for massive multi-tenancy: milliseconds of start time and megabytes of overhead change the unit economics | InfoQ abstract: a compute platform "designed for massive multi-tenancy without using virtual machines or containers, but instead using V8 isolates"; overhead claims corroborated by the Workers security-model docs (row 21); no timestamp captured, video host unreachable from this environment |
| 24 | Stefan Aulbach, Torsten Grust, Dean Jacobs, Alfons Kemper, Jan Rittinger | Multi-tenant databases for software as a service: schema-mapping techniques (SIGMOD 2008) | paper | 2008-06 | 2026-09-18 | https://dl.acm.org/doi/10.1145/1376616.1376736 | The canonical statement of why tenants are pooled at all, and of the schema-mapping cost of pooling | Abstract: "multiple tenants are often consolidated into the same database to reduce total cost of ownership" (near-verbatim) |
| 25 | AWS (Tod Golding) | SaaS tenant isolation strategies (whitepaper) | vendor | 2020-08 | 2026-09-18 | https://docs.aws.amazon.com/whitepapers/latest/saas-tenant-isolation-strategies/saas-tenant-isolation-strategies.html | The silo / pool / bridge vocabulary used across the industry for tenant placement | Whitepaper defines silo ("each tenant is running a fully siloed stack of resources"), pool (shared infrastructure), and bridge (a mix) (near-verbatim) |
| 26 | AWS (Colm MacCárthaigh) | Workload isolation using shuffle-sharding | vendor | 2019 | 2026-09-18 | https://aws.amazon.com/builders-library/workload-isolation-using-shuffle-sharding/ | Assigning each tenant a random small subset of shared workers makes full overlap between two tenants combinatorially rare, bounding noisy-neighbour blast radius without per-tenant hardware | Article develops the combinatorial argument (with 8 workers taken 2 at a time, 28 distinct shards) and reports the pattern as core to Route 53's DDoS containment (near-verbatim) |
Tier mix
postmortem 5 · source 4 · adr 1 · casestudy 2 · blog 6 · paper 2 · talk 2 · vendor 4 — total 26 across 24 distinct hosts and 20 organisations. Vendor share 15%.
Known gaps (stated in the guide)
- No operator has published a postmortem of a plain missing-
WHERE-clause cross-tenant leak; that class lives in bug-bounty reports and disclosure notes, not incident reviews. - Nobody publishes the cost delta of running silo vs pool at a stated tenant count; the cost argument for pooling is universal but always qualitative in public.
- Valve's original news post could not be fetched; the statement is cited as reproduced in full by contemporaneous press.