Pinterest described in 2015 how every object it stores carries a 64-bit identifier with the shard number packed into the high bits, so any object routes to its database without a lookup. Your analytics platform holds 40 TB of event tables for 900 internal tenants keyed by a random UUID, and an engineer proposes adopting the same trick for the warehouse. What does packing the tenant into the key actually buy on the analytical side, and where would copying Pinterest be a mistake?
Show the full answer Hide the answer
The situation they were in
Pinterest's problem was transactional routing. Given an object id, a request had to reach the one MySQL host holding it, with no directory service in the hot path and no second round trip. Their answer was to compose the id rather than look it up: 16 bits of shard number, 10 bits of type and 36 bits of local sequence inside one 64-bit integer, so the routing decision is a bit shift on a number the caller already has.
That is a solution to a problem an analytical platform does not have. A warehouse query does not route to a host; it scans. Copying the mechanic without restating the benefit is how this proposal goes wrong in review.
What packing the tenant into the key buys in a warehouse
The value is not routing, it is that the tenant becomes derivable from the key, so layout and lifecycle can be tenant-shaped without adding a column to every join. Four concrete gains:
- Pruning that survives joins. If tenant is recoverable from the surrogate key, the engine can filter on the key range rather than on a column carried through three joins, and per-file min and max statistics stay narrow.
- Deletion becomes a drop. A departing tenant under a contractual erasure deadline is a
partition drop rather than a
DELETEthat rewrites 40 TB and leaves the old files alive in snapshot history. - Attribution without instrumentation. Bytes scanned per partition is a per-tenant cost number you get from the engine's own statistics, with nothing to tag and nothing to maintain.
- Blast radius. A runaway query is bounded by the partitions its key range touches.
What it costs
The key becomes a contract you can never renegotiate. Two tenants merging after an acquisition means rewriting every row of both, because the tenant is inside the primary key rather than beside it. Splitting a tenant is the same. And a tenant that becomes 30% of the platform is now a single partition with no way to divide it, which is precisely the case Pinterest's design avoids: a logical shard can be moved between hosts, a partition value cannot be split.
Where copying it would be a mistake
- You usually do not own the key. Analytical keys arrive from source systems. Composing your own means maintaining a surrogate mapping for every source, which is a permanent cost against a one-off pruning gain.
- Skew. With 900 tenants where the top five hold roughly 80% of the data, tenant-as-partition gives you five partitions that are each larger than the other 895 combined. Pinterest sized a 16-bit shard space so that shards stay small and movable; a partition key has no equivalent escape.
- The cheaper 90%. Clustering or sorting on an ordinary
tenant_idcolumn delivers most of the pruning, costs one column, and can be changed next quarter.
The decision rule: derive tenancy from the key only when tenants are never merged or split,
tenant appears in the predicate of the large majority of queries, and per-tenant erasure is a
real obligation. Otherwise carry tenant_id as a column and make it the clustering key.
Common weak answers
- "It makes queries faster because there is no lookup." There was never a lookup. The analytical benefit is layout and lifecycle, not routing.
- "Use row-level security instead." That is an access control, not a layout. It stops the wrong person reading a row and does nothing about scan cost, erasure or noisy neighbours.
- "Pinterest runs at enormous scale so this must be right." They solved online routing for a sharded OLTP fleet. The transfer is the principle that addressing schemes are hard to change, not the bit layout.