Swapping the engine under the client: ten years of Pinterest, read from its own repositories

A decade of one company's data platform read from artefacts rather than announcements, built to answer a single question: what actually decides whether replacing a storage or messaging engine is a migration or a rewrite? A reader leaves with a decision tree for when to own the client interface, a failure catalogue in which every entry is a coordination failure rather than a storage failure, a six-rung ladder that ends by breaking a consumer mid-upload on purpose, and a repeatable method for dating any organisation's platform decisions from GitHub alone.

What you get out of it

• Every failure visible in the tracker is a coordination failure, not a storage failure: Secor's exactly-once claim is broken by consumer-group rebalancing in threads from 2020 and still open in 2022, and MemQ's answer was to delete group coordination from the design rather than patch the sink.
• The migration cost of an engine is set by how many build files name its client, which is why PSC, a library that offers 'seamless switching between PubSub backends without code changes', is the load-bearing component of the whole decade.
• Owning the client is not free: PSC cut tags on three major version lines within four days of each other in January 2025, so the abstraction that decoupled applications from the engine became its own compatibility surface.
• Retirement is a periodic sweep, not a decision: Terrapin and PinLater were archived on the same day in February 2018, while Rocksplicator carries an unmaintained notice above a present-tense description of 4,000 hosts and no archive flag at all.
• Pinterest publishes no postmortems and has no advisories filed against its own software, so the only security finding in the corpus is an open issue saying the fleet-wide restart platform installs a no-op authorization filter unconditionally.

Scope

Why this, now. Pinterest's archive register stops in December 2021 while its platform plainly kept changing, and the public fork of Ray now says development has moved to an internal GitHub, so the artefacts an outsider can date are shifting from what a company retires to what it forks.

What it does not cover. The recommendation and ranking stack, the image pipeline, the HBase estate, the cloud bill and all internal operational detail, along with every engineering blog, talk, paper and vendor account: this session's network policy resolved github.com and refused every other host, so the corpus is one host and the guide can establish what was built and when but never how the migrations turned out.