Swapping the engine under the client: ten years of Pinterest, read from its own repositories

The territory

An organisation has to keep replacing the systems that hold and move its data, every few years, while several hundred applications keep running on top of them. What decides whether that is a migration or a rewrite?

archived repositories in the organisation, none pushed to after 2021

Pinterest archived listing, checked 2026-09

90%

claimed cost improvement of MemQ against the Kafka footprint it augments

MemQ README, undated claim

4,000+

hosts in the RocksDB serving tier the README describes in the present tense

rocksplicator README, undated

291

open pull requests on the published ingest agent, mostly unmerged security bumps

singer pull queue, checked 2026-09

Start with the thing that is not there. Pinterest's public archive contains fourteen repositories, and the most recent of them stopped receiving commits in December 2021. Read naively, that says the platform stopped changing five years ago. It did not. In the same period the organisation tagged a 1.0 of a pubsub system it wrote itself, shipped three parallel major versions of a client library, and filled the top of its repository listing with forks of Ray, StarRocks, Druid, Superset and Karpenter. The archive stopped being the place where change is recorded. The record moved to what gets forked.

That is the first finding and it is a method point before it is a Pinterest point. If you are dating another organisation's platform decisions from its archive notices, you are reading a register that organisations stop maintaining once they stop publishing their own systems. The signal that replaces it is the fork: a repository with "forked from" in the header and commits of its own is a company operating somebody else's engine in anger, and the date it appears is the date the build-versus-adopt argument was lost.

The second finding is the shape of the decade. Between 2016 and 2019 Pinterest published the systems it ran: a batch-serving store on HDFS, a job queue on MySQL, a service-discovery toolkit on ZooKeeper, a workflow manager, MySQL fleet tooling, a RocksDB replication library. Between 2018 and 2021 it retired most of them, twice in batches on a single day. From 2021 it publishes almost nothing that stores or moves data, and a great deal that manages things that store and move data: a cluster-management platform, a deploy system, a resource lifecycle framework, and a client library whose stated purpose is "Seamless switching between PubSub backends without code changes".

The third is the one worth taking into a design review. The migrations Pinterest completed cheaply are the ones where an abstraction already sat between the application and the engine, and the failures visible in its issue trackers are almost all failures of coordination rather than of storage. Those two observations are the same observation from opposite ends, and the rest of this guide is the case for that.

Figure 1 · What was published, and what was retired

Every date here is an artefact date rather than a decision date: an archive banner, a release tag or a fork header. The "Data systems" lane, holding what Pinterest built and published itself, stops in 2022 and is replaced by forks of Ray, StarRocks and Druid. The "Control plane" lane, the software that manages whatever engine is underneath, never stops. Compiled from the archived listing, MemQ tags and PSC tags.

Diagram source

Scope, and a hard limit on the corpus

This guide covers Pinterest's ingest, pubsub, serving-store and cluster-management layers as they can be read from public repositories. It does not cover the recommendation and ranking stack, the image pipeline, the HBase estate, the cloud bill, or anything about how any of this is operated internally. The reason is blunt: the network policy for this session resolved github.com and refused every other host, so no engineering blog, conference talk, paper or vendor case study could be fetched, and this method does not allow citing what was not fetched. Everything below is one host and one company's own artefacts. Where that changes what you should believe, the text says so.

How it is actually built

A pipeline with four layers, one of which exists only so that the other three can be replaced without anybody rewriting an application.

Figure 2 · The shape the artefacts describe

The seam that matters is the dashed box: an application talks to PSC, and PSC decides whether the bytes go to Kafka or to MemQ. Reconstructed from the PSC README, MemQ architecture document, Singer README and Orion README.

Diagram source

Read left to right in time rather than top to bottom in layers and the design argument becomes obvious. Secor, whose oldest visible release tag is v0.20 from May 2016, existed to copy the Kafka log into object storage after the fact, and its README still promises that "each message will be saved in exactly one S3 file". MemQ, first tagged publicly in December 2022, removed the copy by making object storage the place the log is written in the first place. The MemQ architecture document is explicit about the mechanism: messages accumulate into a Batch, "Once a Batch is finalized, it is handed to the StorageHandler for upload to the Storage layer (like S3)", and consumers then discover batches through a notification queue and fetch them "directly from the Storage layer". The broker in MemQ is not in the read path for data. It hands out pointers.

That single change is what pays for the rest. The README claims MemQ is "90% more cost effective than our Kafka footprint" and that it "eliminates expensive rebalancing during traffic growth", and the architecture document supplies the reason the economics work: "micro-batching and immutable writes" cut the IOPS the storage layer has to serve, which is what makes an object store viable as primary storage rather than as an archive tier. It also changes recovery. Where a replicated-log broker re-replicates a partition from the start after a fault, MemQ states that "the underlying replicated storage only needs to recover the specific Batch whose replica counts were reduced". Recovery becomes proportional to the damage rather than to the dataset.

None of that would have been adoptable on its own. A team that has three hundred producers and consumers compiled against the Kafka client cannot swap the broker, whatever the broker costs, because the client is in everybody's build file. PSC is the answer to that, and its README says what it is for without ornament: a "generic and extensible client library that allows applications to interact with different backend PubSub systems with a single unified interface", offering "Seamless switching between PubSub backends without code changes", supporting "Apache Kafka and MemQ in Java" with an Apache Flink integration. Neither README says PSC was built so MemQ could be rolled out, and I have no artefact that says so, but the two systems support exactly each other's shape, PSC's only two backends are the two engines in question, and the tags line up: MemQ's first public tag is December 2022 and PSC's is November 2023. Treat the causal claim as inferred, and treat the structural claim, that the swap is only cheap if the client is yours, as the transferable part.

Figure 3 · Who is in the read path

The consumer's bytes never pass through the MemQ broker, which is why storage cost and serving cost can be scaled independently. Drawn from the MemQ architecture document.

Diagram source

The ingest agent is deliberately weaker than the bus

Singer, the log agent that feeds the pipeline, commits to at-least-once delivery using watermark files, and publishes per-agent figures of more than 100MB per second on the thrift format with sub-5ms upload latency. It does not claim exactly-once, which is the honest position for a process that is tailing files on a host that can be terminated. Everything downstream therefore has to be built for duplicates, and is.

Source: Singer README, checked 2026-09-19.

The control plane outlived every engine under it

Orion describes itself as "a generalized pluggable management and automation platform for stateful distributed systems", and its stated design goals include preventing conflicts between human and automated operations on the same cluster. It arrived to replace a Kafka-specific auto-healer, and it is the layer that survives when the engine beneath is swapped, because rolling restarts and remediation are the same problem whatever is being restarted.

Source: Orion README, checked 2026-09-19.

The oldest surviving component is the one nobody standardised

Teletraan, the deploy system, still takes commits in September 2026 after 1,897 of them, and its README claims it "deploys thousands of Pinterest internal services, supports tens of thousands hosts". Every other 2016-era system in the organisation is archived or unmaintained. Deployment is the one area where no external project arrived to make the in-house version embarrassing, which is the real sorting rule behind what survives.

Source: Teletraan README, checked 2026-09-19.

The decisions that matter

Five forks in the road, each with the condition that would have sent Pinterest the other way, and would send you the other way.

Decision: where does the durable copy of the log live?

Chosen

Object storage as the primary store, written directly by the broker as immutable batches (MemQ, tagged from December 2022).
Stated payoff: "90% more cost effective than our Kafka footprint", and IOPS on the storage layer "dramatically reduced" by micro-batching.

Rejected

Replicated broker disks with an after-the-fact copy into object storage, which is what Secor had been doing since the mid-2010s.
It pays twice: once for broker storage and replication, once for the copy job and its failure modes.

Flips when

Consumers need the tail of the log in single-digit milliseconds. A batch is not visible until it is finalised and uploaded, so the design trades tail latency for cost.
Also flips when your object store charges per request rather than per byte at your batch size; the whole argument rests on IOPS per byte.

Decision: who decides which consumer reads which partition?

Chosen

A notification queue publishing pointers, with consumers fetching batches from storage themselves.
MemQ's README claims this "eliminates expensive rebalancing during traffic growth".

Rejected

Broker-coordinated consumer groups with partition assignment and revocation.
The cost is visible in Pinterest's own tracker: events lost during rebalancing in 2020, duplicates on partition revocation still open from 2022.

Flips when

You need strict per-key ordering across a consumer fleet. Group coordination is how that is bought, and pointer-passing gives it up.
Or when your consumers are so few that rebalances are rare and cheap, in which case the coordination protocol never becomes the failure you are debugging.

Decision: automate the operation of the engine, or replace the engine?

Chosen

Both, in that order, and the automation outlived the engine choice. DoctorK auto-healed Kafka; Orion replaced it with something engine-agnostic.
DoctorK's own archive notice says Orion "addresses the shortcomings of DoctorK" and had been "stabily managing our entire kafka fleet for >6months".

Rejected

Extending the Kafka-specific healer into a general platform.
Its design goals name a problem a bolt-on healer cannot solve: humans and automation issuing conflicting operations against the same cluster.

Flips when

You operate exactly one kind of stateful cluster and expect to for years. A general control plane is a bet on heterogeneity you may not have.

Figure 4 · Whether to build the client abstraction

The test is not how good the abstraction is, it is how many build files name the engine's client. Terminal nodes are actions rather than opinions.

Diagram source

Decision	Chosen	Rejected	Because, as stated in the artefact	Flips when	Evidence
Durable log storage	Object store, written directly	Broker disks plus a copy job	"90% more cost effective than our Kafka footprint"	You need the tail in single-digit milliseconds	MemQ README
Consumer coordination	Notification queue, direct fetch	Broker-managed consumer groups	"eliminates expensive rebalancing during traffic growth"	You need per-key ordering across the fleet	MemQ architecture
Client coupling	One in-house interface over both backends	The engine's own client in every application	"Seamless switching between PubSub backends without code changes"	Fewer than roughly ten teams import the client	PSC README
Cluster operations	Engine-agnostic control plane	Extending the Kafka-specific healer	Orion "addresses the shortcomings of DoctorK"	You run one kind of cluster and always will	DoctorK archive notice
Analytics and ML engines	Fork and track upstream	Build in-house, as with Terrapin	No stated reason; inferred from the fork headers	Your access pattern has no upstream equivalent	starrocks fork
Where the fork is developed	An internal GitHub, with the public repo as a conduit	Developing in public	"Pull requests and branch pushes here will not be picked up"	You want outside contributors more than release control	ray fork notice

The last row deserves a sentence on its own, because it is the decision with the widest blast radius for people outside Pinterest. The public fork of Ray now carries a notice saying that development, code review and wheel releases "have moved to Pinterest's internal GitHub", and that the public repository is kept "for reference, and as the staging point for contributions back to ray-project/ray". That is a coherent position for the company and a trap for anyone treating the public repository as a view of what Pinterest runs. The same caution applies to Singer, where 291 pull requests sit open, and to Rocksplicator, whose README describes a 4,000-host production estate in the present tense directly beneath a line saying the project is no longer maintained.

What broke in production

Pinterest has published no postmortem that this corpus can reach. What it has published is a decade of issue threads and pull-request queues filed by people running the software, which is a different and weaker instrument. Read these as operator reports.

The weakness is worth being precise about, because it changes what you may conclude. An incident review tells you blast radius, detection delay and the structural fix. A bug report tells you that a failure mode exists and that somebody hit it. None of the four entries below carries a duration or a customer impact, and the GitHub advisory database has nothing filed against any Pinterest-authored package, so the security record is silent too. What survives that filter is still useful: the failures cluster into two classes, and neither one is about storage.

Operator report

Coordination, not storage, is what loses the record

AssumptionSecor's README states that "each message will be saved in exactly one S3 file", and the uploader was written to make that true against S3's consistency model.

What happenedThe guarantee is broken upstream of storage. Issue 395, "Secor loses events during kafka rebalancing", and issue 1010, "Kafka offset checking logic in message write might lead to message loss", both locate the defect in the order of offset commit relative to upload during a partition handover. Issue 2316, "Duplicate message when partition revoking or rebalancing", has been open since August 2022.

Blast radiusUnstated. What is dateable is persistence: the same failure class appears in threads from 2020 and is still open in the tracker in 2026.

FixNot in Secor. The architectural answer arrived as a different system: MemQ removes group rebalancing from the design entirely and hands consumers pointers instead of partitions.

Design ruleAn exactly-once claim made by a component that does not own the offset commit is a claim about the happy path. Test it by revoking a partition mid-upload, which is the one case your load test does not generate.

Sourcesecor issue 395, issue 1010, issue 2316

Operator report

The published copy is not the operated copy

AssumptionA repository with recent commits from company engineers reflects what that company runs.

What happenedSinger carries 291 open pull requests, most of them automated security upgrades. One, opened in February 2026 and still open, proposes moving the Kafka client from 2.3.1 to 4.2.0. Meanwhile human feature work continues to land, including an OTLP log sink proposed in September 2026. The repository is simultaneously alive and abandoned, depending on which queue you look at.

Blast radiusFalls entirely on outside adopters. Secor has 536 forks and Singer 41, and anyone building from the published branch inherits the unmerged dependency set.

FixNone visible. The Ray fork takes the opposite and more honest route, stating plainly that development has moved elsewhere and that pushes here "will not be picked up".

Design ruleBefore adopting a company's open-source infrastructure, diff the dependency set in the published branch against current releases. A stale pinned client in an otherwise active repository means you are reading a mirror, and mirrors do not accept your bug reports.

Sourcesinger pull queue, ray fork notice

Operator report

The control plane ships with the door open

AssumptionAn operations platform that can perform rolling restarts and node replacement across a Kafka fleet will authenticate the caller.

What happenedOrion issue 404, filed in May 2026 and open at the time of checking, is titled "Security hardening: replace unconditional NoopAuthorizationFilter with configurable/fail-closed auth". The published control plane installs a no-op authorization filter unconditionally.

Blast radiusUnknown, and unknowable from here. Orion has three open issues in total and no visible maintainer reply on this one.

FixProposed in the issue, not merged. No advisory has been filed against any Pinterest package.

Design ruleThe blast radius of a control plane is the union of every cluster it can act on, so its authorization default should be fail-closed even when the network is assumed private. A remediation platform with open auth is a fleet-wide restart primitive available to anything that can reach the port.

Sourceorion issue 404, GitHub advisory database

Operator report

Retirement happens in batches, and sometimes without a banner

AssumptionAn unarchived repository with a present-tense README is maintained.

What happenedTerrapin and PinLater were archived on the same day, 14 February 2018. Rocksplicator was never archived at all: its README opens "Rocksplicator is an archived project that is no longer actively maintained or supported by Pinterest" and then describes, in the present tense, nine services across nearly thirty clusters and over four thousand hosts. Pinball's notice names no successor.

Blast radiusRocksplicator has 690 stars and 118 forks; the discrepancy between the flag and the notice is only visible to someone who opens the README rather than filtering the organisation by archive status.

FixNot applied. The archive flag and the README disagree as of the check date.

Design ruleDo not run a dependency audit on the archive flag alone. Batch archive dates say retirement is a periodic cleanup rather than a per-project decision, which means the gap between a project dying and being marked dead is measured in years.

Sourcerocksplicator README, terrapin, pinball

Figure 5 · The rebalance window, which is where the records go

Nothing in this path touches object storage, which is why adding durability to the sink never fixed it. Reconstructed from the mechanism described in secor issue 1010 and issue 2316; neither thread is an incident review, so treat the ordering as the reporters' account.

Diagram source

The pattern under all four

Two of these failures are about coordination between processes that each believe they own a resource, and two are about the gap between what a repository says and what the company runs. Neither class is fixed by better storage, and neither is visible from an architecture diagram. Both are visible from an issue tracker in about ten minutes, which is the cheapest due diligence available on any dependency you are considering.

Numbers you can plan against

Every figure below is copied from a Pinterest repository. Almost none of them carries a date, which is itself the most important thing to know about them.

Metric	Value	System	Context	Status	As of	Source
Cost against the Kafka footprint	90% better	MemQ	Claimed for the workloads MemQ carries, not for all of Kafka	Claimed, undated	checked 2026-09	README
Serving hosts	4,000+	Rocksplicator	Nine online services, nearly thirty clusters	Claimed, undated	checked 2026-09	README
Data processed	tens of PB/day	Rocksplicator	Across the same estate	Claimed, undated	checked 2026-09	README
Agent throughput	>100 MB/s	Singer	Thrift log format, per agent	Claimed, undated	checked 2026-09	README
Agent throughput	20 MB/s	Singer	Text log format, per agent	Claimed, undated	checked 2026-09	README
Upload latency	<5 ms	Singer	Log upload, no percentile given	Claimed, undated	checked 2026-09	README
Queue throughput ceiling	1,000 QPS	PinLater	Per MySQL shard, the stated limit before moving to Redis	Claimed, archived 2018	2018-02	README
Deploy fleet	tens of thousands	Teletraan	Hosts, across thousands of internal services	Claimed, undated	checked 2026-09	README
Archived repositories	14	Organisation	Out of 100 public repositories	Measured from the listing	2026-09-19	listing
Open pull requests	291	Singer	Oldest shown dates from August 2025	Measured from the queue	2026-09-19	queue
Open issues	0	PSC	On the library that is the seam of the whole design	Measured from the tracker	2026-09-19	tracker
Time from first tag to 1.0	2 yr 8 mo	MemQ	Derived: 0.2.17 on 2022-12-06 to 1.0.0 on 2025-08-05	Derived from tag dates	2026-09-19	tags

Read these carefully

Eight of the twelve rows are undated claims in a README, which is the weakest numeric evidence there is: nobody rewrites a README when the fleet doubles. The 90% cost figure in particular is a self-reported comparison with no methodology, no workload description and no baseline date, and it compares MemQ against the Kafka footprint it explicitly says it "augments" rather than replaces. Use it as evidence that decoupled storage changed the economics enough for one company to build and keep a second pubsub system, not as a number to put in your own business case. The two figures worth planning against are structural rather than numeric: recovery after a storage fault is proportional to the batch rather than the partition, and read capacity scales without touching broker capacity. Both are design properties you can verify in a prototype rather than take on trust.

The evidence wall

Every source behind this page, graded. One host, thirty artefacts, no engineering blogs, talks or papers, for the reason given in section 01.

Decision record Pinterest2021-12

DoctorK archive notice naming Orion as successor

The clearest decision record in the corpus. It states that Orion "addresses the shortcomings of DoctorK" and had been "stabily managing our entire kafka fleet for >6months" before the older project was archived on 16 December 2021.

Carry forwardRetire the old system only after the replacement has carried the whole fleet for a stated period, and put that period in the notice.

https://github.com/pinterest/DoctorK

Decision record Pinterestchecked 2026-09

PSC README: one interface over several backends

Describes a "generic and extensible client library that allows applications to interact with different backend PubSub systems with a single unified interface", listing "Seamless switching between PubSub backends without code changes" as a feature and naming Kafka and MemQ as the supported backends.

Carry forwardThe migration cost of an engine is set by how many build files name its client, so the abstraction has to exist before the migration is proposed.

https://github.com/pinterest/psc

Design document Pinterestchecked 2026-09

MemQ architecture document: batches, pointers, direct reads

The in-repo design document for the write and read paths. Batches are uploaded to storage on fill or timer; consumers discover them through a notification queue and fetch them "directly from the Storage layer"; micro-batching and immutable writes are named as the reason an object store is affordable as primary storage.

Carry forwardTaking the broker out of the read path is what lets read capacity and storage cost scale independently.

https://github.com/pinterest/memq/blob/master/docs/architecture.md

Design document Pinterestchecked 2026-09

MemQ recovery model after a storage fault

The same document states that the replicated storage layer "only needs to recover the specific Batch whose replica counts were reduced due to faults", and notes that migrating workloads off Kafka "required strict validation on data loss".

Carry forwardImmutable, addressable units make recovery proportional to the damage rather than to the dataset; that is the operational argument for the design, not cost.

https://github.com/pinterest/memq/blob/master/docs/architecture.md

Decision record Pinterestchecked 2026-09

Orion README and its design goals

Defines Orion as "a generalized pluggable management and automation platform for stateful distributed systems" and names conflict between human and automated operations on the same cluster as a first-class problem to solve.

Carry forwardIf humans and automation can both act on a cluster, arbitration between them belongs in the control plane rather than in a runbook.

https://github.com/pinterest/orion

Case study Pinterestchecked 2026-09

MemQ README, with the cost claim

States that MemQ "augments Kafka at Pinterest", uses "a decoupled storage and serving architecture similar to Apache Pulsar and Facebook Logdevice", handles GB/s, and is "90% more cost effective than our Kafka footprint".

Carry forwardThe word is augments, not replaces. A second bus for the workloads that suit it is a cheaper answer than a migration of everything.

https://github.com/pinterest/memq

Operator report Pinterest2020-06

secor issue 395: events lost during rebalancing

Titled "Secor loses events during kafka rebalancing" and closed in June 2020. The failure is located in the consumer-group handover rather than in the upload to object storage.

Carry forwardWhen a sink promises exactly-once but does not own the offset commit, the promise covers the steady state only.

https://github.com/pinterest/secor/issues/395

Operator report Pinterest2022-08

secor issue 2316: duplicates on partition revocation

"Duplicate message when partition revoking or rebalancing", filed in August 2022 and still open at the check date, four years later and two years after the related loss issue was closed.

Carry forwardAn open issue with this lifetime is a statement that the fix is architectural, not a patch; treat it as a design constraint on anything you build on top.

https://github.com/pinterest/secor/issues/2316

Operator report Pinterest2020-05

secor issue 1010: offset-check ordering

"Kafka offset checking logic in message write might lead to message loss". The reporter places the window between the offset check and the durable write.

Carry forwardThe ordering of commit and durability is the whole guarantee; write it down explicitly in any sink you build.

https://github.com/pinterest/secor/issues/1010

Operator report Pinterest2026-05

orion issue 404: a no-op authorization filter

"Security hardening: replace unconditional NoopAuthorizationFilter with configurable/fail-closed auth", opened 27 May 2026 and open at the check date, on the platform that performs rolling restarts across the Kafka fleet.

Carry forwardAudit the authorization default of every tool that can act on a whole fleet, and require fail-closed regardless of network assumptions.

https://github.com/pinterest/orion/issues

Operator report Pinterest2026-09

singer pull queue: 291 open, mostly security bumps

Includes an automated proposal to move the Kafka client from 2.3.1 to 4.2.0, opened in February 2026 and still open, alongside human feature work merged as recently as September 2026.

Carry forwardJudge a repository's maintenance by its dependency queue rather than its commit graph; the two can tell opposite stories.

https://github.com/pinterest/singer/pulls

Source Pinterest2026-09

Organisation repository listing, sorted by update

One hundred public repositories, with the recently touched ones dominated by forks: Ray, StarRocks, Superset, Druid, Karpenter and Restate all appear on the first page.

Carry forwardSort any organisation's repositories by last update and count the forks; the ratio dates the shift from building infrastructure to operating it.

https://github.com/orgs/pinterest/repositories

Source Pinterest2026-09

Archived repository listing: fourteen, none after 2021

The complete archive register for the organisation. The newest entry, DoctorK, last received a push in December 2021.

Carry forwardAn archive register that stops is not evidence the platform stopped; it is evidence the organisation stopped publishing the things it retires.

https://github.com/orgs/pinterest/repositories?q=archived%3Atrue

Source Pinterest2018-02

terrapin, archived 14 February 2018

"A low latency serving system providing random access over large data sets, generated by Hadoop jobs and stored on HDFS clusters", built on HFiles. Archived the same day as PinLater.

Carry forwardBatch archive dates mean retirement is a periodic sweep; the project died well before the banner.

https://github.com/pinterest/terrapin

Source Pinterest2018-02

pinlater, archived 14 February 2018

A Thrift job-scheduling service backed by MySQL or Redis, recommending MySQL up to "no more than 1000 QPS per shard". The queue state lived in a relational database.

Carry forwardA queue whose state is in your primary database inherits that database's sharding limits as its throughput ceiling.

https://github.com/pinterest/pinlater

Source Pinterest2018-11

kingpin, archived 17 November 2018

Service discovery and configuration for a Thrift service architecture, with ZooKeeper holding dynamic serversets and Amazon S3 holding configuration data.

Carry forwardObject storage as a configuration substrate predates its use as a log substrate by years; the appetite was there long before the latency was acceptable.

https://github.com/pinterest/kingpin

Source Pinterest2019-06

mysql_utils, archived 26 June 2019

Backup, restore, failover and replication tooling for the MySQL estate that held pins, boards and credentials, with a README warning in capitals that the tools will not work outside Pinterest without substantial adaptation.

Carry forwardFleet tooling encodes one company's service discovery and configuration; publishing it is a gesture, not a gift.

https://github.com/pinterest/mysql_utils

Source Pinterest2019-12

pinball, archived 11 December 2019 with no successor named

A workflow manager with a thousand stars, retired under the line "This project is no longer actively maintained by Pinterest" and no forwarding address.

Carry forwardThe absence of a named successor in a retirement notice is the most common form of the notice, and the least useful to everyone downstream.

https://github.com/pinterest/pinball

Source Pinterestchecked 2026-09

rocksplicator: unmaintained, and describing a live fleet

A C++ suite for RocksDB replication, request routing and Helix-driven cluster management. The README declares the project archived and unmaintained, then describes nine services, nearly thirty clusters, over four thousand hosts and tens of petabytes a day in the present tense.

Carry forwardRead the README before the archive flag; they disagree more often than anyone expects.

https://github.com/pinterest/rocksplicator

Source Pinterestchecked 2026-09

secor: the Kafka log copied into object storage

Persists Kafka logs to S3, Google Cloud Storage, Azure Blob Storage and Swift, claiming each message is saved in exactly one file. The most widely adopted thing in the organisation: 1.9k stars, 536 forks, 3,338 commits.

Carry forwardThe system with the most outside users is a decade-old sink, not any of the newer platform pieces; adoption follows narrow scope, not ambition.

https://github.com/pinterest/secor

Source Pinterestchecked 2026-09

singer: the ingest agent and its honest guarantee

A logging agent uploading to Kafka with at-least-once delivery via watermark files, claiming over 100MB/s on the thrift format and sub-5ms upload latency, deployable as a Kubernetes daemonset or sidecar.

Carry forwardAn agent that tails files on an ephemeral host cannot offer exactly-once; design the downstream for duplicates rather than arguing about it.

https://github.com/pinterest/singer

Source Pinterest2022-12 to 2026-04

MemQ release tags

Tags run from 0.2.17 in December 2022 to 1.0.0 in August 2025 and 1.0.2 in April 2026. Two years and eight months of pre-1.0 releases while carrying production traffic.

Carry forwardVersion numbers on internally-motivated infrastructure track internal confidence, not production readiness; date the tags instead.

https://github.com/pinterest/memq/tags

Source Pinterest2023-11 to 2026-04

PSC release tags: three major lines at once

Tags 2.3.3, 4.0.0 and 3.3.1 were all cut within four days of each other at the end of January 2025, with the newest line reaching 4.2.0 in April 2026.

Carry forwardAn abstraction that decouples applications from an engine becomes its own compatibility surface, and you will maintain several of its major versions at once.

https://github.com/pinterest/psc/tags

Source Pinterest2023-12 to 2026-08

PSC closed-unmerged pull requests, including tiered storage

Twenty-seven closed without merging. Two of them, from January and June 2025, are tiered-storage integration attempts left in draft, which is the upstream engine growing the capability MemQ was built to provide.

Carry forwardWatch for the upstream absorbing your differentiator; the abandoned integration branch is where that argument is recorded.

https://github.com/pinterest/psc/pulls?q=is%3Apr+is%3Aclosed+is%3Aunmerged

Source Pinterest2026-09

PSC issue tracker: empty

Zero issues on the library that the whole design depends on, against 1.9k stars and an active issue list on the ten-year-old sink beside it.

Carry forwardPublished infrastructure with no issue traffic has no outside operators, so its public bug record cannot be used as evidence of maturity.

https://github.com/pinterest/psc/issues?q=is%3Aissue

Source Pinterestchecked 2026-09

teletraan: the survivor

"Teletraan is Pinterest's deploy system. It deploys thousands of Pinterest internal services, supports tens of thousands hosts." Still receiving commits in September 2026 after 1,897 of them.

Carry forwardIn-house software survives where no industry standard arrived to replace it; that, not quality, is the sorting rule visible in the archive.

https://github.com/pinterest/teletraan

Source Pinterestchecked 2026-09

The Ray fork, and the notice that closes the record

Forked from ray-project/ray, carrying a notice that development, review and wheel releases "have moved to Pinterest's internal GitHub", that pushes here "will not be picked up", and that the repository exists as a staging point for contributions upstream.

Carry forwardThe endpoint of the fork-and-track strategy is a private fork with a public conduit; plan your own dependency intelligence for a world where that is normal.

https://github.com/pinterest/ray

Source Pinterestchecked 2026-09

The StarRocks fork

Forked from StarRocks/starrocks, the analytics engine Pinterest now runs rather than builds, in the same pattern as the Druid and Superset forks beside it.

Carry forwardThe build-versus-adopt line moved engine by engine, and the fork header is the dated artefact that marks where it moved.

https://github.com/pinterest/starrocks

Source Pinterestchecked 2026-09

slate: the newest thing published is a control plane

"Slate is a framework that solves Resource Lifecycle Management enabling us to create a unified Platform as a Service for infrastructure." One star, sixty-eight commits, no outside community.

Carry forwardThe layer a mature platform team still writes itself is resource lifecycle and policy, because that is the part no upstream can supply.

https://github.com/pinterest/slate

Source GitHub2026-09

Advisory database: nothing against Pinterest's own software

Every advisory matching the query concerns third-party WordPress and Joomla plugins. None names a repository in the Pinterest organisation.

Carry forwardA silent advisory record is not a safety record. Combine it with the issue tracker, where the only security finding in this corpus actually sits.

https://github.com/advisories?query=pinterest

Build a miniature, then productionise it

Six rungs. The line between a toy and something production-shaped is crossed at rung four, where you stop testing the happy path.

Put a log behind an interface you own

Write a producer and consumer interface with four methods, back it with a local Kafka, and forbid the vendor client from appearing anywhere except one adapter package. Enforce it with a build rule, not a convention.

Done when: grepping the application for the vendor package name returns only the adapter. Teaches: the abstraction is a boundary you enforce mechanically, or it is a comment.

Add a second backend that is obviously worse

Implement the same interface over local files or a bucket. The point is not to build a good engine, it is to discover which parts of the vendor client leaked into your interface, which will be offsets, partitions and rebalance callbacks.

Done when: the same application binary runs against both backends by config alone. Teaches: the shape of the leak, which is what the PSC design is mostly about.

Move durability to the object store

Batch on size or a timer, upload immutable objects, and publish a pointer to a small notification topic. Let the consumer read the object directly rather than through your broker.

Done when: your broker process can be killed mid-stream and consumers still drain everything already uploaded. Teaches: why taking the broker out of the read path changes the cost model.

Break it at the handover, on purpose

Kill a consumer while it holds an unfinished upload and an uncommitted offset. Do it under load, repeatedly. Count duplicates and gaps against a known input set.

Done when: you can state your delivery guarantee as a number of duplicates per thousand kills, not as an adjective. Teaches: the exact failure the Secor issues describe, on your own code, where you can fix it.

Make the swap a config change and prove it

Run one workload across both backends simultaneously, compare the two output sets, and only then cut over. MemQ's design document notes that migrating off Kafka "required strict validation on data loss"; build that validator before you need it.

Done when: a dual-write comparison runs continuously and alerts on divergence. Teaches: the migration's real cost is the verification, not the code.

Give the control plane an authorization default

Add the operations that make it production-shaped, which are rolling restart, node replacement and a lock that stops a human and an automation acting at once. Then make the authorization filter fail closed and write the test that proves it.

Done when: an unauthenticated call to the restart endpoint is refused in the default configuration. Teaches: why Orion issue 404 is the most consequential open item in this whole corpus.

Keep hunting

These are the queries that produced the material above, against a single host. They work on any organisation, and they are most of what you need when the engineering blogs are unreachable.

Dating an organisation's platform decisions

https://github.com/orgs/<org>/repositories?q=archived%3Atrue&type=public
https://github.com/orgs/<org>/repositories?type=public&sort=updated
https://github.com/<org>/<repo>/tags

Finding the failures nobody wrote up

repo:<org>/<repo> is:issue "data loss" OR duplicate OR rebalanc
repo:<org>/<repo> is:pr is:closed is:unmerged
repo:<org>/<repo> is:issue sort:reactions-+1-desc
https://github.com/advisories?query=<org>

Separating the mirror from the real repository

https://github.com/<org>/<repo>/pulls?q=is%3Apr+is%3Aopen+sort%3Acreated-asc
"no longer" "maintained" path:README.md org:<org>
"moved to" "internal" path:README.md org:<org>

Reading the design argument out of the repo

path:docs/architecture.md org:<org>
"addresses the shortcomings of" org:<org>
"more cost effective than" path:README.md

One caveat on the method, learned the hard way in this session. A single-host corpus can establish what a company published and when, and it can establish the mechanism of a design where an in-repo document describes it. It cannot establish outcome. Nothing here tells you whether the MemQ migration went well, how much of the Kafka fleet remains, or what any of it costs. If your network reaches further than this one did, take these dates and mechanisms and go looking for the engineering-blog account and the conference talk that describe the same systems; the artefacts give you the questions to ask them.

References

Pinterest, public repositories sorted by last update GitHub organisation listing. Checked 2026-09-19.
Pinterest, archived public repositories GitHub organisation listing, 14 results. Checked 2026-09-19.
pinterest/terrapin Archived 14 February 2018. Checked 2026-09-19.
pinterest/pinlater Archived 14 February 2018. Checked 2026-09-19.
pinterest/kingpin Archived 17 November 2018. Checked 2026-09-19.
pinterest/mysql_utils Archived 26 June 2019. Checked 2026-09-19.
pinterest/pinball Archived 11 December 2019. Checked 2026-09-19.
pinterest/DoctorK Archived 16 December 2021, naming Orion as successor. Checked 2026-09-19.
pinterest/orion Cluster management platform. Checked 2026-09-19.
pinterest/orion, issue tracker Issue 404 opened 27 May 2026. Checked 2026-09-19.
pinterest/rocksplicator RocksDB replication and cluster management. Checked 2026-09-19.
pinterest/secor Kafka log persistence to object storage. Checked 2026-09-19.
pinterest/secor, issue 395 "Secor loses events during kafka rebalancing", closed June 2020. Checked 2026-09-19.
pinterest/secor, issue 1010 Offset-check ordering and message loss, closed May 2020. Checked 2026-09-19.
pinterest/secor, issue 2316 Duplicates on partition revocation, open since August 2022. Checked 2026-09-19.
pinterest/memq Object-store-backed pubsub. Checked 2026-09-19.
pinterest/memq, architecture document In-repo design document. Checked 2026-09-19.
pinterest/memq, tags 0.2.17 (2022-12-06) to 1.0.2 (2026-04-13). Checked 2026-09-19.
pinterest/psc PubSub client abstraction. Checked 2026-09-19.
pinterest/psc, tags 2.3.0 (2023-11-01) to 4.2.0 (2026-04-17). Checked 2026-09-19.
pinterest/psc, closed-unmerged pull requests 27 results including tiered-storage drafts. Checked 2026-09-19.
pinterest/psc, issue tracker No results. Checked 2026-09-19.
pinterest/singer Logging agent. Checked 2026-09-19.
pinterest/singer, pull requests 291 open. Checked 2026-09-19.
pinterest/teletraan Deploy system. Checked 2026-09-19.
pinterest/teletraan, tags v1.0.0 tagged 14 January 2016. Checked 2026-09-19.
pinterest/secor, tags Oldest visible tag v0.20, 11 May 2016; the list is paginated, so earlier tags may exist. Checked 2026-09-19.
pinterest/ray Fork of ray-project/ray, development moved internal. Checked 2026-09-19.
pinterest/starrocks Fork of StarRocks/starrocks. Checked 2026-09-19.
pinterest/slate Resource lifecycle framework. Checked 2026-09-19.
GitHub advisory database, query "pinterest" No advisory against Pinterest-authored software. Checked 2026-09-19.