Evidence ledger
30 sources
Checked 19 Sep 2026
Evidence ledger
One row per claim in Swapping the engine under the client: ten years of Pinterest, read from its own repositories: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Guide: Swapping the engine under the client: ten years of Pinterest, read from its own repositories
Category: data-platforms · Research date: 2026-09-19 · All links fetched 2026-09-19.
Corpus limit, stated up front. This session's network egress policy resolved github.com and no
other host. Every attempt to reach an engineering blog, a conference site, a paper host or a status
page was refused by the proxy. So the corpus is one host: repositories, README files, in-repo design
documents, release tags, issue threads and pull-request queues published by Pinterest and, in two
cases, the GitHub advisory database. No engineering-blog post, talk, paper or vendor case study is
cited here, because none could be fetched, and this skill does not permit citing from memory. What
that costs the guide is set out in the page itself.
One row per claim. The right-hand column is copied text, not paraphrase.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Public repositories, sorted by last update | source | live listing | 2026-09-19 | https://github.com/orgs/pinterest/repositories?q=&type=public&sort=updated | The organisation's most recently touched repositories are dominated by forks of other people's infrastructure. | Listing reports 100 public repositories; the first page includes ray ("Ray is a unified framework for scaling AI and Python applications"), starrocks, superset, druid, karpenter-provider-aws and restate, each updated between Jul and Sep 2026. |
|
| 2 | Archived public repositories | source | live listing | 2026-09-19 | https://github.com/orgs/pinterest/repositories?q=archived%3Atrue&type=public | Only fourteen repositories carry the archive flag, and none were pushed to after 2021. | "Total Count: 14 repositories"; newest last-push date in the list is DoctorK, Dec 15 2021. | |
| 3 | terrapin | source | archived 2018-02-14 | 2026-09-19 | https://github.com/pinterest/terrapin | The batch-serving store Pinterest built on Hadoop was retired in early 2018. | "This repository was archived by the owner on Feb 14, 2018."; "Terrapin is a low latency serving system providing random access over large data sets, generated by Hadoop jobs and stored on HDFS clusters." | |
| 4 | pinlater | source | archived 2018-02-14 | 2026-09-19 | https://github.com/pinterest/pinlater | The asynchronous job system was retired on the same day as Terrapin, and had put queue state in MySQL. | "This repository was archived by the owner on Feb 14, 2018. It is now read-only."; "services should default to use the MySQL backend as long as the QPS is in the lower to mid range (no more than 1000 QPS per shard)". | |
| 5 | kingpin | source | archived 2018-11-17 | 2026-09-19 | https://github.com/pinterest/kingpin | Pinterest was storing configuration in object storage and coordinating through ZooKeeper by the mid-2010s. | Archive banner: Nov 17, 2018. "KingPin is the Python toolset used at Pinterest for facilitating service oriented architecture and application configuration management."; ZooKeeper for "dynamic serversets", "Amazon S3 stores configuration data". | |
| 6 | mysql_utils | source | archived 2019-06-26 | 2026-09-19 | https://github.com/pinterest/mysql_utils | The MySQL fleet tooling was retired in 2019 with an explicit warning that it was never portable. | Archive banner: Jun 26, 2019. "THESE TOOLS WILL NOT JUST WORK!!!"; "Pinterest has historically used MySQL to store some of our most important data". | |
| 7 | pinball | source | archived 2019-12-11 | 2026-09-19 | https://github.com/pinterest/pinball | The in-house workflow manager was retired with no named successor. | Archive banner: Dec 11, 2019. "This project is no longer actively maintained by Pinterest."; 1.0k stars, 151 commits. | |
| 8 | DoctorK | adr | archived 2021-12-16 | 2026-09-19 | https://github.com/pinterest/DoctorK | The Kafka auto-healing service was replaced by a general cluster-management platform, and the reason is stated. | Archive banner: Dec 16, 2021. "Orion addresses the shortcomings of DoctorK and also adds new features like topic management, rolling restarts, rolling upgrades, stuck consumer remediation etc. Orion has been stabily managing our entire kafka fleet for >6months." | |
| 9 | orion | adr | active | 2026-09-19 | https://github.com/pinterest/orion | What replaced DoctorK is not a Kafka tool but a general control plane for stateful systems. | "Orion is a generalized pluggable management and automation platform for stateful distributed systems."; design goals include "Operational Conflict Prevention" between human and automated operations; 113 stars, 331 commits. | |
| 10 | orion issues #404, #21, #36 | postmortem | 2021-03-02 to 2026-05-27 | 2026-09-19 | https://github.com/pinterest/orion/issues | The published cluster-management control plane ships with authorization effectively disabled, reported and still open. | Issue #404, opened May 27 2026, open: "Security hardening: replace unconditional NoopAuthorizationFilter with configurable/fail-closed auth". Three open issues in total; #21 "build and deploy issue" open since Mar 2 2021. | |
| 11 | rocksplicator | source | README notice, undated | 2026-09-19 | https://github.com/pinterest/rocksplicator | Pinterest ran a large in-house stateful serving tier built on RocksDB, and it is now unmaintained without carrying an archive flag. | "Important: Rocksplicator is an archived project that is no longer actively maintained or supported by Pinterest."; "Currently, we have 9 different online services based on rocksplicator running at Pinterest, which consist of nearly 30 clusters, over 4000 hosts and process tens of PB data per day." | |
| 12 | secor | source | active | 2026-09-19 | https://github.com/pinterest/secor | Pinterest has been copying the Kafka log into object storage since the mid-2010s, and claims exactly-once placement. | "Secor is a service persisting Kafka logs to Amazon S3, Google Cloud Storage, Microsoft Azure Blob Storage and Openstack Swift."; "each message will be saved in exactly one S3 file"; 1.9k stars, 3,338 commits, 536 forks. | |
| 13 | secor issue #395, "Secor loses events during kafka rebalancing" | postmortem | closed 2020-06-03 | 2026-09-19 | https://github.com/pinterest/secor/issues/395 | Consumer-group rebalancing, not storage, is the mechanism that breaks the exactly-once claim. | Issue title: "Secor loses events during kafka rebalancing". Closed Jun 3, 2020. | |
| 14 | secor issue #2316, duplicate on revoke | postmortem | open since 2022-08-15 | 2026-09-19 | https://github.com/pinterest/secor/issues/2316 | The same failure class was still open four years later. | Issue title: "Duplicate message when partition revoking or rebalancing". State: open, filed Aug 15, 2022. | |
| 15 | secor issue #1010, offset-check ordering | postmortem | closed 2020-05-22 | 2026-09-19 | https://github.com/pinterest/secor/issues/1010 | The loss window is in the order of the offset commit relative to the upload. | Issue title: "Kafka offset checking logic in message write might lead to message loss". | |
| 16 | memq | casestudy | active, tags from 2022-12-06 | 2026-09-19 | https://github.com/pinterest/memq | Pinterest built a pubsub system whose storage layer is an object store, and states a cost multiple against its Kafka fleet. | "MemQ is a new PubSub system that augments Kafka at Pinterest. It uses a decoupled storage and serving architecture similar to Apache Pulsar and Facebook Logdevice; however, it relies on a pluggable replicated storage layer i.e. Object Store / DFS / NFS for storing data."; "90% more cost effective than our Kafka footprint"; "eliminates expensive rebalancing during traffic growth". | |
| 17 | memq docs/architecture.md |
adr | in-repo design document | 2026-09-19 | https://github.com/pinterest/memq/blob/master/docs/architecture.md | Consumers read batches out of object storage directly; the broker only hands out pointers. | "Once a Batch is finalized, it is handed to the StorageHandler for upload to the Storage layer (like S3)."; "Batches are discovered using the Notification Queue and retrieved directly from the Storage layer."; "micro-batching and immutable writes to create an architecture where the number of Input/output Operations Per Second (IOPS) necessary on the storage layer are dramatically reduced". | |
| 18 | memq docs/architecture.md, recovery |
adr | in-repo design document | 2026-09-19 | https://github.com/pinterest/memq/blob/master/docs/architecture.md | Immutable batches change what recovery costs after a storage fault. | "the underlying replicated storage only needs to recover the specific Batch whose replica counts were reduced due to faults in case of storage failures"; "Migrating workloads from Kafka to MemQ required strict validation on data loss". | |
| 19 | memq tags | source | 2022-12-06 to 2026-04-13 | 2026-09-19 | https://github.com/pinterest/memq/tags | MemQ took until 2025 to tag a 1.0 and is still being released. | Oldest tag 0.2.17, Dec 6 2022; 1.0.0, Aug 5 2025; newest 1.0.2, Apr 13 2026. | |
| 20 | psc | adr | active | 2026-09-19 | https://github.com/pinterest/psc | The client library, not the broker, is where Pinterest put the seam that makes an engine swappable. | "PubSub Client (PSC) is a generic and extensible client library that allows applications to interact with different backend PubSub systems with a single unified interface."; features include "Seamless switching between PubSub backends without code changes"; supports "Apache Kafka and MemQ in Java". | |
| 21 | psc tags | source | 2023-11-01 to 2026-04-17 | 2026-09-19 | https://github.com/pinterest/psc/tags | The abstraction layer carries three live major versions at once, which is a maintenance cost of its own. | Tags include 2.3.3 (Jan 28 2025), 4.0.0 (Jan 30 2025) and 3.3.1 (Jan 31 2025) within four days; newest 4.2.0, Apr 17 2026. | |
| 22 | psc closed-unmerged pull requests | source | 2023-12-05 to 2026-08-13 | 2026-09-19 | https://github.com/pinterest/psc/pulls?q=is%3Apr+is%3Aclosed+is%3Aunmerged | Tiered storage was explored inside the client abstraction and abandoned in draft. | 27 closed-unmerged pull requests; #49 and #50, "tiered storage integration", by jfzunigac, Jan and Jun 2025, both closed in WIP/draft state. | |
| 23 | psc issues | source | live listing | 2026-09-19 | https://github.com/pinterest/psc/issues?q=is%3Aissue | The abstraction has no public operator community; the issue tracker is empty. | "Issues 0"; the search returns "No results". | |
| 24 | singer | source | active | 2026-09-19 | https://github.com/pinterest/singer | The ingest agent commits to at-least-once, not exactly-once, and publishes per-agent throughput figures. | "a high performance logging agent for uploading logs to Kafka"; at-least-once delivery using watermark files; ">100MB/second with thrift log format"; "<5ms log uploading latency". | |
| 25 | singer pull requests | postmortem | oldest open 2025-08-14 | 2026-09-19 | https://github.com/pinterest/singer/pulls | The published copy of the agent has drifted from the operated one: security bumps sit unmerged for months. | 291 open pull requests, mostly Snyk security upgrades; #527 "[Snyk] Security upgrade org.apache.kafka:kafka-clients from 2.3.1 to 4.2.0", opened Feb 17 2026, still open; #541 "Add OTLP log sink", Sep 10 2026. | |
| 26a | teletraan tags | source | 2016-01-14 | 2026-09-19 | https://github.com/pinterest/teletraan/tags | The deploy system dates from January 2016, which fixes the start of the decade this guide covers. | v1.0.0, "make conf file optional", tagged 14 January 2016; only two tags exist. | |
| 26b | secor tags | source | 2016-05-11 | 2026-09-19 | https://github.com/pinterest/secor/tags | Secor's oldest visible tag is May 2016; the list is paginated, so the project is older than that but not datable from here. | Oldest tag on the first page: v0.20, 11 May 2016. Newest: v0.29, 17 July 2020. | |
| 26 | teletraan | source | active | 2026-09-19 | https://github.com/pinterest/teletraan | The deploy system is the one 2016-era component that never got replaced. | "Teletraan is Pinterest's deploy system. It deploys thousands of Pinterest internal services, supports tens of thousands hosts, and has been running in production for over many years."; 1,897 commits; last updated Sep 18 2026. | |
| 27 | ray (fork of ray-project/ray) | source | notice live | 2026-09-19 | https://github.com/pinterest/ray | Pinterest now develops its fork of someone else's engine in private and uses the public repository only as an upstream conduit. | "forked from ray-project/ray"; "This repository is no longer where Pinterest's Ray fork is developed. Development, code review, and wheel releases have moved to Pinterest's internal GitHub. Pull requests and branch pushes here will not be picked up. This repository is kept for reference, and as the staging point for contributions back to ray-project/ray." | |
| 28 | starrocks (fork of StarRocks/starrocks) | source | active | 2026-09-19 | https://github.com/pinterest/starrocks | The analytics engine is rented rather than built, on the same fork-and-track pattern. | Header shows "forked from StarRocks/starrocks"; upstream README lists Pinterest among users, citing "Delivering faster analytics at Pinterest". | |
| 29 | slate | source | active | 2026-09-19 | https://github.com/pinterest/slate | The newest infrastructure software Pinterest publishes is a control plane, not a data system. | "Slate is a framework that solves Resource Lifecycle Management enabling us to create a unified Platform as a Service for infrastructure."; 1 star, 68 commits. | |
| 30 | GitHub | Advisory database, query "pinterest" | source | listing | 2026-09-19 | https://github.com/advisories?query=pinterest | There is no published vulnerability record for Pinterest-authored software; every hit is a third-party plugin. | All results concern WordPress and Joomla plugins such as "Pinterest Automatic Pin" and "weblizar-pinterest-feeds"; none names a repository under the pinterest organisation. |
Tier mix
| Tier | Rows |
|---|---|
| postmortem | 5 |
| source | 19 |
| adr | 5 |
| casestudy | 1 |
| blog, paper, talk, vendor | 0 |
Five rows are graded postmortem on the strength of being operator-filed reports of production
behaviour, not because Pinterest published an incident review. It publishes none in this corpus.
The blog, paper, talk and vendor tiers are empty for the reason given at the top: the hosts
that carry that material were unreachable from this session.
Distinct hosts
One: github.com. The guide says so in the hero and in the scope note, because a single-host corpus
is a real limit on what can be concluded, not a detail to bury.