Evidence ledger
36 sources
Checked 14 Sep 2026
Evidence ledger
One row per claim in Deciding what to stop building: ten years of Spotify, read from its own archive notices: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Field guide: Deciding what to stop building: ten years of Spotify, read from its own archive notices Category: platform-and-infrastructure · Research date: 2026-09-14 · All links fetched 2026-09-14.
Corpus limit, stated up front. This session's network egress policy reached github.com,
raw.githubusercontent.com, repo1.maven.org, pypi.org, registry.npmjs.org, crates.io,
hub.docker.com and cloud.google.com. It did not reach engineering.atspotify.com,
backstage.io, spotify.github.io, www.usenix.org, arxiv.org or any video host. Every claim
below therefore comes from a repository, a package registry, an advisory, or the two
Spotify-sourced posts that Google published on its own blog. There are no conference talks and no
peer-reviewed papers in this corpus, and that absence is reported in the guide rather than papered
over.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Spotify | spotify/helios repository | source | archived 2024-09-12 | 2026-09-14 | https://github.com/spotify/helios | Helios was built because no open-source orchestrator existed, and was abandoned when one did | "This project was created when there were no open source container orchestration frameworks. Since the advent of Kubernetes and other tools, we've stopped using helios at Spotify and have now switched to other tools like Kubernetes." |
| 2 | Spotify | helios README, master branch | source | archived 2024-09-12 | 2026-09-14 | https://raw.githubusercontent.com/spotify/helios/master/README.md | The sunset notice is the first heading of the README, above the description | "## Status: Sunset" |
| 3 | Spotify | spotify/helios tags | source | 2014-05-30 onward | 2026-09-14 | https://github.com/spotify/helios/tags?after=0.8.0 | Helios was public from May 2014, so the in-house orchestrator predates Kubernetes 1.0 | Oldest tags listed: 0.0.23 "May 30, 2014", 0.0.24 "Jun 4, 2014" |
| 4 | Spotify | com.spotify:helios-client on Maven Central | source | latest 0.9.283 | 2026-09-14 | https://repo1.maven.org/maven2/com/spotify/helios-client/maven-metadata.xml | Helios shipped 0.0.33 through 0.9.283 as a published Java artifact | <latest>0.9.283</latest><release>0.9.283</release> |
| 5 | Spotify | spotify/apollo repository | source | archived 2021-09-28 | 2026-09-14 | https://github.com/spotify/apollo | Archiving was a publishing decision: development continued inside Spotify | "most of its development has recently been done internally leveraging Apollo's module system ... Apollo development will continue internally in Spotify." |
| 6 | Spotify | spotify/styx repository | source | archived 2023-07-12 | 2026-09-14 | https://github.com/spotify/styx | The OSS repo was discontinued while the system was running more than 10,000 production workflows | "We decided to discontinue the Styx oss repo."; "more than 10000 production workflows" |
| 7 | Spotify | spotify/styx README | source | archived 2023-07-12 | 2026-09-14 | https://github.com/spotify/styx | Styx ran Luigi-orchestrated jobs in Docker containers on Kubernetes, on Datastore and Bigtable | "schedule batch data processing jobs in Docker containers on Kubernetes"; "Google Cloud Datastore, Google Cloud Bigtable and Google Container Engine" |
| 8 | Spotify | spotify/heroic repository | source | archived 2021-03-27 | 2026-09-14 | https://github.com/spotify/heroic | The in-house time series database was frozen, not migrated in public | "This repo is no longer actively maintained. While it should continue to work and there are no major known bugs, we will not be improving Heroic or releasing new versions." |
| 9 | Spotify | heroic build.gradle, master | source | last change 2021 | 2026-09-14 | https://raw.githubusercontent.com/spotify/heroic/master/build.gradle | The final published Heroic build pins log4j-core 2.13.3, inside the Log4Shell range | dependencySet(group: 'org.apache.logging.log4j', version: '2.13.3') with entries log4j-core, log4j-api, log4j-1.2-api |
| 10 | GitHub Advisory Database | GHSA-jfh8-c2jp-5v3q, CVE-2021-44228 | postmortem | 2021-12-10 | 2026-09-14 | https://github.com/advisories/GHSA-jfh8-c2jp-5v3q | log4j-core 2.13.3 falls inside the affected range, 258 days after Heroic was archived | Affected: "≥ 2.13.0, < 2.15.0"; severity Critical, CVSS 10.0 |
| 11 | Docker Inc | spotify/heroic image on Docker Hub | source | last updated 2021-03-26 | 2026-09-14 | https://hub.docker.com/v2/repositories/spotify/heroic/ | The archived system is still distributable and has been pulled thousands of times | pull_count: 6462, last_updated: 2021-03-26 |
| 12 | Docker Inc | spotify/luigi image on Docker Hub | source | last updated 2018-02-06 | 2026-09-14 | https://hub.docker.com/v2/repositories/spotify/luigi/ | Artefacts outlive their maintenance: an image last built in 2018 has 16,791 pulls | pull_count: 16791, last_updated: 2018-02-06 |
| 13 | Spotify | spotify/docker-client repository | source | archived 2022-03-21 | 2026-09-14 | https://github.com/spotify/docker-client | A live internal dependency was pinned by an already-dying project's build file | "The version of docker-client we're using is whatever helios has in its pom.xml. At this point, we're not developing or accepting new features or even fixing non-critical bugs." |
| 14 | Spotify | spotify/dockerfile-maven repository | source | archived 2022-03-31 | 2026-09-14 | https://github.com/spotify/dockerfile-maven | A widely forked build plugin (2.7k stars, 474 forks) was archived in the same sweep | "At this point, we're not developing or accepting new features or even fixing non-critical bugs." |
| 15 | Spotify | spotify/spydra repository | source | archived 2022-03-31 | 2026-09-14 | https://github.com/spotify/spydra | The Hadoop-to-Dataproc bridge was retired once the migration ended | "Ephemeral Hadoop clusters using Google Compute Platform"; "Note This project is inactive." |
| 16 | Spotify | spotify/zoltar repository | source | archived 2023-10-18 | 2026-09-14 | https://github.com/spotify/zoltar | In-house model serving for TensorFlow, XGBoost and scikit-learn was retired | "Common library for serving TensorFlow, XGBoost and scikit-learn models in production." |
| 17 | Spotify | spotify org, archived repositories | source | listing as of 2026-09-14 | 2026-09-14 | https://github.com/orgs/spotify/repositories?q=archived%3Atrue | 163 of Spotify's public repositories are archived | Listing header: 163 archived repositories; entries include helios (Sep 12, 2024), styx (Jul 12, 2023), zoltar (Sep 11, 2023), dns-java (Aug 1, 2024), talos (Aug 6, 2024) |
| 18 | Spotify | spotify org, archived repositories page 2 | source | listing as of 2026-09-14 | 2026-09-14 | https://github.com/orgs/spotify/repositories?q=archived%3Atrue&page=2 | Retirement happens in batches: eight repositories carry a March 2022 date | scanctl, lingon, hdfs2cass, docgenerator, netty-zmtp, spydra all "Updated Mar 31, 2022"; cassandra-medusa and comet "Mar 29, 2022" |
| 19 | Spotify | spotify org, active repositories | source | listing as of 2026-09-14 | 2026-09-14 | https://github.com/orgs/spotify/repositories?q=archived%3Afalse&sort=updated | 126 repositories are active, and the recent ones are product libraries and agent tooling rather than infrastructure | Total 126; recent entries confidence-sdk-js, confidence-resolver (Rust), pedalboard, ads-agentic-tools, flink-on-k8s-operator |
| 20 | Spotify | spotify/flink-on-k8s-operator | source | active, beta | 2026-09-14 | https://github.com/spotify/flink-on-k8s-operator | Spotify adopted and now maintains a Google-originated operator, running Flink and Beam on its own Kubernetes | "an open source fork of GoogleCloudPlatform/flink-on-k8s-operator with several new features and bug fixes"; 227 stars, 932 commits |
| 21 | Spotify | spotify/dbeam repository | source | production since 2017-08 | 2026-09-14 | https://github.com/spotify/dbeam | A fourth exit door exists: software declared finished rather than abandoned | "Mature, maintained and used in production since August 2017"; "No major features or development planned" |
| 22 | Spotify | spotify/luigi repository | source | active | 2026-09-14 | https://github.com/spotify/luigi | The 2012 workflow library outlived the 2016 scheduler built to run it | Maintained by Spotify's data team; 18.8k stars, 4,320 commits |
| 23 | PyPI | luigi release history | source | 2012-10-21 to 2026-05-07 | 2026-09-14 | https://pypi.org/pypi/luigi/json | Luigi has shipped 87 releases across fourteen years and still releases in 2026 | first ('1.0', '2012-10-21'), last ('3.8.1', '2026-05-07'), 87 releases |
| 24 | PyPI | annoy release history | source | 2014-03-24 to 2023-06-14 | 2026-09-14 | https://pypi.org/pypi/annoy/json | Annoy's last release is six weeks before its successor's first | first ('1.0', '2014-03-24'), last ('1.17.3', '2023-06-14') |
| 25 | PyPI | voyager release history | source | 2023-07-28 to 2024-12-13 | 2026-09-14 | https://pypi.org/pypi/voyager/json | Voyager took over from Annoy in mid-2023 | first ('1.2.5', '2023-07-28'), last ('2.1.0', '2024-12-13') |
| 26 | Spotify | spotify/voyager repository | source | 2023 onward | 2026-09-14 | https://github.com/spotify/voyager | The replacement is justified on recall, not on memory layout, and runs at high query volume | "like Annoy, but with much higher recall"; "queried hundreds of millions of times per day to power numerous user-facing features" |
| 27 | Google Cloud | Spotify's journey to the cloud: why Spotify migrated its event delivery system from Kafka to Google Cloud Pub/Sub | blog | 2016-03-03 | 2026-09-14 | https://cloud.google.com/blog/products/gcp/spotifys-journey-to-cloud-why-spotify-migrated-its-event-delivery-system-from-kafka-to-google-cloud-pubsub | The pre-migration stack and its scale: Kafka 0.8, HDFS, Crunch-MapReduce, 700k events per second | "700,000 events per second"; load tested at "2,000,000 messages per second"; previous stack "Kafka", "HDFS", "Crunch-MapReduce for ETL into AVRO" |
| 28 | Google Cloud | Spotify's experiments with stream processing on Google Cloud Dataflow | blog | 2016-03-11 | 2026-09-14 | https://cloud.google.com/blog/products/gcp/spotifys-experiments-with-stream-processing-on-google-cloud-dataflow | Dataflow replaced the Kafka, HDFS and Crunch-MapReduce pipeline, with windows and watermarks as the new operational surface | Lists what was replaced ("Kafka", "HDFS", "Crunch-MapReduce") and the topics of the migration: "window and watermark to work with late arriving data" |
| 29 | Spotify | com.spotify artefacts on Maven Central | source | directory listing | 2026-09-14 | https://repo1.maven.org/maven2/com/spotify/ | The Java-era estate is legible as a single directory: apollo-*, crtauth, dns, docker-client, crunch, dataproc-java-submitter, bigtable-autoscaler, heroic, scio, dbeam | Directory entries include apollo-api/, crtauth/, dns/, crunch/, dataproc-java-submitter/, bigtable-autoscaler/, docker-client/ |
| 30 | Backstage / CNCF | backstage/backstage repository | source | CNCF incubation | 2026-09-14 | https://github.com/backstage/backstage | The one platform component Spotify kept building is now a CNCF incubating project with 34.4k stars | "hosted by the Cloud Native Computing Foundation as an Incubation level initiative"; 34.4k stars, 7.6k forks |
| 31 | Backstage | ADR001: Architecture Decision Record log | adr | 2020 | 2026-09-14 | https://raw.githubusercontent.com/backstage/backstage/master/docs/architecture-decisions/adr001-add-adr-log.md | The project's first recorded decision is to record decisions, and it names its own failure mode | "A decision was made to store ADRs in a log in the project repository."; Risks: "People stop adding ADRs to the log and context gets lost." |
| 32 | Backstage | ADR002: Default Software Catalog File Format | adr | 2020 | 2026-09-14 | https://raw.githubusercontent.com/backstage/backstage/master/docs/architecture-decisions/adr002-default-catalog-file-format.md | The open-source catalogue is a port of an internal Spotify system, modelled on Kubernetes objects | "Internally at Spotify, a homegrown software catalog system is used heavily and forms a core part of Backstage"; "draws heavy inspiration from the kubernetes object format" |
| 33 | Backstage | Backstage Enhancement Proposals, README | adr | process doc | 2026-09-14 | https://raw.githubusercontent.com/backstage/backstage/master/beps/README.md | The project added a KEP-style proposal process on top of ADRs, with ownership as the merge condition | "The merging of a BEP means that it is approved for implementation and has an owner that is responsible for said implementation." |
| 34 | Backstage | BEP 0003: Auth Architecture Evolution | adr | created 2024-01-28, status implemented | 2026-09-14 | https://raw.githubusercontent.com/backstage/backstage/master/beps/0003-auth-architecture-evolution/README.md | Four years after open sourcing, the default trust model was inverted to deny by default | "Built-in protection of Backstage instances such that it is safe to deploy Backstage directly towards the internet."; "It will no longer be a requirement to either set up protection of your Backstage instance or not do so and risk exposing your instance to malicious actors." |
| 35 | Backstage | BEP 0003, token size finding | adr | 2024-01-28 | 2026-09-14 | https://raw.githubusercontent.com/backstage/backstage/master/beps/0003-auth-architecture-evolution/README.md | Ownership data in the user token grew until it affected reliability, and had to be moved behind a service | "Separate out the ownership information out of the Backstage user tokens, since user tokens have been growing large enough to have an impact on performance and reliability." |
| 36 | Backstage | BEP 0002: Dynamic Frontend Plugins | adr | created 2024-01-17, status provisional | 2026-09-14 | https://raw.githubusercontent.com/backstage/backstage/master/beps/0002-dynamic-frontend-plugins/README.md | The build-time plugin model is a known adoption barrier, and the fix has been provisional for over two years | status: provisional; "A Backstage installation currently requires quite a lot of care to maintain, meaning it may not be worth the investment for smaller organizations." |
| 37 | CNCF | cncf/toc issue search for Backstage | casestudy | 2020-09-08, 2022-03-15, 2024-09-13 | 2026-09-14 | https://github.com/cncf/toc/issues?q=backstage | The donation timeline: sandbox in September 2020, incubation in March 2022, governance review opened September 2024 | #476 "Add Backstage as CNCF Sandbox project" merged Sep 8 2020; #717 "Propose Backstage to CNCF Incubation Stage" merged Mar 15 2022; #1598 "[Governance Review]: Backstage" open since Sep 13 2024 |
| 38 | Backstage | Security advisories, listing pages 1 to 7 | postmortem | 2021-06-03 to 2026-09-11 | 2026-09-14 | https://github.com/backstage/backstage/security/advisories | 69 advisories are published, and 37 of them carry a single date, 28 August 2026 | Pages 1-3 and most of page 4 dated "Aug 28, 2026"; page 7 dated Jun 3 2021 through Feb 14 2023; page 8 empty |
| 39 | Backstage | GHSA-g2v8-7jhw-pp8p, Sensitive information exposure in Scaffolder | postmortem | 2026-08-28 | 2026-09-14 | https://github.com/backstage/backstage/security/advisories/GHSA-g2v8-7jhw-pp8p | The most severe published defect is a cross-user read of template execution data, CVSS 9.6 | "An authenticated Backstage user who can read another user's Scaffolder task may receive internal execution data. In deployments where that data contains credentials for an external service, this may permit disclosure and unauthorized changes in that external service." |
| 40 | Backstage | Advisory listing page 7 (2021 to 2023) | postmortem | 2021-11-26, 2023-06-21 | 2026-09-14 | https://github.com/backstage/backstage/security/advisories?page=7 | The template engine has been the recurring soft spot since 2021 | GHSA-2g8g-63j4-9w3r "RCE vulnerability affecting v1beta3 templates in @backstage/plugin-scaffolder-backend", High, Nov 26 2021 |
| 41 | Backstage | Advisory listing pages 5 and 6 (2024 to 2026) | postmortem | 2023-06-21 to 2026-03-11 | 2026-09-14 | https://github.com/backstage/backstage/security/advisories?page=6 | The same two features keep producing findings across five years | GHSA-wg6p-jmpc-xjmr "Insecure sandbox in Backstage Scaffolder plugin", High, Jun 21 2023; GHSA-6jr7-99pf-8vgf "Arbitrary code execution via MkDocs hooks in TechDocs", High, Jan 30 2026 |
| 42 | npm | @backstage/plugin-scaffolder-backend registry metadata | source | 2020-06-17 to 2026-08-28 | 2026-09-14 | https://registry.npmjs.org/@backstage/plugin-scaffolder-backend | The remediation release for the 37-advisory day is dated the same day | created 2020-06-17; 2,144 versions; latest 4.1.0 published 2026-08-28 |
| 43 | npm | @backstage/backend-common registry metadata | source | 2020-04-30 to 2024-09-17 | 2026-09-14 | https://registry.npmjs.org/@backstage/backend-common | The old backend core is deprecated in the registry, with 1,347 published versions behind it | latest 0.25.0 (2024-09-17), deprecated: "This package is deprecated, please follow the deprecation instructions for the exports you still use" |
| 44 | npm | @backstage/core registry metadata | source | 2020-03-26 to 2021-06-24 | 2026-09-14 | https://registry.npmjs.org/@backstage/core | Backstage's first public package dates the open sourcing, and was itself superseded inside fifteen months | created 2020-03-26 (0.1.1-alpha.0); last version 0.7.14 on 2021-06-24 |
| 45 | Backstage | Issue 26665, timeout at 14k user entities | source | opened 2024-09-13, closed not planned | 2026-09-14 | https://github.com/backstage/backstage/issues/26665 | Adopters find the catalogue's scale ceiling, and the finding is closed unresolved | "every fetch request will timeout" at roughly 14,000 user entities; 9,000 entities fine, 21,000 total entities frozen; closed as not planned |
| 46 | Backstage | Issue 27907, slow facet queries | source | opened 2024-11-28, closed not planned | 2026-09-14 | https://github.com/backstage/backstage/issues/27907 | Catalogue facet queries cost seconds at a few thousand entities | over 2,000 components; facet queries "1-3 seconds, occasionally reaching 6 seconds"; catalog page load 6.5 seconds; closed as not planned |
| 47 | Backstage | PR 20409, continued MySQL support | source | closed unmerged 2024-01-31 | 2026-09-14 | https://github.com/backstage/backstage/pull/20409 | A second database engine was rejected on maintenance cost, not on capability | Maintainer Rugvip: "why support clustering of this data at all? It's never expected to be any particularly large volume of data"; complexity "not maintainable long-term" |
| 48 | Backstage | PR 32623, queue service | source | opened 2026-02-02, closed unmerged | 2026-09-14 | https://github.com/backstage/backstage/pull/32623 | A core async-job service was held back for want of a first internal consumer | Maintainer: "I do think we should aim for something that could conceivably replace some of the catalog processing queues" |
| 49 | Backstage | Closed unmerged pull requests by comment count | source | 2021 to 2026 | 2026-09-14 | https://github.com/backstage/backstage/pulls?q=is%3Apr+is%3Aclosed+is%3Aunmerged+sort%3Acomments-desc | The rejected-design record is substantial and concentrated on core architecture | #15519 "[PRFC] Rethink GraphQL plugin", 141 comments, closed Aug 29 2023; #29196 "[BEP 0006] Scaffolder Action Rollback (via checkpoints)", 90 comments, closed May 7 2026 |
| 50 | Backstage | backstage/community-plugins repository | source | 2024 onward | 2026-09-14 | https://github.com/backstage/community-plugins | Roughly 60 plugins were moved out of the core monorepo into per-workspace ownership and releases | "each plugin workspace has its own changesets and isolated releases"; 9,702 commits, 697 forks |
| 51 | Red Hat | redhat-developer/rhdh | casestudy | active | 2026-09-14 | https://github.com/redhat-developer/rhdh | A downstream distribution shipped dynamic plugins while the upstream proposal stayed provisional | "an enterprise-grade Internal Developer Portal based on Backstage"; dynamic-plugins-root directory; "formerly known as janus-idp/backstage-showcase" |
| 52 | Backstage | ADOPTERS.md | casestudy | living document | 2026-09-14 | https://raw.githubusercontent.com/backstage/backstage/master/ADOPTERS.md | Roughly 290 organisations list themselves as adopters, Spotify included as its own first entry | Spotify's row: "Main interface towards all of Spotify's infrastructure and technical documentation."; 290 table rows |
| 53 | Backstage | Releases listing | source | v1.54.7 and v1.49.7 both 2026-09-11 | 2026-09-14 | https://github.com/backstage/backstage/releases | The project maintains several release lines at once and patches them on the same day | v1.54.7 and v1.49.7 both dated September 11; v1.55.0-next.2 September 8 |
| 54 | Backstage | Frontend system documentation | vendor | current | 2026-09-14 | https://raw.githubusercontent.com/backstage/backstage/master/docs/frontend-system/index.md | The migration to the new frontend system is recommended while the enabling proposal is still provisional | "We recommend migrating your frontend plugins to the new frontend system." |
| 55 | Backstage | Backend system documentation | vendor | current | 2026-09-14 | https://raw.githubusercontent.com/backstage/backstage/master/docs/backend-system/index.md | The backend was re-architected around plugins, modules and services with split deployments | "how to split your backend into multiple deployments for better scalability and security isolation" |
| 56 | Spotify | spotify/XCMetrics | casestudy | active | 2026-09-14 | https://github.com/spotify/XCMetrics | The build-metrics platform is a rare published internal scale figure | "collected almost 1 million builds and over 10 billion steps from all Spotify iOS applications" |
| 57 | crates.io | confidence_resolver crate | source | created 2026-01-19 | 2026-09-14 | https://crates.io/api/v1/crates?q=confidence-resolver | Spotify's newest published infrastructure component is a Rust crate, first released in 2026 | "created_at":"2026-01-19", "downloads":72115, "num_versions":25, "description":"Local resolver for Confidence feature flags" |
| 58 | Backstage | Roadmap 2026 issue | source | opened 2025-12-09 | 2026-09-14 | https://github.com/backstage/backstage/issues/32083 | Direction is now set by community input rather than by the originating company | "Backstage continues to grow through community collaboration, and this issue is an open invitation to help shape where we focus our collective efforts in 2026." |
What is not in this corpus, and what that costs
- No outage postmortem. Spotify publishes none that this session could reach, and the Backstage project publishes security advisories rather than incident reviews. Every entry in the failure section of the guide is therefore an advisory, a bug report from an adopter, or a defect visible in a build file. Blast radius is mostly unknown as a result, and the guide says so on each card.
- No talks and no papers. Both are hosted where this session cannot reach.
- No internal architecture. The music serving path, the recommendation stack, the storage layer behind them, and anything Spotify never published, are absent. The guide's claims are about the platform layer that leaves a public trace.
- One vendor-hosted pair of sources. The two 2016 posts are Google's write-ups of Spotify
engineering posts. They are graded
blogbecause the technical content is Spotify's, and they are the only pre-2020 narrative evidence in the corpus.