Evidence ledger
32 sources
Checked 04 Oct 2026
Evidence ledger
One row per claim in Replacing the engine room in public: ten years of Sentry, read from its RFCs, releases and self-host floor: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Field guide: Replacing the engine room in public: ten years of Sentry, read from its RFCs, release notes and self-host floor. Research date 2026-10-04. Every row was fetched in this session.
Corpus limit, stated up front. This session's network policy reached github.com,
raw.githubusercontent.com and nothing else. sentry.io, blog.sentry.io,
develop.sentry.dev, infoq.com, arxiv.org, usenix.org and web.archive.org all returned
an egress block when tested. Sentry's engineering blog, its conference talks, its developer
documentation and every published figure about sentry.io's own event volume are therefore absent
from this ledger, and so is any paper. What remains is the repository record: RFCs, source
files, release notes, changelog entries, pull requests and operator-filed issues. That record is
a good witness for what shipped and when, a usable witness for why, and a poor witness for how
large the production system is.
Tiers follow the skill's hierarchy: postmortem, source, adr, casestudy, blog,
paper, talk, vendor.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Sentry | RFC 0002: Sentry Architecture Vision (Living Document) | adr | 2022-07-21 | 2026-10-04 | https://github.com/getsentry/rfcs/blob/main/text/0002-new-architecture.md | The 2022 architecture programme was opened by a scaling limit, not a product requirement, and it named the replacements that followed | "We are running into scaling limitations on our current infrastructure and as such some larger decisions have to be made about the future of our codebase." |
| 2 | Sentry | RFC 0002: Sentry Architecture Vision (Living Document) | adr | 2022-07-21 | 2026-10-04 | https://github.com/getsentry/rfcs/blob/main/text/0002-new-architecture.md | The programme explicitly included leaving pickle, leaving RabbitMQ for Kafka, and replacing buffers with Kafka and ClickHouse | Workstreams listed: extracting celery tasks into a separate sentry_pipeline package, "replacing pickle serialization with formats like JSON", "potentially migrating from RabbitMQ to Kafka", "phasing out buffer systems in favor of Kafka and ClickHouse" |
| 3 | Sentry | RFC 0072: Centralized Schema Repository for Kafka Topics | adr | 2023-02-01 | 2026-10-04 | https://github.com/getsentry/rfcs/blob/main/text/0072-kafka-schema-registry.md | Kafka became the inter-service contract, and schema disagreement had already caused production incidents | "Kafka is increasingly used as the message bus between services at Sentry. Kafka topics and their schemas are usually not internal to any one service, but part of the contract between services." Incidents named affecting post-processing, transaction consumption and replay |
| 4 | Sentry | RFC 0072: Centralized Schema Repository for Kafka Topics | adr | 2023-02-01 | 2026-10-04 | https://github.com/getsentry/rfcs/blob/main/text/0072-kafka-schema-registry.md | A schema registry service was rejected because it would have to run in every deployment flavour, including open source | Option B (a separate schema service, "potentially using Confluent Schema Registry") rejected for infrastructure overhead and the cost of keeping it available "in all regions, open source, dev, CI and single tenant installations" |
| 5 | Sentry | RFC 0119: Make it easier to use Rust code from Sentry/Python | adr | 2023-10-25 | 2026-10-04 | https://github.com/getsentry/rfcs/blob/main/text/0119-rust-in-sentry.md | Rust is pulled into the Python monolith through one private bindings package rather than per-crate public wheels | Chosen: a single Sentry-specific bindings package in a separate repository (ophio) using PyO3 and maturin, for "Only a single Python extension module to build / care about with fixed overhead" and "Ability to move functionality from Python to Rust more fine-grained" |
| 6 | Sentry | RFC 0119: Make it easier to use Rust code from Sentry/Python | adr | 2023-10-25 | 2026-10-04 | https://github.com/getsentry/rfcs/blob/main/text/0119-rust-in-sentry.md | Publishing the bindings publicly was rejected on maintenance grounds | Rejected alternative would "still require maintaining a public SemVer API", depend on PyPI publishing, and add "multiple smaller Python extension modules will increased the fixed per-module overhead" |
| 7 | Sentry | getsentry/rfcs, closed-unmerged pull requests | source | 2023-01 to 2026-03 | 2026-10-04 | https://github.com/getsentry/rfcs/pulls?q=is%3Apr+is%3Aclosed+is%3Aunmerged | Twenty RFCs were withdrawn rather than decided, and the withdrawals cluster on client-side and product proposals | Listing includes #115 Custom Metrics in SDKs, #111 Combined Dynamic Sampling, #87 Per Category Abuse Rate Limiting, #64 Proposal to modify RFC workflow state process |
| 8 | Sentry | RFC PR #115: Custom Metrics in SDKs | source | opened 2023-10-03, closed 2024-11-12 | 2026-10-04 | https://github.com/getsentry/rfcs/pull/115 | A whole ingestion path was proposed, built against, and then abandoned in public with one sentence | philippsuess on closing: "We can close this as we haven't moved forward with metrics in the SDKs." Proposal required that "SDKs are expected to aggregate locally for windows of up to 10 seconds" |
| 9 | Sentry | getsentry/relay README | source | n/a | 2026-10-04 | https://github.com/getsentry/relay/blob/master/README.md | The ingest edge is a separate Rust process that, in processing mode, writes to Kafka instead of to the application | "The Sentry Relay is a service that pushes some functionality from the Sentry SDKs as well as the Sentry server into a proxy process." Processing mode "producing events to Kafka topics instead of forwarding" |
| 10 | Sentry | getsentry/relay CHANGELOG | source | 24.8.0 to 26.7.0 | 2026-10-04 | https://github.com/getsentry/relay/blob/master/CHANGELOG.md | The edge grew a disk-backed SQLite buffer in 2024 and the older in-memory spooling was then deleted | 24.9.0: "Allow creation of SqliteEnvelopeBuffer from config, and load existing stacks from db on startup." (#3967); 24.11.2: "Remove old disk spooling logic, default to new version." (#4303) |
| 11 | Sentry | getsentry/snuba README | source | n/a | 2026-10-04 | https://github.com/getsentry/snuba/blob/master/README.rst | Search and aggregation were moved off the relational store onto ClickHouse | "Snuba was originally developed to replace a combination of Postgres and Redis to search and provide aggregated data on Sentry errors." |
| 12 | Sentry | Snuba architecture overview | source | n/a | 2026-10-04 | https://github.com/getsentry/snuba/blob/master/docs/source/architecture/overview.rst | ClickHouse was chosen for a balance of real-time performance and replication, and Kafka is the only ingestion input | "Clickhouse was chosen as backing storage because it provides a good balance of the real time performance Snuba needs, its distributed and replicated nature, its flexibility in terms of storage engines and consistency guarantees." Ingestion "is ingested through input streams (only Kafka topics today)" |
| 13 | Sentry | getsentry/taskbroker README | source | n/a | 2026-10-04 | https://github.com/getsentry/taskbroker/blob/main/README.md | The replacement task system is a Rust broker with a local SQLite store, built against head-of-line blocking | "Taskbroker provides a Kafka consumer, RPC interface, and inflight task storage that form the core engine of asynchronous task execution at Sentry." Purposes: avoiding head-of-line blocking, out-of-order execution, per-task acknowledgements; workers call GetTask and SetTaskStatus over gRPC |
| 14 | Sentry | self-hosted release 25.6.0 | source | 25.6.0 (June 2025) | 2026-10-04 | https://github.com/getsentry/self-hosted/releases/tag/25.6.0 | Taskbroker was announced to self-hosters as the Celery replacement before the switch was complete | "Taskbroker is a new service (written in Rust) that aims to replace Celery, which is backed by Redis PubSub. In the next release, Taskbroker will fully replace Celery, taking over the roles of the worker and cron containers." |
| 15 | Sentry | self-hosted CHANGELOG | source | 23.11.0 to 26.9.0 | 2026-10-04 | https://github.com/getsentry/self-hosted/blob/master/CHANGELOG.md | The Celery replacement was announced, then held back for self-hosted, then resumed over more than a year | 25.6.0: "feat: Add taskbroker + worker + scheduler" (#3738); 25.7.0: "feat: Continue using celery in self-hosted for now" (#3845); 25.9.0: "feat(tasks): Remove taskworker option override and add worker healthcheck" (#3933); 26.7.0: "Port subscriptions consumers to tasks" (#4395) |
| 16 | Sentry | self-hosted release 25.8.0 | source | 25.8.0 (Aug 2025) | 2026-10-04 | https://github.com/getsentry/self-hosted/releases/tag/25.8.0 | Operators were given an explicit option to stay on the old task system during the switch | "make sure you have SENTRY_OPTIONS[\"taskworker.enabled\"] = False so your jobs continue running on Celery." |
| 17 | Sentry | self-hosted docker-compose.yml at tag 9.1.2 | source | tag 9.1.2 | 2026-10-04 | https://github.com/getsentry/self-hosted/blob/9.1.2/docker-compose.yml | The shipped deployment was seven containers before the replacements began | Services: smtp, memcached, redis, postgres, web, cron, worker. Total 7 |
| 18 | Sentry | self-hosted docker-compose.yml at tag 20.12.1 | source | tag 20.12.1 (Dec 2020) | 2026-10-04 | https://github.com/getsentry/self-hosted/blob/20.12.1/docker-compose.yml | By the end of 2020 the shipped deployment carried Zookeeper, Kafka, ClickHouse, Snuba consumers, Symbolicator and Relay | 30 services including zookeeper, kafka, clickhouse, snuba-api, snuba-consumer, snuba-replacer, symbolicator, relay, ingest-consumer, post-process-forwarder, cron, worker |
| 19 | Sentry | self-hosted docker-compose.yml on master | source | n/a | 2026-10-04 | https://github.com/getsentry/self-hosted/blob/master/docker-compose.yml | The shipped deployment is now 51 containers, Zookeeper is gone, and cron and worker have been replaced by taskbroker, taskscheduler and taskworker | 51 services including pgbouncer, seaweedfs, 18 snuba-* consumers, relay, taskbroker, taskscheduler, taskworker, launchpad-taskworker, uptime-checker, vroom; no zookeeper, no cron, no worker |
| 20 | Sentry | self-hosted README at tag 20.12.1 | source | tag 20.12.1 (Dec 2020) | 2026-10-04 | https://github.com/getsentry/self-hosted/blob/20.12.1/README.md | The documented memory floor at the end of 2020 was 2400 MB | "You need at least 2400MB RAM" |
| 21 | Sentry | self-hosted install/_min-requirements.sh | source | n/a | 2026-10-04 | https://github.com/getsentry/self-hosted/blob/master/install/_min-requirements.sh | The current hard floor is 14 GB of RAM and 4 CPUs, halved only for an errors-only install | MIN_RAM_HARD=14000 and MIN_CPU_HARD=4 by default; MIN_RAM_HARD=7000 and MIN_CPU_HARD=2 when COMPOSE_PROFILES="errors-only" |
| 22 | Sentry | Commit history of install/_min-requirements.sh | source | 2021-08-18 to 2025-08-14 | 2026-10-04 | https://github.com/getsentry/self-hosted/commits/master/install/_min-requirements.sh | The floor became a hard install-time gate in August 2024, and a reduced profile was added seven months later | Commits: "Mandate minimum requirements for ram/cpu (#3275)" dated Aug 17, 2024; "fix: more leeway for minimum RAM (#3290)" dated Aug 23, 2024; "Minimum requirements for 'errors-only' profile (#3634)" dated Mar 27, 2025 |
| 23 | Sentry | self-hosted PR #3634: Minimum requirements for 'errors-only' profile | source | merged 2025-03-27 | 2026-10-04 | https://github.com/getsentry/self-hosted/pull/3634 | The errors-only profile exists because the full product roughly doubles the resource floor, and reviewers doubted the smaller numbers | Author: "Using the errors-only profile, fewer resources are required. About 2 times." Reviewer aldy505: "Kafka and Clickhouse would still take a lot of resources" |
| 24 | Sentry | install/check-minimum-requirements.sh | source | n/a | 2026-10-04 | https://github.com/getsentry/self-hosted/blob/master/install/check-minimum-requirements.sh | The installer refuses to proceed below the floor and separately requires an instruction-set feature for ClickHouse | "Required minimum RAM available to Docker is $MIN_RAM_HARD MB, found $RAM_AVAILABLE_IN_DOCKER MB"; an SSE 4.2 check required for ClickHouse on x86_64 unless SKIP_SSE42_REQUIREMENTS=1 |
| 25 | Sentry | src/sentry/silo/base.py | source | n/a | 2026-10-04 | https://github.com/getsentry/sentry/blob/master/src/sentry/silo/base.py | The same codebase runs as one monolith or as split control and region deployments, and the renaming of region to cell is still visible in the enum | class SiloMode(Enum) with docstring "Defines which \"silo\" component the application is acting as. The default choice is \"monolith\", which assumes that the server is the only \"silo\" in its environment and allows access to all tables and endpoints." Members: MONOLITH, CONTROL, CELL = "REGION" |
| 26 | Sentry | src/sentry/types/cell.py | source | n/a | 2026-10-04 | https://github.com/getsentry/sentry/blob/master/src/sentry/types/cell.py | Deployment topology is now cells grouped into localities, with the monolith kept as a synthetic single cell | "A cell of the Sentry platform, hosted by a region silo."; a locality is "A grouping of one or more cells (e.g. 'us' contains 'us1', 'us2')"; in monolith mode "there exists only the one monolith 'region', which is a dummy object" |
| 27 | Sentry | getsentry/sentry pull requests mentioning silo, oldest first | source | 2022-08-31 | 2026-10-04 | https://github.com/getsentry/sentry/pulls?q=is%3Apr+silo+in%3Atitle+sort%3Acreated-asc | The topology split began at the end of August 2022 and was labelled hybrid from the first commit | #37983 "Experiment with new CI jobs for silo modes" and #38277 "ref(hybrid): Standardize nomenclature around silos", both RyanSkonnord, Aug 31 2022 |
| 28 | Sentry | Commit history of LICENSE.md in getsentry/sentry | source | 2023-11-17 and 2024-02-27 | 2026-10-04 | https://github.com/getsentry/sentry/commits/master/LICENSE.md | The licence of the shipped artefact changed twice inside four months while the architecture work was running | "Relicense under FSL-1.0-Apache-2.0 (#60144)" dated Nov 17 2023; "Upgrade license to FSL-1.1 (#65879)" dated Feb 27 2024 |
| 29 | Sentry | getsentry/sentry repository page | source | n/a | 2026-10-04 | https://github.com/getsentry/sentry | The whole product is still one repository and one history | "111,734 commits" on master; 45.3k stars; description "Developer-first error tracking and performance monitoring" |
| 30 | operators of self-hosted Sentry | Issue #4240: Relay stopped processing new metrics and events | postmortem | opened 2026-03-24 | 2026-10-04 | https://github.com/getsentry/self-hosted/issues/4240 | When the broker stops accepting writes, the Rust edge stops and the operator sees no events rather than an error | Reported symptom: "Sentry stops processing new metrics and events"; relay logs "failed to produce message to Kafka (delivery callback) error=Message production error: MessageTimedOut (Local: Message timed out)". Label "Waiting for: Product Owner" |
| 31 | operators of self-hosted Sentry | Issue #4485: Multiple Sentry consumers repeatedly become unhealthy due to Kafka coordinator/session timeouts | postmortem | opened 2026-08-21 | 2026-10-04 | https://github.com/getsentry/self-hosted/issues/4485 | Consumer-group coordination is a shared failure domain across eight independent consumers | "Consumer group session timed out (in join-state steady) after 45000 ms without a successful response from the group coordinator", with NOT_COORDINATOR, _WAIT_COORD and COORDINATOR_LOAD_IN_PROGRESS across snuba-subscription-consumer-events, process-segments, post-process-forwarder-errors, monitors-clock-tasks, ingest-monitors, generic-metrics-consumer, events-consumer and snuba-commit-log |
| 32 | operators of self-hosted Sentry | Issue #1894: Kafka Error after update to 22.12.0 from 22.11.0 | postmortem | opened 2023-01-02, closed | 2026-10-04 | https://github.com/getsentry/self-hosted/issues/1894 | A monthly version bump can leave consumers crash-looping on offsets that no longer exist | "fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)", Snuba consumers crash-looping after 22.11.0 to 22.12.0 |
| 33 | operators of self-hosted Sentry | Issue #2876: Sentry stopped accepting transaction data | postmortem | opened 2024-03-10, closed as not planned | 2026-10-04 | https://github.com/getsentry/self-hosted/issues/2876 | The pipeline's own telemetry reports the loss as a number, and the thread was closed without a root cause | "Performance page shows zeros for the time period since the update and until now"; "Stats page shows 49k transactions of which 49k are dropped"; ClickHouse logs "Net Exception: Socket is not connected" |
| 34 | Sentry | self-hosted release 25.6.2 | postmortem | 25.6.2 (June 2025) | 2026-10-04 | https://github.com/getsentry/self-hosted/releases/tag/25.6.2 | The release channel itself is the rollback mechanism: two consecutive releases were marked as ones to skip after a database migration failure | "If you are coming from 25.5.1, you might want to skip 25.6.0 and 25.6.1, and upgrade directly to this version." |
| 35 | Sentry | getsentry/rfcs text directory | source | n/a | 2026-10-04 | https://github.com/getsentry/rfcs/tree/main/text | The accepted public decision record is small and skewed to client-side concerns, with gaps in the numbering where proposals were withdrawn | 71 files listed, numbered 0001 to 0157, the majority naming SDK, span, replay and symbolication concerns rather than server architecture |
Categories with no rows, and what that means
- blog, talk, paper, casestudy: zero rows. Not because none exist, but because every host that would carry them was blocked for this session. Anyone extending this guide should start at Sentry's own engineering blog and at its conference talks, and should expect them to contradict or sharpen the inference in section 2, which is reconstructed from artefacts rather than reported.
- vendor: zero rows, deliberately. The only vendor-shaped material in reach was the product README.
- postmortem: five rows, four of them written by operators of the self-hosted edition rather than by Sentry. Sentry's own incident write-ups are published off GitHub and were out of reach. Read the failure section as the operator's view of this architecture, which is the only published view of it.