Ten years of Sentry's data platform, reconstructed from the commits that deleted things: five storage engines retired behind four unchanged interfaces, a task queue that took ten years and five months to remove, and three dual-backend shims whose measured lifetimes are the number nobody budgets for. A reader finishes able to cost the transition period of a storage migration rather than the migration.
Every team that outgrows its first database writes a plan for the new one. Almost nobody writes a plan for the period when both are running. This guide measures that period.
State the problem without naming a product. A system has an interface that the rest of the application calls to read and write some class of data. The engine behind that interface stops fitting, for reasons that are usually about the shape of the data rather than its volume. You write a second implementation of the same interface, run both for a while so you can compare and roll back, then delete the first. The question this guide answers is what the middle step costs, because that is the step that appears in no design document and has no owner.
Sentry is an unusually good subject for that question. The company has published its error-tracking server as source since 2008; the monolith repository carries 111,408 commits as of 30 September 2026, and the self-hosted deployment repository has a commit on 11 February 2016, which gives a clean ten-year window. More usefully, Sentry does its storage swaps as pluggable backends in a directory per interface, so the arrival and the deletion of each engine are both single commits with dates. The transition period is not inferred here. It is subtracted.
Across the four interfaces examined, the engines behind them lasted between 17 and 72 months. The dual-backend shims written to bridge them lasted 4, 66 and 91 months. Two of the three shims outlived the engine they were introduced to retire, and the longest one is still in the tree, being pruned by a commit dated 25 August 2026, seven and a half years after it was added.
What this guide covers: the data platform and the asynchronous work that feeds it, read from public repositories and package registries. What it deliberately does not cover: Sentry's SaaS topology, its capacity, its costs and its published incident narratives, all of which live on hosts this session's egress policy could not reach. Where an incident appears below, it is reconstructed from the commit that fixed it and the pull request that explains it, never from an incident report, and that limit is stated on each card.
Four named interfaces have survived the decade unchanged in purpose. Everything behind them has been replaced at least once, and in one case four times.
The shape below is what the repositories describe today. Read it as two separate
things. The flow from Relay through Kafka to the consumers is the pipeline, and it is the
part that gets rewritten in other languages. The links from the monolith down to the
stores are the interfaces, and they are the part that does not move. Sentry's directory layout says this out loud: the monolith carries
src/sentry/nodestore, src/sentry/tagstore,
src/sentry/search and src/sentry/tsdb, each holding one
subdirectory per engine plus a base class, which is why a swap is a directory and a
deletion rather than a rewrite.
Relay's README describes it as a service that "pushes some functionality from the
Sentry SDKs as well as the Sentry server into a proxy process". Its first commit is
17 January 2018, under the name smith, renamed to semaphore
on 9 May 2018 and later to Relay. The Python binding that lets the monolith call it,
sentry-relay, has been on PyPI since 27 January 2020.
Evidence: relay README, PyPI
Snuba's README states it "was originally developed to replace a combination of Postgres and Redis to search and provide aggregated data on Sentry errors". The architecture document gives the reason for the engine: ClickHouse "provides a good balance of the real time performance Snuba needs, its distributed and replicated nature, its flexibility in terms of storage engines and consistency guarantees".
Evidence: snuba README, architecture overview
Taskbroker consumes task activations from Kafka and "stores them in a SQLite database to avoid head-of-line blocking, enable out-of-order execution and per-task acknowledgements". Workers pull over gRPC. The interesting move is putting a local durable store back in front of workers because Kafka's offset model cannot express a per-task acknowledgement.
Evidence: taskbroker README, first commit 2024-11-01
The consumer layer deserves its own note, because it is the part that was extracted into
a library rather than a service. Arroyo, first committed on 11 June 2021 and published to
PyPI as sentry-arroyo on 29 June 2021, supplies the Kafka consumer and a set
of composable strategies, described in its README as pre-built steps such as
RunTask, Filter, Reduce and
CommitOffsets that are "chained together to form complex message processing
pipelines". By July 2026 the reference deployment runs 27 containers whose names begin
snuba- and a further 14 consumers on the Sentry side. That is the real reason
a consumer framework became a library: the number of consumers grew faster than the number
of services.
multi or
combined backend is the shim, and it is the box that turns out to be permanent.
Reconstructed from the directory layout and git history of
src/sentry/tsdb.One more component belongs in the reference shape even though it is not a data store.
Between September 2022 and September 2023 the monolith grew src/sentry/silo
and then src/sentry/hybridcloud, splitting the single application into a
control plane and regional planes. That change was not driven by load. It is the shape a
product takes when customers require their data to stay in a named jurisdiction, and it
is worth noticing that it arrived through the same mechanism as the storage swaps: a
decorator layer over the existing ORM models rather than a new system.
Four forks with the reasons written down at the time, and the condition that would flip each one.
The chain between those two artefacts is the most instructive thing in this guide. The constraint was named in an RFC dated 21 July 2022. The repository that answers it was created on 1 November 2024. The containers appeared in the reference deployment on 11 June 2025, and the last Celery code left the monolith on 26 September 2025, in a pull request whose body reads "Continue from #100302 and #99677 and remove another chunk of celery related code. Only one consumer and some settings remain." Three years and two months from naming a constraint to deleting the thing it constrained, with a numbered internal ticket reference as the only visible planning artefact.
That RFC also contains the single most useful sentence about in-house infrastructure in the corpus, and it is about a database migration in a deployment tool: "no one knows what alembic is and this actually resulted in some downtime as a recent Freight migration had to be figured out adhoc". The tool that deploys everything else had become a system whose schema nobody could change safely. That is the same failure as the long-lived shim, in a different subsystem.
| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Issue search engine, 2013 to 2015 | Back to Postgres | Solr, then Elasticsearch | Both were added and deleted inside 22 and 17 months respectively; no public statement of the reason survives | git history, src/sentry/search |
| Search and aggregation engine, 2018 | ClickHouse behind Snuba | Postgres plus Redis | Real-time performance, replication, and storage-engine flexibility in one system | Snuba architecture overview |
| Exactly-once ingestion | Deduplicating table engines | Transactional writes | "we can achieve exactly once semantics if we accept eventual consistency" | Snuba architecture overview |
| Task serialization format | Move off pickle | Keeping pickle | "it's not possible for code outside of the Sentry monolith to dispatch tasks" | RFC 0002, 2022-07-21 |
| Kafka coordination | KRaft, ZooKeeper removed | Keeping ZooKeeper | One fewer stateful system in a deployment that had reached 56 containers | self-hosted#3263, 2024-08-13 |
| Change data capture from Postgres | Removed | Keeping wal2json CDC | Deleted alongside the ZooKeeper change; the compose file simply stops carrying it | self-hosted#3260, 2024-08-12 |
| Event body storage | Object storage, 2025 | Bigtable-only or Postgres-only node store | Brings the self-hosted shape in line with an S3 API rather than a cloud-specific one | self-hosted#3498, 2025-09-13 |
| Container base image, 2026 | Distroless | Debian slim images | "hundreds of OS packages that the application itself never uses", each needing CVE triage | RFC 0157, 2026-03-27 |
Two things about that table are worth saying directly. First, the 2013 to 2015 search experiments are the only decisions here with no recorded reason at all: two engines were added and removed, and the argument is not in the repository. Second, the decisions with the best-recorded reasoning are the ones about tooling and packaging, not about data stores. Sentry's public RFC repository opened on 21 July 2022, long after the storage moves that mattered most, and of its 64 merged records the overwhelming majority concern SDK behaviour and product semantics. The biggest architectural changes in this guide were decided somewhere that does not publish.
Sentry numbers its incidents and the numbers leak into public commit messages. None of the narratives below is published; each card is reconstructed from the commit that closed the incident and the pull request that explains the change.
This is worth dwelling on as a method, because it generalises to any company that
develops in the open. Commit subjects in the Snuba, Arroyo and Relay repositories carry
references of the form inc-NNNN. The lowest observed is INC-181, fixed on 28
July 2022; the highest is INC-2179, fixed on 19 May 2026. If the identifiers are
sequential and cover one incident stream, that is roughly two thousand numbered incidents
in forty-six months, which works out near forty a month. That inference is mine and the
numbering scheme is not published, so treat the rate as an order of magnitude and the
individual identifiers as solid. What matters more is the distribution: the fixes cluster
in the consumer framework and the query service, not in the storage engines that the
migrations were about.
creates_event set to true contains an event.No incident in this corpus is attributed to a storage engine losing data, and none is attributed to a dual-backend shim returning inconsistent answers. Both absences are ambiguous. They may mean the interface-and-shim method works, or they may mean that inconsistency between two read paths is not the kind of thing a commit message records. The second reading is more likely, because a shim's whole purpose is to let one side be wrong quietly. If you run one, the reconciliation job that compares the two paths is the only instrument that would tell you.
Every figure below is counted out of a public repository or a package registry on 30 September 2026. None is a vendor claim, and none describes Sentry's own production capacity, which is not public.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Containers in reference deployment | 8 / 7 / 23 / 30 / 38 / 56 / 60 / 69 | self-hosted | Mid-year snapshots for 2016, 2019, 2020, 2021, 2023, 2024, 2025, 2026 | 2026-07 | compose file |
| Minimum RAM enforced by the installer | 3,800 MB → 15,900 MB → 14,000 MB | self-hosted | Hard floor; a soft floor of 7,800 MB existed in 2021 | 2021-03 to 2024-08 | _min-requirements.sh |
| Minimum CPU cores | 2 → 4 | self-hosted | Raised with the RAM floor in August 2024 | 2024-08 | _min-requirements.sh |
| Engine lifetime, node store (Riak) | 72 months | sentry | Added 2013-10-17, deleted 2019-10-14 | 2019-10 | git history |
| Engine lifetime, node store (Cassandra) | 66 months | sentry | Added 2013-11-05, deleted 2019-04-26 | 2019-04 | git history |
| Engine lifetime, issue search (Solr) | 22 months | sentry | Added 2013-10-20, deleted 2015-08-18 | 2015-08 | git history |
| Engine lifetime, issue search (Elasticsearch) | 17 months | sentry | Added 2014-05-20, deleted 2015-10-04 | 2015-10 | git history |
| Engine lifetime, tag store v2 | 22 months | sentry | Added 2017-11-27, deleted 2019-09-30; a full rewrite abandoned | 2019-09 | git history |
| Shim lifetime, tag store multi-backend | 4 months | sentry | Added 2017-12-19, deleted 2018-04-24; the only shim that was removed promptly | 2018-04 | git history |
| Shim lifetime, node store multi-backend | 66 months | sentry | Added 2013-10-17, deleted 2019-04-26 | 2019-04 | git history |
| Shim lifetime, Redis and ClickHouse time series | 91 months, open | sentry | Added 2019-02-28, last pruned 2026-08-25 | 2026-09 | redissnuba.py |
| Celery in the monolith | 126 months | sentry | 2015-04-09 to 2025-09-26; replacement landed 2025-01-31 | 2025-09 | sentry#100326 |
| ClickHouse schema migrations in tree | 252 | snuba | Files under snuba/snuba_migrations | 2026-09 | snuba_migrations |
| Commits in the monolith | 111,408 | sentry | First commit 2008-05-12 | 2026-09-30 | getsentry/sentry |
| Merged RFCs | 64 | rfcs | Repository opened 2022-07-21; proposal numbers reach 157 | 2026-09 | rfcs/text |
| Observed incident identifiers | INC-181 to INC-2179 | snuba, arroyo, relay | Range in public commit subjects; scheme not published | 2022-07 to 2026-05 | commit history |
| Server published as a Python package | 371 releases, ended | PyPI | 2012-01-04 to 2023-07-25, final version 23.7.1 | 2023-07 | pypi.org/project/sentry |
| Python SDK generations | 187 then 351 releases | PyPI | raven ended 2018-12-19; sentry-sdk began 2018-07-26 | 2026-09-28 | pypi.org/project/sentry-sdk |
| Browser SDK generations | 88 then 750 versions | npm | raven-js ended 2019-06-04; @sentry/browser began 2017-12-07 | 2026-09-29 | npmjs.com/@sentry/browser |
Three of these deserve interpretation. The container count is the honest measure of what a decade of storage specialisation costs to operate: the deployment did not get more complex gradually, it jumped from 7 to 23 in a single release in November 2019 and then grew steadily as each new product class arrived, profiling in 2022, replays in 2022, crons in 2023, feedback in 2023, uptime in 2024. The RAM floor is the same fact expressed in money, and it roughly quadrupled in three years while the product's core promise did not change. Both numbers are about the self-hosted artefact rather than the hosted service, so treat them as a lower bound on the real operational surface.
The shim lifetimes are the numbers to carry into your own planning, and the arithmetic is simple subtraction of two commit dates, so check it yourself. Three shims, three very different outcomes: 4 months when the migration was small and recent, 66 months when the engine being retired had two replacements in flight, and 91 months and still counting for the one that spans two data models. The median of the three is the one to plan against, and the median is 66 months. That is not a migration. That is a subsystem you now own.
Every lifetime here is the lifetime of a file in a repository, which is a proxy for the lifetime of a deployed backend and not the same thing. A backend can be dead in production for a year before the directory is deleted, which would make these figures upper bounds, or it can be kept for self-hosted users after the hosted service has moved off it, which makes them incomparable to Sentry's own operations. The incident rate derived from identifier numbering is an inference, not a report. The container counts describe the self-hosted deployment only.
Thirty-one sources, all fetched on 30 September 2026. Every one is a repository artefact or a package registry record, which is both this page's strength and its limit: see the note at the end of this section.
A living informational RFC naming the constraints on the pipeline: RabbitMQ near its safe limit, pickle preventing non-Python task dispatch, buffers that require pickle and cannot be filtered, and symbolication task durations spanning milliseconds to thirty minutes.
Describes the answer to RFC 0002's queue problem: Kafka for transport, a SQLite store
for inflight activations "to avoid head-of-line blocking, enable out-of-order execution
and per-task acknowledgements", and gRPC GetTask and
SetTaskStatus for workers.
The dual-backend shim itself. A method specification table routes each time-series
call to Redis or to Snuba by read or write, with one entry bound to a function called
dont_do_this that raises NotImplementedError. Added
2019-02-28; a commit on 2026-08-25 removes "the dead Redis cmsketch frequency tables".
Git history for four interface directories gives the add and delete date of every engine: five node stores, four issue-search backends, four tag stores. Three of the five node stores and two of the four search backends are gone.
"Right now, if the produce fails for any reason, the consumer backlogs, allow graceful degradation instead." The shared consumer library's dead-letter path could stop the pipeline it protects.
"Pool.join will block for more than timeout seconds if there is a task that takes more than timeout seconds", made worse when the next step rejects messages and batches are reprocessed. Fixed by terminating the pool.
Adds a killswitch that defaults queries to tier 8 rather than tier 1 during ClickHouse overload, which the author acknowledges "will lead to a degraded performance for customers" but prefers to aggressive rate limiting.
The liveness health check covered all ClickHouse clusters, so a non-essential cluster outage removed serviceable capacity. Narrowed to essential clusters only.
Envelopes with creates_event set true but carrying no event, notably
Unreal payloads, were dropped during metrics extraction in point-of-presence relays.
Messages with timestamps outside the retention window are dropped on the EAP items and accepted-outcomes topics; a companion change deletes stored data older than thirty days.
The most candid document in the corpus. Freight "has been in maintenance mode for a long time" because "primary contributors are no longer here", and a Freight schema migration caused downtime because "no one knows what alembic is". ArgoCD, Tekton and Argo Workflows are each rejected with a stated reason.
Chooses a single internal bindings package over per-crate PyPI packages, because "maintaining Python Bindings is itself a huge burden", public packages create SemVer obligations, and publishing had been failing. Existing bindings are noted as Sentry-specific with minimal external use.
Approved decision to replace Debian-based images, on the grounds that standard bases ship "hundreds of OS packages that the application itself never uses", each of which still requires CVE triage.
Sixty-four merged records from 2022-07-21 to 2026-09-25, with proposal numbers reaching 157. Volume peaks in 2023 and thins sharply afterwards, and the merged set is dominated by SDK and product semantics rather than platform architecture.
The single best instrument in this hunt. Mid-year snapshots give the container count across the decade, and the arrival and departure commit of every service: Kafka, ClickHouse, Snuba and Symbolicator on 2019-11-12, Relay on 2020-04-24, Taskbroker on 2025-06-11, object storage on 2025-09-13.
The commit that takes the reference deployment from 7 containers to 23, introducing Kafka, ZooKeeper, ClickHouse, the Snuba services and Symbolicator at once. Dated six days after the switch to the Business Source License.
"Continue from #100302 and #99677 and remove another chunk of celery related code.
Only one consumer and some settings remain." The deletion of celery.py,
ten years and five months after it was added.
The operational counterpart to the Celery deletion. The two containers present since May 2016 leave the reference deployment, replaced by taskbroker, taskworker and taskscheduler which arrived on 2025-06-11.
Removes a stateful coordination service from a deployment that had reached 56 containers. Merged one day after the removal of Postgres change data capture and wal2json.
Change data capture out of Postgres, and the logical decoding plugin behind it, leave the reference deployment with no replacement service appearing in the compose file.
Event bodies move to an S3-compatible object store in the reference deployment, six years after Bigtable became the hosted node store and a decade after the original Postgres one.
The installer's enforced floor: 3,800 MB hard and 7,800 MB soft with 2 cores in March 2021; 15,900 MB and 4 cores in August 2024, trimmed to 14,000 MB six days later; a two-tier scheme from March 2025.
States the origin directly: Snuba "was originally developed to replace a combination of Postgres and Redis to search and provide aggregated data on Sentry errors", and has since absorbed most time-series features. First commit 2018-03-01; a ClickHouse query appears on day one.
Gives the engine rationale and the consistency bargain: ClickHouse for "a good balance of the real time performance Snuba needs", one writer per table, and "exactly once semantics if we accept eventual consistency" through deduplicating table engines.
The consumer framework extracted as a library rather than a service: Kafka backends,
composable strategies such as RunTask, Filter,
Reduce and CommitOffsets, and a stream processor that
schedules work and controls consumer progress.
"A service that pushes some functionality from the Sentry SDKs as well as the Sentry
server into a proxy process." First commit 2018-01-17 as smith, renamed
semaphore on 2018-05-09. The repository now carries explicit rules for AI
usage in a HOWTOAI.md.
371 releases of sentry from 2012-01-04 to 2023-07-25, ending at version
23.7.1. After that the product exists as a fleet of containers rather than something
installable with a package manager.
187 releases of raven ending 2018-12-19; sentry-sdk starts
2018-07-26 and reaches 351 releases by 2026-09-28. The two overlap for five months.
88 versions of raven-js from 2013-12-20 to 2019-06-04; 750 versions of
@sentry/browser from 2017-12-07 to 2026-09-29. The browser SDK reset began
eighteen months before the Python one ended.
205 releases from 2021-06-29 to 2026-09-25, eighteen days after the repository's first commit. A shared consumer framework versioned publicly while being used internally.
93 releases from 2020-01-27 to 2026-09-23. These are the bindings RFC 0119 later describes as a maintenance burden, which is the evidence that the RFC was describing lived experience rather than anticipating a problem.
There are no engineering blog posts, papers or conference talks in this wall, and that is not because none exist. This session's network policy could reach GitHub and the package registries and nothing else, so Sentry's own published narratives, its incident reports and any third-party measurement were unavailable. The consequence is a page built almost entirely from tiers nine to eleven of the evidence hierarchy, which is the strongest material for what was built and the weakest for how it felt to operate. Three distinct hosts is well below what breadth would normally require. Where a reason is quoted here it is quoted from a decision record; where an outcome is described it is reconstructed from a diff. Nothing in this guide should be read as Sentry's account of its own history.
Six rungs. The line between a toy and the real thing is rung four, where you deliberately break the error path.
Take any read-write path in a service you own and put an abstract backend in front of it, with the current store as the first implementation and a second store, in a different engine family, as the second. Select by configuration.
Done when: the test suite passes against both backends with no test changes. Teaches: which of your callers depend on engine behaviour rather than on the interface.
Add a mode that reads from both and logs divergences with the query arguments, without serving the new answer. Run it on real traffic for a week.
Done when: you can state the divergence rate per method and explain every class of difference. Teaches: that most divergences are semantic, not bugs, and that the semantics were never written down.
Build the routing table Sentry's shim uses: each method tagged read or write and pointed at one engine. Then add the piece the original lacks, a counter per legacy read, exported by method name.
Done when: a dashboard shows legacy reads by method and it is the metric you would use to decide the migration is finished. Teaches: a shim without that counter cannot tell you it is done, which is why shims do not end.
Make the dead-letter queue, quarantine table or retry topic unavailable while valid traffic flows. Watch whether consumption stops. Then make the failure absorbed rather than raised, and watch what you lose instead.
Done when: an unavailable DLQ degrades throughput and does not halt it, and you can say what data is lost in that state. Teaches: the choice in arroyo#425, which is that graceful degradation of the error path always costs something and the cost has to be named.
Put a small durable store between a partitioned log and a pool of workers so tasks can complete out of order and be acknowledged individually. Feed it a workload whose durations span three orders of magnitude.
Done when: a single thirty-minute task does not delay thousands of millisecond tasks behind it in the same partition. Teaches: why Sentry put SQLite in a broker in 2024 rather than adding partitions.
Add a switch that answers queries from a coarser or sampled source under load, and measure the error it introduces. Then write the deletion ticket for the shim from rung three: a named owner, a date, and an alert that fires when legacy reads are non-zero after that date.
Done when: the coarse path is exercised in a game day and the deletion ticket exists with a date in it. Teaches: the two things the ten-year record says are missing, a degradation lever built before the incident and a deletion that is somebody's job.
This page was assembled with git and two package registries, not with a search engine. These are the commands that produced it; they work on any company that develops in the open.
git log --reverse --diff-filter=A --format=%ad --date=short -- path/to/backendgit log --diff-filter=D --format=%ad --date=short -- path/to/backendgit log --all --name-only --format= --diff-filter=A -- 'src/*/nodestore/*' | sort -ugit rev-list -1 --before=2020-07-01 HEADgit show <sha>:docker-compose.yml | grep -E '^ [a-z0-9_.-]+:$'git log -S' zookeeper:' --format='%ad %s' --date=short -- docker-compose.ymlgit log -i --grep='inc-[0-9]' -E --format='%ad | %s' --date=shortgit log -i --grep='postmortem\|root cause\|data loss' --format='%ad | %s' --date=shortgit log --format='%ad | %s' --date=short --grep='^Revert'curl -s https://pypi.org/pypi/<pkg>/json | jq -r '.releases | to_entries[] | [.key, (.value[0].upload_time // "")] | @tsv'curl -s https://registry.npmjs.org/<pkg> | jq -r '.time | to_entries[] | [.key, .value] | @tsv'git ls-remote --heads <repo> # unmerged proposal branchesgrep -m1 -i 'Start Date' text/*.mdgrep -ho 'rfcs/pull/[0-9]*' text/*.md | sort -u # numbering gaps are rejectionsgit log --format='%ad' --date=format:%Y -- text/ | sort | uniq -cgit log -p --date=short --format='%ad' -- install/_min-requirements.sh | grep -E '^[+-]MIN_'git log --format='%ad | %s' --date=short -- Dockerfile | tail -40