Every backend you add is a six-year promise: ten years of Grafana Labs in git

The territory

One company, four databases, ten years, and a public record made entirely of commits.

The question behind this guide has nothing to do with observability. It is this: when your system stores data on somebody else's infrastructure, and you decide the storage layer was wrong, how long does it actually take to get rid of the old one? Not to ship the replacement, which is the part every architecture review talks about, but to reach the day when the old code is gone, the configuration flags are gone, and nobody is still writing to it. Grafana Labs has run that experiment four times in public, and because its products are open source, every step of it is in git with a date attached.

The answer, measured across the four products, is between four and six years, and the companies paying that bill are usually surprised by it. Loki's replacement index, which put the index into object storage instead of a database, landed on 28 July 2020. The last of the database backends it replaced, Cassandra, was deleted on 17 April 2026, and the replacement itself was deleted a month later in favour of its own successor. That is sixty-nine months between shipping the better thing and finishing with the worse one, and thirty-two months after the documentation first said those backends "will be removed in a future release", inside a project whose written guarantee is that deprecated features survive two minor releases.

69.6

months from the object-store index shipping in Loki to the last legacy backend being deleted

grafana/loki, 2020-07 to 2026-05

pull requests listed in the one changelog line that removes a single storage engine

Mimir 2.0 changelog, 2022-03

on-disk block formats shipped by Tempo in 38 months, each with its own conversion path

grafana/tempo, 2022-06 to 2025-08

20.4

months for a Kafka write path to spread from the first product to the third

Mimir 2023-12, Loki 2024-09, Tempo 2025-08

What surprised me. I expected the interesting material to be the rewrites. It is not. Every rewrite in this decade is legible, well argued and roughly on the schedule you would guess. The part nobody plans for is the tail: 2026 is the year Grafana Labs deleted BigTable, Cassandra, DynamoDB, the gRPC store, the BoltDB index and the BoltDB shipper from Loki, all inside nine weeks, six years after the thing that replaced them worked. And in the middle of that wave, one removal was reverted eight days after it merged, which is the clearest evidence in the whole record that even a team with changelogs, deprecation banners and a versioning policy cannot reliably tell who is still using a feature.

What this guide covers. The storage and ingestion architecture of the four Grafana Labs telemetry databases (Mimir for metrics, Loki for logs, Tempo for traces, Pyroscope for profiles), the collector consolidation around Alloy, and the fork decisions that hold them together, from 2016 to September 2026. What it deliberately does not cover: the Grafana dashboard application beyond one data point about its frontend, Grafana Cloud's commercial architecture, pricing, and anything about how these systems are operated as a service, because that material lives on the vendor's website rather than in its repositories.

The evidence limit, stated up front

This session could reach GitHub and nothing else. Engineering blogs, talks, papers and vendor documentation sites were refused at the network boundary, including grafana.com, where Grafana Labs publishes both its architecture posts and its own Grafana Cloud incident reviews. So this guide is built from one layer only: git history, changelogs, design proposals, go.mod comments and repository READMEs, in ten repositories cloned in full. That makes every claim here reproducible with git show and none of it dependent on a narrative. It also means the reasons are inferred more often than reported, and it means the first-party incident record is missing; the four incident reviews in section 4 come from two other organisations running this stack. Where a reason is my reconstruction rather than a statement, the sentence says so.

Figure 1 · 2016 to 2021: four products, built one at a time

Six years of additions, each one arriving before the previous generation was retired. Dates are commit and tag dates from grafana/loki and grafana/mimir.

Diagram source

Figure 2 · 2022 to 2026: convergence, then one wave of deletions

Notice that the removals do not track the replacements. Everything deleted in 2026 had a working successor by 2022. Dates from grafana/tempo and grafana/alloy.

Diagram source

Four databases that became one architecture

The shape they all converged on, which components are genuinely shared, and where they still differ.

Read the four repositories side by side and the same seven boxes appear in each, arriving in a different order and under different names. The convergence is not a claim in a blog post; it is visible in the directory names. Mimir, Loki and Tempo each now have a distributor, a partitioned Kafka log in front of the in-memory tier, an ingester or live-store that holds recent data and writes a write-ahead log, a flush that produces immutable blocks in object storage, a compactor that rewrites those blocks, a catalogue object that tells readers what exists without listing the bucket, and a read path built from a query-frontend, a scheduler and a set of stateless queriers in front of gateway processes that fetch block fragments.

The catalogue is the piece worth studying, because it is the one that was designed explicitly and is documented. Cortex's November 2020 design proposal for a bucket index states the problem in numbers: with one block per day per tenant and 400 days of retention, "we would have 400 blocks for a single-tenant cluster, but 4M blocks for a 10K tenants cluster", the cold start of queriers and store-gateways "take tens of minutes", and the listing traffic "represents a significant % of bucket API calls baseline costs, regardless the tenants QPS (costs we have even if the cluster has 0 queries)" (cortexproject/cortex, proposal, November 2020). The fix is a single JSON object per tenant, written by the compactor, so a querier needs one GET and no LIST. Every one of the four products now has its own version of that object: Mimir's bucket index, Loki's index files shipped to the bucket, Tempo's polled blocklist, Pyroscope's metastore. If you are designing anything that keeps immutable files in object storage, this is the component you will need and the one you will build last.

The Kafka log is the newest shared box and the one with the clearest chronology. Mimir added it first, on 12 December 2023, as an experimental path that writes incoming samples to a Kafka-compatible backend before the ingesters consume them. Loki followed on 10 September 2024 with write-ahead log segments carried over Kafka and a partition ring. Tempo was last, on 25 August 2025, under the internal name Rhythm, which introduced a live-store consuming from Kafka in place of the previous ingester. Nothing in the repositories explains the ordering, so the reason is my reconstruction: putting a durable log in front of the ingesters decouples write durability from ingester lifecycle, which is what makes it possible to build blocks in a separate component and to restart or rescale the in-memory tier without a replication quorum. The block-builder and block-builder-scheduler components that appear in Mimir immediately afterwards only make sense on that assumption.

What is genuinely shared

The hash ring over gossip, the compactor, the object-storage block, the per-tenant catalogue, the query-frontend and scheduler split, and now the Kafka partition log. Mimir 3.0 went further and made the scheduler mandatory, removing the option to embed it in the frontend.

Seen in: Mimir 3.0 changelog, Loki #14043, Tempo #5563

Where they still diverge

The on-disk format. Tempo took a general columnar format, Parquet, because its queries search arbitrary span attributes. Mimir kept Prometheus TSDB blocks. Loki is on its third bespoke layout, "dataobj", begun in January 2025 and at 494 commits by September 2026.

Seen in: Tempo #1479, Loki dataobj

What quietly disappeared

External coordination and caching dependencies. Mimir 3.0 removed Redis as a cache backend and made gossip the default for the high-availability tracker, deprecating etcd and Consul for that role. Loki deleted its Cassandra, DynamoDB, BigTable and BoltDB backends in nine weeks in 2026.

Seen in: Mimir 3.0 changelog, Loki #21502

Figure 3 · The shape all four converged on

The only durable store is the bucket; everything else is a cache, a buffer or an index of it. The catalogue object in the middle is what keeps readers from listing the bucket, and it is the component the Cortex bucket-index proposal was written to add.

Diagram source

The same architecture, named four ways. Read from the component directories at each repository's default branch, September 2026.
Concern	Mimir (metrics)	Loki (logs)	Tempo (traces)	Pyroscope (profiles)
Write buffer	Kafka ingest storage, 2023-12	Kafka WAL segments, 2024-09	Kafka live-store, 2025-08	Ingester with segment writer
Durable unit	Prometheus TSDB block	Chunks plus index; dataobj in progress	Parquet block, five versions	Block with Parquet columns
Catalogue	Bucket index JSON per tenant	Index files shipped to the bucket	Polled blocklist	Metastore service
Acceleration	Store-gateway index headers	Bloom planner, builder and gateway	Dedicated attribute columns	Column pruning
Coordination	Hash ring on gossip, default since 3.0	Hash ring on gossip	Hash ring on gossip	Hash ring on gossip

Figure 4 · Four index generations in one product

Loki's index has been rebuilt three times in eight years, and each generation outlived its successor's arrival by years. The dates are the first commit of each generation and the commit that deleted the previous one.

Diagram source

The decisions that matter

Six forks in the road, what was chosen, what was abandoned, and the condition that would flip each one.

Put the index in the object store, or keep it in a database

Chosen

Index as immutable files in the same bucket as the data, from July 2020 in Loki and from the blocks storage work in Cortex
One durable dependency, one consistency model, one bill

Rejected

DynamoDB, Bigtable, Cassandra and BoltDB as index stores, all deleted in April 2026
Bucket listing as the way to discover what exists, replaced by a per-tenant catalogue object

Flips when

Your tenant count stays low. The proposal's own arithmetic is 400 blocks for one tenant against 4M for ten thousand; below a few hundred blocks, listing is free and a catalogue is overhead
You need single-digit millisecond point lookups on the index itself, which object storage will not give you

Fork the dependency, or wait for upstream

Chosen

Fork, repeatedly. Mimir's go.mod replaces Prometheus itself with grafana/mimir-prometheus, plus forks of memberlist, regexp, goautoneg, yaml and the OTLP translator
Loki forks the Thanos object-storage library and the Cassandra driver

Rejected

Carrying patches upstream and waiting. The memberlist comment is explicit: changes "haven't been merged upstream yet for years and we don't expect to change anytime soon"

Flips when

The change can live in a wrapper rather than inside the library
You would have to track upstream's release cadence anyway. The stated reason for the OTLP translator fork is the opposite case: upstream moves faster than Mimir wants to
Watch the exit cost: the Cassandra driver fork is still in Loki's go.mod five months after the Cassandra backend was deleted

Fork the project, or keep sharing the upstream

Chosen

Cortex was relicensed and renamed to Mimir inside a private repository on 5 and 11 August 2021, and released publicly as mimir-2.0.0 on 28 March 2022
Grafana, Loki and Tempo were relicensed to AGPLv3 on one day, 20 April 2021

Rejected

Continuing under shared governance. Cortex is still developed independently and tagged v1.22.0-rc.0 on 12 September 2026

Flips when

The change you need is architectural rather than commercial. Both projects removed the old chunks storage engine within a year of each other, independently, which suggests the fork was not about that
The fork has a seven-month private runway before anyone else can plan for it; if your consumers need longer notice than that, fork in public

A general columnar format, or a bespoke one

Chosen

Tempo took Parquet in June 2022, because trace search is a scan over arbitrary attributes and Parquet gives predicate pushdown and dedicated columns for free
Mimir kept TSDB blocks; Loki is building its own layout for a third time

Rejected

Tempo's own v2 paged block format, retired after 15 months

Flips when

Your read pattern is "fetch this series by key" rather than "find rows matching an arbitrary predicate". A general format costs you a format treadmill: five versions in 38 months, each needing a conversion tool and a deprecation cycle

Own the agent, or give it away

Chosen

One collector. Grafana Agent was renamed to Alloy on 25 March 2024 and reached end of life on 1 November 2025; Promtail support was deleted from Loki on 25 March 2026
The eBPF instrumentation agent, Beyla, was donated to the CNCF OpenTelemetry project, and its maintainers moved upstream

Rejected

A per-signal agent estate. Promtail for logs, the static Agent for metrics and a separate eBPF tool were all folded in or handed over

Flips when

The component's value is adoption rather than differentiation. Donating removes your ability to change it on your schedule, which is precisely why the storage engines were never donated

Buy the product, or keep your implementation

Chosen

Both, in a specific order. Grafana Labs built Phlare from May 2022, acquired Pyroscope on 15 March 2023, archived Phlare, then renamed its own engine: phlaredb became pyroscopedb on 17 August 2023 and Pyroscope 1.0 shipped eleven days later

Rejected

Adopting the acquired storage engine. The acquired code still sits at HEAD three years later in a package named pkg/og

Flips when

The acquisition is for the ingestion format and the users rather than the backend. Keeping your engine and taking their name is cheap; keeping both engines is what leaves a legacy package in your tree for years

Figure 5 · Deciding whether the old backend can go

The branch that actually decides the outcome is the one on the left: whether you can observe use of the feature. Loki's reverted removal of the /api/prom endpoints, eight days after it merged, is what the "no" path looks like in practice.

Diagram source

Decision	Chosen	Rejected	Stated or inferred reason	Evidence
Index location	Files in object storage	Cassandra, DynamoDB, Bigtable, BoltDB	Stated: listing and cold-start cost scale with tenants and blocks	Proposal, 2020-11
Write durability	Kafka partition log ahead of ingesters	Quorum replication between ingesters alone	Inferred from the block-builder components that follow	Mimir #6888, 2023-12
Dependency strategy	Fork Prometheus, memberlist, objstore, regexp	Waiting for upstream merges	Stated in `go.mod` comments	mimir go.mod
Project governance	Fork Cortex, relicence AGPLv3	Shared upstream	Not stated in the repositories; visible only as a rename and a licence commit	Rename, 2021-08-11
Trace block format	Parquet, five versions	Bespoke v2 paged blocks	Inferred: attribute search needs predicate pushdown	Tempo #1479, 2022-06
Collector estate	One collector, Alloy; donate Beyla	Per-signal agents	Stated in the archived READMEs	Agent README
Recent-data reads	Fail the query when an instance lags	Serve possibly incomplete recent results	Stated as a breaking change in Tempo 3.0	Tempo 3.0 changelog

What broke, and what the record does not tell you

Three failure classes visible in the commits, three operator incidents from organisations running this stack, and one honest gap.

The vendor-side failures in this record are not outages, because outages are not what a repository holds. They are fix commits, and they cluster with striking regularity in whichever component was newest at the time. Two classes account for almost all of them. The first is progress committed before durability: a consumer that acknowledges its position in the log before the data it read is safely written. The second is a rewritten query engine that returns a wrong answer instead of an error. Both are classes an architect can design against, and both are invisible to the availability metrics that would normally catch a regression.

The gap worth naming

None of the vendor fix commits state a blast radius. "Fix data loss bug in the experimental ingest storage when a Kafka Fetch is split into multiple requests and some of them return an error" tells you the mechanism and nothing about whether you lost data, for how long, or in which versions. The public record is structurally incapable of answering the question an operator actually has. The only way to know is to work out the first release that contains the fix and compare it with what you ran: the Mimir split-fetch fix first appears in mimir-2.15.0, the Loki offset fix in v3.6.0, and the August 2026 block-builder fix is in no tagged release at all as of 25 September 2026.

Figure 6 · The failure class the new write path keeps producing

The window between the offset commit and the flush is the whole bug. Four fixes across two products, from November 2024 to August 2026, are variations on this single ordering. Reconstructed from the fix commits; no incident report describes the impact.

Diagram source

Source

The consumer commits its place before the data is safe

AssumptionOnce the records are in the consumer's memory, the position in the log can be recorded.

What happenedFour separate fixes across two products: a split Kafka fetch where some responses errored (Mimir, 2024-11), consumers committing an incorrect offset (Loki, 2025-08), partitions fully consumed at startup (Mimir, 2026-01), and single-record jobs losing data silently (Mimir, 2026-08).

Blast radiusNot published. The commit titles are the entire public record; three of the four say "data loss" and one says "silent data loss".

FixCommit after the flush, and treat a partially failed fetch as a failed fetch. In Tempo the same concern is visible as a separate commit decoupling consuming from committing.

Design ruleAny component that records its own progress must record it after the write that makes the data durable, and the test for it is killing the process between the two, not asserting on a happy path.

SourceMimir #16493, 2026-08-28, Loki #18833, 2025-08-13

Source

The new query engine answers, and the answer is wrong

AssumptionA rewritten engine that passes the test suite can be made the default.

What happenedMimir's own query engine produced incorrect results for queries with a common subexpression (2025-06 and 2025-07), corrupted native histograms in two separate code paths (2026-06, 2026-07), and returned incorrect results or panicked when a selector-elimination optimisation applied (2026-03). Loki's rewritten engine silently dropped OR operations (2026-03).

Blast radiusNot published. Note the sequencing: Mimir made its engine the default in 3.0 on 2025-10-31, and correctness fixes continued through 2026.

FixIndividual correctness patches, each naming the optimisation that caused it. Every one of these bugs is in an optimisation pass, not in the evaluation core.

Design ruleOptimisation passes are where query engines lie. Run the new engine shadowed against the old one on production queries and diff the results, and keep doing it after the default flips, because that is when the reports stop arriving as errors.

SourceLoki #21204, 2026-03-19, Mimir 3.0 changelog

Source

The removal that had to be put back

AssumptionAn endpoint deprecated for years, in a release that already carries twenty breaking changes, can be deleted.

What happenedLoki removed the /api/prom endpoints on 11 June 2026 and reverted the removal on 19 June 2026. Four years earlier, upstream Cortex needed a pull request titled "Remove support for chunks storage entirely: Take 2" to finish a removal it had started.

Blast radiusEight days in main, and no released version shipped without the endpoints. The cost was a wasted release cycle rather than an outage.

FixRevert, and leave the endpoints in place. The project's own deprecation record did not predict which consumers still depended on them.

Design ruleDeprecation notices measure your intent, not your users. Before deleting an interface, instrument it per tenant and let the counter, not the changelog, decide the date.

SourceLoki revert #22466, Cortex #4812

Postmortem

The collector migration changed the label set, and the alerts fired

AssumptionSwapping the metrics collector is a deployment change, not a data change.

What happenedDFDS migrated exporters and service monitors from the kube-prometheus stack to a Grafana Cloud collector deployment. The labels exposed by kube-state-metrics changed, alerts started firing, and the fix could not be released because an unrelated infrastructure pipeline change was blocking production.

Blast radiusNoise from 11:27 on 2024-06-06 until the next morning; alarms were silenced overnight. A day later, users reported dashboards missing metrics entirely.

FixRestore the label configuration, and an action item to keep production no more than one release behind the configuration.

Design ruleA collector swap is a schema migration of your alerting contract. Diff the emitted label sets between old and new collector in staging, and do not start one while another change is blocking your rollback path.

SourceDFDS PM2024-003, 2024-06-06

Postmortem

The monitoring system was a single pod with no limits

AssumptionThe Helm chart sets sensible resource requests and limits.

What happenedThree Kubernetes nodes went NotReady in one day at DFDS, each time the one running Prometheus; a kernel message showed the OOM killer invoked. The monitoring system failed at exactly the moment it was needed to diagnose the node failures.

Blast radius2022-01-25, 10:21 to 15:00 UTC, three separate node failures, with SSH access to the nodes unavailable for part of it.

FixRequests and limits set manually, then encoded in infrastructure as code the following morning.

Design ruleThe scale-out architecture in section 2 exists because the single-instance version fails this way. If you are running one instance, the thing to add first is not more storage, it is a resource limit and a second place to look when it dies.

SourceDFDS PM2022-001, 2022-01-25

Postmortem

Telemetry stopped because a person's account was deleted

AssumptionOnce the exporter is up again, the metrics are back.

What happenedA personal Confluent Cloud account was deleted; the Kafka exporter's API key went with it. A new key restored the exporter in an hour, but the key lacked the full set of access control entries, so a subset of metrics stayed missing for another three and a half hours while the incident was already closed.

Blast radius2022-07-07, exporter down 08:58 to 10:09 UTC, partial metric loss until 13:34 UTC.

FixService-account ownership of the key, and documentation for the access control entries it needs.

Design rulePartial recovery of a telemetry pipeline looks identical to full recovery from the outside. Close the incident on a check that compares the metric set against a known list, not on the exporter being up.

SourceDFDS PM2022-007, 2022-07-07

One more absence is worth stating plainly, because it is where a reader's own risk sits. No public account in this evidence base describes a production incident caused by one of these storage migrations. The migrations happened, the data-loss bugs are documented as fixes, and the operator postmortems that exist are about resource limits, credentials and label sets rather than about blocks or indexes. Either the migrations went cleanly, or the failures went unpublished. The commits cannot distinguish those two readings, and neither can I.

Numbers you can plan against

Mostly intervals, because intervals are what a repository measures honestly.

Quantity	Value	What it measures	Kind	As of	Source
Replacement to removal, Loki index	68.6 months	boltdb-shipper shipping to Cassandra backend deleted	Derived from two commit dates	2026-04	#21574
Replacement to removal, first generation	69.6 months	boltdb-shipper shipping to boltdb-shipper deleted	Derived	2026-05	#21678
Documented deprecation to deletion	32.0 months	Loki docs first saying the legacy index stores "will be removed in a future release", to their removal	Derived	2026-04	#9246
Deprecation to deletion, Cortex chunks	13.7 months	Upstream Cortex, deprecation commit to "Take 2" removal	Derived	2022-08	#4812
Deprecation to deletion, Promtail	13.4 months	Docs banners to removal of the agent from Loki	Derived	2026-03	#21245
Rename to end of life, Grafana Agent	19.2 months	Module rename to Alloy, to the stated Agent end-of-life date	Derived, end date stated in the README	2025-11	Agent README
Frontend framework lifetime	146 months	Grafana's initial commit to "Frontend: Remove Angular"	Derived	2025-04	#99760
Written deprecation guarantee	2 minor releases	What Mimir commits to for a deprecated feature	Stated policy	2026-09	about-versioning
Written data-readability guarantee	2 years	How far back a future Mimir will read data written by an older one	Stated policy	2026-09	about-versioning
Cost of removing one storage engine	18 pull requests	The PRs listed in Mimir 2.0's "Remove chunks storage engine" line	Measured from the changelog	2022-03	Mimir 2.0
Block format churn	5 in 38 months	Tempo's Parquet block versions, first commit of each	Measured	2025-08	#1479
Breaking-change commits in Loki	6, 3, 20	Commits marked `!` in 2024, 2025 and 2026 to date. The convention was only adopted in 2024, so the years before are not comparable	Measured, with a caveat	2026-09	loki commits
Blocks at ten thousand tenants	4 million	One block per tenant per day at 400 days retention, the arithmetic that justified the bucket index	Stated in the design proposal	2020-11	Proposal
Cold start before the bucket index	tens of minutes	Queriers and store-gateways with empty caches scanning the bucket	Stated, no distribution given	2020-11	Proposal
Active series handled	1 billion	The scale figure in Mimir's README	Vendor claim from internal testing, not an independent measurement	2026-09	README
Investment in the current rewrite	494 commits	Commits touching `pkg/dataobj` since 2025-01-07	Measured	2026-09	loki dataobj

What to plan against, in one paragraph. If you ship a storage backend to users who operate the software themselves, assume you will be supporting it for five years after you stop recommending it, not for the two releases your policy promises; the gap between the written guarantee and the observed practice in this record is more than a factor of ten. If you ship a new on-disk format, assume a conversion tool, a deprecation cycle and a default flip for every version, and look at Tempo's cadence of one format every fifteen months before you choose a format you will want to evolve. If you put a durable log in front of your ingest path, budget for a class of data-loss bugs rather than a single one; the record here shows four across two products in 21 months, and the last of them is not yet in a release. And if a number matters to your capacity plan, notice which column of that table it came from: the intervals are measurements, the billion series is a claim, and the blast radius of every bug above is unknown.

Read these carefully

Every interval here is derived from two dates in git, which makes it precise about code and silent about adoption. The day a backend is deleted from main is not the day the last cluster stopped using it, and no public data says when that was. Treat these as lower bounds on the real support window.

The evidence wall

Every source behind this page, graded. The mix is unusual: almost all of it is tier-10 repository evidence, because that is the only layer this session could reach.

ADR Cortex maintainers2020-11

Blocks storage bucket index (design proposal)

The one document in this corpus that argues a design rather than recording it. It states the goal as "The querier should be up and running without having to scan the bucket at all (zero startup time)" and gives the tenant arithmetic that makes bucket listing untenable.

Carry forwardAny object-storage-backed store needs a catalogue object before it needs a cache; listing is a cost you pay even at zero queries.

cortexproject/cortex docs/proposals/blocks-storage-bucket-index.md

Source Grafana Labs2020-07

Loki: the index moves into the bucket

The boltdb-shipper commits, the first generation of index-as-files. The replacement for four database backends arrives here, and everything in the 2026 deletion wave is downstream of it.

Carry forwardDate the replacement, not the deprecation: the support window starts the day the better option works.

grafana/loki commit e59adccf

Source Grafana Labs2026-04

Loki deletes Cassandra, DynamoDB, BigTable and BoltDB

Six breaking removals in nine weeks, each a chore! commit that deletes a storage client package, its configuration and its dashboards.

Carry forwardRemovals arrive in waves because they are gated on one release boundary; plan the wave, not the individual deletion.

grafana/loki #21574 and #21502, #21601, #21637, #21638, #21678

Source Grafana Labs2026-06

The removal reverted eight days later

/api/prom endpoints removed on 11 June 2026, revert merged on 19 June 2026. No issue text is reachable from here, so the reason is unstated; the revert itself is the evidence.

Carry forwardIf you cannot count calls per tenant, your deprecation date is a guess and your revert is a matter of time.

grafana/loki #22466

Source Cortex maintainers2022-08

"Remove support for chunks storage entirely: Take 2"

The upstream project deprecated its original storage engine in June 2021 and needed a second attempt to finish deleting it 13.7 months later.

Carry forwardA removal that needs a second attempt is normal; leave room in the release plan for it.

cortexproject/cortex #4812

Source Grafana Labs2022-03

Mimir 2.0 changelog: one line, eighteen pull requests

"Remove chunks storage engine" with eighteen PR numbers after it, followed by pages of removed flags. The cheapest available estimate of what deleting a storage engine costs.

Carry forwardBudget removal in pull requests, not in a ticket. Configuration surface is most of the work.

grafana/mimir CHANGELOG at mimir-2.0.0

Source Grafana Labs2021-08

Cortex renamed to Mimir, in private, seven months early

The licence commit lands 5 August 2021 and the rename 11 August 2021; the first public Mimir tag is 28 March 2022. The fork existed for 7.6 months before anyone outside could plan for it.

Carry forwardWhen you depend on an open-source project, the fork you will have to react to may already exist and be invisible.

grafana/mimir commit fa86354b

Source Grafana Labs2026-09

Mimir's `go.mod`: six forks, with reasons

Prometheus itself is replaced by grafana/mimir-prometheus. The memberlist comment says the changes "haven't been merged upstream yet for years and we don't expect to change anytime soon"; the OTLP translator fork exists so Mimir can move slower than upstream, not faster.

Carry forwardRead a project's replace directives before adopting it: they tell you which dependencies its maintainers no longer trust to move at the right speed.

grafana/mimir go.mod

Source Grafana Labs2026-09

Loki's `go.mod`: the fork that outlived the feature

The Cassandra driver fork, kept "to have gokit logs and Prometheus metrics", is still a replace directive five months after the Cassandra backend was deleted. The Thanos object-storage library is also forked.

Carry forwardRemoving a feature does not remove its dependencies; put the fork on the same deletion checklist as the code.

grafana/loki go.mod

Vendor Grafana Labs2026-09

Mimir's versioning guarantees

"We will keep deprecated features in place for two minor releases." Reading old data is guaranteed for two years. Experimental features carry no compatibility guarantee at all, which matters because the Kafka ingest path shipped as experimental.

Carry forwardCompare the stated window against the observed one before you build a migration plan on the policy.

grafana/mimir about-versioning.md

Source Grafana Labs2023-12

Mimir puts Kafka in the write path

The first commit under pkg/storage/ingest, "Add experimental support to write incoming data to a Kafka-compatible backend". Loki and Tempo follow at nine and twenty months.

Carry forwardWhen one product in a portfolio adopts a pattern, read it as a preview of the others; the convergence here is ordered and roughly annual.

grafana/mimir #6888

Source Grafana Labs2026-08

Four data-loss fixes in the Kafka path

Split fetches with partial errors, incorrect offset commits, partitions fully consumed at startup, and single-record jobs. The most recent, "fix silent data loss for single-record jobs", is in no tagged release as of 25 September 2026.

Carry forwardA durable log in front of ingest moves the correctness problem to offset management; test it by killing processes, not by load.

grafana/mimir #16493

Source Grafana Labs2026-03

The rewritten Loki engine dropped OR

"fix(query-engine): Fix OR operations being silently dropped". The companion class in Mimir is a run of correctness fixes in optimisation passes, several after the new engine became the default.

Carry forwardShadow the new engine against the old one and diff results; a wrong answer does not page anyone.

grafana/loki #21204

Source Grafana Labs2025-10

Mimir 3.0 removes the alternatives

Redis as a cache backend, the read-write deployment mode, the embedded query-scheduler and several HA-tracker backends all go in one release, and the project's own query engine becomes the default.

Carry forwardMajor versions are where optionality is withdrawn. Audit which of your deployment choices are the ones being deleted.

grafana/mimir CHANGELOG at mimir-3.0.0

Source Grafana Labs2022-06

Tempo adopts Parquet, then keeps re-adopting it

vParquet in June 2022, then v2, v3, v4 and v5 by August 2025, with defaults flipping in November 2023, June 2024 and August 2026 and conversion commands added for each step.

Carry forwardA general columnar format buys query flexibility and sells you a migration every fifteen months.

grafana/tempo #1479

Source Grafana Labs2026-05

Tempo 3.0 chooses failing over answering

"Recent data queries guarantee complete results by failing when an instance is lagging", with a 30 second cutoff and fail_on_high_lag defaulted to true.

Carry forwardState explicitly whether a query over recent data may return partial results; make it a configuration flag with a default you can defend.

grafana/tempo CHANGELOG at v3.0.0

Source Grafana Labs2025-11

Grafana Agent's end of life, stated in its own README

"Grafana Agent has reached End-of-Life (EOL) on November 1, 2025." The repository shares its initial commit with Alloy, so the successor is the same codebase renamed on 25 March 2024.

Carry forwardA rename with history preserved is a migration you can diff; a rewrite is not. Check which one you are being offered.

grafana/agent README

Source Grafana Labs2026-03

Promtail deleted from Loki

Deprecation banners in the docs on 12 February 2025, "feat: Remove Promtail support" on 25 March 2026, and the mixin, issue templates and lambda variant removed over the following months.

Carry forwardThe code is the first thing to go and the dashboards are the last; a deprecation is done when the operational furniture is gone too.

grafana/loki #21245

Source Grafana Labs2023-08

Phlare archived, its engine renamed to Pyroscope

The archived README states the acquisition date of 15 March 2023. Five months later phlaredb became pyroscopedb, and the acquired codebase still sits at HEAD in pkg/og.

Carry forwardAfter an acquisition, look at which engine's package names survive; that tells you which team's architecture won.

grafana/phlare README

Source Grafana Labs2026-09

Beyla donated to OpenTelemetry

"Beyla has been donated to the CNCF OpenTelemetry Project, under the project name OpenTelemetry eBPF Instrumentation… All Beyla current maintainers work full time on the upstream repository."

Carry forwardComponents whose value is ecosystem reach get donated; components that differentiate the product do not. Use that test on your own estate.

grafana/beyla README

Source Grafana Labs2025-04

Grafana removes Angular, twelve years in

"Frontend: Remove Angular" deletes the support toggle, the deprecation UI and the framework itself, 146 months after the repository's initial commit.

Carry forwardFramework choices in a plugin ecosystem outlive every other decision in the system, because your users' code is written against them.

grafana/grafana #99760

Vendor Grafana Labs2026-09

Mimir's README scale claim

"Internal testing shows that Grafana Mimir handles up to 1 billion active time series." No methodology, hardware or cost accompanies it in the repository.

Carry forwardTreat a headline scale number with no methodology as a ceiling claim, not a capacity plan.

grafana/mimir README

Postmortem DFDS2024-06

Kube state metrics exposing wrong labels

A collector migration towards Grafana Cloud changed exposed labels, alerts fired, and the fix was blocked by an unrelated unstable infrastructure change, so alarms were silenced overnight.

Carry forwardTreat a collector swap as a change to your alerting contract, and never start one while your rollback path is blocked.

dfds/postmortems PM2024-003

Postmortem DFDS2022-01

Prometheus is down with connection refused errors

Three nodes NotReady in one day, each running the single Prometheus pod, with the OOM killer in the kernel log and no resource requests or limits in the chart.

Carry forwardThe monitoring system fails when you need it; give it limits, and a second place to look.

dfds/postmortems PM2022-001

Postmortem DFDS2022-07

Kafka metrics stopped being exported

A deleted personal account took the exporter's API key with it; the replacement key was missing access control entries, so metrics were partially missing for three and a half hours after the incident was closed.

Carry forwardClose a telemetry incident on a metric-set comparison, not on the exporter being up.

dfds/postmortems PM2022-007

Postmortem Zalando2019-01

Total DNS outage in a Kubernetes cluster

Included as the pattern this guide depends on rather than for its content: an operator keeping incident reviews as markdown in the repository that configures the platform, where anyone can read them years later.

Carry forwardIncident reviews kept in the repository outlive status pages, and they are searchable by the people who inherit the system.

zalando-incubator/kubernetes-on-aws postmortems

Build a miniature, then productionise it

Six rungs. The first three are an evening each; the crossing into production shape happens at rung four.

Write immutable blocks to a bucket and query them

Take any append-only data you have. Buffer it in memory, flush every N seconds into an object named by tenant and time range, and write a reader that answers a time-range query by fetching the objects that overlap.

Done when: a query returns correct results after the writer process has been killed and restarted. Teaches: why the flush boundary, not the write, is the durability boundary.

Add the catalogue object, and delete the LIST

Have the writer maintain one small JSON object per tenant listing the blocks and their time ranges. Change the reader to fetch that object and nothing else before deciding what to read.

Done when: a cold reader answers its first query with exactly one GET before any data fetch, and your object-store request count stops growing with block count. Teaches: the design in the Cortex bucket-index proposal, and why cold start was measured in tens of minutes without it.

Compact, and keep the catalogue honest

Add a compactor that merges small blocks into larger ones, marks the originals for deletion and updates the catalogue. Run it concurrently with the reader.

Done when: a reader that loaded the catalogue before compaction still returns correct results, with no 404 from a deleted block. Teaches: why deletion marks exist and why the catalogue is written by the compactor rather than the writer.

Put a log in front, and try to lose data

Insert Kafka or Redpanda between the ingest API and the buffer. Commit offsets after the flush. Then deliberately write the bug: commit before the flush, kill the process mid-flush, and measure what disappears.

Done when: you can produce the data loss on demand and then make it impossible. Teaches: the exact failure four Grafana Labs fixes address, in a system small enough to see it.

Ship a second on-disk format and migrate

Change the block layout, write a version field, support reading both, add a conversion command, flip the default, and only then delete the reader for the old one.

Done when: a cluster holding both formats answers queries correctly and the conversion is resumable. Teaches: why Tempo needed five formats and a CLI subcommand for each transition, and why format versions outlive their authors' plans.

Run the deprecation drill on yourself

Pick a configuration option in your own system. Add a per-tenant usage counter for it, mark it deprecated, and set a removal date from what the counter shows rather than from the calendar. Keep going until the code, the flags, the dashboards and any forked dependency it pulled in are gone.

Done when: you can state, from telemetry, how many consumers would break today. Teaches: the discipline that turns a five-year tail into a one-year one, and the reason the reverted removal in section 4 happened.

Keep hunting

This page was assembled with git, not a search engine. These are the commands that found the material, and they work on any company that develops in public.

Find what a company removed

git log --format="%ad %h %s" --date=short | grep -iE "chore!|feat!|remove (deprecated|support for)"
git log --diff-filter=D --name-only --format="%ad %h %s" --date=short -- "pkg/**"
git log --format="%ad %s" --date=short | grep -iE "^.* revert" | head -40

Date the architecture, not the announcement

git log --reverse --diff-filter=A --format="%ad %h %s" --date=short -- "pkg/storage/ingest/*"
git log -1 --format=%ad --date=short <tag>
git rev-list --max-parents=0 HEAD

Read the decisions that were written down

git ls-tree -r --name-only HEAD | grep -iE "proposal|adr|rfc|design"
git log --diff-filter=A --name-only -- "docs/proposals/*"
git show <tag>:CHANGELOG.md | grep -iE "BREAKING|\[CHANGE\]" | head -60

Find the forks and the bugs that matter

git show HEAD:go.mod | grep -B2 -A1 "^replace"
git log --format="%ad|%h|%s" --date=short | grep -iE "data loss|silently|incorrect result|corrupt"
git tag --contains <fix-sha> --sort=creatordate | head -3

Two notes on method. First, clone with --filter=blob:none: you get the whole history of commits and trees in a fraction of the space, and blobs are fetched only when you actually read a file. Second, the last command in the fourth block is the one that turns a fix into an operational answer: given the commit that fixed a data-loss bug, the first tag containing it is the first release where you were safe.

References

Repository artefacts, all read on 26 September 2026 from clones made the same day. The full ledger, with the quote supporting each claim, is in sources.md beside this file.

grafana/loki, boltdb-shipper commit e59adccf Grafana Labs, 2020-07-28. Checked 2026-09-26.
grafana/loki, TSDB index commit 8c1095a5 Grafana Labs, 2022-04-11. Checked 2026-09-26.
grafana/loki, first dataobj commit 9a215902 Grafana Labs, 2025-01-07. Checked 2026-09-26.
grafana/loki #21502, remove BigTable storage backend Grafana Labs, 2026-04-16. Checked 2026-09-26.
grafana/loki #21574, remove Cassandra storage backend Grafana Labs, 2026-04-17. Checked 2026-09-26.
grafana/loki #21601, remove DynamoDB storage backend Grafana Labs, 2026-04-20. Checked 2026-09-26.
grafana/loki #21637, remove BoltDB index backend Grafana Labs, 2026-04-21. Checked 2026-09-26.
grafana/loki #21678, remove BoltDB Shipper backend Grafana Labs, 2026-05-15. Checked 2026-09-26.
grafana/loki #9246, docs state the legacy index stores will be removed in a future release Grafana Labs, 2023-08-16. Checked 2026-09-26.
grafana/loki #12742, mark legacy stores deprecated more visibly in the docs Grafana Labs, 2024-04-23. Checked 2026-09-26.
grafana/loki #22466, revert of the /api/prom removal Grafana Labs, 2026-06-19. Checked 2026-09-26.
grafana/loki #21481, remove Simple Scalable Deployment targets Grafana Labs, 2026-06-24. Checked 2026-09-26.
grafana/loki #21245, remove Promtail support Grafana Labs, 2026-03-25. Checked 2026-09-26.
grafana/loki #14043, WAL segment ingestion via Kafka Grafana Labs, 2024-09-10. Checked 2026-09-26.
grafana/loki #18833, dataobj consumers commit incorrect offset Grafana Labs, 2025-08-13. Checked 2026-09-26.
grafana/loki #21204, OR operations silently dropped Grafana Labs, 2026-03-19. Checked 2026-09-26.
grafana/loki go.mod at main Grafana Labs, read at 2026-09-25 head. Checked 2026-09-26.
grafana/mimir, Cortex to Mimir rename commit fa86354b Grafana Labs, 2021-08-11. Checked 2026-09-26.
grafana/mimir CHANGELOG at mimir-2.0.0 Grafana Labs, 2022-03-28. Checked 2026-09-26.
grafana/mimir CHANGELOG at mimir-3.0.0 Grafana Labs, 2025-10-31. Checked 2026-09-26.
grafana/mimir #6888, Kafka-compatible ingest storage Grafana Labs, 2023-12-12. Checked 2026-09-26.
grafana/mimir #16493, silent data loss for single-record jobs Grafana Labs, 2026-08-28. Checked 2026-09-26.
grafana/mimir go.mod at main Grafana Labs, read at 2026-09-25 head. Checked 2026-09-26.
grafana/mimir, About Grafana Mimir versioning Grafana Labs, read at 2026-09-25 head. Checked 2026-09-26.
grafana/mimir README Grafana Labs, read at 2026-09-25 head. Checked 2026-09-26.
cortexproject/cortex, Blocks storage bucket index proposal Marco Pracucci, November 2020, merged 2020-12-09. Checked 2026-09-26.
cortexproject/cortex #4812, remove chunks storage entirely: Take 2 Cortex maintainers, 2022-08-10. Checked 2026-09-26.
grafana/tempo #1479, add Parquet block format Grafana Labs, 2022-06-23. Checked 2026-09-26.
grafana/tempo #5563, Rhythm live-store Grafana Labs, 2025-08-25. Checked 2026-09-26.
grafana/tempo CHANGELOG at v3.0.0 Grafana Labs, 2026-05-28. Checked 2026-09-26.
grafana/alloy, module rename commit 8c2c7a0f Grafana Labs, 2024-03-25. Checked 2026-09-26.
grafana/agent README, end-of-life notice Grafana Labs, read at 2026-03-25 head. Checked 2026-09-26.
grafana/beyla README, donation notice Grafana Labs, read at 2026-09-25 head. Checked 2026-09-26.
grafana/phlare README, archive notice Grafana Labs, archived 2023. Checked 2026-09-26.
grafana/pyroscope #2260, rename phlaredb to pyroscopedb Grafana Labs, 2023-08-17. Checked 2026-09-26.
grafana/grafana #99760, remove Angular Grafana Labs, 2025-04-04. Checked 2026-09-26.
grafana/grafana, AGPLv3 relicence commit 8db3eb90 Grafana Labs, 2021-04-20. Checked 2026-09-26.
DFDS, PM2024-003, Kube state metrics exposing wrong labels DFDS, 2024-06-06. Checked 2026-09-26.
DFDS, PM2022-001, Prometheus is down with connection refused errors DFDS, 2022-01-25. Checked 2026-09-26.
DFDS, PM2022-007, Kafka metrics stopped being exported to Prometheus DFDS, 2022-07-07. Checked 2026-09-26.
Zalando, Total DNS outage in Kubernetes cluster Zalando, 2019-01-07. Checked 2026-09-26.

The territory

Figure 1 · 2016 to 2021: four products, built one at a time

Figure 2 · 2022 to 2026: convergence, then one wave of deletions

Four databases that became one architecture

What is genuinely shared

Where they still diverge

What quietly disappeared

Figure 3 · The shape all four converged on

Figure 4 · Four index generations in one product

The decisions that matter

Put the index in the object store, or keep it in a database

Fork the dependency, or wait for upstream

Fork the project, or keep sharing the upstream

A general columnar format, or a bespoke one

Own the agent, or give it away

Buy the product, or keep your implementation

Figure 5 · Deciding whether the old backend can go

What broke, and what the record does not tell you

Figure 6 · The failure class the new write path keeps producing

Numbers you can plan against

The evidence wall

Blocks storage bucket index (design proposal)

Loki: the index moves into the bucket

Loki deletes Cassandra, DynamoDB, BigTable and BoltDB

The removal reverted eight days later

"Remove support for chunks storage entirely: Take 2"

Mimir 2.0 changelog: one line, eighteen pull requests

Cortex renamed to Mimir, in private, seven months early

Mimir's go.mod: six forks, with reasons

Loki's go.mod: the fork that outlived the feature

Mimir's versioning guarantees

Mimir puts Kafka in the write path

Four data-loss fixes in the Kafka path

The rewritten Loki engine dropped OR

Mimir 3.0 removes the alternatives

Tempo adopts Parquet, then keeps re-adopting it

Tempo 3.0 chooses failing over answering

Grafana Agent's end of life, stated in its own README

Promtail deleted from Loki

Phlare archived, its engine renamed to Pyroscope

Beyla donated to OpenTelemetry

Grafana removes Angular, twelve years in

Mimir's README scale claim

Kube state metrics exposing wrong labels

Prometheus is down with connection refused errors

Kafka metrics stopped being exported

Total DNS outage in a Kubernetes cluster

Build a miniature, then productionise it

Write immutable blocks to a bucket and query them

Add the catalogue object, and delete the LIST

Compact, and keep the catalogue honest

Put a log in front, and try to lose data

Ship a second on-disk format and migrate

Run the deprecation drill on yourself

Keep hunting

References

Mimir's `go.mod`: six forks, with reasons

Loki's `go.mod`: the fork that outlived the feature