When the answer is wrong, not late: ten years of CockroachDB read from its own record

The territory

A system that has been engineered for availability for a decade does not stop failing. Its failures change shape, and the monitoring built for the old shape cannot see the new one. This guide reads one company's public record in bulk to find out what the new shape is.

technical advisories published between October 2018 and May 2026, each with affected version ranges

cockroachdb/docs advisories, counted 2026-09-10

9→4

C and C++ submodule dependencies, v19.1 to v26.1, with the storage engine among those removed

.gitmodules at v19.1.0 and v26.1.0

45→1

public design records written per year: 45 in 2017, one since the start of 2024

cockroachdb/cockroach docs/RFCS, counted 2026-09-10

37/73

advisories whose only published remedy is to upgrade every node

derived from the advisory mitigation sections, 2026-09

Cockroach Labs is an unusually good subject for this question, and not because CockroachDB is unusually good. It is because the company leaves three kinds of artefact in public that most vendors keep private. It publishes numbered technical advisories describing defects that reached production clusters, with affected version ranges and a mitigation. It kept a design-record directory in the product repository for eight years, with a status field on every document, including the rejected ones. And every release is a public git tag, which means any claim about when a subsystem appeared or disappeared can be settled by fetching one file at two tags rather than by trusting a narrative.

Read in date order, the advisory corpus tells a story that the release notes do not. The first advisory, in October 2018, is an authentication bypass. Through 2020 and 2021 the corpus is dominated by the failures a distributed database is supposed to have: a schema change that makes a cluster unavailable, an HTTP endpoint that can be used to take a cluster down, snapshots that corrupt data during replica garbage collection. Then, between 2021 and 2023, fourteen advisories in a row say a version of the same sentence: the database returned the wrong answer, quietly, to a query that succeeded. Nine of them are in 2022 alone. After 2023 that class disappears from the record entirely.

Two things happened in the same window, and the guide argues that they are related. The features shipped between 2020 and 2022 (multi-region tables, partitioning by locality, additional index shapes) each multiplied the number of physically distinct plans the optimiser could choose for the same logical query. And the tests that can catch a wrong plan, rather than a slow one, did not exist in the tree until late 2021 and late 2022. You can watch both happen by fetching files at release tags, which is what section five does.

The finding to take away

Availability failures announce themselves. Correctness failures are announced by nobody, which is why the corpus shows them being found by a machine that runs each query twice and compares. If your system has matured past its availability problems, your next detection investment is not another probe of liveness. It is a second, slower, obviously correct implementation of your own hot path to diff against.

The second finding is about the record itself, and it is the one that surprised the author of this guide. Between the v23.2.0 release (generally available 5 February 2024) and the v24.1.0 release (20 May 2024), the first line of the design-record directory's README changed to read: "This review process is deprecated. Existing docs are left for archival purposes." The link that follows points at an internal Confluence space. Over the same period, PingCAP's competing distributed SQL database kept writing dated design documents in its public repository, seven of them in 2024, five in 2025 and seven in the first eight months of 2026. One company's public architecture record has a last page; the other's does not. If you are evaluating a vendor by reading how it thinks, check whether it is still thinking in public.

What this guide covers. The architecture of the CockroachDB product and the visible practices of the company that builds it, 2016 to 2026, as evidenced by repository artefacts. What it deliberately does not cover. The CockroachDB Cloud service architecture, prices, unit economics, customer deployments, and any comparison of performance between vendors. It also contains no engineering-blog, conference-talk or peer-reviewed evidence at all: the network policy for this session reached code hosts only, so every source here is a repository artefact, and the reasoning published on the company's blog is outside the corpus. Where a motive is stated only on the blog, this guide says the motive is unavailable rather than guessing it.

Figure 1 · Four eras, dated from the repository

Each era is dated by artefacts rather than by announcements: design-record counts per year, the release tag where a file first appears, and advisory publication dates. Sources: docs/RFCS, advisories, releases.yml.

Diagram source

How it is actually built

The shape that three independent distributed SQL implementations share, the four places they diverge, and the two layers CockroachDB took in-house during the decade.

Every distributed SQL system in this corpus has the same five-layer shape, and the shape is not a matter of taste: it falls out of wanting SQL semantics on top of a replicated key-value store. A gateway process parses and plans. A distributed execution layer splits the plan across nodes. A key-value layer maps keys to contiguous ranges, replicates each range with Raft, and elects a leaseholder that serves reads without a round trip. Below that sits a log-structured merge tree per node. Alongside all of it sit the paths that move data out of the transactional boundary: backup, restore, bulk import, and change data capture.

The divergences between vendors are the interesting part, because each one is a decision somebody wrote down. CockroachDB runs the whole stack in a single binary; PingCAP splits the SQL layer (TiDB) from the key-value layer (TiKV) into separate processes, which is why its design-document directory has documents about cross-process request limiters that have no CockroachDB counterpart. YugabyteDB describes its storage layer, DocDB, as "inspired by Google Spanner" and built on a fork of RocksDB, where CockroachDB replaced RocksDB with an implementation of its own. Those two choices are the same decision answered differently, and section three treats it as such.

Two layers changed hands inside CockroachDB during the decade, and both changes are visible as files appearing and disappearing at release tags rather than as announcements. The storage engine came first. pkg/storage/pebble.go is present at the v20.1.0 tag (generally available 12 May 2020), sitting beside pkg/storage/rocksdb.go; at v21.1.0 (18 May 2021) the RocksDB file is gone, and so are the c-deps/rocksdb, c-deps/snappy, c-deps/cryptopp and c-deps/googletest submodules that supported it. Two releases of coexistence, then removal. Pebble's own README states the scope of the replacement rather than the motive: it "inherits the RocksDB file formats" and "intentionally does not aspire to include every feature in RocksDB", targeting only what CockroachDB needs. Inheriting the file format is the load bearing detail. It is what made the swap an in-place upgrade rather than a migration.

The second layer was scheduling. pkg/util/admission/work_queue.go first appears at v21.2.0 (16 November 2021). The design note explains what was being taken away from the Go runtime: the goal is "to shift queueing from inside the goroutine scheduler, where there is no differentiation, into various admission queues, where we can differentiate", and it names the two resources that had actually hurt clusters: CPU overload and read amplification in the LSM. The same note is candid about what admission control still cannot do, which is memory, "because memory is non-preemptible". By v23.1.0 the mechanism has grown a replication arm (the replicated write admission tests), which is admission control applied to Raft followers rather than to the leaseholder alone.

A third change of hands arrived late and is worth watching. Range leases were built on node liveness recorded in a single system range, a design that makes the liveness range a shared fate for the whole cluster. storeliveness/support_manager.go first appears at v24.3.0 (18 November 2024), moving liveness to direct support exchanged between stores. The company also wrote a TLA+ specification for it, and the tla-plus directory contains exactly two specifications: this one and the one for parallel commits. Two formal models in ten years, both for protocol changes whose failure mode is silent rather than loud. That is a deliberate allocation of the most expensive verification technique the company uses, and it is aimed at the same target as the query oracles.

Figure 2 · The common shape, with the layers that changed hands

Solid boxes are common to all three vendors in the corpus. The two boxes marked in violet are layers CockroachDB moved in-house during the decade, dated by the release tag at which the file first appears. The dashed boxes are the paths that leave the transactional boundary, and they carry a disproportionate share of the advisories.

Diagram source

Ranges and leaseholders

Data is split into ranges, each a Raft group, each with a leaseholder that can serve reads locally. Splits were irreversible until 2018: the range-merge design record opens with "Splitting a range in a CockroachDB cluster today is irreversible", and lists load-based splitting as the feature that could not ship until merges existed.

Evidence: range merges design record, 2018-03-30

Closed timestamps

Follower replicas learn a timestamp below which no future write can land, which turns every replica into a legal source for historical reads. The 2018 design record lists three consumers before the feature existed: analytics, reference tables, and recovery of a consistent snapshot after loss of quorum.

Evidence: follower reads design record, 2018-06-03

The data-out paths

Backup, restore, import and changefeeds read committed data and hand it to something that is not the database. Nothing downstream can detect an omission, which is why these paths produce advisories with the word "silent" in them as late as 2026.

Evidence: advisory 161317, 2026-02-11

The decisions that matter

Five forks where the record states both the choice and the reason, plus the condition that would flip each one for a reader in a different situation.

The design records are unusually honest about reversal. One of them opens with a warning that the document is wrong: the range-merge record carries the disclaimer that "the implementation encountered serious hurdles with the initial design proposed here" and that it is "preserved in its original form for posterity", pointing readers at a tech note written a year later. Treat that as the house style rather than an accident. The decisions below are the ones where a reader building something else can extract a rule.

Decision: own the storage engine, or keep using someone else's?

Chosen

Write Pebble in Go, inheriting the RocksDB file format so existing stores can be read in place
Ship it alongside RocksDB for two releases (v20.1, v20.2), then delete the old engine and four C dependencies at v21.1

Rejected

Staying on RocksDB, the choice YugabyteDB made and still describes in its own design docs
The repository states the scope of the replacement, not the motive; the reasoning was published elsewhere and is outside this corpus

Flips when

You can observe and patch the dependency on your own schedule. The one advisory in this corpus that blames the engine directly says "a defect in RocksDB allowed it to violate its guarantees for snapshots", and the remedy was a new release of the whole database
You cannot inherit the on-disk format. Without format compatibility this is a migration, not a swap, and the calculus changes completely

Decision: serializable only, or offer a weaker isolation level?

Chosen

After eight years of serializable-only, implement Read Committed (design record accepted January 2023)
Stated impetus: to "ease application migrations"

Rejected

Holding the line and requiring applications to add retry logic
The record is blunt about why that failed: adding retry logic is "merely inconvenient for new applications" but "often infeasible for existing applications"

Flips when

Your users write new code against your system rather than moving old code onto it. Greenfield users pay the retry tax without noticing; migrating users cannot pay it at all
Note the direction of travel: the guarantee was not weakened, an option was added. The cost is a second concurrency-control path to test forever

Decision: where does the tenant boundary sit, process or address space?

Chosen

Start with separate SQL processes per tenant for the serverless product, metered by a token bucket in "request units"
Add shared-process virtual clusters in 2022: tenant SQL servers instantiated on the fly inside an already-running node process

Rejected

Process-per-tenant as the only shape. The 2022 record lists the reasons plainly, including separating process startup from tenant selection to lower "initial connection latency"

Flips when

Tenant density and cold-start latency dominate your economics, which is when the address-space boundary wins
It flips back the moment a tenant can consume a shared resource the runtime will not arbitrate; that is exactly why admission control had to exist first

Decision: let the language runtime schedule work, or build your own queues?

Chosen

Build admission control in-process from v21.2, with explicit queues for CPU and for storage writes, and priorities across tenants

Rejected

Relying on the Go scheduler, which the tech note dismisses in one clause: inside it "there is no differentiation"
Memory is explicitly left out of scope because it is "non-preemptible"

Flips when

Your nodes are stateless. The note's own justification is that CockroachDB "has stateful nodes", so a hotspot persists until rebalancing rather than being shed to a peer
If load balancers can move work in milliseconds, spend the effort there instead

Decision: keep the design record public, or move it inside?

Chosen

Retire the public process. Between v23.2.0 (February 2024) and v24.1.0 (May 2024) the directory README became "This review process is deprecated", pointing at an internal wiki
Volume had already collapsed: 25 records in 2021, 20 in 2022, 6 in 2023, one since

Rejected

Continuing in public, which is what PingCAP does: dated design documents are still landing in its repository through August 2026

Flips when

Outside contributors write the code. A public record is a coordination tool for people you do not employ; when the hard subsystems are built entirely in-house, its cost stops buying anything internally
For a buyer this flips the other way: from 2024 the only way to learn how this system decides things is to read its commits

Decision	Chosen	Rejected or reversed	Stated reason	Evidence
Storage engine	Pebble, in Go, format-compatible	RocksDB, deleted at v21.1	Scope control: only the features CockroachDB uses	Pebble README
Isolation levels	Add Read Committed, 2023	Serializable only, 2015 to 2023	Retry logic is "often infeasible for existing applications"	Design record, 2023-01-22
Tenant boundary	Shared-process virtual clusters	Process-per-tenant as the only option	Start the process before choosing the tenant, to cut connection latency	Design record, 2022-07-20
Overload control	In-process admission queues, v21.2	The language runtime's scheduler	No differentiation inside the goroutine scheduler	Admission control note
Commit protocol	Parallel commits, 2018	Two sequential rounds of consensus	Cut commit latency "in half, from two rounds of consensus down to one"	Design record, 2018-03-24
Physical layout	Interleaved tables, shipped 2016	Removed: the executor file is gone by v21.1	Not stated in this corpus; the feature required an irreversible schema choice	Executor at v20.2.0, absent at v21.1.0
Design record	Internal review from early 2024	Public design records, 2015 to 2023	"This review process is deprecated"	docs/RFCS README

Figure 3 · When to take a layer in-house

A decision tree derived from the three layers this company insourced and the one dependency class it kept. Every terminal node is an action, not a preference.

Diagram source

What broke in production

Six advisories, chosen to cover the four classes the corpus falls into. Each is a vendor-authored defect disclosure: it names the mechanism precisely and the impact not at all.

Read seventy-three of these in one sitting and one shape recurs so often that it stops looking like coincidence. A performance optimisation skips work that a slower path would have done. Somewhere else in the system, a structural change moves the data the optimisation is reasoning about: a range splits, a lease transfers, a job restarts, a replica is garbage collected. The two meet, and the result is not a crash. It is a success response covering an absence. Garbage collection racing a split deletes live data. A lease transfer racing a slow disk gives a range two leaseholders. A changefeed checkpoint racing a restart marks unscanned spans as scanned.

The second recurring shape is narrower and belongs to one era. Between 2021 and 2023 the query optimiser returned wrong results in fourteen published cases, and the conditions read like a catalogue of the plan space that had just been added: two indexes with matching nullable trailing columns, a partitioned table with a default partition, a lookup join against a table that is regional by row. Each locality and index feature that made a query faster also made the set of legal physical plans larger, and the wrong answers came from the corners of that set.

Advisory

The engine you did not write violates its own guarantee

AssumptionA storage engine's snapshot isolation holds while replicas are garbage collected around it.

What happenedRange deletions from table drops, truncates and rebalancing during decommissioning, concurrent with replica garbage collection, hit a defect that "allowed RocksDB to violate its guarantees for snapshots".

Blast radiusEvery v2.x and v19.x release. The advisory states the corruption "is irrecoverable using regular node failover and must be addressed by restoring a backup". No count of affected clusters is published.

FixPatch releases; strategically, the engine was already being replaced, and RocksDB left the tree eleven months later.

Design ruleA dependency whose defects can only be fixed by shipping your entire product is not a dependency, it is a subsystem you own without control of.

SourceTechnical advisory 48860, 20 May 2020

Advisory

Two leaseholders, and the write that fell between them

AssumptionPromoting a transferred lease from expiration-based to epoch-based cannot move the expiry backwards.

What happenedWith a block device intermittently slow over an extended period, the lease expiration "regressed" during promotion, leaving a window in which another node could claim the same lease. Transactions spanning multiple ranges could lose some of their writes.

Blast radiusv22.2 through v24.2. Single-range transactions and one-phase commits unaffected. Recovery is possible only if a secondary index happens to cover the lost columns.

FixPatch releases across five supported branches. The store-liveness rework, whose first files land one month later at v24.3.0, replaces the liveness substrate this depended on.

Design ruleAny lease or lock whose safety rests on a clock reading must be tested against a slow device, not only against a dead one. Slow is a distinct failure mode from down, and it is the one that produces two owners.

SourceTechnical advisory 131639, 8 October 2024

Advisory

Garbage collection deleted data that was not garbage

AssumptionDeleting a whole key range at once is equivalent to deleting its keys individually, only faster.

What happenedA race between MVCC garbage collection and range splits could, in narrow circumstances, delete live data. The interim mitigation is to turn the optimisation off: SET CLUSTER SETTING kv.gc.clear_range_min_keys = 0, which forces individual key deletions.

Blast radiusv23.1 and later, across seven release series. The advisory says most workloads are unlikely to be affected and that the conditions are narrow, and it offers customers a section titled "Am I Affected?" rather than a number.

FixPatched maintenance releases on every supported branch; managed customers were patched without action.

Design ruleEvery bulk shortcut needs a runtime switch back to the slow path, and the switch has to be safe to flip under load, because it will be flipped during an incident by someone who cannot upgrade today.

SourceTechnical advisory 162085, 19 February 2026

Advisory

The optimisation that dropped events was an optimisation about duplicates

AssumptionA checkpoint written during an initial scan describes which spans have genuinely been scanned.

What happenedA change intended to reduce duplicate events on restart introduced non-determinism elsewhere that "would sometimes incorrectly forward the progress of every span a node was tracking to the lowest checkpoint timestamp when some spans may not have been scanned yet". Consumers received incomplete data.

Blast radiusv22.2 through v23.2.5. The listed symptom is the giveaway: emitted row count lower than the source table's row count, which nobody sees unless they compare.

FixA single pull request; the advisory names it.

Design ruleIf your pipeline's contract is at-least-once, the reconciliation that counts rows is not optional tooling. It is the only detector you have for the class of bug that trades duplicates for omissions.

SourceTechnical advisory 123371, 17 June 2024

Advisory

A join optimisation that returned the wrong rows

AssumptionTwo indexes constrained on their leading columns can be zigzagged together safely.

What happenedWhen the trailing columns were shared and nullable, and held nulls, zigzag joins could return incorrect results. Four major versions were affected, two of which were already out of maintenance support and would never be patched.

Blast radiusv19.2, v20.1, v20.2 and v21.1. The mitigation is to disable the optimisation with a cluster setting, which the advisory notes "may result in poorer performance in some cases".

FixStop planning the join in the cases that were wrong, rather than fixing the execution.

Design ruleNullability is where plan-level shortcuts break. If a rewrite rule is only valid for non-null values, the validity condition belongs in a test that generates nulls on purpose, not in a comment.

SourceTechnical advisory 71655, 2 November 2021

Advisory

The locality feature that corrupted the table it was placing

AssumptionDropping a region is safe once no table declares that region.

What happenedFor tables altered to be regional by row, dropping a region succeeded where it should have failed, because back-references from the table to the region enum were not updated. Later locality changes then failed with an internal error naming an enum value that no longer existed.

Blast radiusv22.1.0 to v22.1.3, four patch releases wide. The advisory includes an exact reproduction against a nine-node demo cluster.

FixTwo pull requests correcting the descriptor back-references.

Design ruleWhen you add a dimension to physical placement, you add a second graph of references that must stay consistent with the first. Model that graph explicitly or it will be maintained by whichever code path happens to run.

SourceTechnical advisory 84144, 19 July 2022

Figure 4 · The seam that produces most of the corpus

Four advisories, one shape: an optimisation that skips work meets a structural change that moves the data underneath it, and the result is a successful operation with something missing. Sources: advisories 48860, 131639, 123371, 162085.

Diagram source

Figure 5 · How a lease promotion loses a write

The mechanism in advisory 131639, drawn from the advisory's own description. Notice that no participant observes an error: the gateway sees two successful writes and commits. Source: Technical advisory 131639, 2024-10-08.

Diagram source

What these documents do not contain

Not one of the 73 advisories states how long the defect existed before discovery, how many clusters were affected, or how it was found. The public record establishes mechanism and version range with real precision and says nothing about blast radius or detection lag. Any figure you see quoted for those is not coming from here.

Numbers you can plan against

Everything quantitative in this guide, with how it was obtained. Counts derived by the author from repository artefacts are marked as derived, and the method is given so you can repeat them.

Measure	Value	Kind	Context	As of	Source
Technical advisories published	73	Derived	Files in the advisories directory, excluding the index page	2026-09	advisories directory
Advisories per year	1, 9, 13, 19, 20, 5, 2, 4	Derived	2018, 2020 to 2026 in order; none in 2019	2026-09	front matter field advisory_date
Advisories about wrong query results	14	Derived	2 in 2021, 9 in 2022, 3 in 2023, none since	2026-09	classified from summary fields
Advisories offering only an upgrade	37 of 73	Derived	Mitigation section names no setting and no workaround	2026-09	mitigation sections
Advisories mitigable by one cluster setting	10 of 73	Derived	Mitigation contains a SET CLUSTER SETTING statement	2026-09	advisory 162085 is one
Public design records	187 dated	Derived	Date-prefixed files in docs/RFCS; 194 files in total	2026-09	docs/RFCS
Design records per year, peak and now	45 then 1	Derived	45 in 2017; 6 in 2023; one in 2024, none in 2025, one in 2026	2026-09	filename date prefixes
Peer vendor design documents, 2024 to 2026	19	Derived	PingCAP TiDB: 7 in 2024, 5 in 2025, 7 by August 2026	2026-09	tidb docs/design
C and C++ submodules	9 then 4	Measured	v19.1.0 versus v26.1.0; rocksdb, snappy, cryptopp and googletest all leave at v21.1.0	2026-09	.gitmodules by tag
Documented cluster settings	172 to 387	Derived	Rows in the generated settings table, v21.2.0 to v26.1.0; earlier tags use a different format and are not comparable	2026-09	generated settings table
Production releases per year	11 to 83	Measured	2017 versus 2025, from the release register; 393 production releases in total	2026-09	releases.yml
Major versions per year	2 then 3 to 4	Measured	Two per year through 2023; v24.1, v24.2 and v24.3 in 2024, four series dated in 2025	2026-09	support policy and release register
Formal specifications in the tree	2	Measured	Parallel commits and store liveness, the two protocol changes with silent failure modes	2026-09	tla-plus directory

The dating method is worth stealing, because it works on any public repository and it does not depend on anybody's blog staying online. Pick a file that only exists when a subsystem exists, then fetch it at successive release tags and record the first tag that returns a response. The table below is the result for the layers this guide discusses, and every row can be re-derived in about a minute.

Subsystem	Marker file	First tag present	That release went GA	Check
Pebble storage engine	pkg/storage/pebble.go	v20.1.0	2020-05-12	RocksDB still present at v20.2.0
Declarative schema changer	pkg/sql/schemachanger/scplan/plan.go	v21.1.0	2021-05-18	same tag drops four C dependencies
Admission control	pkg/util/admission/work_queue.go	v21.2.0	2021-11-16	work_queue.go at v21.2.0
Query oracle: ternary logic partitioning	pkg/cmd/roachtest/tests/tlp.go	v21.2.0	2021-11-16	tlp.go
Query oracles: cost fuzz and unoptimised diff	pkg/cmd/roachtest/tests/costfuzz.go	v22.2.0	2022-12-06	unoptimized_query_oracle.go
Replication admission control	pkg/util/admission/replicated_write_admission_test.go	v23.1.0	2023-05-15	admission control note
Store liveness	pkg/kv/kvserver/storeliveness/support_manager.go	v24.3.0	2024-11-18	support_manager.go at v24.3.0

Put the two tables beside each other and the argument of this guide is visible as arithmetic. The oracles that compare a query's result against the same query run with optimiser rules disabled arrive at v21.2.0 and v22.2.0. The wrong-answer advisories run from 2021 to 2023 and then stop. The corpus cannot prove that the first caused the second, and no document in it claims so, so treat the link as inferred from timing rather than reported. What is not inferred is the shape of the tool: the unoptimised query oracle runs each generated statement twice, once normally and once with optimiser rules disabled with probability one half or one, and compares the two result sets, on nine-node geo-distributed clusters among others. That is a second, deliberately slower implementation used as an oracle, which is the technique this guide would have an architect steal.

Read these carefully

The advisory counts measure disclosure, not defect rate. A year with fewer advisories may be a better year or a quieter one, and nothing in the corpus separates the two. The classification into failure classes is the author's, applied to the summary field of each advisory; the raw material is in the ledger beside this page so the classification can be disputed.

Everything here is self-reported by one vendor about its own product. There is no independent measurement in this guide, no cost figure, and no benchmark. The three vendors named are compared only on what they publish and how, never on performance.

The evidence wall

Every source behind this page, graded. The mix is unusual and the reason is stated at the end of section one: this hunt reached code hosts only, so there are no blogs, talks or papers here at all.

Postmortem Cockroach Labs2020-05

Technical advisory 48860: data corruption with snapshots and delete range

Range deletions concurrent with replica garbage collection could corrupt data permanently, because of a defect in the third-party storage engine. Recovery requires restoring a backup.

Carry forwardA defect you can only fix by shipping your whole product is a subsystem you own without controlling.

raw.githubusercontent.com/cockroachdb/docs/main/src/current/advisories/a48860.md

Postmortem Cockroach Labs2024-10

Technical advisory 131639: lease expiration regression loses writes

Sustained disk slowness during lease promotion could move a lease expiry backwards, allowing two leaseholders and losing writes in multi-range transactions.

Carry forwardTest leases against a slow device, not a dead one; slowness is what produces two owners.

raw.githubusercontent.com/cockroachdb/docs/main/src/current/advisories/a131639.md

Postmortem Cockroach Labs2026-02

Technical advisory 162085: garbage collection races range splits

A race between MVCC garbage collection and range splits could delete live data. The interim mitigation disables the bulk-delete optimisation with one cluster setting.

Carry forwardEvery bulk shortcut needs a switch back to the slow path that is safe to flip under load.

raw.githubusercontent.com/cockroachdb/docs/main/src/current/advisories/a162085.md

Postmortem Cockroach Labs2024-06

Technical advisory 123371: changefeeds drop events during initial scan

A change made to reduce duplicates on restart caused span progress to be forwarded past unscanned spans. Consumers received incomplete data, detectable only by counting rows.

Carry forwardRow-count reconciliation is the only detector for a pipeline that trades duplicates for omissions.

raw.githubusercontent.com/cockroachdb/docs/main/src/current/advisories/a123371.md

Postmortem Cockroach Labs2021-11

Technical advisory 71655: zigzag joins produce incorrect results

Wrong answers when two indexes shared nullable trailing columns holding nulls. Fixed by no longer planning the join in those cases; mitigated by a setting that costs performance.

Carry forwardA rewrite rule that is only valid for non-null values needs a test that generates nulls deliberately.

raw.githubusercontent.com/cockroachdb/docs/main/src/current/advisories/a71655.md

Postmortem Cockroach Labs2022-07

Technical advisory 84144: regional by row tables at risk of corruption

Dropping a region succeeded when it should have failed, leaving stale references to a region enum value and breaking later locality changes. Includes a full reproduction.

Carry forwardAdding a placement dimension adds a reference graph that somebody has to own explicitly.

raw.githubusercontent.com/cockroachdb/docs/main/src/current/advisories/a84144.md

Postmortem Cockroach Labs2026-02

Technical advisory 161317: silent data loss importing AVRO

An import defect could silently drop rows, affecting versions from v20.1 onward. The latest member of the class this guide calls the data-out paths, arriving in 2026.

Carry forwardBulk ingest needs a post-load count check, because the loader's success code proves nothing about completeness.

raw.githubusercontent.com/cockroachdb/docs/main/src/current/advisories/a161317.md

Postmortem Cockroach Labs2018-10 to 2026-05

The advisory index: 73 disclosures in one list

The index page is the corpus. Sorted by date it shows the shift from availability and authentication defects to wrong answers and silent omissions, and back to a lower overall rate after 2023.

Carry forwardRead a vendor's defect disclosures in bulk and in order before you read its architecture page.

raw.githubusercontent.com/cockroachdb/docs/main/src/current/advisories/index.md

ADR Cockroach Labs2023-01

Design record: Read Committed isolation

The document that reverses eight years of serializable-only, and says why: applications built for Read Committed "often struggle" to move, and adding retry logic is "often infeasible for existing applications".

Carry forwardMigration friction, not benchmark numbers, is what forces a guarantee to become an option.

raw.githubusercontent.com/cockroachdb/cockroach/master/docs/RFCS/20230122_read_committed_isolation.md

ADR Cockroach Labs2018-03

Design record: parallel commits

Cuts commit latency "in half, from two rounds of consensus down to one" by writing a STAGED status in parallel with the final batch, and introduces a recovery process for transactions abandoned in that state.

Carry forwardEvery latency shortcut across a commit boundary creates a new intermediate state that somebody must be able to resolve.

raw.githubusercontent.com/cockroachdb/cockroach/master/docs/RFCS/20180324_parallel_commit.md

ADR Cockroach Labs2018-03

Design record: range merges, with its own disclaimer

Opens by stating that splitting a range was irreversible, and carries a note that the implementation "encountered serious hurdles with the initial design proposed here", preserved "for posterity".

Carry forwardA decision record that is annotated when reality diverges is worth ten that are quietly deleted.

raw.githubusercontent.com/cockroachdb/cockroach/master/docs/RFCS/20180330_range_merges.md

ADR Cockroach Labs2018-06

Design record: follower reads

Introduces closed timestamps so that non-leaseholder replicas can serve consistent historical reads, and names three downstream consumers before any of them existed.

Carry forwardA mechanism that publishes a safe-to-read boundary is reusable; features that need one keep appearing.

raw.githubusercontent.com/cockroachdb/cockroach/master/docs/RFCS/20180603_follower_reads.md

ADR Cockroach Labs2022-07

Design record: shared-process deployments for virtual clusters

Tenant SQL servers start dynamically inside an already-running node process, with a server registry and routing to avoid one listener per tenant. Written for the managed service and reused for on-premises multitenancy.

Carry forwardTenant isolation is a dial, not a switch; the same product moved it from process to address space when latency mattered more.

raw.githubusercontent.com/cockroachdb/cockroach/master/docs/RFCS/20220720_shared_process_deployments.md

ADR Cockroach Labs2021-06

Design record: distributed token bucket for tenant cost control

Rate limits tenant work against a budget in "request units" tied directly to a dollar amount. Notable for what it points at: the motivating document is an internal-only record in a private repository.

Carry forwardWhen the money model moves into a private repository, the public design record stops explaining the system's incentives.

raw.githubusercontent.com/cockroachdb/cockroach/master/docs/RFCS/20210604_distributed_token_bucket.md

ADR Cockroach Labs2015-07

Design record: segmented storage, rejected

One of the rejected records that make the directory worth reading. The status field distinguishes completed, rejected, obsolete, postponed and superseded, so the corpus records paths not taken as well as those taken.

Carry forwardA status vocabulary that includes obsolete and superseded is what lets a record stay honest for a decade.

raw.githubusercontent.com/cockroachdb/cockroach/master/docs/RFCS/20150729_segmented_storage.md

ADR Cockroach Labs2024-05

The design-record README, after the process was retired

"This review process is deprecated. Existing docs are left for archival purposes." The replacement process lives on an internal wiki. The change lands between the v23.2.0 and v24.1.0 tags.

Carry forwardCheck whether a vendor's public reasoning is still being written before you rely on it for a decision.

raw.githubusercontent.com/cockroachdb/cockroach/master/docs/RFCS/README.md

ADR Cockroach Labs2021 to 2024

Admission control tech note

States the goal as moving queueing out of the goroutine scheduler into queues that can differentiate, names CPU overload and LSM read amplification as observed problems in real clusters, and excludes memory because it is non-preemptible.

Carry forwardOverload control is only worth building where the resource is preemptible; say which resources you are not protecting.

raw.githubusercontent.com/cockroachdb/cockroach/master/docs/tech-notes/admission_control.md

Source Cockroach Labs2022 onward

Unoptimised query oracle and cost fuzz roachtests

Generated statements are run twice, once normally and once with optimiser rules disabled with probability one half or one, and the result sets compared. Setups include nine-node geo-distributed clusters and multi-region schemas.

Carry forwardBuild a deliberately slow reference path and diff against it; it is the only cheap detector of silent wrongness.

raw.githubusercontent.com/cockroachdb/cockroach/master/pkg/cmd/roachtest/tests/unoptimized_query_oracle.go

Source Cockroach Labsv19.1 to v26.1

.gitmodules across eighteen release tags

Nine submodules at v19.1.0 including rocksdb, snappy, cryptopp and googletest; four at v26.1.0. The storage engine and its supporting C libraries leave together at v21.1.0.

Carry forwardDependency manifests at tags are the cheapest honest timeline of what a project stopped depending on.

raw.githubusercontent.com/cockroachdb/cockroach/v19.1.0/.gitmodules

Source Cockroach Labsv24.3.0

Store liveness support manager

The first tag where liveness support is exchanged between stores rather than read from a single system range. It arrives one month after the advisory about lease expiration regression.

Carry forwardWhen a shared range is the substrate for safety, its failure mode is the cluster's failure mode.

raw.githubusercontent.com/cockroachdb/cockroach/v24.3.0/pkg/kv/kvserver/storeliveness/support_manager.go

Source Cockroach Labs2026-09

Generated cluster settings table

The public switch surface, generated from the code at each release: 172 rows at v21.2.0, 387 at v26.1.0. Only ten of the 73 advisories can be mitigated by flipping one of them.

Carry forwardCounting your switches is easy; knowing which failure each one buys you back is the work.

raw.githubusercontent.com/cockroachdb/cockroach/master/docs/generated/settings/settings.html

Source Cockroach Labs2026-09

Two TLA+ specifications, and only two

Parallel commits and store liveness. Both are protocol changes whose failure mode is a silent loss of a guarantee rather than an outage.

Carry forwardSpend formal methods where the failure is invisible, not where it is merely expensive.

raw.githubusercontent.com/cockroachdb/cockroach/master/docs/tla-plus/StoreLiveness/StoreLiveness.tla

Source Cockroach Labs2020 onward

Pebble: README and the RocksDB differences document

States that Pebble inherits the RocksDB file formats and targets only the features CockroachDB needs, and documents the implementation divergences in detail. Its own design records continue after the main product's stopped.

Carry forwardFormat compatibility is what turns a rewrite into an upgrade; design for it before you decide to rewrite.

raw.githubusercontent.com/cockroachdb/pebble/master/docs/rocksdb.md

ADR PingCAP2018 to 2026

TiDB design documents, still being written

113 dated design documents in the public repository, including seven in 2024, five in 2025 and seven by August 2026. The contrast that dates the closure of the other vendor's public record.

Carry forwardCompare vendors on whether they still argue in public, not on whether they once did.

raw.githubusercontent.com/pingcap/tidb/master/docs/design/README.md

ADR Yugabyte2026-09

YugabyteDB design documents, with named owners

A design directory whose index lists a reference engineer per feature, and which describes DocDB as inspired by Spanner and built on RocksDB: the storage-engine decision answered the other way.

Carry forwardTwo competent teams split on owning the storage engine; the deciding variable is whether you can patch it on your own schedule.

raw.githubusercontent.com/yugabyte/yugabyte-db/master/architecture/design/README.md

Vendor Cockroach Labs2026-09

Release register and support policy

636 release entries, 393 of them production, with dates. Two major versions a year through 2023, then three or four, under a policy that splits releases into Regular and Innovation with different support phases.

Carry forwardA faster major-version cadence moves upgrade cost onto the operator; read the support phases before agreeing to it.

raw.githubusercontent.com/cockroachdb/docs/main/src/current/releases/release-support-policy.md

Build a miniature, then productionise it

Six rungs that turn this reading into a practice for your own system. The first three are an afternoon each; the crossing into production shape happens at rung four.

Read a defect corpus the way this guide did

Pick a vendor you depend on that publishes advisories or security bulletins. Extract the date and summary of every one into a table, sort by date, and classify each into at most five failure classes of your own naming.

Done when: you can state which class is growing and which has disappeared. Teaches: that a vendor's risk profile is a time series, not a reputation.

Date your own dependencies from tags

Choose three subsystems in a repository you rely on, pick a marker file for each, and fetch that path at successive release tags until you find the first tag where it exists. Record the dates against the project's release register.

Done when: you have a dated adoption timeline nobody wrote for you. Teaches: archaeology from artefacts beats archaeology from announcements.

Inventory the seams in your own system

List every optimisation that skips work (a cache, a checkpoint, a bulk path, a lease, a fast path for the common case) and beside each one write the structural change that could move its data underneath it: a rebalance, a restart, a deploy, a scale-in.

Done when: every row has a named concurrent event, or an explicit argument for why none exists. Teaches: where your own silent failures will come from.

Build one differential oracle

Take the most valuable read path you own and implement it a second time, as slowly and obviously as you like. Run both against generated inputs in continuous integration and compare results, not timings.

Done when: the oracle has failed once for a real reason and you fixed the fast path. Teaches: that correctness needs an independent implementation, because tests written from the same understanding share its blind spots.

Give every shortcut a switch, then rehearse using it

Add a runtime setting that forces the slow path for each row from rung three. Then run a game day in which somebody flips one under load and you measure what it costs.

Done when: the cost of each switch is written down beside it. Teaches: a mitigation nobody has exercised is a mitigation nobody will reach for at three in the morning.

Write the advisory you would want to receive

Take your last real defect and write it up in the shape used here: affected versions, mechanism, mitigation, and the one thing this corpus never provides, which is detection lag and blast radius. Circulate it internally.

Done when: a colleague who was not involved can tell whether they were affected. Teaches: how much of an incident's value is lost when nobody writes down when it started and who it touched.

Keep hunting

The commands that produced this page. They work against any public repository, and they do not depend on a search engine, which is the point.

Get the record cheaply

git clone --depth=1 --filter=blob:none --sparse https://github.com/cockroachdb/docs.git
git sparse-checkout set src/current/advisories src/current/_data
git clone --depth=1 --filter=blob:none --sparse https://github.com/cockroachdb/cockroach.git && git sparse-checkout set docs

Date a subsystem from release tags

for t in v21.1.0 v22.1.0 v23.1.0 v24.1.0; do curl -o /dev/null -w "$t %{http_code}\n" https://raw.githubusercontent.com/OWNER/REPO/$t/PATH; done
curl -s https://raw.githubusercontent.com/OWNER/REPO/$tag/.gitmodules | grep "path ="

Mine the decision record

grep -m1 -i "^- Status:" docs/RFCS/*.md | sort -t: -k3
ls docs/RFCS | sed -E 's/^([0-9]{4}).*/\1/' | sort | uniq -c
head -3 docs/RFCS/README.md

Turn advisories into a time series

grep -h "^advisory_date:" src/current/advisories/*.md | cut -c16-19 | sort | uniq -c
grep -l "SET CLUSTER SETTING" src/current/advisories/*.md | wc -l
grep -h "^summary:" src/current/advisories/*.md | grep -ci "incorrect result"

References

Cockroach Labs, Technical Advisories index cockroachdb/docs repository. Checked 2026-09-10.
Cockroach Labs, Technical Advisory 30821: authentication bypass for internal RPCs Published 2018-10-01. Checked 2026-09-10.
Cockroach Labs, Technical Advisory 48860: data corruption with snapshots and delete range Published 2020-05-20. Checked 2026-09-10.
Cockroach Labs, Technical Advisory 71655: zigzag joins could produce incorrect results Published 2021-11-02. Checked 2026-09-10.
Cockroach Labs, Technical Advisory 84144: regional by row tables at risk of corruption Published 2022-07-19. Checked 2026-09-10.
Cockroach Labs, Technical Advisory 123371: changefeeds could drop events during the initial scan Published 2024-06-17. Checked 2026-09-10.
Cockroach Labs, Technical Advisory 131639: disk slowness with lease transfers can lose writes Published 2024-10-08. Checked 2026-09-10.
Cockroach Labs, Technical Advisory 144650: bulk write operations could succeed without ingesting all data Published 2025-04-28. Checked 2026-09-10.
Cockroach Labs, Technical Advisory 161317: silent data loss importing AVRO Published 2026-02-11. Checked 2026-09-10.
Cockroach Labs, Technical Advisory 162085: MVCC garbage collection races range splits Published 2026-02-19. Checked 2026-09-10.
Cockroach Labs, docs/RFCS README Text changed to "deprecated" between the v23.2.0 and v24.1.0 tags. Checked 2026-09-10.
Cockroach Labs, design record: parallel commit Start date 2018-03-24, status completed. Checked 2026-09-10.
Cockroach Labs, design record: range merges Start date 2018-03-30, with a disclaimer added after implementation. Checked 2026-09-10.
Cockroach Labs, design record: follower reads Start date 2018-06-03, status completed. Checked 2026-09-10.
Cockroach Labs, design record: distributed token bucket for tenant cost control Start date 2021-06-04. Checked 2026-09-10.
Cockroach Labs, design record: shared-process deployments for virtual clusters Start date 2022-07-20, status completed. Checked 2026-09-10.
Cockroach Labs, design record: Read Committed isolation Start date 2023-01-22, status completed. Checked 2026-09-10.
Cockroach Labs, design record: segmented storage, status rejected Start date 2015-07-29. Checked 2026-09-10.
Cockroach Labs, tech note: admission control Author Sumeer Bhola, undated in the file. Checked 2026-09-10.
Cockroach Labs, tech note: range merges Last update stated as 10 April 2019. Checked 2026-09-10.
Cockroach Labs, unoptimized query oracle roachtest Copyright header 2022; first present at the v22.2.0 tag. Checked 2026-09-10.
Cockroach Labs, .gitmodules at v19.1.0 Compared against v20.2.0, v21.1.0, v22.1.0, v24.1.0 and v26.1.0. Checked 2026-09-10.
Cockroach Labs, release register 636 entries, 393 production releases, dated. Checked 2026-09-10.
Cockroach Labs, Pebble: implementation differences from RocksDB With the Pebble README's statement of scope. Checked 2026-09-10.
PingCAP, TiDB design documents 113 dated documents, most recent August 2026. Checked 2026-09-10.
Yugabyte, YugabyteDB design documents Index with a named reference engineer per feature. Checked 2026-09-10.