The object store became the quorum: ten years of Elasticsearch

The territory

A search cluster keeps three things on the machines that serve queries: the data, the record of what has been written, and the authority to say who owns a shard. Each one pins a machine to an identity, and identity is what makes a cluster expensive to grow, slow to shrink, and impossible to reshape.

2,090

commits from a four-year private fork that entered the public repository in one April 2026 migration

elastic/elasticsearch, 2026-04-29

2y 9m

master election ran on a multipart-upload consensus protocol before conditional writes replaced it

commits 49d5cd7f2 to 2074dc4ea, 2023-02 to 2025-12

3,000

shards allowed per node when the data lives in the object store, against 1,000 when it lives on the disk

cluster settings reference, 2026-10

31 / 225

muted tests on the main branch that name the newly merged stateless plugin

muted-tests.yml, 2026-10-12

Two organisations have taken a mature, shard-per-Lucene-index search engine and rebuilt its durability on object storage while it was in production, and both wrote the work down in public git history. Elastic did it in a private fork named elasticsearch-serverless from September 2022, and merged that fork, with its history, into the public repository on 2026-04-29. OpenSearch did it in the open from April 2022 under the names segment replication and remote store. The two lineages share a codebase up to early 2021, faced the same forces, and answered them differently in ways that are still visible in both repositories this month.

The surprise in the record is not that the object store won. It is the order of events. Elasticsearch shipped master election through an object-store register in April 2023, built on a compare-and-exchange operation that S3 could not actually perform: the implementation was a consensus protocol assembled out of multipart uploads, upload-id ordering and aborts, whose own code comment cites the FLP impossibility result and concedes that "in theory this whole thing is provably impossible anyway but in practice it'll eventually work with sufficient retries". That protocol carried production leadership for two years and nine months, until conditional writes arrived and the register became a single conditional PUT in Elasticsearch 9.3.0 (2026-01-28). The architecture was committed to before the primitive it needed existed.

Figure 1 · What moved, and when

Each era is a precondition for the next: nothing can be fenced until every write has a name, and nothing can be elected through a blob store until the blob store can refuse a write. Dates from tags and commits in elastic/elasticsearch.

Diagram source

Scope

This guide covers durability, coordination and shard mobility in Elasticsearch from 2015 to October 2026, and the same problem in the OpenSearch lineage. It does not cover relevance, vector search, the ES|QL query engine, Kibana, ingest, or the licence argument beyond the dated licence files. It contains no latency, throughput or cost figures, because this session's network reached only git hosts and package registries: every engineering blog, conference recording and digital library was refused. Section 5 says exactly which numbers are therefore missing, and the ledger in sources.md names the two primary accounts a reader with open egress should read first.

How it is actually built

The common shape across both lineages is a write tier that acknowledges after an upload, a read tier that is a cache in front of the store, and a lease object that decides who may speak. The divergence is in what still lives on a node.

In the stateful engine, the durability boundary is a local file. Elasticsearch's own internals guide states it plainly: "the translog is always persisted and fsync'ed on disk before acknowledging writes back to the user", with the fsync amortised across a batch so that one flush covers many documents. Replication is what makes that local file survive the machine: the primary assigns a sequence number, replicas apply the same number, and the global checkpoint, the highest sequence number processed by every in-sync copy, is the safety line that governs translog truncation, soft-delete retention and peer recovery.

In the stateless engine, that entire mechanism is replaced by one sentence, also from the internals guide: "in serverless Elasticsearch, there is no replication process needed. The same reliability and fault tolerance are instead achieved by uploading the translog to a blob store and relying on the store's durability and fault tolerance guarantees." An index-role node batches operations into a node-level NodeTranslogBuffer, uploads them, and only then acknowledges; a StatelessCommitService batches Lucene commits for upload. A search-role node holds no primary and no replica in the old sense: it reads byte ranges of Lucene files out of the store into a shared blob cache. The disk is a cache layer, in the guide's words, and the roles that used to describe data tiers are disabled.

Figure 2 · Reference architecture, both lineages

Every acknowledgement crosses the store twice: once to put the operations there, once to confirm the writer still owns the shard. Reconstructed from Elastic's distributed-area internals guide and its object-store requirements document.

Diagram source

The register, not the quorum

Leadership is a value in the store. The elected master periodically writes a heartbeat object carrying the current term and a timestamp; a candidate reads it, and if it is older than a threshold that defaults to 30 seconds, starts an election by atomically overwriting the term. The voting configuration contains one node, itself, so committing a cluster state needs only the master's own acknowledgement and the pre-vote phase contacts no peers.

Attested by: internals guide, Serverless Elections

The gate on the ack

Deleting replication deletes the thing that used to stop a displaced primary: a replica with a higher term refusing its requests. The replacement is a read. Before acknowledging a translog upload a node verifies its local cluster state against the authoritative blob, which "prevents a node that has fallen behind on cluster-state updates from durably acknowledging writes it should no longer be handling".

Attested by: commit b715087a5, 2023-08-08

The cache that must be warm

A read tier in front of an object store is only as fast as its cache, so cache population became a service of its own: online prewarming lands in April 2025, and an idle indexing shard can be hollowed out entirely, evicted from compute on a time-to-live and recovered later from the store by a hollow engine. The compute saving and the cold-read penalty are the same mechanism seen from two sides.

Attested by: commit 654e85e39, 2024-12-06, commit fa0925518, 2025-04-11

The contract with the store is longer than the contract with the kernel

The most useful artefact in this excavation is not code. It is docs/internal/s3-object-store-requirements.md, a requirements document that enumerates what Elasticsearch needs from an S3-compatible store and, for each requirement, what breaks when the store does not provide it. Five of its clauses are load-bearing for the architecture rather than merely for snapshots. Cross-node read-after-write consistency is on the hot path, because "after writeBlobAtomic uploads a commit, a different node's shard recovery process immediately lists and reads blobs to discover new commits", and the document is explicit that "there is no polling or retry logic for data blobs". Listing promptness is equally hard: a blob missing from a recovery listing means "a commit is invisible and the shard operates on stale data", while a blob missing from a garbage-collection listing means "stale data accumulates indefinitely". Then there is the register clause, which is where coordination lives, and the server-side copy clause, which is where resharding lives.

A reader running on anything other than AWS S3 should treat that file as a compatibility checklist with teeth. Elasticsearch 9.2.0's release highlight says that earlier versions "had a small risk of inadvertently overwriting an object in an AWS S3 snapshot repository which, if it happened, might corrupt the repository contents", fixed by using conditional writes. The setting that disables them is called unsafely_incompatible_with_s3_conditional_writes, logs a warning that the repository "may lead to corruption", was documented as deprecated on the day it was introduced, and now carries an annotation marking it for removal in the next major version. The window for S3-compatible stores without conditional writes is closing inside one major release.

Figure 3 · Two ways to get a linearizable register out of a blob store

The fallback is six round trips, depends on the store's own clock and upload-id ordering, and provides no protection against two writers overwriting an ordinary blob. Source: S3BlobContainer.java at v9.3.0.

Diagram source

The decisions that matter

Four forks, each with a stated reason in the record, and the condition under which the rejected option is the right one for a reader's system.

Decision: where does durability live, in replicas or in the store?

Chosen

Elastic, from 2023-04: upload the translog to the object store and acknowledge after the upload. The internals guide states that no replication process is needed and the store's durability guarantees replace it.
The write tier and the read tier then scale independently, which the guide names as the point of the role split.

Rejected

Document-level replication to in-sync copies, the mechanism the project spent 2015 to 2019 making correct.
OpenSearch rejected the inverse: it kept replicas and changed what they replicate, shipping segment replication as an index-level setting in April 2022 and remote-backed storage two months later.

Flips when

The acknowledgement budget is smaller than a durable object PUT, or the store does not give cross-node read-after-write. Then the store can hold segments, but the write-ahead record has to stay replicated.
You need query capacity from the copies you are already paying for: replicas give read throughput, a single object store does not.

Decision: what makes a leader, a vote among peers or a lease in the store?

Chosen

A single-node voting configuration plus a heartbeat object. Candidates read the heartbeat, and a term older than the threshold, 30 seconds by default, licenses an election by CAS on the term.
Cluster-state publication needs one acknowledgement, the master's own.

Rejected

The quorum-based coordination layer the same project built through 2018 and shipped in 7.0 (2019-04-10) specifically to end split brain.
OpenSearch kept it, and uses the store for cluster-state publication and recovery rather than for election.

Flips when

The store's CAS is not linearizable, or its availability is lower than your cluster's target. A lease register makes the store a hard dependency of leadership, not just of data.
You cannot tolerate a leadership gap of roughly one heartbeat threshold: a quorum detects a dead leader faster than a stale timestamp does.

Decision: build the CAS primitive yourself, or wait for the store to ship it?

Chosen

Build it. The multipart-upload consensus protocol shipped 2023-02-28 and became the foundation of master election and lease management.
Replace it when possible: conditional-write CAS landed 2025-12-15 and shipped in 9.3.0, with the fallback retained behind a setting named for its unsafety.

Rejected

Waiting for the object store, or putting a consensus service beside it. No external coordination service appears anywhere in the stateless design in this record.
Pretending the fallback is equivalent: it gives no protection against two writers overwriting an ordinary blob, and its own comment concedes it has no liveness proof.

Flips when

Your store already offers compare-and-swap. Then the whole protocol collapses into one conditional PUT and you should not write the other 400 lines.
Your store offers neither CAS nor correct multipart listing. Then the honest answer is that leadership cannot live there at all.

Decision: is a shard count permanent?

Chosen

2018: permanent in practice. The documentation commit of 2018-03-16 explains that because a shard is a Lucene index, moving even 5% of documents "comes with a much higher cost than with a key-value store", so splits are multiplicative, local, and done with hard links.
2026: reshardable. The stateless plugin carries a live split path, and the store's server-side copy moves commit files "between shard containers without transferring data through the node".

Rejected

Consistent hashing with 1/N key movement, rejected in 2018 because the cost model of a search index is not the cost model of a key-value store.
Reindexing into a new index, which the same 2018 text recommends for append-only data behind an alias.

Flips when

The segment files stop being local. Server-side copy inside the store is what turns resharding from a data movement into a metadata operation, which is why the capability appears in the stateless lineage and not in the stateful one.

Figure 4 · Can you move durability off your nodes?

The two upper gates are properties of the store and are not negotiable by engineering effort; the two lower ones are budget decisions. Derived from the requirements and internals documents cited above.

Diagram source

Decision	Chosen	Rejected	Because	Evidence
Unit of replication	Elastic: nothing, the store is the copy. OpenSearch: segments	Per-document replication to in-sync copies	Replication couples write cost to copy count and ties a shard to a machine	OpenSearch ee7b73197, 2022-04-25
Who fences a displaced writer	Elastic: a lease read before the ack. OpenSearch: a CAS fence object per term	A replica that has learned a higher term	Requiring a replica for safety "blocks zero-replica remote-backed indices from recovering automatically on node loss"	OpenSearch 98da9b425, 2026-09-01
Where cluster state is authoritative	Elastic: a register in the store. OpenSearch: on master-eligible nodes, with the store as a publication path	Elastic rejected its own 2018 quorum; OpenSearch rejected store-side election	One linearizable register replaces a vote; a publication path does not	OpenSearch d375e4cfe, 2023-09-04
Compute and storage separation	Elastic: node roles, `index` and `search`. OpenSearch: a warm tier with a file cache	Elastic rejected keeping data roles; OpenSearch rejected splitting them	Role split is what allows independent scaling; a tier keeps the existing cluster shape	OpenSearch ee6052b64, 2026-04-17
Who may run the architecture	Merge the private fork into the public repository, under the Elastic Licence	Keeping it in a private repository indefinitely	Self-managed deployments were in scope: six modules moved to `x-pack/plugin/`, serverless-only files were stripped	commit 6caac0157, 2026-04-23

The fork comparison is the part a reader can act on fastest, because it shows the same force producing two shapes. Both lineages concluded that durability belongs in object storage, within four months of each other, from the same codebase. Both then spent four years discovering that the real problem was not durability but authority: who is allowed to acknowledge a write. Elastic answered in 2023 with a lease read before every acknowledgement. OpenSearch answered in September 2026 with a per-term fence object written only by compare-and-swap, and its commit message says exactly what the previous answer cost: fencing through a replica "makes at least one replica a safety requirement", and without it "the new primary would begin acknowledging seconds after node-left while the old one is very likely still alive and unaware". Two independent teams, two mechanisms, one conclusion: once the data is in a shared store, the fence has to be in the shared store too.

Figure 5 · One codebase, two answers

The lineages agree on durability and disagree on who holds a shard, which is why only one of them can reshard a live index and only one of them can serve reads from replicas. Dates from both repositories' histories.

Diagram source

What broke in production

Elastic publishes a known-issues document inside the repository, which is the closest thing to a postmortem archive this excavation could reach. Three failure classes account for everything in it that matters to this architecture.

Class 1: the write path can bypass the invariant the engine spent a decade building

Postmortem

Backpressure made the primary stop replicating, silently

AssumptionAnything acknowledged on a primary has been handed to the replication machinery, so sequence numbers and the global checkpoint are enough to keep copies identical.

What happenedA bulk request being processed exactly as the node crossed its indexing-pressure limit could "process some of the bulk items on the primary shard without replicating them, causing the contents of any replicas to diverge from those of the primary".

Blast radiusUnacknowledged items intermittently returned by searches; translog trimming blocked, so disk grows; recoveries that "attempt to replay all operations starting from the ones that were not replicated". Introduced in 9.5.0 (2026-07-29), fixed in 9.5.3.

FixA code fix in the pressure path. Note that the documented recommendation is an urgent upgrade, with guidance on determining whether a cluster was affected.

Design ruleOverload paths are where invariants go to die. Any shed, defer or fast-fail branch in a write path needs the same correctness review as the happy path, because it is the branch that skips steps by design.

SourceElastic known issues, 9.5.0 to 9.5.2

Postmortem

The storage saving returned documents that did not match

AssumptionTurning indexing off for fields, which is what the cheap log and columnar index modes do by default, is a storage decision with no semantic consequences.

What happenedBoolean queries combining a positive clause with must_not on non-indexed fields "can return false-positive matches" through a Lucene bulk-scorer iterator defect. Time-series and columnar data streams were affected because they disable indexing on all fields.

Blast radiusWrong answers, not errors, on the index modes adopted for cost reasons. A partial fix in 9.5.1 covered term and range paths but missed multi-value terms; the full fix needed a Lucene upgrade to 10.5.1 in 9.5.2.

FixUpstream dependency upgrade, in two attempts.

Design ruleEvery storage saving is a removal of redundancy, and redundancy is what your correctness tests were accidentally relying on. Price a cost-reduction feature with a correctness test matrix, not only with a storage benchmark.

SourceElastic known issues, 9.5.0 and 9.5.1

Class 2: the displaced writer

This is the failure mode the architecture creates. When durability is a shared object store, two nodes can both believe they own a shard and both write durably, because the store does not know about shards. Both lineages hit it, and the dates show how long it took each to find the bottom of it: Elastic moved the durability boundary to the store on 2023-04-04 and shipped the fix for stale acknowledgements on 2023-08-08, four months later; OpenSearch moved segments to the store in June 2022 and replaced replica-based fencing with an object-store fence in September 2026, four years later.

Figure 6 · The write that must not be acknowledged

The upload succeeds, which is the problem: the store accepts bytes from a node that no longer owns the shard, so the gate has to be a separate read of the authority object before the client is told anything. Reconstructed from the consistency-service description in the internals guide and commit b715087a5.

Diagram source

Source code

A stale indexing shard acknowledged a write

AssumptionIf the upload to the store succeeds, the write is safe to acknowledge.

What happenedA node whose cluster state was behind could still complete an upload and acknowledge, because nothing in the upload path knows that the shard has moved.

Blast radiusFound inside the private fork before the managed service was generally available, so the public record shows the fix and not an outage.

FixA consistency service that verifies local cluster state against the authoritative blob before an acknowledgement, blocking writes while the node is behind.

Design ruleDurability is not authority. A successful write to shared storage proves bytes landed, never that the writer was entitled to land them.

Sourcecommit b715087a5, 2023-08-08

Decision record

Fencing through a replica blocked zero-replica recovery

AssumptionA replica that has learned a newer primary term is a sufficient fence for a partitioned primary.

What happenedThat made a replica a safety requirement. With zero replicas, "the new primary would begin acknowledging seconds after node-left while the old one is very likely still alive and unaware", so automatic recovery on node loss was blocked rather than unsafe.

Blast radiusFour years of remote-backed indices paying for a replica they did not need for durability. Reported by the maintainers in the commit that changed it, not by an incident.

FixOne small mutable object per primary term, written only by compare-and-swap, where "the fencing token is the CAS chain, not the term: every acknowledged translog upload advances it". A takeover creates its own object with create-if-absent and unconditionally deletes every lower-term object.

Design ruleIf a safety property depends on a component being present, it is a capacity decision masquerading as a safety decision. Move the fence to the thing that is always there.

SourceOpenSearch commit 98da9b425, 2026-09-01

Class 3: the store under you is not the store you tested against

Postmortem

Two writers, one blob, a corrupt repository

AssumptionConcurrent writers to a snapshot repository are prevented well enough by higher-level coordination.

What happenedElastic's own 9.2.0 highlight says earlier versions "had a small risk of inadvertently overwriting an object in an AWS S3 snapshot repository which, if it happened, might corrupt the repository contents".

Blast radiusUnquantified in the public record, which is itself the finding: a corrupt repository is discovered at restore time, which is the worst possible detection point.

FixConditional writes for ordinary blob writes in 9.2.0 and for the register in 9.3.0, with the non-conforming path behind a setting that warns, is deprecated, and is annotated for removal.

Design ruleWhen your durability layer gains a primitive, the migration is not optional for anyone who depended on the absence of that primitive. Audit your storage vendor against the engine's requirements file, not against the S3 marketing surface.

SourceElasticsearch 9.2.0 release highlight

Postmortem

A node could not parse its own persisted cluster state

AssumptionPersisted cluster state survives a patch upgrade, because it is the project's own format.

What happenedUpgrading 9.1.10 to 9.2.4 could produce a "fatal exception while booting Elasticsearch" with a CorruptStateException from the metadata parser, on the node_shutdown custom's nodes field.

Blast radiusNode fails to start, so the failure is a rolling upgrade that stops rolling. Documented against 9.2.4 in the known-issues file.

FixA parser fix in a later patch. The relevant point for an architect is that the file naming the failure ships inside the product's own repository.

Design ruleAny state you persist in your own format is a schema with a compatibility obligation, including the parts that are internal bookkeeping. Give it the same upgrade tests as user data, and keep a documented path for reading the previous shape.

SourceElastic known issues, 9.2.4

What is missing from this catalogue

No incident in the reachable record describes the stateless architecture failing in production: no lost lease, no split master, no cold-cache brownout, no object-store outage propagating into a cluster. That is not evidence the architecture is safe. It is evidence that the operator of the only large deployment of it does not publish incident reviews where this session could read them, and that the self-managed form of it is six months old. Treat the absence as the location of your own risk, and design the first rollout to produce the evidence that does not exist.

Numbers you can plan against

Everything quantitative that the reachable record contains. It is thinner than this kind of guide normally carries, and the gaps are named below rather than filled.

Metric	Value	At	Context	As of	Source
Shard limit per node, data on local disk	1,000	Elastic	Default `cluster.max_shards_per_node`, non-frozen	2026-10	cluster settings
Shard limit per node, data in the store	3,000	Elastic	Frozen tier, disk is a partial cache	2026-10	cluster settings
Leader-liveness threshold before an election	30 s	Elastic	Heartbeat staleness, stateless deployments, default	2026-10	internals guide
Voting configuration size, stateless	1	Elastic	Master acknowledges its own publications	2026-10	internals guide
Time the fallback CAS protocol ran in production	2 y 9.5 m	Elastic	2023-02-28 to 2025-12-15, derived from commit dates	2026-10	49d5cd7f2, 2074dc4ea
Single-upload threshold for a blob	100 MB	Elastic	Repository `buffer_size` default; above it, multipart	2026-10	requirements doc
Store limits the design inherits	5 MB / 10,000 / 5 TB	S3	Minimum part, maximum parts, maximum object	2026-10	requirements doc
Objects per listing or bulk delete	1,000	S3	Pagination is mandatory on recovery and GC paths	2026-10	requirements doc
Commits mentioning the stateless work, per year	14 / 174 / 173 / 104 / 505	Elastic	2022 to 2026 to date, derived from the merged history	2026-10-12	repository history
Commits naming the private serverless fork	2,090	Elastic	Visible in the public repository after the April 2026 migration	2026-10-12	b40e7eb23
Muted tests naming the stateless plugin	31 of 225	Elastic	Lines in `muted-tests.yml` on main	2026-10-12	muted-tests.yml
Repository sizes read for this guide	106,974 / 61,541	Elastic / OpenSearch	Commits to 2026-10-12 in each lineage	2026-10-12	OpenSearch

Read these carefully

Measured: none. No figure in this guide is an independent measurement, because no benchmark, talk or engineering account was reachable. Reported: the defaults and store limits, which come from the project's own documentation and are what the code does, not what a deployment experiences. Derived: the five counts drawn from git history, computed in this session with the commands in section 8; the commit-message counts are keyword matches and overstate slightly. Unknown, and worth knowing: the added acknowledgement latency from uploading rather than fsyncing; the cold-read penalty on a search node with an empty shared cache; the request-count cost of listing-driven recovery at a realistic shard count; the availability coupling between the object store and cluster leadership; how long a hollowed shard takes to serve its first query; and what the 30-second heartbeat threshold costs in write unavailability during a master failover. Every one of those is a number a reader should generate for their own deployment before committing, and none of them is in the public record this session could reach.

The evidence wall

Twenty-six artefacts, all from two git histories and the documents committed inside them. There is no blog, talk or paper tier here: those hosts were refused by this session's egress policy, and the ledger names what a reader with open egress should read first.

Decision record Elastic2026-10

Elasticsearch Requirements for an S3-Compatible Object Store

A clause-by-clause contract with the storage layer, each clause carrying what breaks without it. Names the register as the foundation of coordination, the two CAS paths, the read-after-write and listing requirements on the hot path, and the server-side copy that resharding uses.

Carry forwardWrite this document for your own storage dependency. The clauses you cannot test are the ones that will fail.

raw.githubusercontent.com/elastic/elasticsearch/b2f4a538f/docs/internal/s3-object-store-requirements.md

Decision record Elastic2026-10

Distributed Area Internals, 3,584 lines

The in-repository architecture guide. Sections used here: stateless roles, serverless elections, the write path, primary terms and sequence numbers, checkpoints and gaps, and the translog's acknowledgement rules. It is also where the project distinguishes stateless, the architecture, from serverless, the offering.

Carry forwardInternal design docs committed beside the code age with the code; a wiki does not.

raw.githubusercontent.com/elastic/elasticsearch/b2f4a538f/docs/internal/DistributedArchitectureGuide.md

Postmortem Elastic2026-10

Known issues, 9.5.0 to 9.5.2: replicas diverge under indexing pressure

The clearest correctness failure in the record: crossing the indexing-pressure limit mid bulk request could leave items on the primary unreplicated, with divergent replicas, blocked translog trimming and replay-everything recoveries. Introduced by a named pull request, fixed in a named pull request two patch releases later.

Carry forwardReview every load-shedding branch as a correctness change, because it is one.

raw.githubusercontent.com/elastic/elasticsearch/b2f4a538f/docs/release-notes/known-issues.md

Postmortem Elastic2026-10

Known issues, 9.5.0 and 9.5.1: false positives on non-indexed fields

Time-series and columnar index modes disable indexing on all fields to save storage, and a Lucene bulk-scorer defect turned that into wrong query results for a specific boolean shape. The first fix covered two query paths and missed a third.

Carry forwardA cost feature that removes a data structure needs a correctness matrix, not a storage chart.

raw.githubusercontent.com/elastic/elasticsearch/b2f4a538f/docs/release-notes/known-issues.md

Postmortem Elastic2026-10

Known issues, 9.2.4: CorruptStateException on boot after upgrade

A patch upgrade from 9.1.10 to 9.2.4 could leave a node unable to parse its own persisted cluster metadata, failing at startup inside GatewayMetaState.start. The entry also carries two GCS-credential regressions from the same series, each with a workaround encoded as a base64 policy override.

Carry forwardInternal persisted state is a versioned schema; test its upgrade path like a public API.

raw.githubusercontent.com/elastic/elasticsearch/b2f4a538f/docs/release-notes/known-issues.md

Source code Elasticv9.3.0

S3BlobContainer.java, the compare-and-exchange operation

Both CAS paths in one file: conditional headers when the store supports them, and the multipart protocol when it does not, complete with upload-id ordering, position-based back-off, a clock-skew warning against the store's own initiation timestamps, and a comment citing FLP to explain why liveness is best effort.

Carry forwardIf your coordination rests on a protocol with no liveness proof, say so in the code, and keep the retry budget generous.

raw.githubusercontent.com/elastic/elasticsearch/v9.3.0/.../S3BlobContainer.java

Source code Elasticv9.3.0

S3Repository.java, the unsafety switch

The setting unsafely_incompatible_with_s3_conditional_writes defaults to false, and turning it on logs that the repository "may lead to repository corruption" with a pointer to the compatibility documentation.

Carry forwardName an escape hatch after its risk. Operators read setting names, not release notes.

raw.githubusercontent.com/elastic/elasticsearch/v9.3.0/.../S3Repository.java

Vendor doc Elastic2025-10

Release highlight, 9.2.0: conditional writes prevent repository corruption

States that earlier versions carried a small risk of overwriting an object in an S3 repository and corrupting its contents, and that 9.2.0 uses conditional writes to prevent it. The companion 9.3.0 bundle carries the register change as pull request 139228.

Carry forwardRelease highlights are where a project admits what the old design could lose.

raw.githubusercontent.com/elastic/elasticsearch/b2f4a538f/docs/release-notes/changelog-bundles/9.2.0.yml

Decision record Elastic2026-04-29

Commit b40e7eb23, "Migrate stateless modules with full file history"

The migration of the private elasticsearch-serverless fork into the public repository: six modules into x-pack/plugin/, a test framework and a QA module, history preserved through a two-step git-filter-repo rename, licence headers rewritten to the Elastic Licence, and the fork point named.

Carry forwardA fork merged back with history is a four-year design record handed to the public in one commit. Read the history, not the diff.

github.com/elastic/elasticsearch/commit/b40e7eb23

Source code Elastic2026-04-23

Commit 6caac0157, removing out-of-scope modules from the migration

Names the files that stayed serverless-only, including ServerlessStatelessPlugin.java and the split-state transport actions, which shows that the public plugin is a subset of what the managed service runs.

Carry forwardWhen a vendor open-sources an architecture, diff the module list before assuming parity.

github.com/elastic/elasticsearch/commit/6caac0157

Source code Elastic2026-10

StatelessPlugin.java licence header

The merged architecture carries an Elastic Licence 2.0 header, inside a repository whose top-level licence became a triple licence including AGPL-3.0 in 8.16.0. Public source, restricted use.

Carry forward"In the public repository" and "open source" are now independent properties. Check the file header.

raw.githubusercontent.com/elastic/elasticsearch/b2f4a538f/.../StatelessPlugin.java

Decision record Elastic2018-03-16

Commit 708c06896, "Explain why Elasticsearch doesn't support incremental resharding"

A documentation commit written as a design rationale: consistent hashing is the usual answer, but a shard is a Lucene index, so moving 5% of documents costs far more than in a key-value store. Hence multiplicative, local splits using hard links.

Carry forwardWrite down why you refused a feature. Eight years later it is the clearest statement of which constraint changed.

github.com/elastic/elasticsearch/commit/708c06896

Source code Elastic2015 to 2016

Primary terms, sequence numbers and the global checkpoint

Commits b364cf54b (2015-10-09), 5fb0f9a88 (2015-11-03) and 484432592 (2016-06-06) give every operation a name and a term, and define the safety line across in-sync copies. Ten years later the same term is what a fence compares against.

Carry forwardYou cannot fence what you cannot name. Identity for writes is the prerequisite for every later move.

github.com/elastic/elasticsearch/commit/484432592

Source code Elastic2018 to 2019

Soft deletes and shard history retention leases

Commit ac84879a7 (2018-04-19) keeps document history in Lucene rather than the translog; commit c0f8c8917 (2019-01-07) turns the right to retain that history into an explicit, expiring lease. Operation-based recovery and cross-cluster replication both depend on it.

Carry forwardRetention is a resource. Make somebody hold a lease on it, with an expiry, or it grows without an owner.

github.com/elastic/elasticsearch/commit/c0f8c8917

Source code Elastic2018-08

The Zen2 coordination layer

Commits 2176184db and 785b6e824 build the discovery and publication pipeline of the quorum-based coordination layer that shipped in 7.0 (tag dated 2019-04-10). It is the mechanism the stateless deployment later sets aside in favour of a lease register.

Carry forwardA consensus layer built for peer nodes is not wasted when state moves: it stays the answer for every deployment without a linearizable store.

github.com/elastic/elasticsearch/commit/2176184db

Source code Elastic2021-02-05

Commit 50f4a0b77, partial searchable snapshots for a frozen tier

The first production tier whose local disk is only a partial cache of data that lives in the object store, with allocation restricted to nodes that have a cache (commit 1a432a26d, 2021-03-04). The pattern the stateless engine later generalises.

Carry forwardShip the cache-backed tier at the cold end first. It teaches the eviction and allocation problems at low stakes.

github.com/elastic/elasticsearch/commit/50f4a0b77

Source code Elastic2022-09-28

Commit e3f86aa0d, "Add stateless skeleton project"

The first commit of the stateless work, in the private fork, now visible in the public history. The index and search roles arrive three weeks later, behind a feature flag.

Carry forwardFour years from skeleton to self-managed availability is the honest schedule for rebuilding a storage engine's durability.

github.com/elastic/elasticsearch/commit/e3f86aa0d

Source code Elastic2023-04-04

Commit 3aeeb4584, "Sync translog to object before acking write"

The commit that moves the durability boundary from a local fsync to an object-store upload. Everything in the failure catalogue's second class follows from this one line of design.

Carry forwardLocate the exact commit where your acknowledgement crosses a new boundary. That is the date your failure modes changed.

github.com/elastic/elasticsearch/commit/3aeeb4584

Source code Elastic2023-04-17

Commit da214475a, heartbeat leadership into production

The atomic-register pre-vote collector and the blob-store heartbeat service move from test scaffolding into production code, three weeks after the project taught a leader to stand down when its heartbeat write fails (commit 3416ce0df).

Carry forwardA leader that cannot write its lease must resign. Make that explicit before you make the lease authoritative.

github.com/elastic/elasticsearch/commit/da214475a

Source code Elastic2024-12 to 2025-05

Hollow shards and online prewarming

Commits 654e85e39 and 9325e5ca9 make an idle indexing shard evictable from compute on a time-to-live and recoverable through a hollow engine; commits fa0925518 and 93c893a76 make cache population a service with its own interface.

Carry forwardThe compute saving and the cold-start penalty are the same feature. Budget for the warming path in the same design review.

github.com/elastic/elasticsearch/commit/654e85e39

Source code Elastic2026-08-18

Commit 05ef24f05, marking the S3 escape hatch for removal

The non-conditional-write setting "was documented as deprecated as soon as it was introduced" and now carries an annotation scheduling it for removal in the next major version. The compatibility window for stores without conditional writes is explicit.

Carry forwardDate your compatibility shims at birth. An undated shim becomes a permanent supported configuration.

github.com/elastic/elasticsearch/commit/05ef24f05

Source code Elastic2021 to 2024

LICENSE.txt at v7.10.2, v7.11.0 and v8.16.0

Apache-2.0 based through 7.10.2, a dual Server Side Public Licence and Elastic Licence from 7.11.0 (2021-02-10), and a triple licence adding AGPL-3.0 from 8.16.0 (2024-11-08). The licence boundary is the fork boundary, and the fork is the comparison this guide depends on.

Carry forwardRead a licence change as an architectural event: it determines who else will solve your problem in public.

raw.githubusercontent.com/elastic/elasticsearch/v7.11.0/LICENSE.txt

Source code OpenSearch2022-04-25

Commit ee7b73197, replication type as an index setting

The fork's first move is to make replication type configurable per index, introducing segment replication beside document replication rather than removing replication. Feature flags arrive in the same commit.

Carry forwardPer-index switching lets a storage change ship incrementally; a node-role split does not.

github.com/opensearch-project/OpenSearch/commit/ee7b73197

Source code OpenSearch2022-06 to 2022-09

Remote store: segments, then translog

Commit fb1375987 uploads segments to a remote store after refresh; commit 57a4aedb0 changes replica recovery for remote-translog indices. Same conclusion as the other lineage, two months apart, with the shard still owned by a data node.

Carry forwardMoving segments is the easy half. The write-ahead log and the fence are where the design is decided.

github.com/opensearch-project/OpenSearch/commit/fb1375987

Decision record OpenSearch2026-09-01

Commit 98da9b425, object-store fencing token

A commit message written as a design document: why replica-based fencing made a replica a safety requirement, why the fencing token is the CAS chain rather than the term, why a takeover claims the fence before reading its restore point, and why deleting lower-term objects is what makes a takeover certain.

Carry forwardDeterminism from the key space beats determinism from winning a race. Design fences so the loser discovers it lost.

github.com/opensearch-project/OpenSearch/commit/98da9b425

Source code OpenSearch2026-04 to 2026-05

WritableWarm tiered storage

Commits ee6052b64 and bf7bfedc0 build a Lucene directory and index-input layer over object storage with a file cache, plus prefetch settings and per-query metrics. Compute and storage separation as a tier, not as a cluster shape.

Carry forwardIf you cannot change your cluster's node roles, a cache-backed directory implementation buys most of the storage economics.

github.com/opensearch-project/OpenSearch/commit/ee6052b64

Build a miniature, then productionise it

Six rungs against any S3-compatible store. The line between toy and real is rung four, where a second writer appears.

A register with one conditional PUT

Implement compare-and-exchange on a single object using If-Match on the ETag and If-None-Match: * for creation. Two processes, one key, a term counter.

Done when: two concurrent writers race 1,000 times and the number of successful increments equals the final term value. Teaches: the one primitive the whole architecture rests on.

The same register without conditional writes

Reimplement it with the multipart protocol: initiate, upload the proposed value as a part, list in-flight uploads, order by upload id, abort the losers, complete. Add the position-based back-off.

Done when: the race test passes and you can state the round-trip count of each path. Teaches: what a missing primitive costs, and why the setting that selects this path is named for its unsafety.

Lease-based leadership

One heartbeat object carrying term and timestamp. A candidate takes over when the heartbeat is older than a threshold; a leader that fails to write its heartbeat stands down.

Done when: killing the leader produces exactly one new leader within the threshold, and a paused leader resigns rather than continuing. Teaches: that the threshold is your failover latency and your split-brain window at the same time.

Acknowledge after the upload, and fence it

A tiny append-only log: batch writes, upload the batch, then read the lease and only then acknowledge. Now start a second writer that believes it owns the log.

Done when: the displaced writer's upload succeeds and its acknowledgement does not, and your test asserts on what the client was told. Teaches: durability is not authority. This is the rung that crosses into real.

A reader that discovers work by listing

A second process that lists the log prefix, reads new objects by range, and serves queries from a local cache with eviction. Measure first-query latency with a cold cache and a warm one.

Done when: you can quote cold and warm latency, and the request count per recovery at 10 and 1,000 objects. Teaches: listing and range reads are the real hot path, and pagination limits are architecture.

Break the store, then price it

Inject the failures the requirements document names: stale reads after write, a blob missing from a listing, a 500 on complete, clock skew on upload timestamps. Then count requests and bytes for a day of synthetic traffic.

Done when: each injected fault produces a named, logged outcome rather than a hang, and you have a per-request cost figure for your own store. Teaches: which clauses of your storage contract you are actually relying on, and what they cost.

Keep hunting

This guide was assembled with git and a single HTTP client, because nothing else was reachable. These are the commands that produced it, and they work on any project with a public history.

Dating a design change from history alone

git clone --filter=tree:0 --no-checkout <repo>
git log --format='%ad %h %s' --date=short --grep='stateless' -i | tail -20
git log --grep='conditional write' -i --format='%ad %h %s' --date=short
git for-each-ref --sort=creatordate --format='%(creatordate:short) %(refname:short)' refs/tags

Finding the design documents nobody links to

git sparse-checkout set docs && git checkout main
ls docs/internal docs/release-notes
grep -rn -iE 'data loss|corrupt|diverge' docs/release-notes/known-issues.md
curl -s https://raw.githubusercontent.com/<org>/<repo>/<sha>/<path>

Reading a merged fork

git log --grep='filter-repo' --format='%ad %h %s' --date=short
git log --grep='<private-repo-name>#' | wc -l
git log -1 --format='%ad%n%B' --date=iso <migration-sha>

Measuring the cost of a merge

grep -c 'class:' muted-tests.yml
grep -ci stateless muted-tests.yml
git log --format='%ad' --date=short --grep='<feature>' -i --since=2026-01-01 | wc -l

The lesson

Two preconditions, in that order, and three costs. Everything else in this decade follows from them.

The first precondition is identity. You cannot move durability off a machine until every write carries a name and a term, because the only thing that stops a displaced writer is a comparison. Elasticsearch acquired that in late 2015, with primary terms and sequence numbers added for a different reason entirely, which is why the rule it still states, that where two copies share a sequence number "the copy with the higher primary term wins", is the rule a lease check and a fence object both reduce to. A system whose writes are anonymous cannot adopt this architecture at any price.

The second precondition is a primitive. You cannot move authority off your nodes until the layer below can refuse a write, and if you try before then you will build consensus out of whatever the store does offer. That is not a hypothetical: Elasticsearch shipped master election through an object-store register in April 2023 on a protocol made of multipart uploads, upload-id ordering, aborts and the store's own clock, ran it for two years and nine months, and replaced it with a single conditional PUT once the store could do the job. The fallback still exists, is named for its unsafety, and is annotated for deletion. Check your store against the engine's own requirements file before you design around it.

Then price the three costs, because the record shows all three being paid. The acknowledgement now crosses the network instead of a disk controller, which is why the write path batches harder and why a second read of the authority object sits between the upload and the client. The read tier is only as fast as its cache, which is why prewarming became a service with its own interface and metrics. And the storage contract grew longer than the kernel contract: five clauses in that requirements document, including cross-node read-after-write with no retry path, are load-bearing, and the ones marked as untestable are where your risk actually lives.

Carry this into your own design

Delegating durability to a shared store does not remove a consensus problem; it moves it. The question that decides the design stops being "can the store keep my bytes" and becomes "can the store refuse a write". Answer that question first, in writing, and the rest of the architecture, tier split, fencing, resharding, hollowing, follows from the answer rather than from ambition.

References

Elastic, Elasticsearch Requirements for an S3-Compatible Object Store elastic/elasticsearch, main at b2f4a538f. Checked 2026-10-12.
Elastic, Distributed Area Internals elastic/elasticsearch, main at b2f4a538f. Checked 2026-10-12.
Elastic, Elasticsearch known issues elastic/elasticsearch, main at b2f4a538f. Checked 2026-10-12.
Elastic, 9.2.0 changelog bundle and release highlights elastic/elasticsearch, main at b2f4a538f. Checked 2026-10-12.
Elastic, 9.3.0 changelog bundle elastic/elasticsearch, main at b2f4a538f. Checked 2026-10-12.
Elastic, S3BlobContainer.java elastic/elasticsearch, tag v9.3.0. Checked 2026-10-12.
Elastic, S3Repository.java elastic/elasticsearch, tag v9.3.0. Checked 2026-10-12.
Elastic, StatelessPlugin.java elastic/elasticsearch, main at b2f4a538f. Checked 2026-10-12.
Elastic, muted-tests.yml elastic/elasticsearch, main at b2f4a538f. Checked 2026-10-12.
Elastic, Miscellaneous cluster settings elastic/elasticsearch, main at b2f4a538f. Checked 2026-10-12.
Elastic, LICENSE.txt at v7.10.2 elastic/elasticsearch. Checked 2026-10-12.
Elastic, LICENSE.txt at v7.11.0 elastic/elasticsearch, released 2021-02-10. Checked 2026-10-12.
Elastic, LICENSE.txt at v8.16.0 elastic/elasticsearch, released 2024-11-08. Checked 2026-10-12.
Elastic, commit b40e7eb23, Migrate stateless modules with full file history elastic/elasticsearch, 2026-04-29. Verified in a local clone 2026-10-12.
Elastic, commit 6caac0157, Remove extra modules out of migration scope elastic/elasticsearch, 2026-04-23. Verified in a local clone 2026-10-12.
Elastic, commit 708c06896, Explain why Elasticsearch doesn't support incremental resharding elastic/elasticsearch, 2018-03-16. Verified in a local clone 2026-10-12.
Elastic, commit 3aeeb4584, Sync translog to object before acking write elastic/elasticsearch, 2023-04-04. Verified in a local clone 2026-10-12.
Elastic, commit b715087a5, Ensure stale indexing shard does not Ack write elastic/elasticsearch, 2023-08-08. Verified in a local clone 2026-10-12.
Elastic, commit da214475a, Extract AtomicRegisterPreVoteCollector and StoreHeartbeatService to production elastic/elasticsearch, 2023-04-17. Verified in a local clone 2026-10-12.
Elastic, commit 49d5cd7f2, S3 compare-and-exchange implementation elastic/elasticsearch, 2023-02-28. Verified in a local clone 2026-10-12.
Elastic, commit 2074dc4ea, S3 compareAndExchange using conditional writes elastic/elasticsearch, 2025-12-15. Verified in a local clone 2026-10-12.
Elastic, commit 05ef24f05, Mark S3 repo settings for removal elastic/elasticsearch, 2026-08-18. Verified in a local clone 2026-10-12.
Elastic, commit 50f4a0b77, Add partial searchable snapshot support for a frozen tier elastic/elasticsearch, 2021-02-05. Verified in a local clone 2026-10-12.
Elastic, commit 654e85e39, Feature flag for super thin hollow indexing shards elastic/elasticsearch, 2024-12-06. Verified in a local clone 2026-10-12.
Elastic, commit fa0925518, Online prewarming service interface docs and usage in SearchService elastic/elasticsearch, 2025-04-11. Verified in a local clone 2026-10-12.
Elastic, commit 484432592, Introduced Global checkpoints for Sequence Numbers elastic/elasticsearch, 2016-06-06. Verified in a local clone 2026-10-12.
Elastic, commit c0f8c8917, Introduce shard history retention leases elastic/elasticsearch, 2019-01-07. Verified in a local clone 2026-10-12.
Elastic, commit 2176184db, Zen2: introduce gossip-like discovery of master nodes elastic/elasticsearch, 2018-08-06. Verified in a local clone 2026-10-12.
Elastic, commit e3f86aa0d, Add stateless skeleton project elastic/elasticsearch, 2022-09-28. Verified in a local clone 2026-10-12.
OpenSearch, commit 98da9b425, Add object-store fencing token for remote store primary term validation opensearch-project/OpenSearch, 2026-09-01. Verified in a local clone 2026-10-12.
OpenSearch, commit ee7b73197, Adds the replication type index setting opensearch-project/OpenSearch, 2022-04-25. Verified in a local clone 2026-10-12.
OpenSearch, commit fb1375987, Upload segments to remote store post refresh opensearch-project/OpenSearch, 2022-06-10. Verified in a local clone 2026-10-12.
OpenSearch, commit 57a4aedb0, Change behaviour in replica recovery for remote translog enabled indices opensearch-project/OpenSearch, 2022-09-17. Verified in a local clone 2026-10-12.
OpenSearch, commit d375e4cfe, Integrate remote cluster state in publish/commit flow opensearch-project/OpenSearch, 2023-09-04. Verified in a local clone 2026-10-12.
OpenSearch, commit ee6052b64, Implement Directory and IndexInput layers for WritableWarm tiered storage opensearch-project/OpenSearch, 2026-04-17. Verified in a local clone 2026-10-12.
elastic/elasticsearch repository 106,974 commits to 2026-10-12, cloned and queried in this session.
opensearch-project/OpenSearch repository 61,541 commits to 2026-10-12, cloned and queried in this session.