Distributed Systems 12 Oct 2026 36 min read

The object store became the quorum: ten years of Elasticsearch, read from its own repository

How Elasticsearch moved durability, and then leadership, off its nodes into an object store between 2015 and 2026, and how the OpenSearch fork answered the same force differently.

A decade of one search engine's coordination design, reconstructed from two complete git histories, the design documents committed inside them, and a known-issues file that doubles as an incident archive. It shows the two preconditions the move requires (named writes, a linearizable store primitive), the three costs it imposes (network acknowledgement, cache warming, a storage contract longer than the kernel contract), and the failure class it creates (the displaced writer). A reader can take the decision tree, the object-store requirements checklist and the six-rung ladder and decide whether their own system qualifies.

The finding that surprised me

Elasticsearch shipped master election through an object-store register in April 2023 on a compare-and-exchange protocol built out of multipart uploads and upload-id ordering, whose own code comment cites the FLP impossibility result and concedes it works only 'with sufficient retries'; that protocol carried production leadership for two years and nine months until S3 conditional writes replaced it in 9.3.0.

What you get out of it

  • Identity comes first: the primary-term and sequence-number work of 2015-2016, built for replication correctness, is what later makes fencing a displaced writer possible at all, because the rule reduces to comparing a term.
  • In the stateless engine there is no replication process; durability is an object-store upload and the ack waits for it, which converts a disk-controller cost into a network cost and creates the displaced-writer failure class.
  • Deleting replication deletes the fence, so both lineages had to rebuild one in the store: Elastic with a lease read before every acknowledgement in 2023, OpenSearch with a per-term compare-and-swap fence object in 2026.
  • Resharding was documented as economically impossible in 2018 because a shard is a local Lucene index; it becomes feasible once the segments live in the store and the split is a server-side copy.
  • The architecture's contract with the object store is longer than its contract with the kernel: cross-node read-after-write with no retry path, prompt listings, atomic overwrite, server-side copy and a linearizable register, and the setting that tolerates a store without conditional writes is named for its unsafety and annotated for removal.

Scope

Why this, now. In April 2026 Elastic merged its four-year private serverless fork, history intact, into the public Elasticsearch repository, so the stateless architecture and its whole development record became readable at once; and in September 2026 OpenSearch independently moved its fencing to a compare-and-swap object in the store, giving the decade two endings to compare.

What it does not cover. Relevance, vector search, ES|QL, Kibana and ingest; the licensing argument beyond dated licence files; and all latency, throughput, cost and production-scale figures, because this session's egress policy refused every engineering-blog, conference and digital-library host, leaving the blog, talk and paper tiers empty and the SoCC'25 stateless-architecture paper unread.

Open the field guide → Self-contained: it loads nothing at read time, follows your system theme, and prints cleanly.