Digs

Searching X architecture returns material written to be found by that query, which means tutorials and vendor pages. The material worth having was written by an engineer explaining a real system to their peers, and it uses different words. A dig is what comes back from going after that layer on one topic: the architecture reconstructed across real systems, the decisions and the conditions that flip them, what actually broke, and numbers you can plan against. Every claim is graded, linked and dated.

90 field guides 2433 graded sources 419 postmortems 8 categories

Consensus, replication, consistency, coordination, sharding and scheduling.

Distributed Systems 12 Oct 2026

The object store became the quorum: ten years of Elasticsearch, read from its own repository

A decade of one search engine's coordination design, reconstructed from two complete git histories, the design documents committed inside them, and a known-issues file that doubles as an incident archive. It s…

What surprised meElasticsearch shipped master election through an object-store register in April 2023 on a compare-and-exchange protocol built out of multipart upload…

26 sources · 2 organisations · 3 postmortems 36 min read
object-storageconsensuselasticsearchopensearchfencing
Distributed Systems 09 Oct 2026

Choosing the backend

Reconstructs the backend-selection stack from Google's SRE book and Prequal paper, Netflix's edge redesign, Twitter's aperture work, Uber's dynamic subsetting, gRPC's design records (including one subsetting p…

What surprised meThe mature implementations agree the balancer's job is not to find the best backend but to bound how much it trusts every signal that claims to know …

30 sources · 18 organisations · 5 postmortems 30 min read
load-balancingp2csubsettingfeedback-loopsenvoy
Distributed Systems 06 Oct 2026

Serving without the control plane

Kubernetes, Envoy, Eureka and Patroni all answer the same question in their repositories: what may the data plane do with its last instructions when the control plane goes silent? This guide reconstructs the f…

What surprised meEnvoy, Kubernetes and Eureka each ship a numeric threshold (50%, 55%, 15%) above which the system stops believing its own health signal, three indepe…

21 sources · 10 organisations · 4 postmortems 27 min read
control-planedata-planestatic-stabilityleasesservice-discovery
Distributed Systems 24 Sep 2026

Your write hasn't happened here yet

Reconstructs the four-part machine behind read-your-writes (mint a position token at commit, carry it with the client, route to a caught-up replica, keep a bounded escape hatch) from the implementations at Git…

What surprised meEvery layer of the stack independently converged on the same four-part token machine, while the database underneath took three landing attempts acros…

24 sources · 16 organisations · 4 postmortems 28 min read
replicationconsistencyread-replicaspostgresqlmysql
Distributed Systems 22 Sep 2026

How long to wait: timeouts and deadlines in distributed systems

A field guide to timeouts and deadlines: the difference between a local per-call timeout and a propagated deadline, the stack of independent timers every mature system exposes, and the three failure classes th…

What surprised meThe maintainers of the most-used HTTP libraries in the world still cannot agree on whether a default timeout should exist: requests parked the questi…

22 sources · 13 organisations · 3 postmortems 28 min read
timeoutsdeadlinesgrpcresiliencerpc
Distributed Systems 19 Sep 2026

When two hold the lock: what the repositories admit about distributed mutual exclusion

Every mainstream distributed lock ships with a written admission that it cannot guarantee mutual exclusion, and this guide reads those admissions where they live: client-go's package comment, etcd's post-Jepse…

What surprised meThe industry's real safety mechanism is not the lock but deliberate process death: a rejected Kubernetes pull request records the design intent in on…

22 sources · 13 organisations · 3 postmortems 26 min read
distributed-locksleasesfencingleader-electionsplit-brain
Distributed Systems 10 Sep 2026

When the answer is wrong, not late: ten years of CockroachDB read from its own record

A decade of one distributed database company read from artefacts nobody edits for effect: technical advisories with affected version ranges, design records with a status field including the rejected ones, and …

What surprised meThe public design-record process was retired between the February and May 2024 releases and moved to an internal wiki, while a competing vendor's pub…

26 sources · 3 organisations · 8 postmortems 34 min read
distributed-sqlcorrectnesspostmortemsstorage-enginemultitenancy
Distributed Systems 05 Sep 2026

Time cannot go backwards, and other lies production believed

Two postmortems (Cloudflare 2017, Azure leap day 2012), three Linux kernel commits, the Go and Rust standard-library design records, Kubernetes leader election, and the clock architectures of Google, Meta, AWS…

What surprised meThe industry's fix for the leap second is to make every clock deliberately wrong, and no two operators make them wrong the same way: Google smears 24…

23 sources · 14 organisations · 2 postmortems 27 min read
clocksmonotonic-timeleap-secondfencing-tokenshybrid-logical-clocks
Distributed Systems 02 Sep 2026

A cache miss is a protocol, not an event

Reconstructs the miss protocol behind every production cache from Facebook, Slack and Wikimedia postmortems and the implementations at Google, Instagram, Discord, DoorDash, Netflix, memcached, Rails, nginx, Va…

What surprised meIn the worst published cache outages the stampede was triggered by the machinery that manages the cache, not by an expiring key: Facebook's repair pa…

26 sources · 22 organisations · 3 postmortems 26 min read
cachingcache-stampedethundering-herdmemcachedstale-while-revalidate