Distributed systems Simulation Intermediate 3D dissection 96 components 18 Oct 2026

Anatomy of Dynamo

Amazon's Highly Available Key-value Store, dissected: a ring of seven nodes laid out in 3D, the paper's Figure 3 lived through by a seeded cluster, and a research desk that tests the model of how stale its reads are.

Dynamo is the template for leaderless, eventually consistent stores: Cassandra, Riak and Voldemort descend from it. This model runs its design as a seeded discrete-event simulation: seven nodes with four tokens each on an MD5 ring, preference lists of distinct nodes, a coordinator that waits for W of N writes and R of N reads, per-server vector clocks, a sloppy quorum with hinted handoff, read repair, Merkle-tree anti-entropy and gossip membership. The page narrates the paper's Figure 3 with that cluster: five writes to a cart, a node down, a stand-in holding a hint, two clients in conflict, one merge. A research desk then asks how stale a read is. It reproduces the probabilistically bounded staleness model of Bailis et al. (2012) and checks it against the full cluster, relaxes its assumptions one at a time, tests its closed form for versions, and tests the paper's claim that an add to cart is never lost.

The machine

Seven nodes stand in a circle, each a card: for six carts, up to three versions held (colour says which version, so a conflict is two colours in one row) and the hinted replicas it keeps for others. Inside them, the ring of Figure 2 (seven tokens) and the ring the cluster uses (four tokens a node, coloured by owner), with the six carts hashed onto it. Messages are links between the cards, coloured by type. Up the table: Figure 3's clocks and who descends from whom, one stale read replica by replica, two Merkle trees, gossip round by round, the three partitioning strategies of §6.2, and the research results as blocks.

What is real. The run is a seeded discrete-event simulation of the protocol the paper describes, computed in the page by engine.js. test.mjs checks MD5 against node's own, Figure 3 clock by clock, consistent hashing's defining property, the theorem that R + W > N never reads stale across four latency models, convergence of every replica after random crashes and partitions once hinted handoff and anti-entropy run, negative controls that must fail, the closed forms and arithmetic of the Bailis paper (including two of its numbers that its own equation contradicts), and the model against the numbers that paper published: the probabilities right after a commit, Figure 4's, Table 4's t-visibilities and the N sweep. Every paper number is a fact with a verbatim quote.

What to take away

  • A read is stale when its request overtakes the write at a replica, not because replication is slow: the WARS model explains it from four message delays, and the cluster agreed with it in all 132 cells of the first experiment.
  • The model is conservative only in the middle: with a replica as coordinator and W = 1 the write commits at once and reads in the next few milliseconds are up to 56 points staler than predicted; read repair removes the staleness it counts; a failure nobody has noticed leaves 1.7% of reads stale a second later, 2,000 times the model.
  • The paper's claim that an add to cart is never lost does not hold for its own clocks: with four clients on one cart, 4.7% of acknowledged adds were gone from the final cart, because per-server counters cannot order two writes made through the same coordinator.
  • Dotted version vectors fix that only if replicas receive whole sibling sets: with Dynamo's version-at-a-time replication a few adds were still lost (6 of 180 runs), and with state shipping none were (0 of 180).
  • PBS's closed form for versions is exact for random quorums and an upper bound for a real cluster, but a loose one: the cluster reached between 3% and 86% of it.

Sources

More machines

Other working models

Anatomy of Raft: a still of the machine Simulation · 3D Architecture · Distributed systems

Anatomy of Raft

In Search of an Understandable Consensus Algorithm, dissected: five servers laid out in 3D with their terms, votes, logs and state machines, Figures 7 and 8 rebuilt from real blocks, and a research desk that reruns the paper's leader-election measurements.

Raft's Figure 2 as five servers in 3D: timers, votes, logs and messages computed live from a seeded simulation, with the paper's Figures 7 and 8 as real blocks and a research desk that reproduces its…

52 components · Intermediate 17 Oct 2026
Anatomy of Lamport Clocks: a still of the machine Simulation · 3D Architecture · Distributed systems

Anatomy of Lamport Clocks

Time, Clocks, and the Ordering of Events, dissected: an execution laid out in 3D with its logical clocks, vector clocks, every pair of events, the mutual exclusion algorithm and a physical-clock synchroniser, all computed live and checked against what really happened before what.

Lamport's 1978 paper built from its own definitions: the paper's Figure 1 or a seeded execution in 3D, logical and vector clocks on every event, the mutual exclusion algorithm, physical clock skew ag…

45 components · Intermediate 16 Oct 2026

The whole gallery →