intermediate 2 min answer

You are designing the part of a file-sync client that decides which files to upload, download or merge. The last version used threads and its bugs only appeared in the field. Dropbox described rebuilding its engine over about four years around a deterministic core. Design the separation so the logic can be tested before release.

determinismtestingseparation of concernssync enginedropbox
Show the full answer Hide the answer

Requirements that drive the structure

Sync fails in rare interleavings: a file moved while another device edits it, a network drop mid-upload. If a bug needs a specific timing to appear, you can only reproduce it by controlling time. Dropbox's 2020 posts describe the old engine's thread scheduling being left to the operating system, which made integration tests non-reproducible, and describe the new engine's single-threaded design as the key to deterministic, replayable randomized testing, with tens of millions of randomized runs every night.

The design

Split the client into two concerns.

  1. A planner that takes the current picture of local, remote and last-synced state and returns a list of operations to perform. It does no input or output, reads no clock and creates no threads.
  2. An executor that carries out operations against the network and disk, and feeds results back as new state.

Because the planner is a pure function of its inputs, a test is: generate a random starting state with a seeded random number generator, run the planner and executor loop against fakes, and check invariants (it terminates, local and remote converge). A failure prints the seed, and the same seed always reproduces it.

The hard part

Concurrency still exists in the world: uploads are slow. The executor runs them asynchronously, but results re-enter the planner through one ordered queue, so decisions happen one at a time.

What it costs

  • A single decision thread limits how fast huge file sets are planned; you must keep planning cheap and push heavy work (hashing, transfers) to the executor.
  • Writing the fakes and invariant checks is real work, perhaps a few weeks.
  • Every effect must be described as data first, which adds an indirection.

How it fails

If someone sneaks a clock read or a hash-map iteration order dependence into the planner, tests become flaky again; a check that runs each seed twice and compares output catches it.

When this is the wrong answer

When the logic is trivial (a one-way backup) a planner and executor split is overhead.

Where copying it would be a mistake

Dropbox spent roughly four years and could justify it with hundreds of millions of installs. A tool with 20,000 users should not rewrite; extract the planner inside the existing code and test it first.

What I would build first

The planner as a function with a text-printable operation list, and a 1,000-seed nightly test.