Sync Engine

In a conventional application, every user interaction is a network round trip. Opening a record, changing a status, applying a filter — each waits for a server. Careful optimisation reaches perhaps 100 milliseconds on a good connection, and never reaches instant, because the speed of light and the variance of mobile networks are not engineering problems.

A sync engine inverts the arrangement. The client holds a real database containing the user's working set. Reads are local. Writes are applied locally and rendered immediately, then propagated. The server sends deltas as things change elsewhere.

The network stops being on the interaction path, and the application becomes as responsive as a local one because it is one.

Why it matters

The user-visible effect is the obvious argument, and there is a second one that explains why small teams adopt it: after the engine exists, every feature is written against a local database. No endpoint design, no loading states, no cache invalidation, no pagination logic, no request waterfalls, no optimistic-update plumbing per feature.

The engine is a large one-off cost and it reduces the marginal cost of every subsequent feature, which inverts the usual assumption that this is a technique only large teams can afford.

Implementation patterns

• A durable local mutation log for unconfirmed writes, so optimistic changes survive a reload, with reads reflecting confirmed state plus the pending log.
• Rollback and honest notification on server rejection — routinely under-built, because rejections are rare in testing and confusing in production.
• Field-level rather than record-level conflict resolution, so two people editing different fields never conflict. The single most valuable design decision in the whole pattern.
• Operation-based sync for counters, reorderings and assignments, where sending the operation composes and sending the value does not.
• Explicit conflict surfacing for long-form text, the one place silent resolution destroys work.
• Server-side re-authorisation of every arriving mutation. The client's local state is a cache, not an authority, and it may hold data whose access was revoked while it was offline.
• Authorisation embedded in the delta stream, scoping what a client receives — which makes sync a permission-evaluation problem as much as a data-transfer one.
• Versioned client schema with forward migrations and a minimum supported version below which the client wipes and re-syncs. That escape hatch must exist, because some migrations are not worth writing.
• A snapshot path alongside the delta path, for clients that have been away too long for the retained history.

Industry example

Linear is the most-discussed instance: a small team building an issue tracker around a client-side database and delta sync, with the explicit rationale that interaction latency is the product's defining characteristic and cannot be achieved through request-response. The published accounts emphasise the same secondary benefit — that feature development against a local store is faster than against an API — which is what made the investment rational at their size.

The pattern appears wherever per-user data is bounded and latency is the differentiator: note-taking applications, design tools, offline-capable field applications, and collaborative editors, which combine it with convergent data types.

Failure scenarios

• Permission revocation not enforced server-side, so an offline client's stale authority is honoured.
• Optimistic rollback unimplemented, leaving the client showing a state the server rejected.
• Record-level conflict resolution, so unrelated concurrent edits collide unnecessarily.
• No minimum-version escape hatch, forcing every migration to be written for every historical schema.
• Migrations tested only against fresh databases, missing the failures that live in accumulated local state.
• A dataset that does not fit on the client, discovered after the architecture is committed.
• Silent last-writer-wins on text, destroying user work.
• No progressive rendering during a long catch-up, freezing the application exactly when the user returns.

Trade-offs

A sync engine is a substantial piece of infrastructure with a long tail of correctness problems — conflict semantics, permission changes, schema migration on devices you do not control, and clients that have been offline for months. It is not a weekend project, and a partial implementation is worse than none because users trust it with their work.

It also constrains the data model: the user's working set must fit on the client, which rules out large catalogues, analytics, and any query across data the user does not own. Products that grow into that shape have a very expensive problem.

The trade is a large upfront investment and a bounded data model in exchange for instant interaction, offline capability and a much lower per-feature cost thereafter. It is right when latency is genuinely the product's differentiator and per-user data is naturally bounded. It is wrong, and expensively so, when the dataset is open-ended.

Interview question

"We want our app to feel instant, so someone has proposed keeping the whole workspace in a local database. Tell me what you would have to build, then tell me what happens when a user is removed from a project while their laptop is closed."