Architecture Views

When two clients edit the same object in the same 50 milliseconds, what decides the outcome — a data type that cannot disagree, or a process that is allowed to?

Context

There are two honest answers to concurrent editing. The first is a conflict-free replicated data type: clients merge each other's changes directly, convergence is a mathematical property of the data structure, and no server needs authority at all. The second is a single sequencer: one process per document assigns a total order, and everything else follows it. The first makes offline and peer-to-peer trivially correct and pays in metadata that grows with edit history, tombstones that are never safely collectable, undo semantics that stop matching user intent once two people share a history, and conflict outcomes that are correct but sometimes inexplicable to the person looking at the screen. The second gives small intent-shaped operations, a plain append-only history, per-user undo that means what a user expects, and one implementation of the conflict rules — and introduces an authority that has to be elected, leased, fenced and failed over. The product context decides it: a hundred people in one file during a working day, every edit attributable, revocation that must bite in seconds, and a server that must be able to refuse an operation that breaks a document invariant. A server that cannot interpret the document cannot enforce anything about it.

Decision

Each document has exactly one authoritative session at a time. That session holds the document in memory, owns the editor sockets, assigns a monotonic sequence number to every accepted operation, and is the only writer to the document's log. Clients apply their own operations optimistically before the session has seen them and reconcile when the authoritative order differs. Convergence is a property of the sequencer plus a specified set of conflict rules, not of the data type.

How it is realised on AWS

Session hosts run as a memory-optimised pool on AKS across three availability zones. A client's connection is routed to the host currently holding the document's lease; the host validates, sequences and resolves, appends to the document's partition in Cosmos DB, and only then acknowledges. Clients keep an unacknowledged-operation buffer and a parent sequence number on every operation, which is how the host distinguishes a concurrent edit from a stale one without per-object vector clocks.

Options weighed

• ChosenOne leased authoritative sequencer per document, speculative clients: Small operations, plain history, enforceable invariants, one implementation of the conflict rules
• Right elsewhereCRDT merged peer-to-peer or through a dumb relay: Right for offline-first, local-first products and for documents that must sync without a trusted server; wrong where the server must enforce invariants and revoke access mid-session
• RejectedServer-side CRDT with a relay that never interprets operations: Keeps the metadata cost and still cannot refuse an operation on the document's own semantics
• RejectedPessimistic locking per object: Makes 'two designers, one file' a queue; the product is the parallelism

Consequences

What it buys

• An edit renders before the network sees it, so the latency budget is about acknowledgement rather than about how the product feels
• Operations stay small and intent-shaped, which is what makes history, per-user undo and attribution expressible
• The server understands the document, so it can refuse an operation that would create a cycle, an orphan or a structure no client can render

What it costs

• Document authority becomes a distributed-systems problem: a lease, a fence, a re-host path and a split-authority failure class (ADR-03)
• The session host is both socket and sequencer, which concentrates blast radius in the component most often deployed (ADR-05)
• Offline editing is not free the way it is with a CRDT: rebasing buffered work is a designed path with its own refusals (ADR-12)

Choose differently when

If the product were local-first — files that live on a device, sync opportunistically, and have no server-side invariants or revocation requirement — a CRDT would be the right answer and this record would be the wrong one. The decision rests on three product facts: a hundred collaborators in one document, edits that must be attributable, and access that must be withdrawable while the file is open.

Why it holds up over time

The boundary is a statement about where decisions are made, not about a technology. Session hosts, the store behind the log and the socket transport can all be replaced without changing it, and the fencing mechanism that protects it is older than this decade's infrastructure and will outlive the next one's.

What is the unit the platform orders, stores and replays — a change of meaning, or a change of bytes?

Context

A collaborative editor can send three things over the wire. A whole-document write is simplest and makes concurrent editing impossible to merge. A diff of rendered output (geometry, pixels, serialised scene) is easy to produce from a renderer and ties the log permanently to that renderer's version: a history of pixel diffs cannot be reinterpreted, searched, attributed meaningfully or undone per user. An operation that names intent — create this node under that parent, set this property, reorder these siblings — is more work to define and is the only one of the three from which history, merge, undo and attribution all follow naturally. The cost of intent is a vocabulary that has to be versioned, because a client on an older schema must not be allowed to write operations the current schema cannot interpret.

Decision

The platform accepts only intent operations over the document graph: create, delete, reparent, set property, reorder. Every operation carries the document's schema version. A whole-document overwrite is not an edit path; it exists only as an administrative import. Operations that are structurally invalid, reference unknown objects, or would create a cycle are refused with a typed error at admission.

How it is realised on AWS

The operation vocabulary is defined once in the monorepo and compiled into both the client and the session host, so the two can never disagree about what an operation means. Each operation row in Cosmos DB holds the document id, sequence number, epoch, author, operation kind and payload. The schema version is on the document; a client handshake declaring an unsupported version is refused with an actionable error rather than silently tolerated.

Options weighed

• ChosenIntent operations over the document graph, versioned vocabulary: The only option from which merge, history, undo and attribution all follow
• RejectedRendered or serialised scene diffs: Cheap to produce, impossible to reinterpret; ties the permanent record to a renderer version
• RejectedWhole-document writes with optimistic concurrency: Turns every concurrent edit into a conflict and every save into a document-sized write
• Right elsewhereText-style character operations generalised to a canvas: Right for documents that really are sequences; a canvas is a tree with ordered siblings, not a string

Consequences

What it buys

• The four conflict classes can be specified and resolved identically on every client, because they are about structure rather than bytes
• History is readable: a version, an undo and an audit trail all mean something to a human
• A future renderer can reinterpret an old document, because the log records what was meant rather than what was drawn

What it costs

• The operation vocabulary is a versioned interface with its own compatibility rules, and getting it wrong is expensive after launch
• Some edits a renderer finds trivial (a bulk restyle) become many operations, which is why batch limits and plugin quotas exist (ADR-08)
• The server has to understand enough of the document to validate structure, which couples it to the document model

Choose differently when

If the product's documents were opaque binaries the server had no business understanding — a video project file, a CAD assembly behind a proprietary kernel — then shipping diffs and keeping the server dumb would be the right answer, and merge would have to be solved in the client or not at all.

Why it holds up over time

Intent outlives implementations. A log of 'reparent this object' can be applied by a renderer nobody has written yet, which is the property that lets a document from 2026 open in 2036. A log of geometry diffs is tied to the code that produced it.

What stops two hosts believing they own the same document, and where is that enforced?

Context

A single-authority design is only as good as its guarantee of singularity. Orchestration cannot provide it: a container runtime that promises one replica still produces two processes when a node is partitioned rather than dead, and the partitioned one keeps accepting edits it cannot commit. Consensus on every operation would provide it and would put a quorum round inside the acknowledgement budget. The standard answer is older than any of this infrastructure: a lease with a monotonically increasing token, where the store that holds the data rejects writes carrying a stale token. That turns 'exactly one owner at all times' — which no lease can provide — into 'at most one owner can ever write', which is the property that actually matters.

Decision

Document ownership is a 15-second lease held in a strongly consistent store, renewed every 5 seconds, carrying a monotonically increasing epoch. Every append to a document's log carries the writing host's epoch, and the log rejects any append whose epoch is lower than the document's recorded epoch. A brief double-ownership is therefore harmless: the old owner's writes fail, it learns it has been fenced, and it detaches its clients.

How it is realised on AWS

The lease is a single Cosmos DB item per document, claimed and renewed with an ETag-conditional write; the epoch is part of the item and of the document's log head. Appends are conditional on the recorded epoch, so the fence is enforced by the store on the same round trip as the write — no extra latency, and no service in the path that could be bypassed. The session router claims the lease on behalf of a host and then leaves the data path entirely.

Options weighed

• ChosenLease plus monotonic epoch, fenced at the store: Makes transient double-ownership harmless and keeps the acknowledgement path to one round trip
• RejectedConsensus (quorum append) per operation: Correct and too slow: a quorum round inside a 120 ms p99 budget leaves nothing for the rest
• RejectedOrchestrator-guaranteed singleton (StatefulSet, singleton controller): Guarantees scheduling intent, not exclusion; a partitioned pod is still running
• Right elsewhereConsistent-hash placement with no lease: Right for caches and for stateless shards where a double-owner is merely wasteful rather than incorrect

Consequences

What it buys

• A 15-second lease is enough, which keeps coordination off the hot path entirely
• The split-authority failure class resolves to 'nothing the user sees' because the losing write is rejected rather than reconciled
• Re-host is cheap: claim the lease, increment the epoch, rehydrate from checkpoint plus tail — p95 10 s including client reattach

What it costs

• A lease-store outage stops handover, and the platform must then fail closed on saving rather than permit two authorities
• Every append carries an epoch and a conditional check, so the log's write path is slightly more expensive and considerably less generic
• Up to 15 seconds can pass between a host dying and its document becoming writable again — visible as a pause

Choose differently when

If the store behind the log could not enforce a conditional write on a monotonic value, the fence would have to move into a service and this design would change shape — most likely towards consensus, and towards a very different latency target. The decision depends on the store being able to say no.

Why it holds up over time

Fencing tokens are a 1990s idea that every generation of infrastructure rediscovers. Lease durations will move with network reality and the store will be replaced; the property — the data store is the last line of exclusion — will not.

What does the acknowledgement a client receives actually promise — that the server received the edit, or that the edit cannot be lost?

Context

A collaborative editor's implicit contract is that there is no save button because saving always already happened. That contract is broken the moment an acknowledgement means 'received': a host crash between acknowledgement and commit loses work the user was told was safe, and the user has no way of knowing which work. The alternative — acknowledge after durable, replicated commit — puts a store round trip inside the latency budget and makes the store's availability the edit path's availability. The second cost is the one people resist: when the log cannot be written, the honest behaviour is to refuse edits and say so, not to keep accepting them into memory and hope. A buffer that absorbs an outage is indistinguishable from a buffer that loses the outage's work.

Decision

An operation is acknowledged to its author only after it is durably committed and replicated in-region. When the log is unwritable, affected documents enter read-only within 5 seconds and the product says so explicitly; viewing, presence and reading continue. RPO for acknowledged operations is zero in-region.

How it is realised on AWS

Appends go to the document's Cosmos DB partition with strong consistency inside the region, which commits to a replica quorum before returning; the acknowledgement follows that return. Write failures are surfaced as a session state change (`read-only, cannot save`) rather than as a retryable error the client hides, and the client keeps its unacknowledged operations buffered so nothing is lost when the store returns.

Options weighed

• ChosenAcknowledge after durable replicated commit; fail closed on saving: The acknowledgement means what users believe it means
• RejectedAcknowledge on receipt, commit asynchronously: Faster and occasionally dishonest; the loss is silent and unbounded in a crash
• RejectedAcknowledge on receipt but buffer and retry during a store outage: Converts a visible read-only period into an invisible data-loss risk
• DeferredGroup commit with a few milliseconds of batching: A legitimate optimisation under burst; it changes the latency profile, not the contract

Consequences

What it buys

• RPO 0 for everything the user was told was safe, which is the only RPO a user can verify
• A store outage is a visible, bounded degradation rather than a quiet loss discovered later
• The latency budget has one dominant term — the commit — which makes it easy to reason about and to measure

What it costs

• Edit-path availability is bounded by the log's availability, so the log must be zone-redundant and its capacity planned for burst
• Users meet a read-only state they never saw in competing products that buffer silently, and support has to explain it
• Acknowledgement latency cannot be improved below the store's commit latency without weakening the promise

Choose differently when

If the product tolerated losing a few seconds of work in a crash — a brainstorming whiteboard, an ephemeral session — acknowledging on receipt would be defensible and would buy a faster, simpler platform. For a file people are paid to produce, it is not.

Why it holds up over time

The rule survives every change of store: a user may only be told their work is safe when it is. It is also the rule whose violation is invisible until the day it is not, which is why it belongs in the architecture rather than in a service's configuration.

How does anyone know that this build still converges, given that divergence is correct-looking on every individual client?

Context

Convergence bugs are the worst class of defect this platform can ship. They are invisible locally — each client is internally consistent — and they are discovered by a user whose file is quietly wrong, often hours later, with no error anywhere. Ordinary unit tests do not find them, because the failure depends on the order and interleaving of operations across clients. Worse, 'the server decides' is not by itself a convergence guarantee: it guarantees a total order, while the outcome also depends on every client applying the same rules to that order. If one client platform resolves an edit-after-delete differently, the documents diverge while both obey the order.

Decision

The four conflict classes — concurrent set of the same property, edit after delete, concurrent reparent, concurrent reorder within a container — are specified once, with a defined outcome and a defined notification to the losing author. Randomised operation-set replay is a build gate: the same operation sets, delivered in many randomised orders, must produce byte-identical document state across implementations. A build that fails it does not ship, and divergence detected in canary stops the rollout.

How it is realised on AWS

The conflict rules live in shared code compiled into both client and host. CI replays recorded and generated operation sets — 10,000 per run — in randomised orders and asserts identical serialised state, alongside operation fuzzing and an acknowledgement-latency budget check. Canary covers 1% of new sessions with divergence watch on client checksums (ADR-17); a single mismatch stops the line.

Options weighed

• ChosenSpecified conflict classes plus randomised replay as a gate: Turns convergence from a claim into a property test that keeps working as the vocabulary grows
• RejectedTrust the total order and test conflicts by example: Example tests pass while the interleaving that matters goes untried
• DeferredFormal verification of the merge rules: Valuable for the rule set itself; does not cover the implementations that have to agree
• RejectedDetect divergence in production only: Necessary (ADR-17) and far too late on its own

Consequences

What it buys

• A convergence regression is caught by CI rather than by a customer whose file is wrong
• The losing author is always told, which is what keeps users trusting the tool when a conflict does happen
• The gate keeps working as the operation vocabulary grows, because it is a property test rather than a set of fixtures

What it costs

• Release velocity on the session host is bounded by the gate's runtime, and the gate must be fast enough to run on every build
• Shared rule code couples client and server release cycles more tightly than a protocol boundary normally would
• Randomised testing finds rare failures rarely: a flake-free gate requires fixed seeds and reproducible failures

Choose differently when

If the document model were a CRDT with a proven merge, the gate would still be worth having but would be checking an implementation rather than a design. With a sequencer and hand-specified rules, the gate is the only thing standing between a plausible refactor and a divergent fleet.

Why it holds up over time

Property-based replay is independent of the operation vocabulary, the language and the store. It will outlive every component in this set, and it is the cheapest permanent answer to a class of bug that otherwise reaches users.

Should the platform own 300,000 connections, or buy them — and does the answer differ by what the connection carries?

Context

Two honest options exist for real-time transport. A managed fan-out service removes connection scaling, TLS, backpressure, rolling deploys and mass-reconnect handling from the team's problem, and adds a hop between the client and whatever process makes decisions, plus a vendor's quotas and a shedding policy the team does not control. Terminating the connections on the session host itself puts the socket and the sequencer in one process — the lowest possible acknowledgement latency, and a materially harder operational story at 300,000 connections. The traffic, though, is not homogeneous. Editor operations need the sequencer; viewer broadcast and presence need nothing but delivery, and they are three-quarters of the connections and the overwhelming majority of the messages.

Decision

Editor connections terminate on the session host that sequences the document, so an acknowledgement costs one hop. Viewer connections and all presence traffic go through a managed real-time service, published to per-document groups by the session host. A document's viewers therefore consume none of the capacity of the node that orders its edits.

How it is realised on AWS

Front Door terminates TLS and routes an editor's WebSocket to the AKS session host holding the document's lease; the host publishes presence and viewer frames to an Azure Web PubSub group named for the document. Web PubSub carries the 1.5 million messages a second of presence and the viewer fan-out; the session pool carries operations and acknowledgements only.

Options weighed

• ChosenEditors on the session host, viewers and presence on managed fan-out: Spends the one-hop budget where latency is perceived and buys scale where it is not
• RejectedEverything through managed fan-out: Adds a hop to every acknowledgement and hands the shedding policy to a vendor
• Right elsewhereEverything on self-hosted sockets: Right for a team that already operates connection infrastructure at this scale, and wants one policy for all traffic
• RejectedManaged fan-out with a sidecar sequencer: Two moving parts per document and the hop anyway

Consequences

What it buys

• Acknowledgement latency is one hop plus one commit, which is what makes the p99 120 ms target credible
• A popular document with 1,000 viewers does not threaten the node sequencing its edits
• Presence volume scales on someone else's capacity plan, and its cost is a line item rather than a node count

What it costs

• Two transports to operate, monitor and reason about, and two places a client can be connected
• Ordering between a cursor and the edit it precedes is lost across the two planes — accepted deliberately (ADR-07)
• Session hosts must handle rolling deploys with live sockets, which is why releases are drains (ADR-17)

Choose differently when

If the managed service's latency ever fell inside the acknowledgement budget — or if the team found itself operating a connection tier for other reasons anyway — the split would stop paying for itself and one transport would be simpler. The decision is a latency measurement, and should be re-measured, not re-argued.

Why it holds up over time

The principle is to spend a one-hop path where latency is perceived and buy scale where it is not. Which service carries the fan-out is a procurement decision; the split between the two kinds of traffic is architectural.

Is a cursor position a cheap kind of edit, or a different kind of thing altogether?

Context

Presence traffic — cursors, selections, viewports, who is typing — is the highest-volume thing in a collaborative editor and the least valuable per message. At 20 Hz per client across 300,000 clients it dwarfs the operation stream by an order of magnitude. Treated as a lightweight edit it competes with operations for the same buffers, the same backpressure and the same durability machinery, and it is the reason collaborative editors sometimes get slower when more people watch. Treated as sacrificial it becomes the platform's shock absorber: the thing that can be dropped, coalesced or delayed to protect what cannot.

Decision

Presence is lossy and non-durable by contract. It may be dropped, coalesced, rate-limited or reordered; it is never written to the operation log; it is retained for zero seconds beyond delivery; and it is the first thing shed when any part of the system is under load. The server coalesces to at most 10 Hz per user whatever the client sends, and the shedding order — presence, then viewer broadcast, then non-essential telemetry, never edits — is declared rather than emergent.

How it is realised on AWS

The session host coalesces each user's presence to 10 Hz and publishes to the document's Web PubSub group. Coalescing rates, viewport-based filtering and viewer broadcast policy are configuration rather than code, so a cost or load response is a setting change. Presence visibility is filtered by role, so a comment-only participant does not learn what an editor has selected where the sharing model forbids it.

Options weighed

• ChosenSacrificial, coalesced, unlogged presence on its own channel: Protects the edit path and makes presence the cheapest thing to give up
• RejectedPresence as low-priority operations on the edit channel: Keeps cursor-to-edit ordering nobody needs and puts 1.5 M messages/s in the durable path
• DeferredPeer-to-peer presence between clients: Attractive for cost, awkward for permission filtering and for clients behind restrictive networks
• Right elsewhereDurable presence history for session replay: Right for products that sell session recording; here it would turn the cheapest traffic into the largest store

Consequences

What it buys

• The edit path has a shock absorber that costs nothing to give up, and the shedding order is known before an incident rather than chosen during one
• Presence is a privacy non-event: nothing is stored, so nothing leaks later
• Presence cost is controllable without a deploy, which makes it a budget rather than a surprise

What it costs

• Cursors freeze under load, which users notice and occasionally report as an outage — it must be explained in the product, not just in a runbook
• The ordering relationship between a cursor and the edit it precedes is genuinely lost
• A 10 Hz cap is a product compromise: smooth-cursor interpolation moves to the client

Choose differently when

If presence became load-bearing — multiplayer audio, live spatial interaction, a game-like surface where the cursor is the product — it would need its own availability target and this record would have to be split in two. Richer-but-still-sacrificial presence (video pips, reactions) does not change it.

Why it holds up over time

Presence will get richer with every generation of hardware and will stay the traffic that may be dropped. A platform that has already declared its shedding order gets that richness nearly free; one that has not pays for it in acknowledgement latency.

What is the unit of partitioning, and what happens to the design when one document outgrows it?

Context

Every scaling property in this platform follows from the partition key. Partitioning by document gives independent logs, independent sequencers, a failure domain of one document per host, and no cross-document transaction anywhere. It also makes the single document the hard limit: whatever a session process can hold and sequence is the ceiling for a file, and a 500 MB document with 200 editors is a different machine from the median 1.2 MB one. Partitioning by workspace or project would amortise overhead across documents and would couple unrelated files into one failure and contention domain — the classic noisy-neighbour trade, except here the neighbours are in the same room.

Decision

The document is the partition key from edge to storage: routing, session ownership, the operation log, checkpoints, rate limits and cost attribution are all per document. A document's authoritative state must fit in one session process; the product enforces a published ceiling of 500 MB including assets, and the architecture's escape hatch for anything larger is subtree-level sequencing, designed but not built.

How it is realised on AWS

Cosmos DB partition key is the document id, which bounds a document's throughput to one logical partition's budget and keeps hot documents isolated. Session placement hashes on document id through the lease; rate limits, quotas and cost attribution all key on it. Asset bytes are deliberately outside this accounting (ADR-13), which is what keeps a 100 MB image from consuming a document's operation budget.

Options weighed

• ChosenPartition by document; one document per session process: Independent failure and contention domains at the granularity users actually share
• RejectedPartition by workspace or project: Amortises overhead and couples unrelated documents into one contention domain
• DeferredPartition by subtree from the start: The designed escape hatch; it multiplies the number of authorities per document and the conflict rules with it
• RejectedShard the hottest documents across hosts with cross-host ordering: Reintroduces the problem the single sequencer exists to avoid

Consequences

What it buys

• A host failure affects only the documents it held, and a hot document cannot starve another
• Rate limits, quotas and cost all key on the thing users share, which makes abuse and spend explainable
• No feature can quietly require a cross-document transaction, because none exists

What it costs

• The largest documents are the least comfortable, which is the opposite of what enterprise customers expect
• A per-document throughput ceiling exists and must be published, not discovered
• Moving a component library between documents atomically is a new design rather than a new query

Choose differently when

If the real distribution of document sizes shifted — a market where 300 MB files with 150 editors are normal rather than exceptional — subtree-level sequencing would stop being an escape hatch and become the primary design, with a substantially larger conflict-rule surface.

Why it holds up over time

Partitioning by the thing people share is a durable heuristic: it survives changes of store, runtime and transport, and it keeps the blast radius aligned with the user's mental model of 'my file'.

What holds 900 million independent append-only logs, one per document, each read by at most a few hundred clients?

Context

The obvious home for an append-only operation log is a stream broker, and for a few thousand high-volume streams it is the right one. This platform has 900 million logs, almost all of them tiny and idle, a handful of them busy, and every one of them needing a cheap conditional append, a cheap range read from a sequence number, and a TTL. Brokers price and partition for throughput per topic, not for a very large number of independent, mostly-idle, individually addressable logs; the per-topic overhead and the partition count make 900 million of them a non-starter. A partitioned document store inverts that: a partition per document is free when idle, supports a conditional insert that enforces both uniqueness of sequence and the epoch fence (ADR-03), and reads a range by key with no scan.

Decision

The operation log is the system of record and lives as one logical partition per document in a strongly consistent document store, keyed by (document_id, seq), with the epoch on every row and a TTL that implements the 30-day horizon. Checkpoints in object storage are derived. No stream broker sits on the edit path; the change feed of the log is what feeds derived consumers.

How it is realised on AWS

Cosmos DB for NoSQL with the document id as partition key and strong consistency inside the region; appends are conditional writes that fail on a stale epoch or a duplicate sequence. Its change feed drives the search feed, the audit export and the rebuild verifier. Checkpoints and assets live in Blob Storage (RA-GZRS); PostgreSQL holds the control plane, which has entirely different access patterns and recovery requirements.

Options weighed

• ChosenPer-document partition in a strongly consistent document store: A free idle log, a conditional append that fences, and a range read by key
• RejectedStream broker (Kafka, Event Hubs) as the log: Priced and partitioned for throughput per topic, not for 900 million mostly-idle logs
• Right elsewhereRelational table partitioned by document: Right at a smaller document count, where one engine for log and control plane is a real operational saving
• RejectedAppend-only blobs per document: Cheapest storage, no conditional append semantics worth trusting for a fence

Consequences

What it buys

• An idle document costs storage only — no topic, no partition, no consumer group
• The fence and the sequence-uniqueness check ride the same write as the append, so correctness costs no extra latency
• Derived state has one source and one rebuild path, which is why checkpoints and indexes are never backed up

What it costs

• Request-unit economics have to be modelled per operation, and a hot document's throughput is bounded by its logical partition
• The 30-day horizon is a storage and cost decision with a product consequence for very stale clients (ADR-12)
• Replaying a long log is a read-amplification problem the checkpoint cadence exists to bound (ADR-10)

Choose differently when

If document count fell by three orders of magnitude while per-document throughput rose — a handful of enormous shared canvases rather than hundreds of millions of files — a broker or a log-structured service would become the better fit, and the per-document partition would become overhead.

Why it holds up over time

The record is 'an append-only, per-document, replayable log', not a product name. Any store that offers a cheap conditional append and a keyed range read can hold it, which is a property stores keep acquiring rather than losing.

Who materialises the document state — the process that already holds it, or a separate service that reads the log?

Context

Checkpoints decide how fast a cold document opens and how much log a re-host has to replay. Someone has to fold the log into a document state, and there are only three candidates. The session already holds the state in memory, so its checkpoint is nearly free, and it is also the process with the least spare attention during a burst. A background service reading the log is cleanly separated and needs a second implementation of document application — which means a second set of bugs, and the possibility that the two implementations disagree about what the log means, which is exactly the class of defect this platform most fears. A client could do it cheapest of all, and cannot be trusted. The uncomfortable part of choosing the session is that nothing then independently checks the fold.

Decision

The session host writes the checkpoint it already holds, every 30 minutes of activity and on release. A separate rebuild verifier replays the log independently, on a schedule and not only after an incident, and diffs its result against the live checkpoint; a mismatch is an alert. There is one implementation of document application on the write path and one deliberately independent one off it.

How it is realised on AWS

The checkpoint writer serialises the in-memory document to Blob Storage keyed by document and sequence number. The verifier runs as an AKS job driven by the log's change feed, samples documents (all documents above a size or activity threshold, a random sample below it), rebuilds from the previous checkpoint plus the tail, and compares. Rebuild diff rate is a first-class metric on the observability grid.

Options weighed

• ChosenSession writes the checkpoint; independent verifier diffs it: One implementation on the write path, with an independent opinion where a disagreement is cheap
• RejectedBackground service as the only checkpoint writer: Clean separation at the price of a second application implementation on the critical path
• RejectedClient-written checkpoints: Cheapest and untrustworthy: a checkpoint is the document, and the client is a replica
• Right elsewhereNo checkpoints; replay the whole log: Right for short-lived documents; here a year-old file would replay millions of operations to open

Consequences

What it buys

• Cold open is bounded by 30 minutes of log, which is what makes the 2.5 s p95 on a 50 MB document reachable
• A one-way application bug is caught by the verifier rather than by a user whose file is wrong
• Re-host replays at most the same bounded tail, which is what keeps the 10 s re-host target honest

What it costs

• Checkpointing competes with editing on the busiest host, so it must be incremental and interruptible
• The verifier is a second implementation after all — deliberately, and off the write path, but it still has to be maintained
• 30 minutes of cadence against 30 days of retention is a storage bill that scales with active documents

Choose differently when

If checkpointing ever measurably hurt acknowledgement latency on busy documents, the writer would move off the session and the second implementation would become the price of that — paid consciously, with the verifier then comparing two independent folds rather than checking one.

Why it holds up over time

The pattern — materialise from the authoritative log, keep an independent rebuild that disagrees loudly — survives any storage technology, and it is the mechanism that makes a derived store droppable rather than precious.

When a user restores yesterday's version, does the document travel back in time, or does it move forward to a state that looks like yesterday?

Context

Restoring by rewriting history is the intuitive implementation and it breaks three things at once. Every connected client's document version becomes meaningless, so they must all be reloaded. Every reference into the log — a comment anchor, an audit record, a named version, another client's buffered operations — can dangle. And the act itself leaves no trace, so 'who reverted my work' has no answer. Undo has the same shape in a shared document: a global undo stack means one person's undo can reverse someone else's edit, which users experience as sabotage, while a per-user undo must sometimes refuse, because the object it would restore has since been deleted or restructured by someone else.

Decision

A restore is a new forward operation that makes the document equal to an earlier state; the log is append-only and never rewritten. Undo is per user and expressed as a compensating operation, and it is refused with a clear reason when the object it would touch has since been deleted or structurally changed by another user. Named versions, comments and audit records therefore never dangle.

How it is realised on AWS

Restore computes the difference between the current head and the target checkpoint and emits it as operations, so connected clients receive it exactly as they receive any other edit — no reload, no version discontinuity. Undo looks up the user's own most recent operation, checks applicability against current state, and emits the inverse or a typed refusal. Named versions are rows pointing at a sequence number, so naming a version mutates nothing.

Options weighed

• ChosenRestore as a forward operation; append-only history; per-user undo: No dangling references, no forced reloads, and an auditable record of the restore itself
• RejectedTruncate or rewrite the log to the restore point: Breaks live clients, comment anchors and audit, and erases the evidence of the restore
• RejectedGlobal undo stack: Lets one user reverse another's work, which reads as a bug however correct it is
• Right elsewhereBranch on restore, with an explicit merge: Right for code and for products that teach branching; heavy for a design file people expect to be one file

Consequences

What it buys

• A restore is visible, attributable and itself undoable, because it is just another edit
• Live collaborators see a restore as a large edit rather than as a disconnection
• Comment anchors and audit records are permanently valid, since nothing they point at can disappear

What it costs

• A restore of a large document is a large burst of operations, which must be rate-shaped so it does not look like abuse
• Per-user undo has refusal cases the product has to explain in words, which is more interface work than a global stack
• The log grows with restores, which is a storage cost rather than a correctness one

Choose differently when

If the product adopted explicit branching — a design file with named branches and merges — restore would become a branch operation and this record would be rewritten around merge semantics instead of compensation.

Why it holds up over time

Append-only history with compensation is the same decision ledgers and version control made, for the same reason: evidence is worth more than tidiness. It outlives any storage choice.

What happens to an hour of offline editing when the document it was based on has moved on — and what happens after thirty days?

Context

A client that edited offline comes back holding operations whose parent sequence is in the past. Inside the retained log the platform can rebase them: apply the live conflict rules as if they had arrived late, which is technically convergent and can be semantically strange — restyling a frame someone has since redesigned, or editing objects whose meaning has changed. Beyond the log horizon there is nothing left to rebase against, because the operations the client missed have been compacted away. Three outcomes are available: merge and accept the oddity, refuse and discard, or preserve the work somewhere the user can get at it. Only one of those is acceptable for work a person spent an hour on.

Decision

Buffered offline operations are rebased into the authoritative order under the same conflict rules as live editing while the client's version is inside the 30-day retained log. Beyond the horizon, or when the operations cannot be merged without violating document invariants, the work is preserved as a recoverable branch and the user is told. Silent discard is not an available outcome.

How it is realised on AWS

On reattach the client sends its version; the host chooses replay (inside the log), snapshot-plus-tail (behind the log but inside the horizon), or branch (beyond it), and reports which path it took. A branch is written as a separate document derived from the client's buffer plus the last common checkpoint, visible to its author with an explanation. Operations that would violate an invariant are quarantined with the same mechanism and retained for support recovery.

Options weighed

• ChosenRebase within the horizon, recoverable branch beyond it: No silent loss, and the weird-merge case is bounded by a horizon the platform publishes
• RejectedAlways rebase, whatever the age: Requires infinite log retention, and produces merges nobody can explain
• RejectedRefuse and discard beyond a horizon: Cheapest, and the one outcome that loses work a person did
• Right elsewhereAlways ask the user to choose: Right for a developer tool; an interruption most designers should never have to see

Consequences

What it buys

• The 'offline on a train' journey has a defined ending in every case, including the ugly one
• The log horizon becomes a published product property rather than an internal retention setting
• Quarantine gives support something to recover from, instead of an apology

What it costs

• Branches are a product concept the interface must explain, in a product that otherwise has one file
• The client must hold at least 10,000 operations durably, which is real client-side engineering
• Semantically odd merges inside the horizon still happen and will generate support contacts

Choose differently when

If the document model were a CRDT, the horizon would stop mattering and the branch case would disappear — which is the strongest argument available against ADR-01 and is why that record names offline duration as the condition that would flip it.

Why it holds up over time

The rule — never silently lose work a person did, and bound the cases where merging is strange — is independent of merge technology, and is the part of this design a user would still recognise in ten years.

Where do a hundred megabytes of image bytes go, given that the operation that references them must be ordered in milliseconds?

Context

A designer drops a 90 MB image into a frame. If those bytes travel as an operation they enter the ordering path, the log, the checkpoint and every client's catch-up, and one paste stalls a document for everyone in it. If they travel out of band the operation is a few hundred bytes containing a hash, and the bytes move client-to-storage over a pre-signed URL — at the cost of a window in which the document legitimately references content that does not exist yet. The platform must then decide what a document means during that window: blocked until complete, or valid with a placeholder.

Decision

Asset bytes never enter the session or the log. The document holds a content-addressed reference, the bytes move directly between client and object storage over short-lived pre-signed URLs, and an operation referencing an asset that has not finished uploading is accepted — the document renders valid-with-placeholder until the bytes arrive. Assets are deduplicated by hash within a tenant, immutable once written, and scanned before any rendition is served.

How it is realised on AWS

The asset service issues an upload intent with a pre-signed Blob Storage URL scoped to one content hash and a short expiry; Defender for Storage scans on write and the scan state lives on the asset row, so an unscanned asset may be referenced but not served. Renditions are generated asynchronously. A reference never grants access: entitlement is checked against the referencing document's grants on every read.

Options weighed

• ChosenOut of band, content-addressed, placeholder-on-pending: Keeps the ordering path small and the paste instant, at the cost of a defined pending state
• RejectedAssets as operation payloads: One paste becomes everyone's problem, in the log, the checkpoint and every catch-up
• RejectedOut of band but blocking the operation until upload completes: Makes a slow network look like a broken file, and couples an edit to an upload
• Right elsewhereAssets inside the checkpoint: Right where documents are exported as self-contained bundles; here it would multiply storage by every checkpoint

Consequences

What it buys

• A 90 MB paste costs the document a few hundred bytes of ordering work
• Deduplication by hash within a tenant makes component libraries and repeated imagery cheap
• Immutability means a reference is permanently valid, which is what lets checkpoints and exports share bytes

What it costs

• A pending state exists and must be rendered honestly, which is the trough of the review journey
• Access control on assets is a second enforcement point with its own failure mode
• Deleting a document cannot blindly delete its assets: reference counting within the tenant is required

Choose differently when

If documents were small and asset-free — diagrams, text, data visualisations — inlining content would be simpler and the pending state would not need to exist. The decision is driven by the 100 MB asset and the 500 MB document ceiling.

Why it holds up over time

Content addressing and out-of-band bytes are how every system that moves large objects alongside small records ends up working. The storage service will change; the split will not.

How does a permission change reach a connection that is already open and editing?

Context

Most products resolve permission when a document is opened and never again, so a revocation takes effect at the next reload — which, in a tool people leave open for a working day, can be hours. The opposite extreme, checking the sharing service on every operation, puts a control-plane dependency inside a 120 ms acknowledgement budget and makes a sharing outage a product outage. Between them lies the honest answer: bind a capability to the connection at admission, and push grant changes to the sessions that hold affected capabilities. That requires the control plane to know which sessions care, and it requires a decision about what happens when the control plane is unreachable — because during that window the platform can either keep working with stale authority or stop working with correct authority.

Decision

Admission authenticates the user, resolves their effective role, and binds the resulting capability to the connection for its lifetime. Every change to grants, links or membership is pushed to affected live sessions, which downgrade or detach within 10 seconds and refuse operations arriving after authority is withdrawn. During a control-plane outage, existing sessions continue on cached capabilities for at most 5 minutes and new admissions fail closed.

How it is realised on AWS

The sharing service publishes grant-change events; session hosts subscribe for the documents they hold and apply downgrade or detach immediately. Capabilities are cached in Redis with a 5-minute ceiling, and tokens are renewed in-band on the live connection so a connection never outlives its credential. Every grant change, link creation, export and permanent deletion is an audit record.

Options weighed

• ChosenBind at admission, push grant changes, 10 s revocation: No control-plane dependency on the hot path, and revocation is a platform guarantee
• RejectedCheck permission on every operation: Correct and unaffordable: a sharing outage becomes an editing outage
• RejectedResolve at open, re-check at next reload: The common industry behaviour, and the reason admins do not trust 'remove access'
• DeferredShort-lived capability tokens with frequent renewal: A reasonable alternative that trades push machinery for renewal traffic; revocation latency becomes the token lifetime

Consequences

What it buys

• An admin can remove a leaver and see it take effect while the file is open — the journey that justifies the design
• The edit path has no synchronous control-plane dependency, so sharing can fail without stopping work
• Capability is enforced server-side per operation, so a client that keeps its editing affordances gains nothing

What it costs

• Grant-change fan-out is a second push path with its own delivery concerns
• The 5-minute grace window is a stated hole: during a control-plane outage a revocation can be late
• Sessions hold authority state, which makes them slightly less disposable than a stateless worker

Choose differently when

If the product served a single trust domain where revocation urgency was low, resolve-at-open would be adequate and the push path would be unnecessary complexity. Enterprise offboarding is what makes 10 seconds a requirement.

Why it holds up over time

Binding authority at admission and invalidating it by event is the same pattern as session revocation lists and token introspection, and it has outlived several generations of identity technology.

Who can read a customer's document, and what does an engineer debugging a convergence bug get to see?

Context

Debugging a convergence problem is far easier with the document in front of you, and that is exactly what makes it the moment confidentiality is lost. A platform that leaves a routine operator read path open will use it, log it nowhere in particular, and discover during an enterprise security review that it cannot describe who has read what. Closing the path entirely has a real cost: incident response gets harder, and some bugs can only be reproduced from a customer's own document. Customer-managed keys raise the same tension one level higher — they are a genuine control for the customer and a genuine availability risk for the platform, because a revoked key is an unreadable document.

Decision

Document content is confidential to its tenant. There is no routine operator content-access path; break-glass access requires approval, uses a separate identity, and produces an audit record that is itself immutable. Content is encrypted at rest with tenant-scoped keys, with customer-managed keys in a hardware security module available on the enterprise tier. Operation metadata — counts, sizes, latencies, attribution — is exported for analytics; content is excluded by construction rather than by policy.

How it is realised on AWS

Keys live in Key Vault Managed HSM, per tenant, with the enterprise tier pointing at a customer-held key. Private Link between the session tier and every store means no public data path exists. Break-glass is a distinct Entra identity with approval workflow; its use writes to the immutable audit log before the access happens. Debug tooling works from structural traces — operation kinds, sequence numbers, conflict classes — not payloads.

Options weighed

• ChosenTenant-scoped keys, CMK on enterprise, approved and audited break-glass: Closes the routine path and keeps an auditable exception for the cases that need one
• RejectedOperator read access with logging: Logging a path everyone uses is not a control, it is a record of its use
• RejectedNo break-glass at all: Attractive until the first bug that only reproduces on one customer's document
• Right elsewhereEnd-to-end client-side encryption: Right where the server must never interpret content; incompatible with a server that validates document structure (ADR-01)

Consequences

What it buys

• The platform can answer 'who has read this document' with a record rather than an assurance
• Enterprise customers get a key they control, and the blast radius of a platform compromise is bounded per tenant
• Analytics and search feeds carry no content, so a second system can never become the leak

What it costs

• Debugging is harder and slower, and some reproductions require a customer's cooperation
• A customer-managed key is an availability dependency on the customer: a revoked or expired key is an unreadable document
• Per-tenant keys complicate asset deduplication, which must stay within a tenant boundary (ADR-13)

Choose differently when

If the product moved to end-to-end encryption, the server could no longer validate document structure or enforce invariants, and ADR-01 would have to be reopened — which is why these two records are linked rather than independent.

Why it holds up over time

Confidentiality boundaries drawn at the tenant, with approved and recorded exceptions, are what every compliance regime has converged on. The key-management product will change; the boundary should not.

Where does a document live when its editors are in three time zones, and what happens when that region fails?

Context

Active-active editing of one document across regions means two sequencers, which contradicts the central decision of this architecture (ADR-01). It could be made to work with a CRDT, or with cross-region consensus and its latency, and both change the platform rather than extending it. The alternatives are narrower and honest: give a document one home region and pay a distance penalty for far-away editors, or move the home to where the editors are and define what happens during the move. The regional failure case is separate and must be stated rather than implied: asynchronous replication means a declared loss window, and pretending otherwise is the most common lie in multi-region architecture.

Decision

Every document has one home region, pinned by its tenant's residency policy, with one write region for its log. A document's home may be migrated live with an announced pause of up to 3 seconds. Viewers are served from regional read paths. Regional failover is a declared operation with RTO 15 minutes and a replication loss window of up to 5 seconds, stated to users rather than hidden.

How it is realised on AWS

Cosmos DB with a single write region per document's home and asynchronous replication to the paired region; Blob Storage as RA-GZRS; a warm standby session pool and admission tier in the paired region. Migration drains the document's session, moves ownership, increments the epoch and reattaches clients — the same mechanism as a re-host (ADR-03), with a longer pause.

Options weighed

• ChosenOne home region, live migration, declared failover loss: Preserves the single sequencer and tells the truth about what a failover costs
• RejectedActive-active editing across regions: Requires two authorities or cross-region consensus; contradicts ADR-01 or the latency budget
• RejectedSynchronous cross-region commit: RPO 0 everywhere, and an acknowledgement budget nobody would accept
• DeferredFollow-the-sun automatic migration: Attractive once the proportion of genuinely cross-region documents is measured rather than assumed

Consequences

What it buys

• The single-sequencer guarantee holds globally with no new failure mode
• A failover's cost is a published number, which makes it a business decision rather than a surprise
• Viewers — the majority of sessions — are served locally regardless of where the document lives

What it costs

• A distant editor sees up to 400 ms acknowledgements, and will call the product slow
• Migration is a visible pause, and is therefore an operation rather than an optimisation
• The paired region is warm, not continuously exercised, so the first real failover is the first full-scale test

Choose differently when

If a measured majority of documents turned out to be genuinely multi-region — editors on two continents in the same hour, routinely — the cost of the latency tax would exceed the cost of a second authority, and the platform would have to revisit ADR-01 with a CRDT in hand.

Why it holds up over time

Choosing one writer per unit of data and being explicit about replication loss is the durable half of multi-region design. The services change; the arithmetic of distance does not.

How does the platform learn that one client's document no longer matches the server's, given that nothing is obviously broken?

Context

Every other failure class in this platform announces itself: a lease expires, an append fails, a probe goes red. Divergence does not. A client whose state has drifted is internally consistent, renders happily, and keeps accepting edits, and the only witness is a human who eventually notices their file is wrong — usually long after the operations that caused it. Two mechanisms are available. One is detection: have clients prove they agree with the server at known points. The other is prevention of the most likely cause: two code versions applying the same operations differently during a rollout. The second is cheap because sessions are short-lived (ADR-19's lifecycle), so a rollout can wait for documents rather than restarting them.

Decision

Clients periodically report a checksum of their document state at a known sequence number; a mismatch raises an alert within 60 seconds and is repaired by re-snapshotting that client. A live session never changes code version: a rollout drains idle documents onto the new build and leaves busy documents until they release, with a forced release at the next idle window to bound version skew. A single divergence in canary stops the rollout.

How it is realised on AWS

The client sends a hash of its serialised state every few minutes and on key transitions; the session compares against its own state at that sequence. Mismatches are logged with client build, session build and document id, alerted on in Azure Monitor, and repaired by pushing a fresh snapshot to that client. Session hosts are version-pinned for a session's lifetime, and the release pipeline treats divergence rate as a stop-the-line signal.

Options weighed

• ChosenClient checksum reporting plus version-pinned sessions: Detects the silent failure and removes its most likely cause
• RejectedTrust the sequencer and the release gates: Gates catch what they test; production finds the rest, and here production means a user's file
• RejectedFull state comparison on every reconnect: Thorough and expensive: a 60 MB comparison on every reattach
• RejectedHot-swap session code in place: Saves a drain and reintroduces the exact condition this record exists to prevent

Consequences

What it buys

• The one failure class with no natural alarm gets one, within a minute
• Repair is automatic and nearly invisible: a brief reload for one client rather than an incident
• A rollout can never have two implementations applying one document, which removes the most plausible cause of divergence

What it costs

• A small permanent overhead in client message volume and server comparison work
• A long-lived document can hold an old build for hours, so version skew is bounded by policy rather than by deploy time
• A noisy checksum signal would be worse than none, so sampling and thresholds have to be tuned rather than guessed

Choose differently when

If the client and server shared a single compiled implementation of document application — one WebAssembly module used by both — the most likely divergence cause would disappear and checksum reporting could become a sampled audit rather than a continuous signal.

Why it holds up over time

Having the replica prove it agrees with the authority is how storage systems, caches and replicated databases all eventually detect silent corruption. It outlives any particular hash or transport.