# Architecture Decision Record

*Multiplayer Collaborative Canvas · Solution Architecture v1.0 · Microsoft Azure · Platform Architecture · 2026-10*

The argument these decisions serve is summarised in the [Architecture One-Pager](architecture-one-pager).

Seventeen decisions make up this architecture. Everything else across the twenty-three views is either a consequence of one of them or a detail that could be decided differently next quarter without anyone redrawing the set. Each record carries more than the classic context / decision / consequences triple: the forcing question, how the decision is realised on Azure, the alternatives including the ones that are right for a different organisation, the conditions that would flip the choice, why it should still hold as scale and technology change, and the transferable lesson.

> **Status of this document.** This is a design, not a report on a running system. Every rate, latency, interval, limit and retention figure is a stated assumption chosen to be defensible and arguable rather than measured. The operating context assumed throughout is a design-collaboration SaaS with 4 million monthly active editors, 11 million monthly viewers, 300,000 concurrently connected clients at peak, 120,000 operations a second with a 3.3× burst for 60 seconds, 1.5 million presence messages a second after coalescing, and 900 million stored documents whose p99 is 60 MB.

## How to read a record

- **Question:** The forcing question: why a decision was needed at all.
- **Context:** The requirement, the scale and the constraint that make it hard.
- **Decision:** What this architecture does, stated so it can be checked.
- **How it is realised on AWS:** The concrete mechanism: which service or package, configured how, in which subscription.
- **Options weighed:** Chosen, rejected, deferred, or right elsewhere, with the reason for each.
- **Consequences:** What the choice buys and what it costs, both kept visible.
- **Choose differently when:** The conditions that would flip the decision for your system.
- **Why it holds up over time:** What keeps the decision right as scale, staff and technology change.
- **Lesson:** The principle that transfers beyond this platform.

## Decision map

**Authority and convergence**: Who decides what the document is, and how every client comes to agree with that decision.

- ADR-01 · Each document has one authoritative sequencer; every client is a speculative replica
- ADR-02 · Operations describe intent on the document graph, never rendered diffs or whole-document writes
- ADR-03 · Ownership is a lease with a monotonic epoch, and the fence is enforced at the log's conditional write
- ADR-04 · An operation is acknowledged only after durable commit; the platform fails closed on saving
- ADR-05 · Conflict outcomes are specified per class, and convergence is a release gate rather than a claim

**Paths and transport**: Which traffic is allowed to be lost, and where each kind of client terminates.

- ADR-06 · Editors terminate on the sequencing host; viewers and presence leave through managed fan-out
- ADR-07 · Presence is sacrificial: coalesced, unlogged, zero-retention, and the first thing shed
- ADR-08 · The document is the partition key everywhere, and a document fits in one session process

**State and history**: What is the record, what is derived, and what the platform owes a client that was away.

- ADR-09 · The per-document operation log is the system of record, held in a partitioned document store rather than a stream broker
- ADR-10 · The session writes its own checkpoints; an independent verifier disagrees with them
- ADR-11 · Restore is a forward operation; history is never rewritten, and undo is per user
- ADR-12 · Offline work is rebased within the log horizon and preserved as a recoverable branch beyond it

**Assets and large objects**: How bytes that dwarf the document stay out of the path that orders it.

- ADR-13 · Assets move out of band, content-addressed, with the document valid before the bytes arrive

**Identity and trust**: Who may edit, how fast that can be withdrawn, and who can never read.

- ADR-14 · Capability is bound at admission, re-evaluated on every grant change, and revocation bites on a live socket
- ADR-15 · Document content is confidential to its tenant, with no routine operator path and keys the customer can hold

**Scale and operations**: Where a document lives, and how the platform finds out it is wrong before a user does.

- ADR-16 · One home region per document, with announced migration rather than active-active editing
- ADR-17 · Divergence is detected by client checksum reporting, and a release never changes a live session's code

## Technology by capability

Microsoft Azure was chosen for this exercise deliberately, and for two reasons that point the same way. The first is rotation: across this repository's recent use cases AWS and Google Cloud have each carried five of the last sixteen packages and Azure three, and reaching for the same catalogue every time teaches a catalogue rather than architecture. The second is that Azure offers both halves of the tension this topic turns on in first-party form — a managed real-time fan-out service (Web PubSub) sitting beside stateful session hosts the team runs itself on AKS — which makes ADR-06 a real choice on this stack rather than a hypothetical one. Every requirement in ask.md stays vendor-neutral, and each row below names what would be used elsewhere: the architecture should survive a change of cloud, and the point of naming one is to stop the design hiding behind 'a durable log' when the decision is about request units, consistency levels and conditional writes.

| Capability | Choice | Origin | Credible alternative | Why this one | Record |
|---|---|---|---|---|---|
| Document operation log | Cosmos DB for NoSQL, partition key = document id, strong consistency in-region, conditional append on epoch, TTL for the 30-day horizon | Microsoft Azure | DynamoDB with conditional writes; Spanner; a per-document table in PostgreSQL at lower document counts | 900 million mostly-idle logs need a free idle partition and a conditional append, which is a key-space problem rather than a throughput one | ADR-09 |
| Ownership lease and fence | One Cosmos DB item per document with an ETag-conditional claim and a monotonic epoch, checked on every append | Microsoft Azure | etcd or Consul leases; Blob lease; ZooKeeper; DynamoDB conditional writes | The fence must be enforced by the store that holds the data, on the same round trip as the write | ADR-03 |
| Session tier | Memory-optimised AKS node pool across three zones, one authoritative session per document, sockets terminated in-process | Microsoft Azure | EKS or GKE; Service Fabric stateful services; Cloudflare Durable Objects for a smaller estate | The sequencer and the socket share a process, which is what makes a p99 120 ms acknowledgement credible | ADR-01 |
| Presence and viewer fan-out | Azure Web PubSub, one group per document, published by the session host after 10 Hz coalescing | Microsoft Azure | AWS API Gateway WebSockets with IoT Core; Ably or Pusher; self-hosted NATS or Centrifugo | Buys connection scale for the traffic that is allowed to be lost, and keeps 1,000 viewers off the sequencing node | ADR-06 |
| Checkpoints and assets | Blob Storage, RA-GZRS, content-addressed asset containers with pre-signed upload URLs and lifecycle tiering | Microsoft Azure | S3 with pre-signed PUT; Google Cloud Storage; MinIO on-premises | Large immutable objects belong in object storage, out of the ordering path and priced by tier | ADR-13 |
| Asset scanning | Microsoft Defender for Storage malware scanning on write, scan state carried on the asset row | Microsoft Azure | GuardDuty Malware Protection for S3; ClamAV in a scanning pipeline | An unscanned asset must be referenceable but not servable, which is a state on the record rather than a gate on the upload | ADR-13 |
| Control plane store | Azure Database for PostgreSQL Flexible Server, zone-redundant HA, read replicas for admission lookups | Microsoft Azure | Aurora PostgreSQL; Cloud SQL; CockroachDB for multi-region grants | Grants, versions and comments are small, relational, transactional and read on every admission — a different problem from the log | ADR-14 |
| Capability cache | Azure Cache for Redis, 5-minute ceiling on cached capabilities | Microsoft Azure | ElastiCache; Memorystore; in-process caches with a shorter ceiling | Keeps the control plane off the hot path while bounding how stale an authority decision may be | ADR-14 |
| Identity | Microsoft Entra ID for workforce identity, Entra External ID for guest and link access, in-band token renewal on the live connection | Microsoft Azure | Okta or Auth0; Cognito; Keycloak self-hosted | A connection must never outlive its credential, and link viewers must be first-class without accounts | ADR-14 |
| Keys | Key Vault Managed HSM, tenant-scoped keys, customer-managed keys on the enterprise tier | Microsoft Azure | AWS KMS with external key store; Cloud KMS; Thales or Entrust HSM on-premises | Per-tenant confidentiality with an auditable exception path, and a key the customer can hold | ADR-15 |
| Edge | Azure Front Door with WAF, WebSocket pass-through, document-aware routing to the session holding the lease | Microsoft Azure | CloudFront with ALB; Cloud Load Balancing; Envoy with a custom control plane | Admission, abuse control and TLS belong at the edge; routing must respect the lease rather than load | ADR-06 |
| Observability | Azure Monitor with managed Prometheus and Grafana, metadata-only telemetry, divergence checksums as a first-class metric | Microsoft Azure | Amazon Managed Prometheus with Grafana; Cloud Operations; self-hosted Prometheus, Thanos and Grafana | The two signals that matter — divergence and read-only minutes — have no substitute and must be alertable | ADR-17 |
| Build and release | GitHub Actions to Azure Container Registry with attestation, convergence replay as a required check, drain-based rollout on AKS | Microsoft Azure | Azure DevOps; GitLab CI with Argo Rollouts; Cloud Build with Cloud Deploy | Convergence has to be a gate, and a rollout has to be a drain rather than a hot swap | ADR-05 |
| Derived feeds | Cosmos DB change feed into Event Hubs for search indexing and audit export, ADLS Gen2 for the metadata-only analytics export | Microsoft Azure | DynamoDB Streams with Kinesis; Datastream with Pub/Sub; Debezium with Kafka | Derived consumers read the record and never the edit path, so a slow index can never slow an edit | ADR-09 |

## The decisions, and the alternatives that lost

### Authority and convergence

*Who decides what the document is, and how every client comes to agree with that decision.*

#### ADR-01 · Each document has one authoritative sequencer; every client is a speculative replica

**Status:** Accepted  ·  **Shown on views:** 01, 08, 13

*When two clients edit the same object in the same 50 milliseconds, what decides the outcome — a data type that cannot disagree, or a process that is allowed to?*

**Context.** There are two honest answers to concurrent editing. The first is a conflict-free replicated data type: clients merge each other's changes directly, convergence is a mathematical property of the data structure, and no server needs authority at all. The second is a single sequencer: one process per document assigns a total order, and everything else follows it. The first makes offline and peer-to-peer trivially correct and pays in metadata that grows with edit history, tombstones that are never safely collectable, undo semantics that stop matching user intent once two people share a history, and conflict outcomes that are correct but sometimes inexplicable to the person looking at the screen. The second gives small intent-shaped operations, a plain append-only history, per-user undo that means what a user expects, and one implementation of the conflict rules — and introduces an authority that has to be elected, leased, fenced and failed over. The product context decides it: a hundred people in one file during a working day, every edit attributable, revocation that must bite in seconds, and a server that must be able to refuse an operation that breaks a document invariant. A server that cannot interpret the document cannot enforce anything about it.

**Decision.** Each document has exactly one authoritative session at a time. That session holds the document in memory, owns the editor sockets, assigns a monotonic sequence number to every accepted operation, and is the only writer to the document's log. Clients apply their own operations optimistically before the session has seen them and reconcile when the authoritative order differs. Convergence is a property of the sequencer plus a specified set of conflict rules, not of the data type.

**How it is realised on AWS.** Session hosts run as a memory-optimised pool on AKS across three availability zones. A client's connection is routed to the host currently holding the document's lease; the host validates, sequences and resolves, appends to the document's partition in Cosmos DB, and only then acknowledges. Clients keep an unacknowledged-operation buffer and a parent sequence number on every operation, which is how the host distinguishes a concurrent edit from a stale one without per-object vector clocks.

| Option | Verdict | Reasoning |
|---|---|---|
| One leased authoritative sequencer per document, speculative clients | Chosen | Small operations, plain history, enforceable invariants, one implementation of the conflict rules |
| CRDT merged peer-to-peer or through a dumb relay | Right elsewhere | Right for offline-first, local-first products and for documents that must sync without a trusted server; wrong where the server must enforce invariants and revoke access mid-session |
| Server-side CRDT with a relay that never interprets operations | Rejected | Keeps the metadata cost and still cannot refuse an operation on the document's own semantics |
| Pessimistic locking per object | Rejected | Makes 'two designers, one file' a queue; the product is the parallelism |

**What it buys**

- An edit renders before the network sees it, so the latency budget is about acknowledgement rather than about how the product feels
- Operations stay small and intent-shaped, which is what makes history, per-user undo and attribution expressible
- The server understands the document, so it can refuse an operation that would create a cycle, an orphan or a structure no client can render

**What it costs**

- Document authority becomes a distributed-systems problem: a lease, a fence, a re-host path and a split-authority failure class (ADR-03)
- The session host is both socket and sequencer, which concentrates blast radius in the component most often deployed (ADR-05)
- Offline editing is not free the way it is with a CRDT: rebasing buffered work is a designed path with its own refusals (ADR-12)

**Choose differently when.** If the product were local-first — files that live on a device, sync opportunistically, and have no server-side invariants or revocation requirement — a CRDT would be the right answer and this record would be the wrong one. The decision rests on three product facts: a hundred collaborators in one document, edits that must be attributable, and access that must be withdrawable while the file is open.

**Why it holds up over time.** The boundary is a statement about where decisions are made, not about a technology. Session hosts, the store behind the log and the socket transport can all be replaced without changing it, and the fencing mechanism that protects it is older than this decade's infrastructure and will outlive the next one's.

> **Lesson.** Convergence without authority is a property of the data; convergence with authority is a property of an operation. Choose by what the server must be able to refuse, not by which merge algorithm reads better in a paper.

#### ADR-02 · Operations describe intent on the document graph, never rendered diffs or whole-document writes

**Status:** Accepted  ·  **Shown on views:** 10, 12

*What is the unit the platform orders, stores and replays — a change of meaning, or a change of bytes?*

**Context.** A collaborative editor can send three things over the wire. A whole-document write is simplest and makes concurrent editing impossible to merge. A diff of rendered output (geometry, pixels, serialised scene) is easy to produce from a renderer and ties the log permanently to that renderer's version: a history of pixel diffs cannot be reinterpreted, searched, attributed meaningfully or undone per user. An operation that names intent — create this node under that parent, set this property, reorder these siblings — is more work to define and is the only one of the three from which history, merge, undo and attribution all follow naturally. The cost of intent is a vocabulary that has to be versioned, because a client on an older schema must not be allowed to write operations the current schema cannot interpret.

**Decision.** The platform accepts only intent operations over the document graph: create, delete, reparent, set property, reorder. Every operation carries the document's schema version. A whole-document overwrite is not an edit path; it exists only as an administrative import. Operations that are structurally invalid, reference unknown objects, or would create a cycle are refused with a typed error at admission.

**How it is realised on AWS.** The operation vocabulary is defined once in the monorepo and compiled into both the client and the session host, so the two can never disagree about what an operation means. Each operation row in Cosmos DB holds the document id, sequence number, epoch, author, operation kind and payload. The schema version is on the document; a client handshake declaring an unsupported version is refused with an actionable error rather than silently tolerated.

| Option | Verdict | Reasoning |
|---|---|---|
| Intent operations over the document graph, versioned vocabulary | Chosen | The only option from which merge, history, undo and attribution all follow |
| Rendered or serialised scene diffs | Rejected | Cheap to produce, impossible to reinterpret; ties the permanent record to a renderer version |
| Whole-document writes with optimistic concurrency | Rejected | Turns every concurrent edit into a conflict and every save into a document-sized write |
| Text-style character operations generalised to a canvas | Right elsewhere | Right for documents that really are sequences; a canvas is a tree with ordered siblings, not a string |

**What it buys**

- The four conflict classes can be specified and resolved identically on every client, because they are about structure rather than bytes
- History is readable: a version, an undo and an audit trail all mean something to a human
- A future renderer can reinterpret an old document, because the log records what was meant rather than what was drawn

**What it costs**

- The operation vocabulary is a versioned interface with its own compatibility rules, and getting it wrong is expensive after launch
- Some edits a renderer finds trivial (a bulk restyle) become many operations, which is why batch limits and plugin quotas exist (ADR-08)
- The server has to understand enough of the document to validate structure, which couples it to the document model

**Choose differently when.** If the product's documents were opaque binaries the server had no business understanding — a video project file, a CAD assembly behind a proprietary kernel — then shipping diffs and keeping the server dumb would be the right answer, and merge would have to be solved in the client or not at all.

**Why it holds up over time.** Intent outlives implementations. A log of 'reparent this object' can be applied by a renderer nobody has written yet, which is the property that lets a document from 2026 open in 2036. A log of geometry diffs is tied to the code that produced it.

> **Lesson.** Record what the user meant, not what the screen did. Everything a collaborative product later wants — history, undo, attribution, merge — is derivable from intent and nearly impossible to recover from pixels.

#### ADR-03 · Ownership is a lease with a monotonic epoch, and the fence is enforced at the log's conditional write

**Status:** Accepted  ·  **Shown on views:** 15, 19, 23

*What stops two hosts believing they own the same document, and where is that enforced?*

**Context.** A single-authority design is only as good as its guarantee of singularity. Orchestration cannot provide it: a container runtime that promises one replica still produces two processes when a node is partitioned rather than dead, and the partitioned one keeps accepting edits it cannot commit. Consensus on every operation would provide it and would put a quorum round inside the acknowledgement budget. The standard answer is older than any of this infrastructure: a lease with a monotonically increasing token, where the store that holds the data rejects writes carrying a stale token. That turns 'exactly one owner at all times' — which no lease can provide — into 'at most one owner can ever write', which is the property that actually matters.

**Decision.** Document ownership is a 15-second lease held in a strongly consistent store, renewed every 5 seconds, carrying a monotonically increasing epoch. Every append to a document's log carries the writing host's epoch, and the log rejects any append whose epoch is lower than the document's recorded epoch. A brief double-ownership is therefore harmless: the old owner's writes fail, it learns it has been fenced, and it detaches its clients.

**How it is realised on AWS.** The lease is a single Cosmos DB item per document, claimed and renewed with an ETag-conditional write; the epoch is part of the item and of the document's log head. Appends are conditional on the recorded epoch, so the fence is enforced by the store on the same round trip as the write — no extra latency, and no service in the path that could be bypassed. The session router claims the lease on behalf of a host and then leaves the data path entirely.

| Option | Verdict | Reasoning |
|---|---|---|
| Lease plus monotonic epoch, fenced at the store | Chosen | Makes transient double-ownership harmless and keeps the acknowledgement path to one round trip |
| Consensus (quorum append) per operation | Rejected | Correct and too slow: a quorum round inside a 120 ms p99 budget leaves nothing for the rest |
| Orchestrator-guaranteed singleton (StatefulSet, singleton controller) | Rejected | Guarantees scheduling intent, not exclusion; a partitioned pod is still running |
| Consistent-hash placement with no lease | Right elsewhere | Right for caches and for stateless shards where a double-owner is merely wasteful rather than incorrect |

**What it buys**

- A 15-second lease is enough, which keeps coordination off the hot path entirely
- The split-authority failure class resolves to 'nothing the user sees' because the losing write is rejected rather than reconciled
- Re-host is cheap: claim the lease, increment the epoch, rehydrate from checkpoint plus tail — p95 10 s including client reattach

**What it costs**

- A lease-store outage stops handover, and the platform must then fail closed on saving rather than permit two authorities
- Every append carries an epoch and a conditional check, so the log's write path is slightly more expensive and considerably less generic
- Up to 15 seconds can pass between a host dying and its document becoming writable again — visible as a pause

**Choose differently when.** If the store behind the log could not enforce a conditional write on a monotonic value, the fence would have to move into a service and this design would change shape — most likely towards consensus, and towards a very different latency target. The decision depends on the store being able to say no.

**Why it holds up over time.** Fencing tokens are a 1990s idea that every generation of infrastructure rediscovers. Lease durations will move with network reality and the store will be replaced; the property — the data store is the last line of exclusion — will not.

> **Lesson.** Do not try to guarantee that only one process is running. Guarantee that only one process can write, and let the loser find out from the store.

#### ADR-04 · An operation is acknowledged only after durable commit; the platform fails closed on saving

**Status:** Accepted  ·  **Shown on views:** 10, 13, 23

*What does the acknowledgement a client receives actually promise — that the server received the edit, or that the edit cannot be lost?*

**Context.** A collaborative editor's implicit contract is that there is no save button because saving always already happened. That contract is broken the moment an acknowledgement means 'received': a host crash between acknowledgement and commit loses work the user was told was safe, and the user has no way of knowing which work. The alternative — acknowledge after durable, replicated commit — puts a store round trip inside the latency budget and makes the store's availability the edit path's availability. The second cost is the one people resist: when the log cannot be written, the honest behaviour is to refuse edits and say so, not to keep accepting them into memory and hope. A buffer that absorbs an outage is indistinguishable from a buffer that loses the outage's work.

**Decision.** An operation is acknowledged to its author only after it is durably committed and replicated in-region. When the log is unwritable, affected documents enter read-only within 5 seconds and the product says so explicitly; viewing, presence and reading continue. RPO for acknowledged operations is zero in-region.

**How it is realised on AWS.** Appends go to the document's Cosmos DB partition with strong consistency inside the region, which commits to a replica quorum before returning; the acknowledgement follows that return. Write failures are surfaced as a session state change (`read-only, cannot save`) rather than as a retryable error the client hides, and the client keeps its unacknowledged operations buffered so nothing is lost when the store returns.

| Option | Verdict | Reasoning |
|---|---|---|
| Acknowledge after durable replicated commit; fail closed on saving | Chosen | The acknowledgement means what users believe it means |
| Acknowledge on receipt, commit asynchronously | Rejected | Faster and occasionally dishonest; the loss is silent and unbounded in a crash |
| Acknowledge on receipt but buffer and retry during a store outage | Rejected | Converts a visible read-only period into an invisible data-loss risk |
| Group commit with a few milliseconds of batching | Deferred | A legitimate optimisation under burst; it changes the latency profile, not the contract |

**What it buys**

- RPO 0 for everything the user was told was safe, which is the only RPO a user can verify
- A store outage is a visible, bounded degradation rather than a quiet loss discovered later
- The latency budget has one dominant term — the commit — which makes it easy to reason about and to measure

**What it costs**

- Edit-path availability is bounded by the log's availability, so the log must be zone-redundant and its capacity planned for burst
- Users meet a read-only state they never saw in competing products that buffer silently, and support has to explain it
- Acknowledgement latency cannot be improved below the store's commit latency without weakening the promise

**Choose differently when.** If the product tolerated losing a few seconds of work in a crash — a brainstorming whiteboard, an ephemeral session — acknowledging on receipt would be defensible and would buy a faster, simpler platform. For a file people are paid to produce, it is not.

**Why it holds up over time.** The rule survives every change of store: a user may only be told their work is safe when it is. It is also the rule whose violation is invisible until the day it is not, which is why it belongs in the architecture rather than in a service's configuration.

> **Lesson.** Decide what your acknowledgement means, write it down, and make the degraded path honest. Buffering an outage behind a cheerful acknowledgement is the most expensive kindness in distributed systems.

#### ADR-05 · Conflict outcomes are specified per class, and convergence is a release gate rather than a claim

**Status:** Accepted  ·  **Shown on views:** 04, 13, 20

*How does anyone know that this build still converges, given that divergence is correct-looking on every individual client?*

**Context.** Convergence bugs are the worst class of defect this platform can ship. They are invisible locally — each client is internally consistent — and they are discovered by a user whose file is quietly wrong, often hours later, with no error anywhere. Ordinary unit tests do not find them, because the failure depends on the order and interleaving of operations across clients. Worse, 'the server decides' is not by itself a convergence guarantee: it guarantees a total order, while the outcome also depends on every client applying the same rules to that order. If one client platform resolves an edit-after-delete differently, the documents diverge while both obey the order.

**Decision.** The four conflict classes — concurrent set of the same property, edit after delete, concurrent reparent, concurrent reorder within a container — are specified once, with a defined outcome and a defined notification to the losing author. Randomised operation-set replay is a build gate: the same operation sets, delivered in many randomised orders, must produce byte-identical document state across implementations. A build that fails it does not ship, and divergence detected in canary stops the rollout.

**How it is realised on AWS.** The conflict rules live in shared code compiled into both client and host. CI replays recorded and generated operation sets — 10,000 per run — in randomised orders and asserts identical serialised state, alongside operation fuzzing and an acknowledgement-latency budget check. Canary covers 1% of new sessions with divergence watch on client checksums (ADR-17); a single mismatch stops the line.

| Option | Verdict | Reasoning |
|---|---|---|
| Specified conflict classes plus randomised replay as a gate | Chosen | Turns convergence from a claim into a property test that keeps working as the vocabulary grows |
| Trust the total order and test conflicts by example | Rejected | Example tests pass while the interleaving that matters goes untried |
| Formal verification of the merge rules | Deferred | Valuable for the rule set itself; does not cover the implementations that have to agree |
| Detect divergence in production only | Rejected | Necessary (ADR-17) and far too late on its own |

**What it buys**

- A convergence regression is caught by CI rather than by a customer whose file is wrong
- The losing author is always told, which is what keeps users trusting the tool when a conflict does happen
- The gate keeps working as the operation vocabulary grows, because it is a property test rather than a set of fixtures

**What it costs**

- Release velocity on the session host is bounded by the gate's runtime, and the gate must be fast enough to run on every build
- Shared rule code couples client and server release cycles more tightly than a protocol boundary normally would
- Randomised testing finds rare failures rarely: a flake-free gate requires fixed seeds and reproducible failures

**Choose differently when.** If the document model were a CRDT with a proven merge, the gate would still be worth having but would be checking an implementation rather than a design. With a sequencer and hand-specified rules, the gate is the only thing standing between a plausible refactor and a divergent fleet.

**Why it holds up over time.** Property-based replay is independent of the operation vocabulary, the language and the store. It will outlive every component in this set, and it is the cheapest permanent answer to a class of bug that otherwise reaches users.

> **Lesson.** If a defect is invisible on the machine where it happens, no amount of care replaces a gate. Make the property executable and put it in the build.

### Paths and transport

*Which traffic is allowed to be lost, and where each kind of client terminates.*

#### ADR-06 · Editors terminate on the sequencing host; viewers and presence leave through managed fan-out

**Status:** Accepted  ·  **Shown on views:** 02, 16, 17

*Should the platform own 300,000 connections, or buy them — and does the answer differ by what the connection carries?*

**Context.** Two honest options exist for real-time transport. A managed fan-out service removes connection scaling, TLS, backpressure, rolling deploys and mass-reconnect handling from the team's problem, and adds a hop between the client and whatever process makes decisions, plus a vendor's quotas and a shedding policy the team does not control. Terminating the connections on the session host itself puts the socket and the sequencer in one process — the lowest possible acknowledgement latency, and a materially harder operational story at 300,000 connections. The traffic, though, is not homogeneous. Editor operations need the sequencer; viewer broadcast and presence need nothing but delivery, and they are three-quarters of the connections and the overwhelming majority of the messages.

**Decision.** Editor connections terminate on the session host that sequences the document, so an acknowledgement costs one hop. Viewer connections and all presence traffic go through a managed real-time service, published to per-document groups by the session host. A document's viewers therefore consume none of the capacity of the node that orders its edits.

**How it is realised on AWS.** Front Door terminates TLS and routes an editor's WebSocket to the AKS session host holding the document's lease; the host publishes presence and viewer frames to an Azure Web PubSub group named for the document. Web PubSub carries the 1.5 million messages a second of presence and the viewer fan-out; the session pool carries operations and acknowledgements only.

| Option | Verdict | Reasoning |
|---|---|---|
| Editors on the session host, viewers and presence on managed fan-out | Chosen | Spends the one-hop budget where latency is perceived and buys scale where it is not |
| Everything through managed fan-out | Rejected | Adds a hop to every acknowledgement and hands the shedding policy to a vendor |
| Everything on self-hosted sockets | Right elsewhere | Right for a team that already operates connection infrastructure at this scale, and wants one policy for all traffic |
| Managed fan-out with a sidecar sequencer | Rejected | Two moving parts per document and the hop anyway |

**What it buys**

- Acknowledgement latency is one hop plus one commit, which is what makes the p99 120 ms target credible
- A popular document with 1,000 viewers does not threaten the node sequencing its edits
- Presence volume scales on someone else's capacity plan, and its cost is a line item rather than a node count

**What it costs**

- Two transports to operate, monitor and reason about, and two places a client can be connected
- Ordering between a cursor and the edit it precedes is lost across the two planes — accepted deliberately (ADR-07)
- Session hosts must handle rolling deploys with live sockets, which is why releases are drains (ADR-17)

**Choose differently when.** If the managed service's latency ever fell inside the acknowledgement budget — or if the team found itself operating a connection tier for other reasons anyway — the split would stop paying for itself and one transport would be simpler. The decision is a latency measurement, and should be re-measured, not re-argued.

**Why it holds up over time.** The principle is to spend a one-hop path where latency is perceived and buy scale where it is not. Which service carries the fan-out is a procurement decision; the split between the two kinds of traffic is architectural.

> **Lesson.** Decide transport per kind of traffic, not per system. Buying scale for the traffic that may be lost is how you afford to own the path that may not.

#### ADR-07 · Presence is sacrificial: coalesced, unlogged, zero-retention, and the first thing shed

**Status:** Accepted  ·  **Shown on views:** 16, 18, 23

*Is a cursor position a cheap kind of edit, or a different kind of thing altogether?*

**Context.** Presence traffic — cursors, selections, viewports, who is typing — is the highest-volume thing in a collaborative editor and the least valuable per message. At 20 Hz per client across 300,000 clients it dwarfs the operation stream by an order of magnitude. Treated as a lightweight edit it competes with operations for the same buffers, the same backpressure and the same durability machinery, and it is the reason collaborative editors sometimes get slower when more people watch. Treated as sacrificial it becomes the platform's shock absorber: the thing that can be dropped, coalesced or delayed to protect what cannot.

**Decision.** Presence is lossy and non-durable by contract. It may be dropped, coalesced, rate-limited or reordered; it is never written to the operation log; it is retained for zero seconds beyond delivery; and it is the first thing shed when any part of the system is under load. The server coalesces to at most 10 Hz per user whatever the client sends, and the shedding order — presence, then viewer broadcast, then non-essential telemetry, never edits — is declared rather than emergent.

**How it is realised on AWS.** The session host coalesces each user's presence to 10 Hz and publishes to the document's Web PubSub group. Coalescing rates, viewport-based filtering and viewer broadcast policy are configuration rather than code, so a cost or load response is a setting change. Presence visibility is filtered by role, so a comment-only participant does not learn what an editor has selected where the sharing model forbids it.

| Option | Verdict | Reasoning |
|---|---|---|
| Sacrificial, coalesced, unlogged presence on its own channel | Chosen | Protects the edit path and makes presence the cheapest thing to give up |
| Presence as low-priority operations on the edit channel | Rejected | Keeps cursor-to-edit ordering nobody needs and puts 1.5 M messages/s in the durable path |
| Peer-to-peer presence between clients | Deferred | Attractive for cost, awkward for permission filtering and for clients behind restrictive networks |
| Durable presence history for session replay | Right elsewhere | Right for products that sell session recording; here it would turn the cheapest traffic into the largest store |

**What it buys**

- The edit path has a shock absorber that costs nothing to give up, and the shedding order is known before an incident rather than chosen during one
- Presence is a privacy non-event: nothing is stored, so nothing leaks later
- Presence cost is controllable without a deploy, which makes it a budget rather than a surprise

**What it costs**

- Cursors freeze under load, which users notice and occasionally report as an outage — it must be explained in the product, not just in a runbook
- The ordering relationship between a cursor and the edit it precedes is genuinely lost
- A 10 Hz cap is a product compromise: smooth-cursor interpolation moves to the client

**Choose differently when.** If presence became load-bearing — multiplayer audio, live spatial interaction, a game-like surface where the cursor is the product — it would need its own availability target and this record would have to be split in two. Richer-but-still-sacrificial presence (video pips, reactions) does not change it.

**Why it holds up over time.** Presence will get richer with every generation of hardware and will stay the traffic that may be dropped. A platform that has already declared its shedding order gets that richness nearly free; one that has not pays for it in acknowledgement latency.

> **Lesson.** Name the traffic you are willing to lose, before load makes the choice for you. A declared shedding order is a design artefact; an emergent one is an incident report.

#### ADR-08 · The document is the partition key everywhere, and a document fits in one session process

**Status:** Accepted  ·  **Shown on views:** 12, 17, 19

*What is the unit of partitioning, and what happens to the design when one document outgrows it?*

**Context.** Every scaling property in this platform follows from the partition key. Partitioning by document gives independent logs, independent sequencers, a failure domain of one document per host, and no cross-document transaction anywhere. It also makes the single document the hard limit: whatever a session process can hold and sequence is the ceiling for a file, and a 500 MB document with 200 editors is a different machine from the median 1.2 MB one. Partitioning by workspace or project would amortise overhead across documents and would couple unrelated files into one failure and contention domain — the classic noisy-neighbour trade, except here the neighbours are in the same room.

**Decision.** The document is the partition key from edge to storage: routing, session ownership, the operation log, checkpoints, rate limits and cost attribution are all per document. A document's authoritative state must fit in one session process; the product enforces a published ceiling of 500 MB including assets, and the architecture's escape hatch for anything larger is subtree-level sequencing, designed but not built.

**How it is realised on AWS.** Cosmos DB partition key is the document id, which bounds a document's throughput to one logical partition's budget and keeps hot documents isolated. Session placement hashes on document id through the lease; rate limits, quotas and cost attribution all key on it. Asset bytes are deliberately outside this accounting (ADR-13), which is what keeps a 100 MB image from consuming a document's operation budget.

| Option | Verdict | Reasoning |
|---|---|---|
| Partition by document; one document per session process | Chosen | Independent failure and contention domains at the granularity users actually share |
| Partition by workspace or project | Rejected | Amortises overhead and couples unrelated documents into one contention domain |
| Partition by subtree from the start | Deferred | The designed escape hatch; it multiplies the number of authorities per document and the conflict rules with it |
| Shard the hottest documents across hosts with cross-host ordering | Rejected | Reintroduces the problem the single sequencer exists to avoid |

**What it buys**

- A host failure affects only the documents it held, and a hot document cannot starve another
- Rate limits, quotas and cost all key on the thing users share, which makes abuse and spend explainable
- No feature can quietly require a cross-document transaction, because none exists

**What it costs**

- The largest documents are the least comfortable, which is the opposite of what enterprise customers expect
- A per-document throughput ceiling exists and must be published, not discovered
- Moving a component library between documents atomically is a new design rather than a new query

**Choose differently when.** If the real distribution of document sizes shifted — a market where 300 MB files with 150 editors are normal rather than exceptional — subtree-level sequencing would stop being an escape hatch and become the primary design, with a substantially larger conflict-rule surface.

**Why it holds up over time.** Partitioning by the thing people share is a durable heuristic: it survives changes of store, runtime and transport, and it keeps the blast radius aligned with the user's mental model of 'my file'.

> **Lesson.** Choose the partition key to match the unit people collaborate in, then publish the ceiling that choice implies. A limit stated in the product is a feature; the same limit discovered in production is an outage.

### State and history

*What is the record, what is derived, and what the platform owes a client that was away.*

#### ADR-09 · The per-document operation log is the system of record, held in a partitioned document store rather than a stream broker

**Status:** Accepted  ·  **Shown on views:** 02, 11, 12

*What holds 900 million independent append-only logs, one per document, each read by at most a few hundred clients?*

**Context.** The obvious home for an append-only operation log is a stream broker, and for a few thousand high-volume streams it is the right one. This platform has 900 million logs, almost all of them tiny and idle, a handful of them busy, and every one of them needing a cheap conditional append, a cheap range read from a sequence number, and a TTL. Brokers price and partition for throughput per topic, not for a very large number of independent, mostly-idle, individually addressable logs; the per-topic overhead and the partition count make 900 million of them a non-starter. A partitioned document store inverts that: a partition per document is free when idle, supports a conditional insert that enforces both uniqueness of sequence and the epoch fence (ADR-03), and reads a range by key with no scan.

**Decision.** The operation log is the system of record and lives as one logical partition per document in a strongly consistent document store, keyed by (document_id, seq), with the epoch on every row and a TTL that implements the 30-day horizon. Checkpoints in object storage are derived. No stream broker sits on the edit path; the change feed of the log is what feeds derived consumers.

**How it is realised on AWS.** Cosmos DB for NoSQL with the document id as partition key and strong consistency inside the region; appends are conditional writes that fail on a stale epoch or a duplicate sequence. Its change feed drives the search feed, the audit export and the rebuild verifier. Checkpoints and assets live in Blob Storage (RA-GZRS); PostgreSQL holds the control plane, which has entirely different access patterns and recovery requirements.

| Option | Verdict | Reasoning |
|---|---|---|
| Per-document partition in a strongly consistent document store | Chosen | A free idle log, a conditional append that fences, and a range read by key |
| Stream broker (Kafka, Event Hubs) as the log | Rejected | Priced and partitioned for throughput per topic, not for 900 million mostly-idle logs |
| Relational table partitioned by document | Right elsewhere | Right at a smaller document count, where one engine for log and control plane is a real operational saving |
| Append-only blobs per document | Rejected | Cheapest storage, no conditional append semantics worth trusting for a fence |

**What it buys**

- An idle document costs storage only — no topic, no partition, no consumer group
- The fence and the sequence-uniqueness check ride the same write as the append, so correctness costs no extra latency
- Derived state has one source and one rebuild path, which is why checkpoints and indexes are never backed up

**What it costs**

- Request-unit economics have to be modelled per operation, and a hot document's throughput is bounded by its logical partition
- The 30-day horizon is a storage and cost decision with a product consequence for very stale clients (ADR-12)
- Replaying a long log is a read-amplification problem the checkpoint cadence exists to bound (ADR-10)

**Choose differently when.** If document count fell by three orders of magnitude while per-document throughput rose — a handful of enormous shared canvases rather than hundreds of millions of files — a broker or a log-structured service would become the better fit, and the per-document partition would become overhead.

**Why it holds up over time.** The record is 'an append-only, per-document, replayable log', not a product name. Any store that offers a cheap conditional append and a keyed range read can hold it, which is a property stores keep acquiring rather than losing.

> **Lesson.** Match the storage engine to the shape of the key space, not to the word in the requirement. 'Append-only log' describes semantics, not a broker.

#### ADR-10 · The session writes its own checkpoints; an independent verifier disagrees with them

**Status:** Accepted  ·  **Shown on views:** 08, 11, 19

*Who materialises the document state — the process that already holds it, or a separate service that reads the log?*

**Context.** Checkpoints decide how fast a cold document opens and how much log a re-host has to replay. Someone has to fold the log into a document state, and there are only three candidates. The session already holds the state in memory, so its checkpoint is nearly free, and it is also the process with the least spare attention during a burst. A background service reading the log is cleanly separated and needs a second implementation of document application — which means a second set of bugs, and the possibility that the two implementations disagree about what the log means, which is exactly the class of defect this platform most fears. A client could do it cheapest of all, and cannot be trusted. The uncomfortable part of choosing the session is that nothing then independently checks the fold.

**Decision.** The session host writes the checkpoint it already holds, every 30 minutes of activity and on release. A separate rebuild verifier replays the log independently, on a schedule and not only after an incident, and diffs its result against the live checkpoint; a mismatch is an alert. There is one implementation of document application on the write path and one deliberately independent one off it.

**How it is realised on AWS.** The checkpoint writer serialises the in-memory document to Blob Storage keyed by document and sequence number. The verifier runs as an AKS job driven by the log's change feed, samples documents (all documents above a size or activity threshold, a random sample below it), rebuilds from the previous checkpoint plus the tail, and compares. Rebuild diff rate is a first-class metric on the observability grid.

| Option | Verdict | Reasoning |
|---|---|---|
| Session writes the checkpoint; independent verifier diffs it | Chosen | One implementation on the write path, with an independent opinion where a disagreement is cheap |
| Background service as the only checkpoint writer | Rejected | Clean separation at the price of a second application implementation on the critical path |
| Client-written checkpoints | Rejected | Cheapest and untrustworthy: a checkpoint is the document, and the client is a replica |
| No checkpoints; replay the whole log | Right elsewhere | Right for short-lived documents; here a year-old file would replay millions of operations to open |

**What it buys**

- Cold open is bounded by 30 minutes of log, which is what makes the 2.5 s p95 on a 50 MB document reachable
- A one-way application bug is caught by the verifier rather than by a user whose file is wrong
- Re-host replays at most the same bounded tail, which is what keeps the 10 s re-host target honest

**What it costs**

- Checkpointing competes with editing on the busiest host, so it must be incremental and interruptible
- The verifier is a second implementation after all — deliberately, and off the write path, but it still has to be maintained
- 30 minutes of cadence against 30 days of retention is a storage bill that scales with active documents

**Choose differently when.** If checkpointing ever measurably hurt acknowledgement latency on busy documents, the writer would move off the session and the second implementation would become the price of that — paid consciously, with the verifier then comparing two independent folds rather than checking one.

**Why it holds up over time.** The pattern — materialise from the authoritative log, keep an independent rebuild that disagrees loudly — survives any storage technology, and it is the mechanism that makes a derived store droppable rather than precious.

> **Lesson.** If you only have one implementation of the thing that defines your data, build a second one somewhere it can argue cheaply. A disagreement you can see is worth far more than an invariant you assert.

#### ADR-11 · Restore is a forward operation; history is never rewritten, and undo is per user

**Status:** Accepted  ·  **Shown on views:** 03, 12

*When a user restores yesterday's version, does the document travel back in time, or does it move forward to a state that looks like yesterday?*

**Context.** Restoring by rewriting history is the intuitive implementation and it breaks three things at once. Every connected client's document version becomes meaningless, so they must all be reloaded. Every reference into the log — a comment anchor, an audit record, a named version, another client's buffered operations — can dangle. And the act itself leaves no trace, so 'who reverted my work' has no answer. Undo has the same shape in a shared document: a global undo stack means one person's undo can reverse someone else's edit, which users experience as sabotage, while a per-user undo must sometimes refuse, because the object it would restore has since been deleted or restructured by someone else.

**Decision.** A restore is a new forward operation that makes the document equal to an earlier state; the log is append-only and never rewritten. Undo is per user and expressed as a compensating operation, and it is refused with a clear reason when the object it would touch has since been deleted or structurally changed by another user. Named versions, comments and audit records therefore never dangle.

**How it is realised on AWS.** Restore computes the difference between the current head and the target checkpoint and emits it as operations, so connected clients receive it exactly as they receive any other edit — no reload, no version discontinuity. Undo looks up the user's own most recent operation, checks applicability against current state, and emits the inverse or a typed refusal. Named versions are rows pointing at a sequence number, so naming a version mutates nothing.

| Option | Verdict | Reasoning |
|---|---|---|
| Restore as a forward operation; append-only history; per-user undo | Chosen | No dangling references, no forced reloads, and an auditable record of the restore itself |
| Truncate or rewrite the log to the restore point | Rejected | Breaks live clients, comment anchors and audit, and erases the evidence of the restore |
| Global undo stack | Rejected | Lets one user reverse another's work, which reads as a bug however correct it is |
| Branch on restore, with an explicit merge | Right elsewhere | Right for code and for products that teach branching; heavy for a design file people expect to be one file |

**What it buys**

- A restore is visible, attributable and itself undoable, because it is just another edit
- Live collaborators see a restore as a large edit rather than as a disconnection
- Comment anchors and audit records are permanently valid, since nothing they point at can disappear

**What it costs**

- A restore of a large document is a large burst of operations, which must be rate-shaped so it does not look like abuse
- Per-user undo has refusal cases the product has to explain in words, which is more interface work than a global stack
- The log grows with restores, which is a storage cost rather than a correctness one

**Choose differently when.** If the product adopted explicit branching — a design file with named branches and merges — restore would become a branch operation and this record would be rewritten around merge semantics instead of compensation.

**Why it holds up over time.** Append-only history with compensation is the same decision ledgers and version control made, for the same reason: evidence is worth more than tidiness. It outlives any storage choice.

> **Lesson.** Never rewrite the record to change the present. Move forward to the state you want and leave the reason behind you.

#### ADR-12 · Offline work is rebased within the log horizon and preserved as a recoverable branch beyond it

**Status:** Accepted  ·  **Shown on views:** 06, 14, 23

*What happens to an hour of offline editing when the document it was based on has moved on — and what happens after thirty days?*

**Context.** A client that edited offline comes back holding operations whose parent sequence is in the past. Inside the retained log the platform can rebase them: apply the live conflict rules as if they had arrived late, which is technically convergent and can be semantically strange — restyling a frame someone has since redesigned, or editing objects whose meaning has changed. Beyond the log horizon there is nothing left to rebase against, because the operations the client missed have been compacted away. Three outcomes are available: merge and accept the oddity, refuse and discard, or preserve the work somewhere the user can get at it. Only one of those is acceptable for work a person spent an hour on.

**Decision.** Buffered offline operations are rebased into the authoritative order under the same conflict rules as live editing while the client's version is inside the 30-day retained log. Beyond the horizon, or when the operations cannot be merged without violating document invariants, the work is preserved as a recoverable branch and the user is told. Silent discard is not an available outcome.

**How it is realised on AWS.** On reattach the client sends its version; the host chooses replay (inside the log), snapshot-plus-tail (behind the log but inside the horizon), or branch (beyond it), and reports which path it took. A branch is written as a separate document derived from the client's buffer plus the last common checkpoint, visible to its author with an explanation. Operations that would violate an invariant are quarantined with the same mechanism and retained for support recovery.

| Option | Verdict | Reasoning |
|---|---|---|
| Rebase within the horizon, recoverable branch beyond it | Chosen | No silent loss, and the weird-merge case is bounded by a horizon the platform publishes |
| Always rebase, whatever the age | Rejected | Requires infinite log retention, and produces merges nobody can explain |
| Refuse and discard beyond a horizon | Rejected | Cheapest, and the one outcome that loses work a person did |
| Always ask the user to choose | Right elsewhere | Right for a developer tool; an interruption most designers should never have to see |

**What it buys**

- The 'offline on a train' journey has a defined ending in every case, including the ugly one
- The log horizon becomes a published product property rather than an internal retention setting
- Quarantine gives support something to recover from, instead of an apology

**What it costs**

- Branches are a product concept the interface must explain, in a product that otherwise has one file
- The client must hold at least 10,000 operations durably, which is real client-side engineering
- Semantically odd merges inside the horizon still happen and will generate support contacts

**Choose differently when.** If the document model were a CRDT, the horizon would stop mattering and the branch case would disappear — which is the strongest argument available against ADR-01 and is why that record names offline duration as the condition that would flip it.

**Why it holds up over time.** The rule — never silently lose work a person did, and bound the cases where merging is strange — is independent of merge technology, and is the part of this design a user would still recognise in ten years.

> **Lesson.** Decide in advance what happens to work that arrives too late to merge. Every system eventually has some; the ones that discard it quietly are the ones nobody trusts twice.

### Assets and large objects

*How bytes that dwarf the document stay out of the path that orders it.*

#### ADR-13 · Assets move out of band, content-addressed, with the document valid before the bytes arrive

**Status:** Accepted  ·  **Shown on views:** 05, 09, 11

*Where do a hundred megabytes of image bytes go, given that the operation that references them must be ordered in milliseconds?*

**Context.** A designer drops a 90 MB image into a frame. If those bytes travel as an operation they enter the ordering path, the log, the checkpoint and every client's catch-up, and one paste stalls a document for everyone in it. If they travel out of band the operation is a few hundred bytes containing a hash, and the bytes move client-to-storage over a pre-signed URL — at the cost of a window in which the document legitimately references content that does not exist yet. The platform must then decide what a document means during that window: blocked until complete, or valid with a placeholder.

**Decision.** Asset bytes never enter the session or the log. The document holds a content-addressed reference, the bytes move directly between client and object storage over short-lived pre-signed URLs, and an operation referencing an asset that has not finished uploading is accepted — the document renders valid-with-placeholder until the bytes arrive. Assets are deduplicated by hash within a tenant, immutable once written, and scanned before any rendition is served.

**How it is realised on AWS.** The asset service issues an upload intent with a pre-signed Blob Storage URL scoped to one content hash and a short expiry; Defender for Storage scans on write and the scan state lives on the asset row, so an unscanned asset may be referenced but not served. Renditions are generated asynchronously. A reference never grants access: entitlement is checked against the referencing document's grants on every read.

| Option | Verdict | Reasoning |
|---|---|---|
| Out of band, content-addressed, placeholder-on-pending | Chosen | Keeps the ordering path small and the paste instant, at the cost of a defined pending state |
| Assets as operation payloads | Rejected | One paste becomes everyone's problem, in the log, the checkpoint and every catch-up |
| Out of band but blocking the operation until upload completes | Rejected | Makes a slow network look like a broken file, and couples an edit to an upload |
| Assets inside the checkpoint | Right elsewhere | Right where documents are exported as self-contained bundles; here it would multiply storage by every checkpoint |

**What it buys**

- A 90 MB paste costs the document a few hundred bytes of ordering work
- Deduplication by hash within a tenant makes component libraries and repeated imagery cheap
- Immutability means a reference is permanently valid, which is what lets checkpoints and exports share bytes

**What it costs**

- A pending state exists and must be rendered honestly, which is the trough of the review journey
- Access control on assets is a second enforcement point with its own failure mode
- Deleting a document cannot blindly delete its assets: reference counting within the tenant is required

**Choose differently when.** If documents were small and asset-free — diagrams, text, data visualisations — inlining content would be simpler and the pending state would not need to exist. The decision is driven by the 100 MB asset and the 500 MB document ceiling.

**Why it holds up over time.** Content addressing and out-of-band bytes are how every system that moves large objects alongside small records ends up working. The storage service will change; the split will not.

> **Lesson.** Keep the big bytes off the path that has to be ordered. Then decide, explicitly, what your document means while they are still in flight.

### Identity and trust

*Who may edit, how fast that can be withdrawn, and who can never read.*

#### ADR-14 · Capability is bound at admission, re-evaluated on every grant change, and revocation bites on a live socket

**Status:** Accepted  ·  **Shown on views:** 21, 22, 23

*How does a permission change reach a connection that is already open and editing?*

**Context.** Most products resolve permission when a document is opened and never again, so a revocation takes effect at the next reload — which, in a tool people leave open for a working day, can be hours. The opposite extreme, checking the sharing service on every operation, puts a control-plane dependency inside a 120 ms acknowledgement budget and makes a sharing outage a product outage. Between them lies the honest answer: bind a capability to the connection at admission, and push grant changes to the sessions that hold affected capabilities. That requires the control plane to know which sessions care, and it requires a decision about what happens when the control plane is unreachable — because during that window the platform can either keep working with stale authority or stop working with correct authority.

**Decision.** Admission authenticates the user, resolves their effective role, and binds the resulting capability to the connection for its lifetime. Every change to grants, links or membership is pushed to affected live sessions, which downgrade or detach within 10 seconds and refuse operations arriving after authority is withdrawn. During a control-plane outage, existing sessions continue on cached capabilities for at most 5 minutes and new admissions fail closed.

**How it is realised on AWS.** The sharing service publishes grant-change events; session hosts subscribe for the documents they hold and apply downgrade or detach immediately. Capabilities are cached in Redis with a 5-minute ceiling, and tokens are renewed in-band on the live connection so a connection never outlives its credential. Every grant change, link creation, export and permanent deletion is an audit record.

| Option | Verdict | Reasoning |
|---|---|---|
| Bind at admission, push grant changes, 10 s revocation | Chosen | No control-plane dependency on the hot path, and revocation is a platform guarantee |
| Check permission on every operation | Rejected | Correct and unaffordable: a sharing outage becomes an editing outage |
| Resolve at open, re-check at next reload | Rejected | The common industry behaviour, and the reason admins do not trust 'remove access' |
| Short-lived capability tokens with frequent renewal | Deferred | A reasonable alternative that trades push machinery for renewal traffic; revocation latency becomes the token lifetime |

**What it buys**

- An admin can remove a leaver and see it take effect while the file is open — the journey that justifies the design
- The edit path has no synchronous control-plane dependency, so sharing can fail without stopping work
- Capability is enforced server-side per operation, so a client that keeps its editing affordances gains nothing

**What it costs**

- Grant-change fan-out is a second push path with its own delivery concerns
- The 5-minute grace window is a stated hole: during a control-plane outage a revocation can be late
- Sessions hold authority state, which makes them slightly less disposable than a stateless worker

**Choose differently when.** If the product served a single trust domain where revocation urgency was low, resolve-at-open would be adequate and the push path would be unnecessary complexity. Enterprise offboarding is what makes 10 seconds a requirement.

**Why it holds up over time.** Binding authority at admission and invalidating it by event is the same pattern as session revocation lists and token introspection, and it has outlived several generations of identity technology.

> **Lesson.** Decide how long a permission may be wrong, then design the revocation path to that number. 'At next reload' is a number too, and it is usually hours.

#### ADR-15 · Document content is confidential to its tenant, with no routine operator path and keys the customer can hold

**Status:** Accepted  ·  **Shown on views:** 17, 21

*Who can read a customer's document, and what does an engineer debugging a convergence bug get to see?*

**Context.** Debugging a convergence problem is far easier with the document in front of you, and that is exactly what makes it the moment confidentiality is lost. A platform that leaves a routine operator read path open will use it, log it nowhere in particular, and discover during an enterprise security review that it cannot describe who has read what. Closing the path entirely has a real cost: incident response gets harder, and some bugs can only be reproduced from a customer's own document. Customer-managed keys raise the same tension one level higher — they are a genuine control for the customer and a genuine availability risk for the platform, because a revoked key is an unreadable document.

**Decision.** Document content is confidential to its tenant. There is no routine operator content-access path; break-glass access requires approval, uses a separate identity, and produces an audit record that is itself immutable. Content is encrypted at rest with tenant-scoped keys, with customer-managed keys in a hardware security module available on the enterprise tier. Operation metadata — counts, sizes, latencies, attribution — is exported for analytics; content is excluded by construction rather than by policy.

**How it is realised on AWS.** Keys live in Key Vault Managed HSM, per tenant, with the enterprise tier pointing at a customer-held key. Private Link between the session tier and every store means no public data path exists. Break-glass is a distinct Entra identity with approval workflow; its use writes to the immutable audit log before the access happens. Debug tooling works from structural traces — operation kinds, sequence numbers, conflict classes — not payloads.

| Option | Verdict | Reasoning |
|---|---|---|
| Tenant-scoped keys, CMK on enterprise, approved and audited break-glass | Chosen | Closes the routine path and keeps an auditable exception for the cases that need one |
| Operator read access with logging | Rejected | Logging a path everyone uses is not a control, it is a record of its use |
| No break-glass at all | Rejected | Attractive until the first bug that only reproduces on one customer's document |
| End-to-end client-side encryption | Right elsewhere | Right where the server must never interpret content; incompatible with a server that validates document structure (ADR-01) |

**What it buys**

- The platform can answer 'who has read this document' with a record rather than an assurance
- Enterprise customers get a key they control, and the blast radius of a platform compromise is bounded per tenant
- Analytics and search feeds carry no content, so a second system can never become the leak

**What it costs**

- Debugging is harder and slower, and some reproductions require a customer's cooperation
- A customer-managed key is an availability dependency on the customer: a revoked or expired key is an unreadable document
- Per-tenant keys complicate asset deduplication, which must stay within a tenant boundary (ADR-13)

**Choose differently when.** If the product moved to end-to-end encryption, the server could no longer validate document structure or enforce invariants, and ADR-01 would have to be reopened — which is why these two records are linked rather than independent.

**Why it holds up over time.** Confidentiality boundaries drawn at the tenant, with approved and recorded exceptions, are what every compliance regime has converged on. The key-management product will change; the boundary should not.

> **Lesson.** Design the debugging path before the security review does. The question is not whether engineers can read customer data, but whether the platform can say who did.

### Scale and operations

*Where a document lives, and how the platform finds out it is wrong before a user does.*

#### ADR-16 · One home region per document, with announced migration rather than active-active editing

**Status:** Accepted  ·  **Shown on views:** 17, 23

*Where does a document live when its editors are in three time zones, and what happens when that region fails?*

**Context.** Active-active editing of one document across regions means two sequencers, which contradicts the central decision of this architecture (ADR-01). It could be made to work with a CRDT, or with cross-region consensus and its latency, and both change the platform rather than extending it. The alternatives are narrower and honest: give a document one home region and pay a distance penalty for far-away editors, or move the home to where the editors are and define what happens during the move. The regional failure case is separate and must be stated rather than implied: asynchronous replication means a declared loss window, and pretending otherwise is the most common lie in multi-region architecture.

**Decision.** Every document has one home region, pinned by its tenant's residency policy, with one write region for its log. A document's home may be migrated live with an announced pause of up to 3 seconds. Viewers are served from regional read paths. Regional failover is a declared operation with RTO 15 minutes and a replication loss window of up to 5 seconds, stated to users rather than hidden.

**How it is realised on AWS.** Cosmos DB with a single write region per document's home and asynchronous replication to the paired region; Blob Storage as RA-GZRS; a warm standby session pool and admission tier in the paired region. Migration drains the document's session, moves ownership, increments the epoch and reattaches clients — the same mechanism as a re-host (ADR-03), with a longer pause.

| Option | Verdict | Reasoning |
|---|---|---|
| One home region, live migration, declared failover loss | Chosen | Preserves the single sequencer and tells the truth about what a failover costs |
| Active-active editing across regions | Rejected | Requires two authorities or cross-region consensus; contradicts ADR-01 or the latency budget |
| Synchronous cross-region commit | Rejected | RPO 0 everywhere, and an acknowledgement budget nobody would accept |
| Follow-the-sun automatic migration | Deferred | Attractive once the proportion of genuinely cross-region documents is measured rather than assumed |

**What it buys**

- The single-sequencer guarantee holds globally with no new failure mode
- A failover's cost is a published number, which makes it a business decision rather than a surprise
- Viewers — the majority of sessions — are served locally regardless of where the document lives

**What it costs**

- A distant editor sees up to 400 ms acknowledgements, and will call the product slow
- Migration is a visible pause, and is therefore an operation rather than an optimisation
- The paired region is warm, not continuously exercised, so the first real failover is the first full-scale test

**Choose differently when.** If a measured majority of documents turned out to be genuinely multi-region — editors on two continents in the same hour, routinely — the cost of the latency tax would exceed the cost of a second authority, and the platform would have to revisit ADR-01 with a CRDT in hand.

**Why it holds up over time.** Choosing one writer per unit of data and being explicit about replication loss is the durable half of multi-region design. The services change; the arithmetic of distance does not.

> **Lesson.** Multi-region is a question about writers, not about deployments. Decide where the one writer lives, then publish what a failover costs.

#### ADR-17 · Divergence is detected by client checksum reporting, and a release never changes a live session's code

**Status:** Accepted  ·  **Shown on views:** 18, 20, 23

*How does the platform learn that one client's document no longer matches the server's, given that nothing is obviously broken?*

**Context.** Every other failure class in this platform announces itself: a lease expires, an append fails, a probe goes red. Divergence does not. A client whose state has drifted is internally consistent, renders happily, and keeps accepting edits, and the only witness is a human who eventually notices their file is wrong — usually long after the operations that caused it. Two mechanisms are available. One is detection: have clients prove they agree with the server at known points. The other is prevention of the most likely cause: two code versions applying the same operations differently during a rollout. The second is cheap because sessions are short-lived (ADR-19's lifecycle), so a rollout can wait for documents rather than restarting them.

**Decision.** Clients periodically report a checksum of their document state at a known sequence number; a mismatch raises an alert within 60 seconds and is repaired by re-snapshotting that client. A live session never changes code version: a rollout drains idle documents onto the new build and leaves busy documents until they release, with a forced release at the next idle window to bound version skew. A single divergence in canary stops the rollout.

**How it is realised on AWS.** The client sends a hash of its serialised state every few minutes and on key transitions; the session compares against its own state at that sequence. Mismatches are logged with client build, session build and document id, alerted on in Azure Monitor, and repaired by pushing a fresh snapshot to that client. Session hosts are version-pinned for a session's lifetime, and the release pipeline treats divergence rate as a stop-the-line signal.

| Option | Verdict | Reasoning |
|---|---|---|
| Client checksum reporting plus version-pinned sessions | Chosen | Detects the silent failure and removes its most likely cause |
| Trust the sequencer and the release gates | Rejected | Gates catch what they test; production finds the rest, and here production means a user's file |
| Full state comparison on every reconnect | Rejected | Thorough and expensive: a 60 MB comparison on every reattach |
| Hot-swap session code in place | Rejected | Saves a drain and reintroduces the exact condition this record exists to prevent |

**What it buys**

- The one failure class with no natural alarm gets one, within a minute
- Repair is automatic and nearly invisible: a brief reload for one client rather than an incident
- A rollout can never have two implementations applying one document, which removes the most plausible cause of divergence

**What it costs**

- A small permanent overhead in client message volume and server comparison work
- A long-lived document can hold an old build for hours, so version skew is bounded by policy rather than by deploy time
- A noisy checksum signal would be worse than none, so sampling and thresholds have to be tuned rather than guessed

**Choose differently when.** If the client and server shared a single compiled implementation of document application — one WebAssembly module used by both — the most likely divergence cause would disappear and checksum reporting could become a sampled audit rather than a continuous signal.

**Why it holds up over time.** Having the replica prove it agrees with the authority is how storage systems, caches and replicated databases all eventually detect silent corruption. It outlives any particular hash or transport.

> **Lesson.** For the failure that nothing else reports, make the client prove agreement at a known point. And never let two versions of your code own the same data at the same time.

## Every package used, in one table

Eleven terms that mean something specific in this design, and are used loosely almost everywhere else. Where a term has a more common alternative, the alternative and why it was not used are named.

| Package | What it is | What it does here | Considered instead |
|---|---|---|---|
| Operation | One intent-shaped change to the document graph — create, delete, reparent, set property, reorder — carrying the author, the client's parent sequence number and the document schema version. | The unit of admission, ordering, durability, history, undo and attribution. | 'Edit' or 'change', which hide the distinction between intent and a rendered diff — the distinction the whole design rests on (ADR-02). |
| Session | The single authoritative process holding one document in memory, owning its editor sockets and assigning its sequence numbers, for as long as at least one client is connected plus a 60-second grace. | The authority. Exactly one exists per document at a time, and it is the only writer to that document's log. | 'Server' or 'room', which suggest a place rather than an authority with a lease. |
| Epoch | A monotonically increasing integer issued with each ownership claim on a document, recorded on the document and on every appended operation. | The fencing token: the log rejects any append carrying an epoch lower than the recorded one, which is what makes transient double-ownership harmless. | 'Lease generation' or 'term'; the mechanism is the same one consensus systems call a term. |
| Speculative replica | A client that has applied its own operations locally before the authority has accepted them, and must be able to reconcile when the authoritative order differs. | The reason an edit feels instant, and the reason reconciliation is a product behaviour rather than an error path. | 'Optimistic client', which describes the write and not the obligation to reconcile. |
| Checkpoint | A materialised document state in object storage keyed by document and sequence number, written by the session every 30 minutes of activity and on release. | Bounds cold-open latency and re-host replay. Derived, droppable, rebuildable from the log. | 'Snapshot', used interchangeably here; 'save' would be wrong, since nothing about it is the record. |
| Presence | Per-user ephemeral state — cursor, selection, viewport, tool, idle — coalesced server-side to 10 Hz and never written anywhere. | The platform's shock absorber: the first traffic shed under load, with no availability target of its own. | 'Awareness', the term the CRDT literature uses for the same thing. |
| Capability | The effective permission resolved at admission and bound to a connection for its lifetime, re-evaluated on every change to the grants that produced it. | What every operation is authorised against, server-side, and what a revocation withdraws within 10 seconds. | 'Role' or 'permission', which describe the grant rather than the thing attached to a live socket. |
| Catch-up | The path a reconnecting client takes to reach the current head: replay of missed operations inside the retained log, or a fresh checkpoint plus the tail beyond it. | The reconnect latency budget, and the decision point that turns into a branch past the log horizon. | 'Sync', which covers four different mechanisms and hides which one ran. |
| Log horizon | The 30-day window of operations retained past a checkpoint, beyond which missed operations are no longer available to rebase against. | The published limit on how long a client may be offline and still be merged rather than branched. | 'Retention', which sounds like a storage setting rather than a product property (ADR-12). |
| Divergence | A client's document state no longer matching the authority's state at the same sequence number. | The only failure class with no natural alarm, detected by client checksum reporting within 60 seconds and repaired by re-snapshot. | 'Desync', which is also used for a client being behind — a normal, recoverable state rather than a defect. |
| Read-only (degraded) | The declared session state when the operation log cannot be written: viewing and presence continue, edits are refused within 5 seconds and the product says so. | The honest form of a write outage, chosen over buffering edits that might be lost. | 'Offline mode', which implies the client can keep working, which in this state it deliberately cannot. |