Architecture Decision Record

Cloud Development Environment Platform · Solution Architecture v1.0 · open source on raw/bare-metal compute · Platform Architecture · 2026-10

The argument these decisions serve is summarised in the Architecture One-Pager.

Fourteen decisions, grouped by the question they answer, each with the forcing question, the alternatives, what would flip the choice, and why it should outlast the technology it is realised on.

Status of this document. This is a design, not a report on a running system. Every rate, latency, ratio, threshold and retention figure in this package is a stated assumption, chosen to be defensible and arguable rather than measured. They are stated precisely so that a reviewer can disagree with one and follow it to the decision that depends on it. The requirement document marks them as assumptions section by section; the operating context assumed throughout is 250,000 daily active workspaces across 1,800 tenants, 60,000 concurrent at peak, 1,200,000 starts a day with 40,000 an hour at the peak, a workload mix of 65% headless agent sessions and 35% human sessions, 2,400,000 hibernated workspaces retained, and 180 TB of new snapshot data a day — on roughly 1,400 owned bare-metal hosts across two regions, with a resume budget of 2.5 seconds at p95.

How to read a record

• Question: The forcing question: why a decision was needed at all.
• Context: The requirement, the scale and the constraint that make it hard.
• Decision: What this architecture does, stated so it can be checked.
• How it is realised on this stack: The concrete mechanism: which service or package, configured how, in which subscription.
• Options weighed: Chosen, rejected, deferred, or right elsewhere, with the reason for each.
• Consequences: What the choice buys and what it costs, both kept visible.
• Choose differently when: The conditions that would flip the decision for your system.
• Why it holds up over time: What keeps the decision right as scale, staff and technology change.
• Lesson: The principle that transfers beyond this platform.

Decision map

The snapshot boundary: The decision that defines the architecture, and the two that follow from it directly: where a running workspace's filesystem truth lives, and whether memory is part of what gets kept.

• ADR-01 · The workspace is a snapshot in durable storage, late-bound to a disposable host
• ADR-02 · Filesystem truth is a content-addressed object store with a host-local copy-on-write overlay and lazy block fetch
• ADR-03 · Memory state is snapshotted while a workspace is recently used, and decays to disk-only after a week

A machine whose user has root: What isolates a workspace from the host, how a credential reaches it without being left there, and what it is allowed to talk to.

• ADR-04 · One microVM per workspace, with the boundary per workspace rather than per tenant
• ADR-05 · Credentials are brokered over a host side channel, scoped to one workspace, and never resident
• ADR-06 · Egress posture is deny-by-default for agent sessions and allowlisted-with-audit for human sessions

Keeping the promise of a fast start: Prebuilds, warm pools and admission: the three mechanisms that turn a four-minute build into a twenty-five-second restore, and a burst into a refusal rather than a queue.

• ADR-07 · A prebuild is a post-create snapshot keyed by an explicit validity predicate, rebased onto newer commits
• ADR-08 · Capacity pressure surfaces as an explicit, class-aware refusal, and agent creations shed before human resumes
• ADR-09 · An authorised connection resumes the workspace, and the gateway holds the client with progress

Idleness, which is the business model: Deciding when nobody is working, and what to do with the host that was carrying them.

• ADR-10 · Idleness is a composite signal evaluated on the host, with a bounded veto for human workspaces
• ADR-11 · Own bare metal and oversubscribe memory 4:1, with hibernation as the relief valve

Running it: The control plane's relationship to the data plane, how the fleet is patched while it is in use, and the one promise that outranks availability.

• ADR-12 · The control plane holds no workspace content, and a running workspace does not depend on it
• ADR-13 · Host drain is the only maintenance mechanism: hibernate, resume elsewhere, patch
• ADR-14 · Region is a hard residency boundary; cross-region resume is an authorised migration, never a failover

Technology by capability

Open-source components on raw, owned compute were chosen for two reasons pointing the same way. The first is that the topic genuinely belongs here: the architecture's defining requirement is sub-second virtual-machine snapshot and restore with kernel-level tenant isolation, and no managed container service on any hyperscaler exposes that — a platform of this kind necessarily owns its virtualisation layer, which is why every company shipping one does. The second is rotation: across this repository's use cases the three large clouds dominate recent packages, and a design that is genuinely about owning the metal is the right place to practise the alternative. Every row below is replaceable; what is not replaceable is the property each row was chosen for.

The decisions, and the alternatives that lost

The snapshot boundary

The decision that defines the architecture, and the two that follow from it directly: where a running workspace's filesystem truth lives, and whether memory is part of what gets kept.

ADR-01 · The workspace is a snapshot in durable storage, late-bound to a disposable host

Status: Accepted · Shown on views: 07, 08, 12, 16

A development environment is, physically, a machine with a disk. Is a workspace that machine, persisted for safety — or is it a persisted artefact that is occasionally incarnated on a machine?

Context. Both readings produce a working product, and they diverge on every operational question afterwards. If the workspace is a machine, then persistence is a backup, host loss is data loss, kernel patching is a maintenance window, and an idle workspace occupies RAM because that is where it lives. If the workspace is an artefact, then a host is interchangeable, host loss costs the interval since the last snapshot, patching is a scheduling problem, and an idle workspace occupies storage. The second reading is harder in exactly one place — resume — and easier everywhere else, including in the part of the design that decides whether the business works at all.

Decision. A workspace's identity and everything required to run it live in durable, content-addressed storage that no host owns. A host is selected at the moment of incarnation and may be destroyed at any time with no workspace-specific recovery procedure. The control plane records which host carried which incarnation; it never records that a workspace belongs to one.

How it is realised on this stack. The workspace object lives in the control-plane registry as state plus a pointer to its latest snapshot digest. Snapshots are written to an S3-compatible object store inside the region, deduplicated against the prebuild they derive from. The host agent holds no authoritative state: on incarnation it is handed a snapshot reference, and on drain it writes a new one and forgets the workspace. The data model expresses this directly — a workspace has many incarnations and many snapshots, and no host column.

What it buys

• Hibernation becomes the normal state rather than an optimisation, which is what makes the storage-versus-RAM economics in ADR-11 possible
• Host loss costs the snapshot interval, not the workspace; the fleet can be treated as cattle without pretending developers' work is cattle too
• Kernel patching becomes drain-and-resume-elsewhere, so the security cadence stops competing with the availability promise

What it costs

• Resume is now the hardest engineering problem in the platform, and it concentrates the snapshot format, block fetch, memory state, stale-state reconciliation and placement in one latency budget
• Every workspace write is eventually a write to object storage, which makes the storage tier a first-order capacity and cost concern rather than a background one
• Fencing becomes correctness-critical: two incarnations of one workspace is two divergent filesystems with one name

Choose differently when. If the fleet were small enough that hosts could be treated as durable — a single-tenant installation on a dozen machines with a four-hour maintenance window nobody minds — the host-owned design is cheaper to build and the snapshot machinery buys very little. The decision is a function of fleet size, patch cadence and how much idleness the price has to absorb, not of taste.

Why it holds up over time. The durable half is "identity belongs to the artefact, not to the machine currently holding it", which is the same move that made container images, immutable infrastructure and object-storage-backed databases work. Firecracker, the snapshot format, the object store and even the idea of a virtual machine will be replaced inside this platform's life; a design in which the thing a user cares about is addressable independently of the hardware carrying it will survive all of those substitutions.

Lesson. Decide what the user's thing actually is before deciding where it runs. Almost every operational pain in a stateful platform traces back to an identity that was accidentally attached to a host.

ADR-02 · Filesystem truth is a content-addressed object store with a host-local copy-on-write overlay and lazy block fetch

Status: Accepted · Shown on views: 10, 11, 13

A workspace needs a 32 GB disk that survives moving between hosts. Does that disk live on the host's NVMe and replicate out, on a network block device, or in object storage with a local overlay?

Context. Host-local NVMe gives the fastest reads and the worst survivability: a host loss costs whatever has not replicated, and migration means copying a disk. A shared network block device makes migration trivial and puts every workspace's random IO on the network — at 60,000 concurrent workspaces that is a storage network sized for the worst collective moment, and a storage outage becomes a fleet outage. Object storage with a local copy-on-write overlay is the awkward middle that happens to match the access pattern: a workspace reads a small, highly repetitive working set, and writes a delta nobody else needs.

Decision. The authoritative filesystem is a chain of content-addressed blocks in the region's object store, rooted at the prebuild. Each incarnation mounts that chain read-only through a host-local copy-on-write overlay, fetching blocks on demand and streaming the remainder in the background; the overlay is flushed to a new snapshot asynchronously every five minutes and synchronously on hibernation.

How it is realised on this stack. A userspace block device on the host presents the chain to the microVM, backed by a local NVMe cache and a fetcher that pulls 1 MB blocks from the object store on first touch. Deduplication is against the prebuild's block map, so the stored delta approximates the developer's work rather than the image. Declared ephemeral paths are excluded from the flush by default. Integrity is verified on write and on restore, and a restore that cannot be verified fails loudly rather than mounting partially.

What it buys

• Time-to-usable does not scale with disk size, which is what keeps the resume budget achievable for a 32 GB workspace
• A host may be destroyed without a recovery procedure, because nothing authoritative was only there
• Storage cost tracks the developer's own delta — a median of about 2 GB compressed — rather than the whole image

What it costs

• First touch of a cold block is a network read, so the first minutes after a resume are measurably slower than steady state
• The block layer is now a bespoke, correctness-critical component on the critical path of every workspace
• RPO is the flush interval: five minutes of work is at risk on an unplanned host loss, and that has to be told to the user rather than hidden

Choose differently when. If object-storage latency and per-request cost were not what they are — or if a workspace's working set were large and uniform rather than small and repetitive, as it is for a data-science workspace over a multi-terabyte dataset — lazy fetch stops paying and a replicated local disk wins. Measure the working set before assuming this transfers.

Why it holds up over time. "Address content by its hash, cache it near the compute, and fetch the rest only when touched" is the same insight behind container image layers, CDNs and lazy-loading filesystems, and it has outlived several generations of each. The specific block size, cache tier and store will all change; the separation between an immutable chain of truth and a mutable local overlay should not need to.

Lesson. When state must move between machines, make the authoritative copy immutable and content-addressed, then let every machine keep a disposable local view of it. Mutable state that must be shared is the thing to design away, not to distribute faster.

ADR-03 · Memory state is snapshotted while a workspace is recently used, and decays to disk-only after a week

Status: Accepted · Shown on views: 11, 13, 15

Sub-second resume requires restoring a microVM's memory, not booting it. A memory image is one to eight gigabytes per workspace. With 2.4 million hibernated workspaces, is memory part of the workspace or not?

Context. A memory snapshot is the difference between "it feels like it never stopped" and "my editor and dev server restarted". It is also, at scale, the single largest line in the storage bill: keeping an 8 GB image for every workspace anyone ever created is an order of magnitude more storage than keeping their files. The usage distribution rescues the decision — a workspace resumed today is overwhelmingly likely to be resumed again this week, and one untouched for a month is probably never coming back.

Decision. Hibernation writes a memory image alongside the filesystem delta for workspaces whose last resume is inside a decay window of seven days. Past the window, a background job drops the memory image and keeps the filesystem chain, so the workspace still resumes correctly, by cold boot, in seconds rather than sub-second. The window is a tuned parameter, not a constant.

How it is realised on this stack. The host agent pauses the microVM, writes the memory image to the region's object store with the filesystem delta, and records which snapshots carry memory. A lifecycle job in the control plane drops expired memory images and rewrites the pointer. On resume, the presence of a memory image chooses between restore and boot; either way the client sees the same progress stream, and the clock, network topology and brokered credentials are refreshed before the session attaches.

What it buys

• The resume target of p95 ≤ 2.5 s is met for the workspaces that are actually resumed, which is where the product lives
• Storage for the long tail collapses to the filesystem chain, keeping hibernated cost at the assumed $0.004 per workspace-hour
• A dropped memory image degrades the experience and never breaks it, so the decay job is safe to run aggressively

What it costs

• Two resume paths exist, and both have to be tested; the fidelity gate in the delivery pipeline covers both for exactly this reason
• A restored memory image is a machine that believes it is still yesterday — stale sockets, old clock, expired tokens — and all three must be reconciled
• Resume latency now depends on a workspace's recency, which is a harder promise to explain than one number

Choose differently when. If memory images could be stored at a small fraction of today's cost, or deduplicated across workspaces sharing a base image far better than they currently can be, the decay window stops earning its complexity and "always snapshot memory" becomes right. Equally, a tenant that only ever runs short agent sessions should be configured disk-only and save the write.

Why it holds up over time. The transferable property is tiering by recency rather than by type: keep the expensive representation for the population that will use it, and keep a cheaper, still-correct representation for everyone else. That pattern long predates microVMs and will outlast them; only the artefact being tiered changes.

Lesson. When a cache is expensive enough to need a retention policy, derive the policy from the access distribution, and make sure the cheap path is correct rather than merely degraded.

A machine whose user has root

What isolates a workspace from the host, how a credential reaches it without being left there, and what it is allowed to talk to.

ADR-04 · One microVM per workspace, with the boundary per workspace rather than per tenant

Status: Accepted · Shown on views: 08, 15, 20

The platform's normal case is running unreviewed code for a user who has root. What isolates that from the host, and is the boundary drawn around each workspace or around each tenant?

Context. A hardened container runtime is the densest option and shares a kernel with its neighbours, which means the threat model rests on a kernel that was never designed to be an adversarial boundary. A user-space kernel interposition layer narrows the syscall surface at a measurable performance cost and breaks nested containerisation, which this platform is required to support. A virtual machine per workspace gives each tenant's code its own kernel to attack. The per-tenant variant of any of these buys density by making one developer's fork bomb their colleague's problem.

Decision. Each workspace runs in its own Firecracker microVM with its own guest kernel, jailed on the host, with per-workspace CPU, memory, PID, file-descriptor, disk-size, IOPS and bandwidth caps the workspace cannot raise. The boundary is per workspace, not per tenant, and it is identical for human and agent workspaces.

How it is realised on this stack. The host agent launches Firecracker under its jailer with a minimal device model, a per-workspace network namespace, and cgroup and rlimit caps applied before the guest boots. Limits are visible inside the guest so toolchains size themselves sanely. Nested containerisation happens inside the guest kernel, where it is the guest's problem. Resource-exhaustion classes — fork bombs, OOM loops, runaway writes, thousands of sockets — are contained to the one VM, reported with their cause, and never auto-restarted into the same loop.

What it buys

• The isolation claim is structural: an escape reaches a kernel the platform is willing to lose, and reaches no other tenant's code
• Snapshot and restore of a whole machine becomes possible, which is the mechanism the rest of the architecture is built on
• Nested containerisation, the thing developers actually do, works without granting the host any additional trust

What it costs

• Density is lower than a container platform's, and the 4:1 memory oversubscription in ADR-11 exists to win some of it back
• The host agent and the virtualisation layer become the most privileged, most security-critical code in the system
• Kernel patching is now a fleet operation on a cadence set by other people's CVEs

Choose differently when. If every tenant were internal and trusted — a single company's own developers running only reviewed code — the per-workspace kernel buys little and a container platform is cheaper and faster. The moment agents or third parties run code, the calculation reverses, and it has already reversed for most of this platform's traffic.

Why it holds up over time. "Put the trust boundary below the operating system the adversary controls" has survived every generation of multi-tenant compute, from mainframe partitions to microVMs, and the arrival of code written by models rather than people has only strengthened it. The specific hypervisor is replaceable; the principle that the boundary is not the guest's own kernel should not be revisited.

Lesson. Size the isolation boundary to the worst code you will ever be asked to run, not to the code you expect. On this platform the worst case is the normal case.

ADR-05 · Credentials are brokered over a host side channel, scoped to one workspace, and never resident

Status: Accepted · Shown on views: 08, 20, 21

A workspace needs to clone and push. Its user has root, it installs arbitrary dependencies, and an agent inside it may read any file. How does a credential get in?

Context. Anything written to the workspace filesystem or environment is readable by the user, by every dependency they install, and by any agent in the session — so an injected token is an exported token, and the only questions are its scope and its lifetime. The alternative extreme, where the platform proxies every privileged operation and holds the identity itself, shrinks the blast radius to nothing and puts the platform in the path of every authenticated call its users make, including ones it has no business seeing.

Decision. No long-lived credential is ever placed in a workspace. The guest asks over a vsock side channel; the host agent attaches the workspace identity, which the guest cannot assert for itself; the broker checks the repository binding and mints a token scoped to that repository and that workspace, valid for fifteen minutes. Every issuance is recorded in the audit log. There is no network path from the guest to the broker or to the control plane.

How it is realised on this stack. A guest helper answers git credential requests and forwards them over vsock. The host agent presents a workload identity issued per workspace to the broker, which checks the binding in the control plane and mints a forge installation token with push scope for one repository. The token is handed back in memory and is re-brokered on every resume rather than restored from a memory image. Pushed content is scanned for credentials on the egress path, failing the push on a high-confidence match.

What it buys

• A leaked token has a known scope, a known owner and a fifteen-minute window, which turns a breach into an incident rather than a catastrophe
• The guest cannot claim an identity, so scope is enforced where the attacker is not
• Revocation is implicit: stopping a workspace stops its ability to obtain credentials at all

What it costs

• A broker outage stops pushes and clones for every workspace, so it is on the critical path of the work even though it is off the keystroke path
• Fifteen-minute tokens mean re-brokering, which means every tool that caches credentials has to cope with expiry
• The vsock protocol is a new, privileged surface between untrusted code and the most trusted component on the host

Choose differently when. For an operation whose blast radius genuinely cannot be bounded by scope — signing a release, touching production — the rejected proxy option is the right one, and the platform should mediate rather than delegate. The boundary between the two is the value of what the credential unlocks, not the convenience of the tool.

Why it holds up over time. Short-lived, audience-scoped, workload-attested credentials are where the whole industry has been converging for a decade, and nothing about agents changes the direction — it only raises the volume of consumers who cannot be trusted to keep a secret. The forge, the broker and the attestation format will change; "the machine proves what it is, and gets only what that needs, briefly" will not.

Lesson. If a secret must enter a machine you do not trust, make it worth as little as possible: narrow scope, short life, attested requester, recorded issuance. Secrecy is not available, so stop paying for it.

ADR-06 · Egress posture is deny-by-default for agent sessions and allowlisted-with-audit for human sessions

Status: Accepted · Shown on views: 09, 18, 20

A workspace is a machine on the internet running unreviewed code. What may it reach by default, and should the default differ between a human's workspace and an agent's?

Context. Open egress makes every developer's first install work and makes exfiltration a one-line script. Deny-by-default with a package-registry proxy makes exfiltration hard and makes a developer's first hour a sequence of failures the platform has to explain. The two populations differ in a way that matters: a human notices a blocked host and asks for it, while an agent cannot distinguish a blocked host from a broken one, and an agent is also the population most likely to be carrying someone else's instructions.

Decision. Three postures exist per workspace class — open with audit, allowlisted with audit, and deny-by-default with a pull-through registry proxy. Agent sessions default to deny-by-default; human sessions default to allowlisted with audit. Tenants may change both within platform bounds, the active posture is visible inside the workspace, and every verdict is logged with destination, volume and outcome.

How it is realised on this stack. All egress leaves through a per-workspace policy enforcement point on the host, combining an eBPF-level block with a forward proxy for the allowed protocols; package managers are pointed at a pull-through mirror. Workspace-to-workspace and workspace-to-control-plane paths are blocked outright, including between workspaces of the same tenant. The posture is written into the guest's environment so a failure is legible rather than mysterious.

What it buys

• The population running the least-reviewed code gets the tightest boundary, by default rather than by configuration
• A registry proxy turns the most common legitimate egress into a cached, inspectable, cheap path
• Every verdict is evidence, which makes an exfiltration question answerable after the workspace is gone

What it costs

• Two defaults mean code that behaves in a human's workspace and fails in an agent's, which is a confusing class of bug
• The proxy is a single point of friction for every dependency install in the estate, and a visible one when it is slow
• An allowlist is a living artefact; a stale one is indistinguishable from a broken network

Choose differently when. If agents and humans converge — the agent running in the developer's own workspace, under their identity, on their posture — the split stops being coherent and the platform should choose one posture per workspace rather than per class. That convergence is already visible in the fork-from-snapshot feature in phase 3.

Why it holds up over time. The durable claim is that default network reach should be set by what the code is trusted to do, not by what is convenient to the person starting it, and that a mediated path for the common case is what makes a strict default survivable. That holds regardless of what the registries, the protocols or the enforcement mechanism become.

Lesson. A strict default is only sustainable if you also build the paved path for the ninety per cent. Deny-by-default without a working proxy is a policy that gets switched off within a month.

Keeping the promise of a fast start

Prebuilds, warm pools and admission: the three mechanisms that turn a four-minute build into a twenty-five-second restore, and a burst into a refusal rather than a queue.

ADR-07 · A prebuild is a post-create snapshot keyed by an explicit validity predicate, rebased onto newer commits

Status: Accepted · Shown on views: 12, 14

Creating a workspace means cloning, installing and running post-create hooks — minutes of work that is identical for everyone on a branch. What exactly is cached, and what makes the cache safe to use?

Context. Caching the base image alone leaves the slowest part — dependency installation — on the critical path. Caching a layered image plus mounted dependency caches is faster and still runs the install. Snapshotting the workspace after post-create caches everything, and raises the only question that matters: when is that snapshot wrong? An age heuristic answers it badly, because a prebuild from this morning can be wrong and one from last week can be right. The failure that matters is silent: a workspace that differs from CI in a way nobody notices until a build passes locally and fails in the pipeline.

Decision. The prebuild artefact is a snapshot taken immediately after the post-create step, indexed by a validity predicate over the base image digest, the environment definition revision and the lockfile hash. A valid prebuild may be rebased onto a newer commit of the same branch; an invalid one is never used. Prebuild fan-out is rate-limited per tenant, and the most recent valid prebuild per active branch is retained.

How it is realised on this stack. A prebuild job clones, installs through the registry proxy and runs post-create inside an ordinary workspace, then hibernates it and registers the resulting snapshot in the prebuild index with its predicate values. Creation looks up the predicate; a hit restores the snapshot and applies the commit delta, a miss falls back to an inline build and reports that it did so, with the start-latency cost attributed to the cause.

What it buys

• Create from a prebuild meets the assumed p95 of 25 seconds, and a warm-pool claim meets 3 seconds
• Staleness is decidable and inspectable, so a divergence from CI is a bug with a cause rather than folklore
• The same mechanism as hibernation produces the artefact, so there is one snapshot path to maintain and to test

What it costs

• Rebasing a post-create snapshot onto a newer commit is correct until a dependency change makes it wrong, and the lockfile hash is the only thing separating the two cases
• Prebuilds consume fleet capacity for environments nobody may ever open, which is why the fan-out cap and the waste metric both exist
• The predicate couples the platform to how each ecosystem expresses its dependency lock, which is a long tail of small integrations

Choose differently when. In a monorepo where every branch shares one lockfile, the predicate collapses to the definition revision and the whole mechanism simplifies. In a polyglot estate with no lockfiles at all, it degrades toward the layered-cache option, and the honest answer is a slower start rather than a riskier cache.

Why it holds up over time. "Cache the expensive step, and make validity an explicit predicate over inputs rather than a function of time" is the lesson every build system has learned, usually twice. The artefacts and ecosystems will change; a cache whose correctness is decidable from its inputs will keep being the only kind worth trusting.

Lesson. A cache with a wrong answer is worse than no cache, so spend the design effort on invalidation rather than on hit rate. If you cannot state the predicate, you are not caching — you are guessing.

ADR-08 · Capacity pressure surfaces as an explicit, class-aware refusal, and agent creations shed before human resumes

Status: Accepted · Shown on views: 02, 05, 18

A single agent task can ask for forty workspaces in a second, ten times the steady-state creation rate. When the fleet cannot absorb that, who waits, who is refused, and how do they find out?

Context. Queueing is the default behaviour of almost every system built without thinking about it, and it is the worst option here: an orchestrator cannot distinguish a queue from a hang, so it retries, which deepens the queue, and a developer resuming an existing workspace ends up behind forty speculative creations. Refusing is better but only if the refusal is legible — classed, attributable and retryable — and only if the choice of who to refuse is made deliberately rather than by arrival order.

Decision. Admission control sits ahead of placement and enforces per-tenant and per-principal concurrency and spend ceilings before any capacity is committed. Under pressure, agent creations are shed first, then new human creations, and resumes of existing workspaces last. Every rejection is explicit, names its cause, and is marked retryable with a suggested delay.

How it is realised on this stack. The admission controller reads ceilings and current commitments from the control plane through a cached view, consults warm-pool availability per (prebuild, machine class) and host headroom per region, and returns a typed rejection the client surfaces verbatim. Human clients see a message with the cause; agent clients get a structured, retryable error. Shed rate by class is a first-class signal on the observability matrix.

What it buys

• Resume latency is defended under burst, which keeps the promise the product is actually judged on
• An agent orchestrator can implement sane backoff because it is told the truth immediately
• Spend ceilings are enforced where capacity is committed rather than discovered on an invoice

What it costs

• Agents are refused more often than humans by design, which has to be explained to the people paying for agent capacity
• Ceilings are another thing for tenants to tune, and a badly set one looks like a platform fault
• A refusal is only as good as the warm-pool sizing behind it; chronic refusals mean the forecast in ADR-11 is wrong

Choose differently when. If agent sessions became the latency-sensitive population — an interactive agent a developer is watching, rather than a batch fan-out — the shed order inverts, and the class distinction has to be redrawn around interactivity rather than around who is typing. That is a plausible two-year change, and the decision is deliberately expressed as policy rather than as code paths.

Why it holds up over time. "Refuse early, explicitly, and in a chosen order" outlives every scheduler. The durable part is that the shed order encodes a product judgement about whose time matters most at that moment, and that it belongs in configuration where it can be argued with, not buried in a queueing discipline.

Lesson. Decide who you will disappoint before the system has to decide for you. An unstated priority order becomes arrival order, which is the one order nobody chose.

ADR-09 · An authorised connection resumes the workspace, and the gateway holds the client with progress

Status: Accepted · Shown on views: 02, 04, 13

A developer clicks a hibernated workspace. Does the client ask for a resume and then poll, or does the connection itself start the resume and stay open?

Context. A poll-based flow is easy to build and makes the common case feel like a page load with a loading screen. It also triples the round trips, needs a client state machine, and tends to fail in the two cases that matter most: a laptop waking on a different network, and a client that gives up while the resume is nearly done. Holding the connection open puts the complexity in the gateway, where the platform controls it, and lets the resume report its own progress.

Decision. Connecting to a hibernated workspace with sufficient authority is the resume request. The edge authorises, asks the control plane once, and holds the client connection open with a progress stream naming the current phase, up to the cold-start deadline. Gateways are stateless with respect to workspace identity, so losing one costs a reconnect rather than a workspace.

How it is realised on this stack. Envoy terminates the connection and routes to a session proxy that subscribes to the incarnation's progress events while the host agent restores. The client receives phase updates — authorised, placed, restoring memory, fetching blocks, attached — and a terminal when the guest agent reports ready. Client network changes reconnect to the same workspace inside the reconnection window, and a dropped connection is explicitly not a workspace event.

What it buys

• The wait has a cause attached, which is the difference between a slow platform and a broken one
• Gateway loss and client network changes degrade to a reconnect, not to a lost session
• Start-latency telemetry is attributable by phase, because the phases are already being streamed

What it costs

• The edge carries tens of thousands of long-lived connections, so its capacity planning is connection-bound rather than request-bound
• A resume that exceeds the deadline must fail in a way that leaves the workspace resumable, which is an extra path to get right
• Holding connections during a burst consumes edge capacity exactly when admission is already refusing work

Choose differently when. If clients converged on one well-behaved implementation under the platform's control, polling would be perfectly adequate and would simplify the edge considerably. With four transports and third-party agent runtimes, that convergence is not available.

Why it holds up over time. The durable idea is that the system should absorb the complexity of waiting rather than exporting it to every client, and that a wait with a named cause is a different product from a wait without one. Transports will change; this division of labour should not.

Lesson. Put the waiting where you can instrument it. A progress stream is not a UX nicety — it is the only honest source of start-latency attribution you will ever get.

Idleness, which is the business model

Deciding when nobody is working, and what to do with the host that was carrying them.

ADR-10 · Idleness is a composite signal evaluated on the host, with a bounded veto for human workspaces

Status: Accepted · Shown on views: 04, 15, 19

Hibernation is what makes the economics work, and a hibernation at the wrong moment is the worst bug the platform can ship. Who decides that nobody is working?

Context. Client heartbeats are the easiest signal and the most wrong: a developer running a forty-minute build is not typing, and a closed laptop lid with a background agent still working looks identical to an abandoned tab. Guest process activity knows about the build and can be gamed by anything that polls. Host-level network and disk activity is unforgeable and cannot tell a compile from a crypto miner. The two errors are asymmetric in kind rather than in size: a false hibernation destroys trust in the platform, while a missed hibernation destroys the margin quietly.

Decision. Idleness is a composite evaluated by the host agent: absence of client input, absence of meaningful guest process activity, and absence of disk and network work, all over a tenant-configured window with a shorter default for agent sessions. A human workspace may request a bounded veto, measured in minutes and logged; an agent workspace may not. Hibernation accuracy is reported as a product metric, not only as a cost one.

How it is realised on this stack. The guest agent reports process and terminal activity as advisory input; the host agent combines it with cgroup CPU, block IO and egress counters, which the guest cannot forge. A long-running foreground process holds the veto while it reports progress, up to a cap. False hibernations and missed idleness are counted on the observability matrix, and a tenant's timeout is policy read from cache rather than from the control plane.

What it buys

• Long builds and long test runs survive, which is what makes aggressive default timeouts acceptable at all
• The signal is unforgeable in the part that costs money, and informative in the part that protects the developer
• Hibernation accuracy becomes measurable, so the oversubscription ratio in ADR-11 can be tuned against evidence

What it costs

• Three inputs and a veto is substantially more logic than a timeout, on the path that runs for every workspace continuously
• A workspace that polls can keep itself alive longer than it deserves, and the ceiling on the veto is the only thing bounding that
• Tenants can set timeouts that are economically wrong for them, and the platform has to show them the cost rather than refuse

Choose differently when. If a reliable, unforgeable signal of human attention existed at the client — and it may, from the editor itself — the composite could collapse toward it and the veto would become unnecessary. Until then the host is the only place a signal cannot be faked.

Why it holds up over time. Every platform that bills for utilisation eventually has to decide what "in use" means, and the durable answer is: measure at a boundary the user cannot forge, take advisory input from where the knowledge is, and treat the two error directions as different products rather than as a single threshold. That survives any change in what runs inside.

Lesson. When a cost control can also break the product, make its two failure directions explicit and measure both. A single threshold hides the fact that you are trading trust against margin.

ADR-11 · Own bare metal and oversubscribe memory 4:1, with hibernation as the relief valve

Status: Accepted · Shown on views: 16, 18, 19

Does the platform own hosts, rent bare metal, or run on standard cloud instances — and how much may it promise that it does not physically have?

Context. Standard cloud instances are the easiest to start with and the hardest to make this architecture work on: nested virtualisation is restricted or slow, local NVMe is not guaranteed, and the price per workspace-hour at the target margin does not close once the hypervisor tax is paid twice. Rented bare metal removes the technical obstacles and keeps a commercial dependency on a supplier's inventory during exactly the bursts that matter. Owning the fleet buys the density and the price and brings hardware failure and kernel patching in-house. Separately, a workspace at 8 GB nominal is idle most of the day it is open, so a fleet sized for the nominal sum is a fleet mostly holding nothing.

Decision. The platform runs on owned bare-metal hosts in its own racks, with local NVMe and hardware virtualisation, and oversubscribes memory 4:1 against measured workspace idleness. Hibernation is the declared relief valve: under memory pressure the host agent hibernates the least recently active workspaces rather than letting the host degrade. The ratio is a tuned parameter with a published headroom signal, not an assumption.

How it is realised on this stack. Hosts are 128-core, 1 TB machines carrying about 40 nominal workspaces each, in two regions. The host agent watches resident memory and block pressure, and hibernates on a declared ladder — idle first, then least recently active, never a workspace whose veto is held. Admission accounts for committed rather than nominal memory, and oversubscription headroom is a fleet-level signal with its own alert.

What it buys

• The assumed $0.085 per active workspace-hour and $0.004 hibernated are achievable, which is the whole commercial argument
• Hardware virtualisation and local NVMe are guaranteed, so the snapshot and lazy-fetch designs are not fighting the substrate
• Capacity decisions are the platform's own, which matters most during the bursts a supplier would also be absorbing for someone else

What it costs

• Kernel patching, firmware, disk failure and rack capacity become the platform team's problem, with the fleet operations that implies
• 4:1 is only safe while hibernation works; a bug in ADR-10 becomes an availability incident within a working day
• Burst headroom is physical, so a 10× agent burst beyond forecast ends in refusals rather than in a larger bill

Choose differently when. If a provider offered bare-metal capacity with guaranteed snapshot-grade local NVMe at a price close to owning it, the deferred option becomes the better one and the ratio stops being the platform's risk. Equally, if measured idleness fell — because agents, which are rarely idle, kept growing as a share of the fleet — the ratio must fall with it, and the published headroom signal is what makes that visible before it hurts.

Why it holds up over time. The durable claim is that a platform may promise more than it has only when it owns a correct, fast mechanism for taking it back. That is as true of memory here as it is of thin-provisioned storage or an airline seat; the mechanism is what makes the promise honest, and a design that oversubscribes without one is simply wrong at a time it has not met yet.

Lesson. Oversubscription is a bet you make on a distribution, settled by a mechanism. Build the mechanism first, measure the distribution continuously, and publish the headroom so the bet is visible to the people who will be paged for it.

Running it

The control plane's relationship to the data plane, how the fleet is patched while it is in use, and the one promise that outranks availability.

ADR-12 · The control plane holds no workspace content, and a running workspace does not depend on it

Status: Accepted · Shown on views: 07, 08, 12

What may the control plane know, and what must keep working when it is gone?

Context. It is tempting to let the control plane see everything: it already knows every workspace, so routing logs, port metadata, even file lists through it makes features easy. Each of those makes the control plane a bigger target, a bigger backup problem and a bigger outage. Separately, if a running workspace needs the control plane for its keystrokes to work, then the management plane's availability becomes the product's availability, and the product's is the harder number.

Decision. The control plane holds records only — workspaces, tenants, seats, policy, placements, prebuild validity, quotas and events. No filesystem data, memory image or workspace log transits control-plane storage. Running workspaces and established sessions survive a total control-plane outage; creation, resume and policy change stall, and policy is read by the data plane from a cached view rather than synchronously.

How it is realised on this stack. Host agents and edge gateways hold a cached projection of the policy and placement they need, refreshed from the event stream and safe to run stale for a bounded period. Workspace content goes directly from the host agent to the object store. Audit and lifecycle events flow out on Kafka, so evidence is durable without the control plane being in the data path.

What it buys

• A control-plane outage degrades to "no new workspaces" rather than to "nobody can work"
• The backup and residency problem collapses onto two stores — the registry and the snapshot store — instead of five
• The most attractive target in the system holds no source code

What it costs

• Cached policy can be stale, so a revocation or a posture change has a bounded propagation delay that must be stated
• Debugging crosses a boundary deliberately: support cannot read a workspace's contents, which makes some tickets slower
• Reconciliation after an outage has to be designed, because hosts will have made decisions the control plane has not seen

Choose differently when. If a tenant required immediate, synchronous enforcement of a policy change — a legal hold, or an urgent revocation — the cached model is insufficient for that path, and it is handled by a kill switch propagated on the event stream rather than by making the whole design synchronous.

Why it holds up over time. Separating a control plane from a data plane, and refusing to let the control plane hold the payload, has survived every generation of networking, storage and compute platform. The specific caches and projections will change; the rule that the thing you can least afford to lose should not depend on the thing you change most often will not.

Lesson. Decide what must keep working when your own management system is down, and then make sure it does not call it. A platform whose availability is bounded by its admin tooling has two products and one number.

ADR-13 · Host drain is the only maintenance mechanism: hibernate, resume elsewhere, patch

Status: Accepted · Shown on views: 06, 15, 17

The isolation boundary is a kernel, and kernels need patching on somebody else's schedule. How does a fleet carrying 60,000 live workspaces get patched?

Context. The honest options are a maintenance window, live migration, or drain-and-resume. A window asks developers to stop working on a security team's schedule, and in practice means patching slips. Live migration of a running microVM keeps sessions warm and is a substantially harder engineering problem, with a correctness tail — in-flight devices, open sockets, memory convergence under load — that shows up as corruption rather than as slowness. Drain-and-resume reuses the mechanism the platform already has to be correct for host loss and idleness.

Decision. Patching, hardware replacement and rebalancing all use one mechanism: mark the host draining so it accepts no new placements, hibernate its workspaces in rate-limited waves, resume them on other hosts, then patch. Live migration is explicitly not built. Resume success rate under drain is a release gate for the platform's own changes.

How it is realised on this stack. The host agent refuses placements when draining and walks its workspaces in waves, honouring vetoes up to their cap. Each workspace hibernates with memory state where it has it and resumes elsewhere immediately, so the developer sees a pause of a few seconds. Drain waves are rate-limited fleet-wide so that storage and the edge are not saturated by evacuation traffic, and the fleet-wide drain state is an operational signal.

What it buys

• The security cadence stops competing with the availability promise, which is what makes a per-workspace kernel sustainable
• Every maintenance operation exercises the resume path, so the platform's most critical mechanism is continuously proven in production
• Hardware failure needs no separate procedure: an unplanned loss is a drain that did not get to say goodbye

What it costs

• Developers experience a short pause during a drain, and a drain is therefore always visible to someone
• Evacuation concentrates snapshot writes and block fetches, so storage must be sized for drain traffic rather than for steady state
• A drain slower than a CVE window creates pressure to terminate workspaces instead, which is exactly the behaviour the design exists to make unnecessary

Choose differently when. If live migration became reliable and cheap to operate — as it did for conventional virtualisation over roughly a decade — the pause disappears and this decision should be revisited for the human-session population. It would never be worth building for agent sessions, which are gone before a migration completes.

Why it holds up over time. The durable move is to make the exceptional path the routine one, so that the mechanism you depend on in a crisis is the mechanism you ran a thousand times this week. That principle has outlasted every specific recovery technology, and it is the reason this platform does not have a separate disaster procedure.

Lesson. If you have a recovery mechanism, use it for maintenance too. A path exercised only during incidents is a path that fails during incidents.

ADR-14 · Region is a hard residency boundary; cross-region resume is an authorised migration, never a failover

Status: Accepted · Shown on views: 11, 16

A region fails. Its workspaces are unavailable and their content is intact inside it. Should the platform resume them somewhere else?

Context. Automatic cross-region resume is what an availability-first design does, and it requires workspace content to exist outside its region, which is the exact thing a residency promise says will not happen. The content in question is the tenant's source code and working state, and most enterprise tenants chose the region deliberately. Meanwhile the availability math is less compelling than it looks: a regional failure makes workspaces unavailable for the duration, which is costly, while a silent residency breach is a contractual and regulatory event that cannot be undone by restoring service.

Decision. Workspace content and snapshots never leave their region. A region loss makes its workspaces unavailable until the region returns; there is no automatic failover. Migrating a workspace to another region is an explicit, tenant-authorised operation that copies content across the boundary with its own audit record. Residency outranks availability, and the asymmetry is stated in the tenant contract rather than discovered during an incident.

How it is realised on this stack. The tenant's region is an attribute enforced on every placement, snapshot write and route; the object store is region-local with no cross-region replication configured. Global services hold identity and the image catalogue only, and the catalogue is mirrored per region. The migration path is an API call with an authorisation step, producing a copy in the target region and an audit event naming who authorised it.

What it buys

• The residency claim is enforced by configuration rather than by promise: there is no cross-region copy to accidentally enable
• The backup and key-management story stays regional, which keeps customer-managed keys meaningful
• A failure during a regional incident is unavailability, which is explainable, rather than a breach, which is not

What it costs

• A regional failure costs the region's developers their working day, and the platform has no technical answer to that during the event
• Capacity cannot be shared across regions during a burst, so each region must carry its own headroom
• Tenants who would have preferred replication must opt into a migration they have to authorise

Choose differently when. A tenant with no residency constraint and a strong availability requirement should be offered the deferred option, and the design is deliberately shaped so that replication is an added capability rather than a removed restriction. The default stays pinned, because the failure direction of the default is the one that can be undone.

Why it holds up over time. When two promises conflict, the durable rule is to keep the one whose breach cannot be repaired. Regulatory and contractual commitments about where data lives have only hardened over the last decade, while availability expectations are negotiable and measurable; that asymmetry is likely to outlast every specific regulation.

Lesson. Rank your promises explicitly, and let the ranking show up in the architecture rather than in an incident review. The promise you quietly break is always the one you never ranked.

Every package used, in one table

These terms are used precisely in this package. Several are used loosely in the wider literature on cloud development environments, and the difference matters when reading the decision records.