Architecture Views

A development environment is, physically, a machine with a disk. Is a workspace that machine, persisted for safety — or is it a persisted artefact that is occasionally incarnated on a machine?

Context

Both readings produce a working product, and they diverge on every operational question afterwards. If the workspace is a machine, then persistence is a backup, host loss is data loss, kernel patching is a maintenance window, and an idle workspace occupies RAM because that is where it lives. If the workspace is an artefact, then a host is interchangeable, host loss costs the interval since the last snapshot, patching is a scheduling problem, and an idle workspace occupies storage. The second reading is harder in exactly one place — resume — and easier everywhere else, including in the part of the design that decides whether the business works at all.

Decision

A workspace's identity and everything required to run it live in durable, content-addressed storage that no host owns. A host is selected at the moment of incarnation and may be destroyed at any time with no workspace-specific recovery procedure. The control plane records which host carried which incarnation; it never records that a workspace belongs to one.

How it is realised on this stack

The workspace object lives in the control-plane registry as state plus a pointer to its latest snapshot digest. Snapshots are written to an S3-compatible object store inside the region, deduplicated against the prebuild they derive from. The host agent holds no authoritative state: on incarnation it is handed a snapshot reference, and on drain it writes a new one and forgets the workspace. The data model expresses this directly — a workspace has many incarnations and many snapshots, and no host column.

Options weighed

• ChosenWorkspace as a snapshot, host late-bound: Makes hibernation normal, host loss a latency event and patching routine; pays with a resume path that has to be engineered to hundreds of milliseconds
• RejectedWorkspace as a long-lived VM with backups: Simplest resume in the world — there isn't one — and it pays in every other section: lost work on host loss, maintenance windows developers feel, and a fleet billing for idle RAM
• Right elsewhereWorkspace as a pure container rebuilt from the definition each time: Right for a platform whose environments are genuinely stateless, such as ephemeral CI runners; wrong here because a developer's uncommitted work is the asset

Consequences

What it buys

• Hibernation becomes the normal state rather than an optimisation, which is what makes the storage-versus-RAM economics in ADR-11 possible
• Host loss costs the snapshot interval, not the workspace; the fleet can be treated as cattle without pretending developers' work is cattle too
• Kernel patching becomes drain-and-resume-elsewhere, so the security cadence stops competing with the availability promise

What it costs

• Resume is now the hardest engineering problem in the platform, and it concentrates the snapshot format, block fetch, memory state, stale-state reconciliation and placement in one latency budget
• Every workspace write is eventually a write to object storage, which makes the storage tier a first-order capacity and cost concern rather than a background one
• Fencing becomes correctness-critical: two incarnations of one workspace is two divergent filesystems with one name

Choose differently when

If the fleet were small enough that hosts could be treated as durable — a single-tenant installation on a dozen machines with a four-hour maintenance window nobody minds — the host-owned design is cheaper to build and the snapshot machinery buys very little. The decision is a function of fleet size, patch cadence and how much idleness the price has to absorb, not of taste.

Why it holds up over time

The durable half is "identity belongs to the artefact, not to the machine currently holding it", which is the same move that made container images, immutable infrastructure and object-storage-backed databases work. Firecracker, the snapshot format, the object store and even the idea of a virtual machine will be replaced inside this platform's life; a design in which the thing a user cares about is addressable independently of the hardware carrying it will survive all of those substitutions.

A workspace needs a 32 GB disk that survives moving between hosts. Does that disk live on the host's NVMe and replicate out, on a network block device, or in object storage with a local overlay?

Context

Host-local NVMe gives the fastest reads and the worst survivability: a host loss costs whatever has not replicated, and migration means copying a disk. A shared network block device makes migration trivial and puts every workspace's random IO on the network — at 60,000 concurrent workspaces that is a storage network sized for the worst collective moment, and a storage outage becomes a fleet outage. Object storage with a local copy-on-write overlay is the awkward middle that happens to match the access pattern: a workspace reads a small, highly repetitive working set, and writes a delta nobody else needs.

Decision

The authoritative filesystem is a chain of content-addressed blocks in the region's object store, rooted at the prebuild. Each incarnation mounts that chain read-only through a host-local copy-on-write overlay, fetching blocks on demand and streaming the remainder in the background; the overlay is flushed to a new snapshot asynchronously every five minutes and synchronously on hibernation.

How it is realised on this stack

A userspace block device on the host presents the chain to the microVM, backed by a local NVMe cache and a fetcher that pulls 1 MB blocks from the object store on first touch. Deduplication is against the prebuild's block map, so the stored delta approximates the developer's work rather than the image. Declared ephemeral paths are excluded from the flush by default. Integrity is verified on write and on restore, and a restore that cannot be verified fails loudly rather than mounting partially.

Options weighed

• ChosenContent-addressed object store + local CoW overlay + lazy fetch: Resume time becomes independent of disk size, host loss costs the flush interval, and the dominant cost is storage rather than a storage network
• RejectedHost-local NVMe with asynchronous replication: Fastest steady state; makes migration a disk copy and host loss a recovery procedure, which contradicts ADR-01 directly
• RejectedShared network block device per workspace: Cleanest migration story and the worst blast radius: a storage fabric incident stops every running workspace, and random IOPS at this concurrency is the whole capacity plan
• Right elsewhereDistributed filesystem shared across hosts: Right where workspaces genuinely share state, such as HPC scratch; here it buys coordination nobody asked for

Consequences

What it buys

• Time-to-usable does not scale with disk size, which is what keeps the resume budget achievable for a 32 GB workspace
• A host may be destroyed without a recovery procedure, because nothing authoritative was only there
• Storage cost tracks the developer's own delta — a median of about 2 GB compressed — rather than the whole image

What it costs

• First touch of a cold block is a network read, so the first minutes after a resume are measurably slower than steady state
• The block layer is now a bespoke, correctness-critical component on the critical path of every workspace
• RPO is the flush interval: five minutes of work is at risk on an unplanned host loss, and that has to be told to the user rather than hidden

Choose differently when

If object-storage latency and per-request cost were not what they are — or if a workspace's working set were large and uniform rather than small and repetitive, as it is for a data-science workspace over a multi-terabyte dataset — lazy fetch stops paying and a replicated local disk wins. Measure the working set before assuming this transfers.

Why it holds up over time

"Address content by its hash, cache it near the compute, and fetch the rest only when touched" is the same insight behind container image layers, CDNs and lazy-loading filesystems, and it has outlived several generations of each. The specific block size, cache tier and store will all change; the separation between an immutable chain of truth and a mutable local overlay should not need to.

Sub-second resume requires restoring a microVM's memory, not booting it. A memory image is one to eight gigabytes per workspace. With 2.4 million hibernated workspaces, is memory part of the workspace or not?

Context

A memory snapshot is the difference between "it feels like it never stopped" and "my editor and dev server restarted". It is also, at scale, the single largest line in the storage bill: keeping an 8 GB image for every workspace anyone ever created is an order of magnitude more storage than keeping their files. The usage distribution rescues the decision — a workspace resumed today is overwhelmingly likely to be resumed again this week, and one untouched for a month is probably never coming back.

Decision

Hibernation writes a memory image alongside the filesystem delta for workspaces whose last resume is inside a decay window of seven days. Past the window, a background job drops the memory image and keeps the filesystem chain, so the workspace still resumes correctly, by cold boot, in seconds rather than sub-second. The window is a tuned parameter, not a constant.

How it is realised on this stack

The host agent pauses the microVM, writes the memory image to the region's object store with the filesystem delta, and records which snapshots carry memory. A lifecycle job in the control plane drops expired memory images and rewrites the pointer. On resume, the presence of a memory image chooses between restore and boot; either way the client sees the same progress stream, and the clock, network topology and brokered credentials are refreshed before the session attaches.

Options weighed

• ChosenMemory snapshot inside a decay window, disk-only after: Buys sub-second resume where it is actually used, and bounds the storage bill by recency rather than by population
• RejectedAlways snapshot memory: Simplest rule and the best experience; multiplies snapshot storage several times over for workspaces that will never be opened again
• RejectedNever snapshot memory — always cold boot: Halves the storage design and loses the product: a resumed workspace with a dead dev server and no open files is a new workspace wearing an old name
• Right elsewhereCheckpoint memory inside the guest, application by application: Right for a platform that controls the applications; here the guest is the user's, and the platform deliberately knows nothing about what runs in it

Consequences

What it buys

• The resume target of p95 ≤ 2.5 s is met for the workspaces that are actually resumed, which is where the product lives
• Storage for the long tail collapses to the filesystem chain, keeping hibernated cost at the assumed $0.004 per workspace-hour
• A dropped memory image degrades the experience and never breaks it, so the decay job is safe to run aggressively

What it costs

• Two resume paths exist, and both have to be tested; the fidelity gate in the delivery pipeline covers both for exactly this reason
• A restored memory image is a machine that believes it is still yesterday — stale sockets, old clock, expired tokens — and all three must be reconciled
• Resume latency now depends on a workspace's recency, which is a harder promise to explain than one number

Choose differently when

If memory images could be stored at a small fraction of today's cost, or deduplicated across workspaces sharing a base image far better than they currently can be, the decay window stops earning its complexity and "always snapshot memory" becomes right. Equally, a tenant that only ever runs short agent sessions should be configured disk-only and save the write.

Why it holds up over time

The transferable property is tiering by recency rather than by type: keep the expensive representation for the population that will use it, and keep a cheaper, still-correct representation for everyone else. That pattern long predates microVMs and will outlast them; only the artefact being tiered changes.

The platform's normal case is running unreviewed code for a user who has root. What isolates that from the host, and is the boundary drawn around each workspace or around each tenant?

Context

A hardened container runtime is the densest option and shares a kernel with its neighbours, which means the threat model rests on a kernel that was never designed to be an adversarial boundary. A user-space kernel interposition layer narrows the syscall surface at a measurable performance cost and breaks nested containerisation, which this platform is required to support. A virtual machine per workspace gives each tenant's code its own kernel to attack. The per-tenant variant of any of these buys density by making one developer's fork bomb their colleague's problem.

Decision

Each workspace runs in its own Firecracker microVM with its own guest kernel, jailed on the host, with per-workspace CPU, memory, PID, file-descriptor, disk-size, IOPS and bandwidth caps the workspace cannot raise. The boundary is per workspace, not per tenant, and it is identical for human and agent workspaces.

How it is realised on this stack

The host agent launches Firecracker under its jailer with a minimal device model, a per-workspace network namespace, and cgroup and rlimit caps applied before the guest boots. Limits are visible inside the guest so toolchains size themselves sanely. Nested containerisation happens inside the guest kernel, where it is the guest's problem. Resource-exhaustion classes — fork bombs, OOM loops, runaway writes, thousands of sockets — are contained to the one VM, reported with their cause, and never auto-restarted into the same loop.

Options weighed

• ChosenOne microVM per workspace: Kernel-level isolation with sub-second snapshot and restore, which is also what ADR-01 and ADR-03 depend on; pays about 100–200 ms of start time and some density
• RejectedHardened container per workspace, shared host kernel: Densest and fastest to start, and makes every host kernel CVE a multi-tenant incident in a platform whose whole purpose is running untrusted code
• RejectedUser-space kernel interposition: A real improvement over plain containers, with a syscall tax and no clean story for nested containers or memory snapshots
• Right elsewhereOne VM per tenant, containers inside it: Right for a single-team private deployment where the tenant already trusts itself; here it makes noisy-neighbour and fork-bomb containment a tenant-internal problem the platform cannot fix

Consequences

What it buys

• The isolation claim is structural: an escape reaches a kernel the platform is willing to lose, and reaches no other tenant's code
• Snapshot and restore of a whole machine becomes possible, which is the mechanism the rest of the architecture is built on
• Nested containerisation, the thing developers actually do, works without granting the host any additional trust

What it costs

• Density is lower than a container platform's, and the 4:1 memory oversubscription in ADR-11 exists to win some of it back
• The host agent and the virtualisation layer become the most privileged, most security-critical code in the system
• Kernel patching is now a fleet operation on a cadence set by other people's CVEs

Choose differently when

If every tenant were internal and trusted — a single company's own developers running only reviewed code — the per-workspace kernel buys little and a container platform is cheaper and faster. The moment agents or third parties run code, the calculation reverses, and it has already reversed for most of this platform's traffic.

Why it holds up over time

"Put the trust boundary below the operating system the adversary controls" has survived every generation of multi-tenant compute, from mainframe partitions to microVMs, and the arrival of code written by models rather than people has only strengthened it. The specific hypervisor is replaceable; the principle that the boundary is not the guest's own kernel should not be revisited.

A workspace needs to clone and push. Its user has root, it installs arbitrary dependencies, and an agent inside it may read any file. How does a credential get in?

Context

Anything written to the workspace filesystem or environment is readable by the user, by every dependency they install, and by any agent in the session — so an injected token is an exported token, and the only questions are its scope and its lifetime. The alternative extreme, where the platform proxies every privileged operation and holds the identity itself, shrinks the blast radius to nothing and puts the platform in the path of every authenticated call its users make, including ones it has no business seeing.

Decision

No long-lived credential is ever placed in a workspace. The guest asks over a vsock side channel; the host agent attaches the workspace identity, which the guest cannot assert for itself; the broker checks the repository binding and mints a token scoped to that repository and that workspace, valid for fifteen minutes. Every issuance is recorded in the audit log. There is no network path from the guest to the broker or to the control plane.

How it is realised on this stack

A guest helper answers git credential requests and forwards them over vsock. The host agent presents a workload identity issued per workspace to the broker, which checks the binding in the control plane and mints a forge installation token with push scope for one repository. The token is handed back in memory and is re-brokered on every resume rather than restored from a memory image. Pushed content is scanned for credentials on the egress path, failing the push on a high-confidence match.

Options weighed

• ChosenBrokered, workspace-scoped, minutes-long tokens over vsock: Accepts that the token is exfiltratable within its window and makes the window and the scope the defence; keeps git ergonomics intact
• RejectedInjected environment variables or mounted credential files: Trivial to build and indistinguishable from publishing the credential to every dependency in the tree
• RejectedNo credentials at all — proxy every privileged operation: Smallest blast radius by far; puts the platform in the path of every authenticated call, breaks tools that expect to speak to the forge, and makes the platform a far more attractive target
• RejectedInherit the user's own forge token: What most first versions do; one compromised dependency then reaches every repository that user can touch

Consequences

What it buys

• A leaked token has a known scope, a known owner and a fifteen-minute window, which turns a breach into an incident rather than a catastrophe
• The guest cannot claim an identity, so scope is enforced where the attacker is not
• Revocation is implicit: stopping a workspace stops its ability to obtain credentials at all

What it costs

• A broker outage stops pushes and clones for every workspace, so it is on the critical path of the work even though it is off the keystroke path
• Fifteen-minute tokens mean re-brokering, which means every tool that caches credentials has to cope with expiry
• The vsock protocol is a new, privileged surface between untrusted code and the most trusted component on the host

Choose differently when

For an operation whose blast radius genuinely cannot be bounded by scope — signing a release, touching production — the rejected proxy option is the right one, and the platform should mediate rather than delegate. The boundary between the two is the value of what the credential unlocks, not the convenience of the tool.

Why it holds up over time

Short-lived, audience-scoped, workload-attested credentials are where the whole industry has been converging for a decade, and nothing about agents changes the direction — it only raises the volume of consumers who cannot be trusted to keep a secret. The forge, the broker and the attestation format will change; "the machine proves what it is, and gets only what that needs, briefly" will not.

A workspace is a machine on the internet running unreviewed code. What may it reach by default, and should the default differ between a human's workspace and an agent's?

Context

Open egress makes every developer's first install work and makes exfiltration a one-line script. Deny-by-default with a package-registry proxy makes exfiltration hard and makes a developer's first hour a sequence of failures the platform has to explain. The two populations differ in a way that matters: a human notices a blocked host and asks for it, while an agent cannot distinguish a blocked host from a broken one, and an agent is also the population most likely to be carrying someone else's instructions.

Decision

Three postures exist per workspace class — open with audit, allowlisted with audit, and deny-by-default with a pull-through registry proxy. Agent sessions default to deny-by-default; human sessions default to allowlisted with audit. Tenants may change both within platform bounds, the active posture is visible inside the workspace, and every verdict is logged with destination, volume and outcome.

How it is realised on this stack

All egress leaves through a per-workspace policy enforcement point on the host, combining an eBPF-level block with a forward proxy for the allowed protocols; package managers are pointed at a pull-through mirror. Workspace-to-workspace and workspace-to-control-plane paths are blocked outright, including between workspaces of the same tenant. The posture is written into the guest's environment so a failure is legible rather than mysterious.

Options weighed

• ChosenSplit default: deny-by-default for agents, allowlisted for humans: Puts the strictest posture where the risk and the tolerance for refusal are both highest; costs two postures to reason about
• RejectedOne posture for everything, deny-by-default: Cleanest security story and the largest friction bill; it also means a human's workspace behaves differently from the laptop they are replacing, in ways they feel hourly
• RejectedOne posture for everything, open with audit: What most of this product category ships; it makes the platform the easiest exfiltration path in the tenant's estate
• DeferredPer-repository declared allowlists in the environment definition: The right long-term answer — the repository knows what it needs — and it requires a migration the tenant estate is not ready for

Consequences

What it buys

• The population running the least-reviewed code gets the tightest boundary, by default rather than by configuration
• A registry proxy turns the most common legitimate egress into a cached, inspectable, cheap path
• Every verdict is evidence, which makes an exfiltration question answerable after the workspace is gone

What it costs

• Two defaults mean code that behaves in a human's workspace and fails in an agent's, which is a confusing class of bug
• The proxy is a single point of friction for every dependency install in the estate, and a visible one when it is slow
• An allowlist is a living artefact; a stale one is indistinguishable from a broken network

Choose differently when

If agents and humans converge — the agent running in the developer's own workspace, under their identity, on their posture — the split stops being coherent and the platform should choose one posture per workspace rather than per class. That convergence is already visible in the fork-from-snapshot feature in phase 3.

Why it holds up over time

The durable claim is that default network reach should be set by what the code is trusted to do, not by what is convenient to the person starting it, and that a mediated path for the common case is what makes a strict default survivable. That holds regardless of what the registries, the protocols or the enforcement mechanism become.

Creating a workspace means cloning, installing and running post-create hooks — minutes of work that is identical for everyone on a branch. What exactly is cached, and what makes the cache safe to use?

Context

Caching the base image alone leaves the slowest part — dependency installation — on the critical path. Caching a layered image plus mounted dependency caches is faster and still runs the install. Snapshotting the workspace after post-create caches everything, and raises the only question that matters: when is that snapshot wrong? An age heuristic answers it badly, because a prebuild from this morning can be wrong and one from last week can be right. The failure that matters is silent: a workspace that differs from CI in a way nobody notices until a build passes locally and fails in the pipeline.

Decision

The prebuild artefact is a snapshot taken immediately after the post-create step, indexed by a validity predicate over the base image digest, the environment definition revision and the lockfile hash. A valid prebuild may be rebased onto a newer commit of the same branch; an invalid one is never used. Prebuild fan-out is rate-limited per tenant, and the most recent valid prebuild per active branch is retained.

How it is realised on this stack

A prebuild job clones, installs through the registry proxy and runs post-create inside an ordinary workspace, then hibernates it and registers the resulting snapshot in the prebuild index with its predicate values. Creation looks up the predicate; a hit restores the snapshot and applies the commit delta, a miss falls back to an inline build and reports that it did so, with the start-latency cost attributed to the cause.

Options weighed

• ChosenPost-create snapshot with an explicit predicate, rebased onto newer commits: Turns a four-minute create into a twenty-five-second restore and makes staleness a decidable question rather than a guess
• RejectedLayered image plus mounted dependency caches: Simpler to reason about and leaves the install on the critical path, which is most of the four minutes
• RejectedFull snapshot per (repository, commit, definition revision): The best hit rate available and a fan-out bill proportional to commit volume across every active branch in the estate
• RejectedAge-based cache invalidation: What is easy to build, and the source of the exact failure this decision exists to prevent

Consequences

What it buys

• Create from a prebuild meets the assumed p95 of 25 seconds, and a warm-pool claim meets 3 seconds
• Staleness is decidable and inspectable, so a divergence from CI is a bug with a cause rather than folklore
• The same mechanism as hibernation produces the artefact, so there is one snapshot path to maintain and to test

What it costs

• Rebasing a post-create snapshot onto a newer commit is correct until a dependency change makes it wrong, and the lockfile hash is the only thing separating the two cases
• Prebuilds consume fleet capacity for environments nobody may ever open, which is why the fan-out cap and the waste metric both exist
• The predicate couples the platform to how each ecosystem expresses its dependency lock, which is a long tail of small integrations

Choose differently when

In a monorepo where every branch shares one lockfile, the predicate collapses to the definition revision and the whole mechanism simplifies. In a polyglot estate with no lockfiles at all, it degrades toward the layered-cache option, and the honest answer is a slower start rather than a riskier cache.

Why it holds up over time

"Cache the expensive step, and make validity an explicit predicate over inputs rather than a function of time" is the lesson every build system has learned, usually twice. The artefacts and ecosystems will change; a cache whose correctness is decidable from its inputs will keep being the only kind worth trusting.

A single agent task can ask for forty workspaces in a second, ten times the steady-state creation rate. When the fleet cannot absorb that, who waits, who is refused, and how do they find out?

Context

Queueing is the default behaviour of almost every system built without thinking about it, and it is the worst option here: an orchestrator cannot distinguish a queue from a hang, so it retries, which deepens the queue, and a developer resuming an existing workspace ends up behind forty speculative creations. Refusing is better but only if the refusal is legible — classed, attributable and retryable — and only if the choice of who to refuse is made deliberately rather than by arrival order.

Decision

Admission control sits ahead of placement and enforces per-tenant and per-principal concurrency and spend ceilings before any capacity is committed. Under pressure, agent creations are shed first, then new human creations, and resumes of existing workspaces last. Every rejection is explicit, names its cause, and is marked retryable with a suggested delay.

How it is realised on this stack

The admission controller reads ceilings and current commitments from the control plane through a cached view, consults warm-pool availability per (prebuild, machine class) and host headroom per region, and returns a typed rejection the client surfaces verbatim. Human clients see a message with the cause; agent clients get a structured, retryable error. Shed rate by class is a first-class signal on the observability matrix.

Options weighed

• ChosenExplicit, class-aware refusal ahead of placement: Protects the resume path, which is the product, and gives orchestrators something they can act on; costs a refusal that some callers will experience as a hard failure
• RejectedQueue and wait with backpressure: Looks friendlier and converts a capacity problem into a latency problem nobody can see, right behind the keystroke path
• RejectedFirst-come-first-served with no classes: Defensibly fair and lets forty speculative agent sandboxes delay a developer who is already mid-task
• RejectedAdmit everything, let the hosts degrade: The implicit choice in an oversubscribed fleet with no admission layer; it converts a bounded refusal into unbounded slowness for everyone

Consequences

What it buys

• Resume latency is defended under burst, which keeps the promise the product is actually judged on
• An agent orchestrator can implement sane backoff because it is told the truth immediately
• Spend ceilings are enforced where capacity is committed rather than discovered on an invoice

What it costs

• Agents are refused more often than humans by design, which has to be explained to the people paying for agent capacity
• Ceilings are another thing for tenants to tune, and a badly set one looks like a platform fault
• A refusal is only as good as the warm-pool sizing behind it; chronic refusals mean the forecast in ADR-11 is wrong

Choose differently when

If agent sessions became the latency-sensitive population — an interactive agent a developer is watching, rather than a batch fan-out — the shed order inverts, and the class distinction has to be redrawn around interactivity rather than around who is typing. That is a plausible two-year change, and the decision is deliberately expressed as policy rather than as code paths.

Why it holds up over time

"Refuse early, explicitly, and in a chosen order" outlives every scheduler. The durable part is that the shed order encodes a product judgement about whose time matters most at that moment, and that it belongs in configuration where it can be argued with, not buried in a queueing discipline.

A developer clicks a hibernated workspace. Does the client ask for a resume and then poll, or does the connection itself start the resume and stay open?

Context

A poll-based flow is easy to build and makes the common case feel like a page load with a loading screen. It also triples the round trips, needs a client state machine, and tends to fail in the two cases that matter most: a laptop waking on a different network, and a client that gives up while the resume is nearly done. Holding the connection open puts the complexity in the gateway, where the platform controls it, and lets the resume report its own progress.

Decision

Connecting to a hibernated workspace with sufficient authority is the resume request. The edge authorises, asks the control plane once, and holds the client connection open with a progress stream naming the current phase, up to the cold-start deadline. Gateways are stateless with respect to workspace identity, so losing one costs a reconnect rather than a workspace.

How it is realised on this stack

Envoy terminates the connection and routes to a session proxy that subscribes to the incarnation's progress events while the host agent restores. The client receives phase updates — authorised, placed, restoring memory, fetching blocks, attached — and a terminal when the guest agent reports ready. Client network changes reconnect to the same workspace inside the reconnection window, and a dropped connection is explicitly not a workspace event.

Options weighed

• ChosenResume-on-connect, connection held with a progress stream: One round trip, a legible wait, and reconnection semantics the platform controls; costs long-lived connections through the edge
• RejectedExplicit resume call then client polling: Simplest server; every client reimplements the wait, and the two hard cases — sleeping laptops and impatient clients — are reimplemented badly each time
• RejectedPre-resume on dashboard view: Would make most resumes feel instant and would pay for every workspace a developer looked at and did not open
• RejectedKeep everything running — no hibernation at all: The only way to make resume free, and it deletes the business model described in ADR-11

Consequences

What it buys

• The wait has a cause attached, which is the difference between a slow platform and a broken one
• Gateway loss and client network changes degrade to a reconnect, not to a lost session
• Start-latency telemetry is attributable by phase, because the phases are already being streamed

What it costs

• The edge carries tens of thousands of long-lived connections, so its capacity planning is connection-bound rather than request-bound
• A resume that exceeds the deadline must fail in a way that leaves the workspace resumable, which is an extra path to get right
• Holding connections during a burst consumes edge capacity exactly when admission is already refusing work

Choose differently when

If clients converged on one well-behaved implementation under the platform's control, polling would be perfectly adequate and would simplify the edge considerably. With four transports and third-party agent runtimes, that convergence is not available.

Why it holds up over time

The durable idea is that the system should absorb the complexity of waiting rather than exporting it to every client, and that a wait with a named cause is a different product from a wait without one. Transports will change; this division of labour should not.

Hibernation is what makes the economics work, and a hibernation at the wrong moment is the worst bug the platform can ship. Who decides that nobody is working?

Context

Client heartbeats are the easiest signal and the most wrong: a developer running a forty-minute build is not typing, and a closed laptop lid with a background agent still working looks identical to an abandoned tab. Guest process activity knows about the build and can be gamed by anything that polls. Host-level network and disk activity is unforgeable and cannot tell a compile from a crypto miner. The two errors are asymmetric in kind rather than in size: a false hibernation destroys trust in the platform, while a missed hibernation destroys the margin quietly.

Decision

Idleness is a composite evaluated by the host agent: absence of client input, absence of meaningful guest process activity, and absence of disk and network work, all over a tenant-configured window with a shorter default for agent sessions. A human workspace may request a bounded veto, measured in minutes and logged; an agent workspace may not. Hibernation accuracy is reported as a product metric, not only as a cost one.

How it is realised on this stack

The guest agent reports process and terminal activity as advisory input; the host agent combines it with cgroup CPU, block IO and egress counters, which the guest cannot forge. A long-running foreground process holds the veto while it reports progress, up to a cap. False hibernations and missed idleness are counted on the observability matrix, and a tenant's timeout is policy read from cache rather than from the control plane.

Options weighed

• ChosenComposite signal on the host, bounded guest veto: Gets the build case right without trusting the guest, and bounds the damage a lying guest can do to the margin
• RejectedClient heartbeats only: Trivial, and it hibernates the developer who is watching a test suite run, which is the single most trust-destroying failure available
• RejectedGuest-reported liveness only: Knows the most about what is happening and is forgeable by anything with a while-loop, which is every workspace
• Right elsewhereFixed timeout with no activity detection: Right for a pure agent-sandbox platform where sessions are short and uniform; wrong for human workspaces, where it is the same failure as heartbeats with extra steps

Consequences

What it buys

• Long builds and long test runs survive, which is what makes aggressive default timeouts acceptable at all
• The signal is unforgeable in the part that costs money, and informative in the part that protects the developer
• Hibernation accuracy becomes measurable, so the oversubscription ratio in ADR-11 can be tuned against evidence

What it costs

• Three inputs and a veto is substantially more logic than a timeout, on the path that runs for every workspace continuously
• A workspace that polls can keep itself alive longer than it deserves, and the ceiling on the veto is the only thing bounding that
• Tenants can set timeouts that are economically wrong for them, and the platform has to show them the cost rather than refuse

Choose differently when

If a reliable, unforgeable signal of human attention existed at the client — and it may, from the editor itself — the composite could collapse toward it and the veto would become unnecessary. Until then the host is the only place a signal cannot be faked.

Why it holds up over time

Every platform that bills for utilisation eventually has to decide what "in use" means, and the durable answer is: measure at a boundary the user cannot forge, take advisory input from where the knowledge is, and treat the two error directions as different products rather than as a single threshold. That survives any change in what runs inside.

Does the platform own hosts, rent bare metal, or run on standard cloud instances — and how much may it promise that it does not physically have?

Context

Standard cloud instances are the easiest to start with and the hardest to make this architecture work on: nested virtualisation is restricted or slow, local NVMe is not guaranteed, and the price per workspace-hour at the target margin does not close once the hypervisor tax is paid twice. Rented bare metal removes the technical obstacles and keeps a commercial dependency on a supplier's inventory during exactly the bursts that matter. Owning the fleet buys the density and the price and brings hardware failure and kernel patching in-house. Separately, a workspace at 8 GB nominal is idle most of the day it is open, so a fleet sized for the nominal sum is a fleet mostly holding nothing.

Decision

The platform runs on owned bare-metal hosts in its own racks, with local NVMe and hardware virtualisation, and oversubscribes memory 4:1 against measured workspace idleness. Hibernation is the declared relief valve: under memory pressure the host agent hibernates the least recently active workspaces rather than letting the host degrade. The ratio is a tuned parameter with a published headroom signal, not an assumption.

How it is realised on this stack

Hosts are 128-core, 1 TB machines carrying about 40 nominal workspaces each, in two regions. The host agent watches resident memory and block pressure, and hibernates on a declared ladder — idle first, then least recently active, never a workspace whose veto is held. Admission accounts for committed rather than nominal memory, and oversubscription headroom is a fleet-level signal with its own alert.

Options weighed

• ChosenOwned bare metal, 4:1 memory oversubscription, hibernation as relief: Gets the snapshot capabilities, the density and the price; takes on hardware operations and the risk that the idleness signal is wrong
• DeferredRented bare metal from a provider: The right first step for a smaller fleet and the natural overflow path for a burst; kept as the stated expansion mechanism rather than the base
• RejectedStandard cloud instances, no oversubscription: Operationally simplest and breaks both the snapshot strategy and the price; it is also what makes most competitors' free tiers unprofitable
• Right elsewhereDedicated host per tenant: Right for a regulated tenant who will pay for it, and offered as such; wrong as the default because it discards the idleness that pays for everything

Consequences

What it buys

• The assumed $0.085 per active workspace-hour and $0.004 hibernated are achievable, which is the whole commercial argument
• Hardware virtualisation and local NVMe are guaranteed, so the snapshot and lazy-fetch designs are not fighting the substrate
• Capacity decisions are the platform's own, which matters most during the bursts a supplier would also be absorbing for someone else

What it costs

• Kernel patching, firmware, disk failure and rack capacity become the platform team's problem, with the fleet operations that implies
• 4:1 is only safe while hibernation works; a bug in ADR-10 becomes an availability incident within a working day
• Burst headroom is physical, so a 10× agent burst beyond forecast ends in refusals rather than in a larger bill

Choose differently when

If a provider offered bare-metal capacity with guaranteed snapshot-grade local NVMe at a price close to owning it, the deferred option becomes the better one and the ratio stops being the platform's risk. Equally, if measured idleness fell — because agents, which are rarely idle, kept growing as a share of the fleet — the ratio must fall with it, and the published headroom signal is what makes that visible before it hurts.

Why it holds up over time

The durable claim is that a platform may promise more than it has only when it owns a correct, fast mechanism for taking it back. That is as true of memory here as it is of thin-provisioned storage or an airline seat; the mechanism is what makes the promise honest, and a design that oversubscribes without one is simply wrong at a time it has not met yet.

What may the control plane know, and what must keep working when it is gone?

Context

It is tempting to let the control plane see everything: it already knows every workspace, so routing logs, port metadata, even file lists through it makes features easy. Each of those makes the control plane a bigger target, a bigger backup problem and a bigger outage. Separately, if a running workspace needs the control plane for its keystrokes to work, then the management plane's availability becomes the product's availability, and the product's is the harder number.

Decision

The control plane holds records only — workspaces, tenants, seats, policy, placements, prebuild validity, quotas and events. No filesystem data, memory image or workspace log transits control-plane storage. Running workspaces and established sessions survive a total control-plane outage; creation, resume and policy change stall, and policy is read by the data plane from a cached view rather than synchronously.

How it is realised on this stack

Host agents and edge gateways hold a cached projection of the policy and placement they need, refreshed from the event stream and safe to run stale for a bounded period. Workspace content goes directly from the host agent to the object store. Audit and lifecycle events flow out on Kafka, so evidence is durable without the control plane being in the data path.

Options weighed

• ChosenRecords-only control plane, data plane runs independently: Decouples the product's availability from the management plane's and keeps the highest-value target small; costs cache staleness and a reconciliation story
• RejectedControl plane in the data path: Makes features easy and ties a developer's keystrokes to a Kubernetes control plane's worst day
• RejectedContent inspection in the control plane for support features: Would make debugging a tenant's workspace much easier, and makes the platform a repository of every tenant's source by accident
• Right elsewhereFully decentralised, no control plane: Right for a single-user tool; here quotas, policy and residency need a consistent arbiter

Consequences

What it buys

• A control-plane outage degrades to "no new workspaces" rather than to "nobody can work"
• The backup and residency problem collapses onto two stores — the registry and the snapshot store — instead of five
• The most attractive target in the system holds no source code

What it costs

• Cached policy can be stale, so a revocation or a posture change has a bounded propagation delay that must be stated
• Debugging crosses a boundary deliberately: support cannot read a workspace's contents, which makes some tickets slower
• Reconciliation after an outage has to be designed, because hosts will have made decisions the control plane has not seen

Choose differently when

If a tenant required immediate, synchronous enforcement of a policy change — a legal hold, or an urgent revocation — the cached model is insufficient for that path, and it is handled by a kill switch propagated on the event stream rather than by making the whole design synchronous.

Why it holds up over time

Separating a control plane from a data plane, and refusing to let the control plane hold the payload, has survived every generation of networking, storage and compute platform. The specific caches and projections will change; the rule that the thing you can least afford to lose should not depend on the thing you change most often will not.

The isolation boundary is a kernel, and kernels need patching on somebody else's schedule. How does a fleet carrying 60,000 live workspaces get patched?

Context

The honest options are a maintenance window, live migration, or drain-and-resume. A window asks developers to stop working on a security team's schedule, and in practice means patching slips. Live migration of a running microVM keeps sessions warm and is a substantially harder engineering problem, with a correctness tail — in-flight devices, open sockets, memory convergence under load — that shows up as corruption rather than as slowness. Drain-and-resume reuses the mechanism the platform already has to be correct for host loss and idleness.

Decision

Patching, hardware replacement and rebalancing all use one mechanism: mark the host draining so it accepts no new placements, hibernate its workspaces in rate-limited waves, resume them on other hosts, then patch. Live migration is explicitly not built. Resume success rate under drain is a release gate for the platform's own changes.

How it is realised on this stack

The host agent refuses placements when draining and walks its workspaces in waves, honouring vetoes up to their cap. Each workspace hibernates with memory state where it has it and resumes elsewhere immediately, so the developer sees a pause of a few seconds. Drain waves are rate-limited fleet-wide so that storage and the edge are not saturated by evacuation traffic, and the fleet-wide drain state is an operational signal.

Options weighed

• ChosenDrain, hibernate, resume elsewhere: One mechanism for patching, host loss, idleness and rebalancing, so the hardest path in the platform is also the best-tested one
• RejectedLive migration of running microVMs: Best possible experience — no pause at all — at the price of a correctness tail that manifests as a corrupted workspace rather than a slow one
• RejectedScheduled maintenance windows: Operationally trivial and it asks developers to stop working, which is how patching slips in every organisation that tries it
• RejectedTerminate and let workspaces be recreated: Acceptable only on a platform whose workspaces are stateless, which this one deliberately is not

Consequences

What it buys

• The security cadence stops competing with the availability promise, which is what makes a per-workspace kernel sustainable
• Every maintenance operation exercises the resume path, so the platform's most critical mechanism is continuously proven in production
• Hardware failure needs no separate procedure: an unplanned loss is a drain that did not get to say goodbye

What it costs

• Developers experience a short pause during a drain, and a drain is therefore always visible to someone
• Evacuation concentrates snapshot writes and block fetches, so storage must be sized for drain traffic rather than for steady state
• A drain slower than a CVE window creates pressure to terminate workspaces instead, which is exactly the behaviour the design exists to make unnecessary

Choose differently when

If live migration became reliable and cheap to operate — as it did for conventional virtualisation over roughly a decade — the pause disappears and this decision should be revisited for the human-session population. It would never be worth building for agent sessions, which are gone before a migration completes.

Why it holds up over time

The durable move is to make the exceptional path the routine one, so that the mechanism you depend on in a crisis is the mechanism you ran a thousand times this week. That principle has outlasted every specific recovery technology, and it is the reason this platform does not have a separate disaster procedure.

A region fails. Its workspaces are unavailable and their content is intact inside it. Should the platform resume them somewhere else?

Context

Automatic cross-region resume is what an availability-first design does, and it requires workspace content to exist outside its region, which is the exact thing a residency promise says will not happen. The content in question is the tenant's source code and working state, and most enterprise tenants chose the region deliberately. Meanwhile the availability math is less compelling than it looks: a regional failure makes workspaces unavailable for the duration, which is costly, while a silent residency breach is a contractual and regulatory event that cannot be undone by restoring service.

Decision

Workspace content and snapshots never leave their region. A region loss makes its workspaces unavailable until the region returns; there is no automatic failover. Migrating a workspace to another region is an explicit, tenant-authorised operation that copies content across the boundary with its own audit record. Residency outranks availability, and the asymmetry is stated in the tenant contract rather than discovered during an incident.

How it is realised on this stack

The tenant's region is an attribute enforced on every placement, snapshot write and route; the object store is region-local with no cross-region replication configured. Global services hold identity and the image catalogue only, and the catalogue is mirrored per region. The migration path is an API call with an authorisation step, producing a copy in the target region and an audit event naming who authorised it.

Options weighed

• ChosenHard residency, migration only on tenant authorisation: Keeps the promise that is hardest to recover from breaking, and accepts a worse availability number during a regional failure
• RejectedAutomatic cross-region resume: Better availability during a rare event, achieved by replicating every tenant's source code out of the region they chose
• DeferredTenant-selectable: replicate or pin: The honest long-term answer, and it doubles the storage model and the failure modes; deferred until a tenant asks and will pay for it
• Right elsewhereGlobal, region-agnostic placement: Right for a consumer product with no residency obligations; impossible for the enterprise tenants this platform is sized for

Consequences

What it buys

• The residency claim is enforced by configuration rather than by promise: there is no cross-region copy to accidentally enable
• The backup and key-management story stays regional, which keeps customer-managed keys meaningful
• A failure during a regional incident is unavailability, which is explainable, rather than a breach, which is not

What it costs

• A regional failure costs the region's developers their working day, and the platform has no technical answer to that during the event
• Capacity cannot be shared across regions during a burst, so each region must carry its own headroom
• Tenants who would have preferred replication must opt into a migration they have to authorise

Choose differently when

A tenant with no residency constraint and a strong availability requirement should be offered the deferred option, and the design is deliberately shaped so that replication is an added capability rather than a removed restriction. The default stays pinned, because the failure direction of the default is the one that can be undone.

Why it holds up over time

When two promises conflict, the durable rule is to keep the one whose breach cannot be repaired. Regulatory and contractual commitments about where data lives have only hardened over the last decade, while availability expectations are negotiable and measurable; that asymmetry is likely to outlast every specific regulation.