Cloud Development Environment Platform  ·  View 06 of 21  ·  People and journeys

Journey — Patching 1,800 Hosts Without Anyone Noticing

The operator journey that is only possible because a workspace does not belong to a host.

Editable source SVG draw.io All views
Platform SRE on call for the fleet Goal — Land a kernel fix on every host this week Trigger — A kernel CVE with a public exploit Done when — Fleet patched, no developer ticket, no workspace lost 1 · Plan 2 · Drain ◆ moment of truth 3 · Patch 4 · Return 5 · Prove What they do Picks a drain rate Marks hosts draining Rolls the kernel Returns to pool Reads the evidence What the platform does Shows live workspaces Hibernates each one Resumes elsewhere Refuses new placements Rebalances warm pools Reports resume success How it feels Confident Fine Exposed Where it hurts No idea who is mid-build A pause looks like a crash What answers it Liveness per workspace Snapshot, not terminate Resume on any host Patching is scheduling 99.9% resume success Journey — Patching 1,800 Hosts Without Anyone Noticing This journey only exists because the workspace is a snapshot: on a host-owned design the same CVE becomes a maintenance window. v 1.0 · owner Platform Architecture · date 2026-10

What this settles

  • Kernel patching is a scheduling problem, not a maintenance window: hosts are drained in waves, workspaces hibernate and resume elsewhere, and developers see a pause (ADR-13).
  • The operator's trough is not knowing who is mid-build, which is why per-workspace liveness is an operational signal and not only a cost signal.
  • Resume success rate is the evidence that the drain worked, which makes it a release gate for the platform's own changes (view 17).

Assumptions

  • 1,800 hosts; drain waves rate-limited so that no more than a small fraction of concurrent workspaces pause at once.
  • Resume success ≥ 99.9% of attempts; a failed resume leaves the workspace resumable.

Risks

  • A drain that is slower than the CVE window pushes operators toward terminating workspaces, which is the one thing the design is meant to make unnecessary.
  • Evacuation concentrates snapshot writes and block fetches; the storage tier must be sized for drain traffic, not only for steady state.