An engineering organisation of 400 moved the inner loop into cloud development environments - the repository only builds inside the platform's container image and the IDE attaches to a remote workspace. At 09:40 on a Tuesday the workspace control plane stops scheduling new workspaces and existing ones cannot be restarted. It stays down for two hours. Walk through what happens, and what would make it survivable.

Minute by minute

09:40. Nothing visible. Engineers with a live workspace keep working; the control plane is only in the path for creation and restart.

09:55. The first casualties are people who were already restarting - after an out-of-memory kill, a dependency change, or simply closing a laptop lid. They cannot get back in and they file tickets.

10:20. Attrition does the damage. Workspaces are routinely reaped on idle timeouts of 30 to 60 minutes, so the population of working engineers decays even though nothing new has failed. Anyone who steps into a meeting returns to nothing.

10:40. The platform team discovers its own tooling runs in the same workspaces.

11:40. Restoration. Then 400 people create workspaces at once, each pulling a multi-gigabyte image, and the recovery becomes its own capacity event.

Where it amplifies

The serious version is not lost productivity, it is the inner loop becoming a dependency of incident response. If a production incident starts at 10:00, the engineers who must write and test the fix have no place to build it. A workspace outage is then a reliability incident for every service in the estate, which is not how it was risk-assessed when it was bought.

What the business sees

If roughly 60% of 400 engineers are actively coding at that hour, two hours is around 480 engineer-hours, a quarter of a person-year for one morning. That number buys attention; the dependency-of-incident-response argument is what buys the fix.

What stops it

• Keep the top repositories buildable on a laptop, tested in CI weekly so the claim stays true. This is the only measure that survives a total control-plane loss.
• Pin the toolchain in a lockfile rather than only in an image, so the local path and the remote path are the same inputs and the fallback is not a different build.
• Decouple the data plane from the control plane: a running workspace should survive without it, and idle reaping should fail safe by keeping workspaces alive when the scheduler is unreachable.
• Keep the break-glass deploy and rollback path outside the workspace plane, so an incident can be mitigated by someone with only a terminal.
• Cache workspace images on the node pool so recovery is not a thundering herd against the registry.

When this is the wrong answer

In a regulated estate where source code must not land on an engineer's machine, "make it build on a laptop" is not available. The fallback is then a second control plane in another region with pre-baked images and a rehearsed cutover, which is genuinely expensive. Below about 50 engineers, the honest answer is that cloud development environments are not worth their control plane at all - a scripted local setup and a shared remote box for the hard cases cost less than the failure mode described here.