Roblox's 2021 outage lasted 73 hours after a service-discovery and key-value cluster degraded under contention. Trace the failure chain, and identify the three architectural properties that turned a degradation into a three-day outage.

The failure chain

1. A streaming feature enabled on the service-discovery cluster changed the read pattern, increasing load on the cluster's write path under a specific contention condition.
2. Contention on a lock inside the cluster caused latency to rise rather than the system to fail — requests were served, slowly.
3. Every internal service depended on that cluster for discovery and configuration, so slow discovery became slow everything. Services began retrying, which added load to the already-contended cluster.
4. The cluster's own data grew as a result of the degraded state, making recovery slower than a simple restart.
5. The telemetry system depended on the same cluster, so as it degraded, the engineers lost the ability to see what was happening — during the period when seeing it mattered most.
6. Recovery required restoring from a known-good state and carefully re-admitting load, because a full-fleet reconnect would have reproduced the contention instantly.

The three properties that turned degradation into a three-day outage

• A universal dependency with no bulkheads. One cluster served every service in the fleet. Its blast radius was the company. There were no cells, so there was no partial failure available — only total.
• Circular observability. The tools needed to diagnose the failure ran on the failing substrate. Engineers were blind exactly when they needed sight, which converts a technical problem into a search problem and multiplies time-to-restore.
• No graceful path back in. Recovery meant reconnecting a very large fleet to a cluster whose failure mode was contention under load. Without throttled re-admission the recovery attempt is the same load spike that caused the outage, so each attempt fails and consumes hours.

Underneath all three: the failure was gradual degradation, not a crash. Health checks that ask "is it up?" return yes throughout. Retries — the standard correct response to a transient error — made it worse, because the dependency was saturated rather than broken.

What the architecture should have

Cell-based isolation with a discovery and configuration control plane per cell, so a contention bug affects one cell's worth of users. Observability on an independent path — a separate telemetry stack with its own storage, plus a minimal break-glass signal that survives everything. Caching of discovery results with a long stale-serving window, so services survive a discovery outage using last-known-good endpoints rather than failing. Throttled re-admission as a designed, rehearsed capability. And retry budgets with circuit breakers, so that a saturated dependency sees load fall rather than rise.