Roblox 2021: A Coordination Layer as a Single Point of Failure
also called Roblox 73-Hour Outage
A performance problem in the shared service-discovery cluster took the entire platform down for 73 hours, and its novelty made it extremely difficult to diagnose.
What happened
In late October 2021 Roblox was down for approximately 73 hours. Their published postmortem describes a compounding failure centred on their HashiCorp Consul cluster, which provided service discovery and configuration for essentially every service on the platform.
Two contributing factors combined. A newly enabled streaming feature in Consul increased load on the write path in a way that produced contention under Roblox's particular usage pattern. Separately, a performance characteristic in Consul's underlying storage engine — related to how it managed its internal free list — degraded write performance as the workload grew.
Neither had been seen at this combination of scale and configuration before, which is why diagnosis took so long. The engineers were debugging a novel interaction, under pressure, in a system that everything else depended on.
The architectural point
Consul was doing what it was designed to do: provide a single, consistent view of service topology. That centralisation is the value — and it means the coordination layer's availability is an upper bound on the platform's availability.
Service discovery, configuration stores, feature flag services, identity providers and secret managers all share this property. They are small, they feel like infrastructure rather than architecture, and everything depends on them.
The transferable lesson
Identify your shared coordination dependencies and treat their availability requirement as higher than any individual service's. If forty services each targeting 99.9% all hard-depend on one component, that component needs to be substantially better than 99.9% or none of them can meet their target.
Then reduce the dependency where you can. Cache the configuration locally with a long fallback so a control-plane outage does not stop the data plane. Ensure services can continue with their last known good topology rather than failing when discovery is unavailable. And be cautious about enabling new features in a component that everything depends on — the blast radius of a configuration change there is the whole platform.