A web application evaluates feature flags in the browser. A flag is turned on for 10% of users and the new code throws during application start-up, so those users see a blank page. Turning the flag off does not help them. Why not, and what is the structural fix?
Show the full answer Hide the answer
The trigger
The flag was evaluated during start-up, and the new code path ran during start-up too. The exception happens before the application reaches the point where it would fetch fresh flag values, so the process that would learn the flag is now off never runs. The user reloads, the cached flag value is read again, the same code throws again.
Whether the cache is a bundled default, localStorage, or a value embedded in the server-rendered HTML, the outcome is the same: the switch is behind the crash.
Why it propagated instead of being contained
Three properties turned a 10% experiment into a 10% outage:
- The flag gated code on the boot path. A flag around a feature the user must click is naturally safe, because the application is already running when it is read. A flag read during initialisation has no such protection.
- Nothing measured start-up success. Error monitoring lives inside the application, and an application that dies during boot often fails to report it, so the dashboards showed a drop in errors along with a drop in traffic.
- The rollout had no automatic halt. A 10% cohort with no health gate is a 10% blast radius for as long as nobody notices, and the affected users cannot file a report through a product that will not load.
Why detection lagged
The metric that would have caught it in minutes is the ratio of application-ready events to page views, segmented by flag variant. Almost no team has it, and it is a few lines of code: emit a beacon from the HTML before the application boots, and another once the application is interactive. The gap between them is the boot failure rate, and it is measurable even when the application is the thing that broke.
The structural fix
- Never let a flag decide anything before the failure-detection path exists. Emit the "boot started" beacon first, from inline script, before any application code.
- A launch-failure counter in local storage. Increment before boot, clear when the application is interactive. On the second or third consecutive failure, ignore all cached flags and boot the known-good configuration. This is the client-side circuit breaker, and it is what actually rescues the stuck users.
- Make flag values fetched fresh on boot with a short timeout and a safe default, rather than read from a cache that can pin a bad value indefinitely. Cache for offline, but treat a failed boot as a reason to distrust the cache.
- Gate the rollout on a health signal, not a schedule. 1%, then 10%, with an automatic halt on a rise in boot failures. A ramp with no gate is a schedule, not a rollout.
- Structure the code so the flag guards a lazily-loaded module. If the new code is in a chunk that loads on demand, a failure to load it cannot take the application down, and the fallback is the old path.
The general lesson
A control that lives inside the thing it controls is not a control. The same shape recurs in mobile applications with kill switches read after initialisation, in configuration systems that need the service they configure, and in observability stacks that depend on the infrastructure they observe. The test is a single question: if this fails as badly as it can, is the mechanism that would stop it still running?
When this is less of a concern
A flag evaluated on the server, or one that gates a lazily-loaded feature, does not need this machinery. Server-side evaluation means a bad flag produces a bad response the server can stop sending, with no cached client state involved, which is a strong argument for keeping risky flags on the server and sending the client a decision rather than the rules. The boot-path defences are for the small set of flags that genuinely must be evaluated in the browser before the application runs.
The trade is worth naming. Client evaluation costs you this failure class and buys an instant decision with no round trip; server evaluation costs a request per decision and a cache to invalidate. Choose client evaluation for presentation, and server evaluation for anything that can stop the application starting. The lesson generalises past the browser: since the Knight Capital deployment of 2012, every medium has produced its own version of the same finding, which is that a switch depending on the code it governs is not a switch.