Flag Propagation Time

An incident starts. Someone turns off the flag guarding the new checkout flow. The error rate falls by half over the next ten minutes and does not reach zero for an hour.

"Off" is not a state of the system; it is a state that propagates. A server-side flag takes effect on the next request. A client-side flag takes effect the next time the client asks - and there are several reasons the client may not ask for a long time.

Why it matters

A flag's specification is not "on or off". It is "on or off, with a worst-case propagation time T", and a team that does not know T cannot use the flag as a control. That distinction decides what the flag is for:

• T of seconds: usable as a kill switch during an incident.
• T of minutes: usable for progressive rollout, not for incident response.
• T unknown: usable for neither, whatever the runbook says.

A kill switch with a 30-minute tail is not a kill switch, and discovering that during an incident is the worst possible moment.

Implementation patterns

Four delays stack, and each has a countermeasure:

• Session length. A user with the tab open already evaluated the flag. Re-evaluate on visibility change and on route change, which is the cheapest opportunity to refresh and costs nothing.
• SDK polling interval, commonly 30-60 seconds by default and sometimes fetch-at-initialisation only. Shorten it for safety-critical flags, or push.
• Cache TTL on the flag payload, often served through a CDN. Set it deliberately and low for the control flags; a misconfigured TTL turns minutes into hours.
• Build-time decisions. Anything compiled into the bundle rather than read at runtime does not change until the user loads a new bundle. Audit for this; it is the delay with no upper bound.

Two further practices: push rather than poll for the small number of flags that are genuinely safety controls - a server-sent event or websocket channel - and enforce anything that matters on the server as well, because a client flag controls what is displayed and only a server check controls what is possible.

Industry example

Every published account of progressive delivery treats propagation as a first-class property, which is why platforms expose streaming flag updates alongside polling: the polling default is correct for the large majority of flags and wrong for the handful used to stop an incident. The recurring industry lesson from flag-related outages published since 2015 is the same one - the flag was turned off promptly and the impact continued, because the mechanism people trusted as a control was a cache with a TTL.

Failure scenarios

• The kill switch that does not kill. Impact continues for the length of T while the incident channel believes the mitigation is applied.
• The default on failure. When the flag service is unreachable, the client falls back to a hard-coded value. If that value is "on", a flag-service outage becomes an unplanned feature launch.
• Partial propagation during a rollback, so some users run new client code against rolled-back server behaviour - a state nobody tested.
• Long-lived tabs. Users who never reload hold a value for days, and they are disproportionately the heaviest users.
• Stale bundles. A flag read at build time, so the flag console shows off and the behaviour is on.
• T never measured, so the runbook's estimate is folklore.

Trade-offs

Short propagation is not free. Aggressive polling multiplies requests to the flag service by the number of active clients divided by the interval, which is a real load and a real bill at scale; a push channel is another persistent connection per client with its own scaling and failure behaviour. The sensible split is push for the few flags that are controls and polling for the rest, which keeps the cost proportional to the value.

When not to use it

For presentation-only flags - a colour, a layout variant, a copy change - the tail is harmless and the default polling interval is correct. Measuring and optimising T there is effort spent on a risk that does not exist.

The distinction is simply whether the flag protects anything. If it does, it belongs on the server, with the client flag as a user-experience nicety rather than the mechanism. A client-side flag should never be the only thing standing between a user and an action they are not entitled to take.

Interview question

Q: Your incident runbook says "disable the flag" as the first mitigation for the new payments flow. What would you check before trusting that step, and what would you change if the answer is unsatisfactory?

What a strong answer covers: asking where the flag is evaluated and measuring T end to end rather than assuming it · enumerating the four delays and which apply · confirming there is a server-side enforcement point, since a client flag only changes what is shown · checking the fallback value when the flag service is unavailable, because that is the behaviour under the correlated failure · proposing a push channel for the small set of control flags while leaving the rest on polling for cost reasons · and adding T to the runbook as a stated number, so the incident commander knows how long to wait before concluding the mitigation failed.

Quick check

Quiz: Why can a client-side flag not be a security control? Because it decides what the client displays, not what the server permits - and its new value takes an unbounded time to reach clients. Only a server-side check controls what is possible.

Flashcard: What are the four delays that stack in client flag propagation? — Session length · SDK polling interval · cache TTL on the flag payload · anything decided at build time rather than read at runtime.