Flag Change Provenance

Feature flags are built for velocity: change a value, the fleet picks it up within seconds, nobody deploys. That property is exactly right for a release toggle and exactly wrong for a value with financial or regulatory consequence.

The moment someone asks "which orders were priced under the old value", the flag system's usual answer — a change log with a timestamp — turns out to be insufficient. A flag flipped at 14:00 reaches instances over anything from 30 seconds to several minutes depending on cache expiry and evaluation frequency, and some services evaluate once per session rather than per request. For every artefact near the boundary, replaying the change log against timestamps produces a guess, and a guess is not evidence.

Provenance therefore has two halves, and teams routinely build only the first.

Why it matters

Because the question that arrives is never the one the audit log answers. "Who changed it" is a governance question and a change log satisfies it. "What was applied to this order" is a reconstruction question, and it can only be answered by data recorded at the time the decision was made.

The asymmetry is that the second requirement is cheap if designed in and impossible to retrofit. Once a million orders have been priced without a configuration version stamped on them, no amount of later tooling recovers it.

Implementation patterns

• Classify flags by kind. Release toggles (short-lived, engineering-owned, deleted after rollout), operational switches (owned by whoever is on call), and business configuration. Different permissions, different approval, different retention. Most estates put all three in one system with one permission model, which is the root cause.
• Append-only change log with actor, timestamp, previous value, new value and a reason; no in-place edits, no deletion.
• Second-person approval for the business-configuration class, enforced by the platform rather than by convention.
• Stamp the version on the artefact. The order records the price and the identifier of the configuration version that produced it. This is the half that makes reconstruction a join rather than an argument.
• Publish change events, so downstream systems can react and so the change is visible in the same timeline as deploys and incidents.
• Set retention from the business requirement, which in finance is commonly 7 years — far beyond what flag platforms are built for, and the strongest signal that the value belongs elsewhere.

Industry example

The pattern is the same one used in production for pricing and rating engines in regulated industries, where the rate card in force is versioned and its identifier is written onto every transaction, precisely so that a dispute years later is resolved by reading the record rather than reconstructing it. Payment and billing platforms take the same approach with API versions pinned per account: the artefact carries the version that produced it, rather than the system inferring it from a clock.

Failure scenarios

• Reconstruction by timestamp fails exactly on the disputed orders, because propagation and caching blur the boundary.
• A console flip with no record, because the flag platform allowed a direct edit outside the approved path.
• Retention expiry, where the flag history is 90 days and the question arrives at 18 months.
• Cache divergence during the change window, so two instances priced concurrently under different values with nothing to distinguish the outputs.
• Permission sprawl, where anyone who can toggle a release flag can also change a price, discovered during an access review.

Trade-offs

Provenance costs velocity and storage. Second-person approval turns a five-second change into a request; stamping versions onto artefacts adds a column and a lookup to a hot write path; keeping history for years means a store that is not the flag platform. In exchange, a class of question becomes answerable by query instead of by reconstruction — and the cost is paid on a small minority of flags, provided the classification is done.

When not to use it

Do not apply this to release toggles. They are short-lived, they are deleted after rollout, and the worst outcome of a wrong one is a rollback. Forcing approvals, immutable history and artefact stamping onto them slows down the thing flags exist to make fast, and teams respond by routing around the system. The trigger is consequence, not caution: if the worst outcome of a wrong value is a number someone has to defend to a regulator, a customer or a finance team, it needs provenance and probably does not belong in the flag platform at all.

Interview question

Q: A feature flag controls a pricing rule, and finance asks who changed it, when, and which orders were priced under the new value. Walk me through what that requirement does to your design.

What a strong answer covers: recognising it is business configuration rather than a flag; classifying flags into three kinds with different controls; the append-only log with actor and previous value plus second-person approval; and above all the point that reconstruction by timestamp fails because of propagation delay and caching, so the configuration version must be stamped onto each order at the time of pricing; plus the retention argument for moving the value out of the flag platform.

Quick check

Quiz: Why can a flag change log not answer "which orders were priced under the old value"? Because the change reaches instances over a propagation and cache window, and some services evaluate once per session, so timestamps near the change do not determine which value was applied.

Flashcard: What are the two halves of flag provenance? An append-only change log (who, when, from what, to what) for governance, and the configuration version stamped onto every artefact produced, because only the second makes reconstruction a join rather than a guess.