Prompt Provenance Record

A product team keeps prompts in a database table, edited through an internal interface, with no version history. Outputs are logged with the user identifier and the response text. An A/B test assigns users between a current row and an experiment row and produces a clear winner.

The result is worthless. The rows are edited in place, so an edit during the experiment silently changed one arm mid-flight, and nothing in the log distinguishes before from after. The experiment still produces a number, and the number will be believed - which is why running it is worse than not running it.

A prompt provenance record is the fix and it is small: with every output, store the prompt version identifier, the model name and version, the sampling parameters, and the identifiers of any retrieved context. Then "which prompt produced this answer" has an answer.

Why it matters

Three capabilities depend on it and none is available without it.

Diagnosis. Yesterday's worst response is the starting point of most quality work. Without provenance, the investigation ends at "we think someone changed the prompt", and the team learns nothing from its own incidents.

Experimentation. Attribution of outcome to arm is the definition of an experiment. A mutable prompt row cannot provide it, so the organisation makes decisions on numbers that mean nothing.

Rollback. With immutable versions and a pointer, reverting is moving the pointer - seconds. Without them, it is reconstructing text from memory or from a screenshot in a chat thread, during an incident.

There is a fourth benefit that is often what finally justifies the work: it separates your changes from the provider's. A model updated behind an unversioned alias produces exactly the same symptom as a bad prompt edit, and only a record carrying both versions tells the two apart.

Implementation patterns

• Immutable versions with a content hash. An edit creates a new version; nothing is updated in place. The hash gives a stable identifier for free and makes accidental duplicates obvious.
• A named pointer per environment - live, canary - so deployment is moving a pointer and rollback is moving it back.
• Record the full set on every inference: prompt version, model name, model version (never a floating alias alone), temperature and other sampling parameters, tool or function schema version, and retrieved document identifiers.
• Keep a promotion step between authoring and serving. Product managers are frequently the right authors, because a prompt encodes product intent; the discipline needed is a gate after the editor, not a code review before it.
• Gate promotion on a small regression set - 30 to 60 cases with expected properties - which is the cheapest control available and catches the instruction-deleted-by-accident class immediately.
• Store the rendered prompt, or enough to reconstruct it, when templates interpolate retrieved content: the template version alone does not determine what the model saw.
• Make the identifier visible in support tooling, so a complaint can be traced without an engineer.

Industry example

Every mature prompt-management product converges on the same primitives - versioned prompts, an environment pointer, and a logged link from output to version - and the pattern is borrowed directly from feature-flag and configuration-management practice, where immutable versions plus an environment pointer have been standard since the mid-2010s. The same requirement arrives from the compliance direction: model-risk and AI governance expectations published since 2023 ask what produced a given decision, and an answer requires exactly this record.

Failure scenarios

• In-place edits during an experiment, producing an unanalysable result that is reported as a win.
• Model alias drift. latest moves, behaviour changes, and the investigation blames the prompt because the model version was never recorded.
• Prompt text logged instead of a version identifier. Storage balloons, comparisons become string diffs, and nobody can group outputs by version.
• Retrieval not recorded. The prompt version is known and what the model actually read is not, so a retrieval regression looks like a prompt regression.
• Provenance recorded but not retained. Logs expire in 7 days and the quality question arrives after 30.
• Rollback by copy-paste, reinstating a prompt that is subtly different from the one that worked, during an incident.

Trade-offs

The honest cost is friction in the authoring loop, which is exactly where product teams feel it. Keep the self-service editor and put the gate after it, or the mutable path will be recreated somewhere nobody is looking.

When not to use it

For a throwaway prototype that is not serving users and whose outputs nobody will revisit, the record buys nothing.

For a single-prompt internal tool with one maintainer and no experimentation, version control over the file is already provenance, and building the records is over-engineering - the lesson transfers as "never edit the live prompt without a commit".

And where the prompt is genuinely user-authored - the user is the prompt, as in a general assistant - there is no version to record; the provenance that matters there is the model version, the system prompt version and the retrieval set.

Interview question

Q: Your team reports that output quality dropped yesterday afternoon. Prompts are edited through an internal interface and outputs are logged with the response text. What can you establish, and what do you change?

What a strong answer covers: that with no version recorded almost nothing can be established - including whether the prompt changed at all - and saying so plainly rather than speculating · immutable versions with a content hash plus an environment pointer as the change, with the version identifier recorded on every inference · recording model version explicitly, because an alias moving produces the identical symptom · retrieval identifiers, because a retrieval regression masquerades as a prompt regression · a promotion gate with a small regression set, while keeping product managers as authors · log retention long enough to outlive the questions · and the immediate action: stop any running experiment, because its result cannot be attributed.

Quick check

Quiz: Why is an A/B test between two mutable prompt rows worse than no test? Because edits during the experiment silently change an arm and the logs cannot distinguish before from after, so it still produces a confident number that will be acted on.

Flashcard: Which identifiers belong on every inference record? - Prompt version, model name and explicit model version (never a floating alias alone), sampling parameters, tool schema version and retrieved document identifiers; the model version is what separates your regression from the provider's.