The fastest change in the stack is a block of text
How organisations change the system prompt of a production LLM feature: where the prompt lives, who may change it, what gates the change, and how it is rolled back.
A field guide to the change lifecycle of the production prompt, the highest-leverage, lowest-friction change surface in an LLM product. Built from the published incidents of OpenAI, xAI, DPD, Cursor and Air Canada, the prompt-pipeline write-ups of Uber, GitLab, GitHub, Discord, DoorDash, LinkedIn and GoDaddy, and the diffs of the one public production prompt repository, it gives an architect the reference change pipeline, the four decisions with their flip conditions, a failure catalogue in three classes, and a six-rung ladder from repo convention to governed rollout.
The industry is running the same experiment in both directions at once: xAI re-coupled prompt changes to code's controls after three incidents in twelve months, while GitLab's design doc decouples prompts from code releases; both are right, because the control was never the deploy path but the evaluation gate, which is exactly the gate OpenAI's sycophancy launch shows can be green while the product regresses.
What you get out of it
- xAI's three 2025 prompt incidents share one root cause and teach the escalation: revert without reform (February), reform circumvented (May), reform passed by an approved-but-unevaluated line (July).
- The prompt is also the fastest rollback lever: OpenAI's first sycophancy mitigation was a system prompt patch pushed Sunday night, a day before the model rollback completed.
- A gate only protects the properties it measures: sycophancy shipped through green offline evals and positive A/B tests, and formatting alone can move task accuracy by up to 76 points (ICLR 2024).
- Even the most transparent operators publish prompt diffs without rationale: fourteen commits in xAI's repo all read 'Updated grok prompts', and no operator publishes the why beside the what.
- Prompt churn is metered: cached input costs 0.1x on Claude and half price on OpenAI, and a one-character prompt edit resets the cache, so edit cadence is a cost variable, not just a quality one.
Scope
Why this, now. Between February 2025 and July 2026 the public record acquired what it never had before: real prompt-change postmortems, a public production prompt repo with visible diffs, and design documents from GitLab and Uber that disagree about the deploy path while converging on the eval gate.
What it does not cover. Prompt-injection and jailbreak defence, retrieval quality in RAG systems, agent orchestration above the model, and regressions caused by infrastructure changes under an unchanged prompt (covered by the 2026-09-08 guide in this collection).
Other field guides
When the model quietly gets worse
A field guide to the production incident where answer quality drops while availability, latency and error dashboards stay green, because the change w…
23 sources · 17 organisations · 4 postmortemsThe token stream stays up and the answers get worse
A field guide to the capacity decision in production LLM serving, reconstructed from the systems of Anthropic, OpenAI, Meta, DeepSeek, Moonshot AI, C…
22 sources · 16 organisations · 3 postmortemsOwn the layer above the engine: ten years of LinkedIn's data platform
Reconstructs a decade of data-platform decisions at LinkedIn from artefacts rather than announcements: archive notices, dated release listings on Mav…
22 sources · 6 organisations · 5 postmortems