Prompt Regression Suite
A set of test cases with expected properties, run against a prompt on every change, to detect quality regressions before deployment.
A prompt is production logic, and changing it changes behaviour in ways that are not visible by reading the diff. Without automated evaluation, prompt changes are deployed on the basis of someone having tried three examples.
A regression suite holds representative inputs with assertions about the output. The assertions are rarely exact-match; they are properties: the output parses as the expected schema, contains a required field, does not contain a prohibited claim, cites a source, stays within a length, or scores above a threshold on a rubric evaluated by a model.
What makes it work in practice:
Cases drawn from production, especially failures. Every incident should add a case, so the suite encodes what has actually gone wrong.
Run on every prompt change in CI, with a pass threshold. Non-determinism means individual cases can flap, so the gate is usually an aggregate score rather than every case passing.
Run against model version changes too, because a provider updating a model behind the same name is a change you did not make and did not choose.
The related discipline: prompts versioned in the repository with the code that uses them, not edited in a console. A prompt changed in production with no record is an untracked deployment.