LLM Evaluation
also called Evals
A repeatable measurement of whether an AI system's outputs are good enough, on cases that reflect the actual task.
The reason it matters architecturally: an LLM feature has no deterministic test. A prompt change that improves three examples can silently degrade thirty, and without a measurement there is no way to know before users do.
A workable eval suite is a curated dataset of real inputs with expected properties, a scoring method appropriate to the task — exact match where possible, a rubric-driven model judge where not, human review for the highest-stakes slice — and a threshold enforced in CI.
Two disciplines that determine whether it stays useful: keep a held-out set that is never used for prompt tuning, or the suite becomes an overfit; and feed production failures back into the dataset continuously, since real failure modes are always stranger than the ones you invent.