Evaluation & MLOps
Benchmarks, LLM-as-judge, red-teaming, model registries, drift detection and observability.
14concepts
144flashcards
115minutes of reading
- 01 Arena Elo and Preference Ranking How pairwise human votes become a leaderboard through the Bradley-Terry model, what the Elo framing gets wrong, and the selection effects that distort arena rankings.
- 02 Custom Evals and LLM-as-Judge Why every production team eventually builds its own eval set, and how to use LLM judges without being fooled by their well-documented biases.
- 03 Epistemic and Aleatoric Uncertainty in LLMs Separating "the model does not know" from "the question has several right answers", why token-level entropy conflates the two, and how semantic clustering fixes it.
- 04 Eval-Driven Development How teams turn production failures into a regression suite, why the first fifty examples matter more than the framework, and the discipline that keeps an eval set honest as the product changes.
- 05 HELM and Holistic Evaluation Why a single accuracy number is gameable, and how Stanford's HELM, BIG-bench, and lm-evaluation-harness push evaluation toward a multi-axis picture.
- 06 LLM Observability Tooling How tracing an LLM app captures the full fan-out of model and tool calls behind one user request, and how LangSmith, Langfuse, Helicone, and Phoenix differ in what they instrument.
- 07 Model Registry, Lineage, and Reproducibility The infrastructure that answers "which dataset, code, and hyperparameters produced this checkpoint?" - and why you only miss it the first time you cannot reproduce a model.
- 08 Production Monitoring and Drift Detection How to catch silent regressions in deployed LLMs by monitoring input drift, output quality, and per-user randomised experiments before users tell you something is broken.
- 09 Public Benchmarks - MMLU, GPQA, HumanEval, MATH A tour of the academic benchmarks that anchor frontier model launches, and why most of them are saturating, contaminated, or both.
- 10 Selective Prediction and Abstention Letting a system answer only when it is likely to be right, measuring the result with a risk-coverage curve rather than accuracy, and choosing the threshold from the cost of being wrong.
- 11 Agentic Benchmarks: SWE-bench and Its Descendants What changes when a benchmark task requires many steps in a real environment, why execution-based grading is the whole point, and the contamination and harness confounds that make agent scores hard to compare.
- 12 Conformal Prediction for LLMs A distribution-free procedure that converts any confidence score into prediction sets with a finite-sample coverage guarantee, and what that guarantee does and does not promise once the predictor is a language model.
- 13 Error Bars for Evals: Sampling, Clustering and Paired Tests Why a benchmark score is an estimate with a standard error, how clustering inflates it, and why paired comparison is the single highest-leverage change to an eval report.
- 14 Red-Teaming and Adversarial Evaluation Why benign benchmark scores do not predict how a deployed model behaves under attack, and the human and automated methods used to find the failures first.