Evaluation & MLOps
Benchmarks, LLM-as-judge, red-teaming, model registries, drift detection and observability.
11concepts
57flashcards
91minutes of reading
- 01 Arena Elo and Preference Ranking How pairwise human votes become a leaderboard through the Bradley-Terry model, what the Elo framing gets wrong, and the selection effects that distort arena rankings.
- 02 Custom Evals and LLM-as-Judge Why every production team eventually builds its own eval set, and how to use LLM judges without being fooled by their well-documented biases.
- 03 Eval-Driven Development How teams turn production failures into a regression suite, why the first fifty examples matter more than the framework, and the discipline that keeps an eval set honest as the product changes.
- 04 HELM and Holistic Evaluation Why a single accuracy number is gameable, and how Stanford's HELM, BIG-bench, and lm-evaluation-harness push evaluation toward a multi-axis picture.
- 05 LLM Observability Tooling How tracing an LLM app captures the full fan-out of model and tool calls behind one user request, and how LangSmith, Langfuse, Helicone, and Phoenix differ in what they instrument.
- 06 Model Registry, Lineage, and Reproducibility The infrastructure that answers "which dataset, code, and hyperparameters produced this checkpoint?" - and why you only miss it the first time you cannot reproduce a model.
- 07 Production Monitoring and Drift Detection How to catch silent regressions in deployed LLMs by monitoring input drift, output quality, and per-user randomised experiments before users tell you something is broken.
- 08 Public Benchmarks - MMLU, GPQA, HumanEval, MATH A tour of the academic benchmarks that anchor frontier model launches, and why most of them are saturating, contaminated, or both.