Evaluation & MLOps
Benchmarks, LLM-as-judge, red-teaming, model registries, drift detection and observability.
14concepts
144flashcards
115minutes of reading
- 01 Agentic Benchmarks: SWE-bench and Its Descendants What changes when a benchmark task requires many steps in a real environment, why execution-based grading is the whole point, and the contamination and harness confounds that make agent scores hard to compare.
- 02 Conformal Prediction for LLMs A distribution-free procedure that converts any confidence score into prediction sets with a finite-sample coverage guarantee, and what that guarantee does and does not promise once the predictor is a language model.
- 03 Error Bars for Evals: Sampling, Clustering and Paired Tests Why a benchmark score is an estimate with a standard error, how clustering inflates it, and why paired comparison is the single highest-leverage change to an eval report.
- 04 Red-Teaming and Adversarial Evaluation Why benign benchmark scores do not predict how a deployed model behaves under attack, and the human and automated methods used to find the failures first.