Code Generation Benchmarks and pass@k

Codex solved 28.8% of HumanEval's problems with one sample per problem and 70.2% when it was allowed 100 (Chen et al., 2021, Evaluating Large Language Models Trained on Code, arXiv:2107.03374). Same model, same problems, a 41-point gap. A code benchmark score is meaningless until you know how many attempts it grants and how the number was estimated.

Execution instead of similarity

HumanEval is 164 hand-written Python problems, each a function signature and docstring with hidden unit tests. Its contribution was methodological: a completion counts as correct if it passes the tests, not if it resembles a reference solution. Match-based metrics such as BLEU reward code that looks right, and code that looks right but fails one edge case is simply wrong. MBPP followed the same pattern at larger scale, 974 entry-level tasks, where the largest model tested reached 59.6% few-shot (Austin et al., 2021, Program Synthesis with Large Language Models, arXiv:2108.07732).

Execution changes what a score means. The benchmark is now as good as its tests, and a problem's tests are a sample of its specification.

The unbiased pass@k estimator

pass@k is the probability that at least one of \(k\) independent samples for a problem is correct, averaged over problems. If a problem's per-sample success probability is \(p\), the target is \(1 - (1-p)^k\).

Two obvious estimators are poor. Drawing exactly \(k\) samples and checking whether any passed is unbiased but has high variance. Estimating \(\hat{p} = c/n\) from \(n\) samples and plugging in gives \(1 - (1-\hat{p})^k\), which is biased low: the function \(f(p) = 1-(1-p)^k\) is concave, so by Jensen's inequality \(\mathbb{E}[f(\hat{p})] \le f(p)\).

Chen et al. instead draw \(n \ge k\) samples (they used \(n = 200\)), count the \(c\) that pass, and compute

The derivation is short. Choose \(k\) of the \(n\) samples uniformly without replacement. The chance that all \(k\) are incorrect is the number of all-incorrect subsets, \(\binom{n-c}{k}\), over all subsets, \(\binom{n}{k}\). Any fixed size-\(k\) subset of i.i.d. draws is itself an i.i.d. sample of size \(k\), so its "at least one correct" indicator has expectation \(1-(1-p)^k\); the formula averages that indicator over every subset, which keeps the expectation and cuts the variance. It is a U-statistic. For numerical stability it is computed as \(1 - \prod_{i=n-c+1}^{n} (1 - k/i)\).

A small case shows the bias. With \(n = 10\), \(c = 2\), \(k = 5\): \(\binom{8}{5}/\binom{10}{5} = 56/252 \approx 0.222\), so the estimate is \(0.778\). The plug-in gives \(1 - 0.8^5 \approx 0.672\), more than ten points lower.

Saturation and contamination

Two forces have eroded HumanEval. The first is weak tests. EvalPlus expanded the tests roughly 80-fold and found enough previously undetected wrong code to cut pass@k by up to 19.3 to 28.9% (Liu et al., 2023, Is Your Code Generated by ChatGPT Really Correct?, arXiv:2305.01210). On its leaderboard, o1-preview scores 96.3 on HumanEval and 89.0 on HumanEval+ (EvalPlus leaderboard).

The second is resolution. With 164 problems and a true rate of 0.9, the standard error is \(\sqrt{0.9 \times 0.1/164} \approx 0.023\), a 95% interval of about \(\pm 4.6\) points. Leaderboard gaps of two points between frontier models sit inside that band, although a paired comparison on the same problems narrows it.

Behind both sits contamination. The problems have been public since 2021 and appear in training corpora. LiveCodeBench responds by time-stamping: it collects problems from LeetCode, AtCoder and Codeforces with release dates and scores models only on problems published after their training cutoff. Its authors found DeepSeek-Instruct and GPT-4o performing considerably worse on problems released after their release and cutoff dates respectively, the signature of memorised older problems (Jain et al., 2024, LiveCodeBench, arXiv:2403.07974). Filtering training data is covered in benchmark decontamination; repository-scale evaluation in SWE-bench and agentic coding evaluation.

When it breaks

pass@k assumes a free oracle. pass@100 credits the model if any sample is right, but a deployed system must pick one, using tests, ranking or execution feedback. The field disagrees on what to report. One view holds pass@1 is the only deployment-relevant number; the other argues pass@k measures latent capability that test-time search and reinforcement learning can convert into pass@1, which is why it keeps appearing in reasoning-model papers.

Temperature is part of the estimand. The best sampling temperature grows with \(k\), because diversity helps coverage and hurts single-shot accuracy. A pass@1 from greedy decoding and a pass@1 estimated from 200 samples at temperature 0.8 are different quantities that share a name.

The estimator needs \(n \ge k\) and independent samples. Reporting pass@10 from 10 samples is legal but noisy; reusing samples across prompts or deduplicating before counting breaks independence.

Passing tests is not correctness. Every function-level benchmark inherits the coverage of its tests, and a model tuned against a benchmark learns its blind spots too.