Evaluating Robot Policies
Why a reported 70 percent success rate on a robot task usually carries an error bar of plus or minus 15 points, and what protocol changes make two policies actually comparable.
A paper reports 72% success on a manipulation task and a competing paper reports 78%. On a language benchmark that gap would be meaningful. In robotics it is usually noise, and the reason is arithmetic that the field routinely skips.
Robot evaluation is a sequence of Bernoulli trials. With \(n\) trials and observed success rate \(\hat{p}\), the standard error is
At \(\hat{p} = 0.7\) and the common \(n = 20\) trials, the standard error is 0.102, so a 95% interval spans roughly 50% to 90%. Two policies differing by six points at \(n = 20\) are statistically indistinguishable. Detecting a genuine 10-point difference near 70% at conventional power needs on the order of 300 trials per condition, which at two minutes per trial with manual resets is more than ten hours of human labour per policy.
That is the whole problem. The measurement is expensive in exactly the resource, human attention, that scales worst.
The confounds that do not appear in the number
Initial state distribution. Success depends on where the object started. A human who places the object slightly differently for the new policy, having watched it fail once, has invalidated the comparison. Marked positions, randomised-but-recorded starts, and a shared reset script are the minimum controls.
Temporal drift. Lighting changes over a day, cameras get bumped, cable tension changes, grippers wear. Evaluating policy A this morning and policy B this afternoon confounds policy with time of day. Interleaving trials, ABAB rather than AAAABBBB, removes most of it and is nearly free.
Success is a judgement call. "Did it place the cup on the plate" has edge cases: the cup landed on the rim, the plate moved, the gripper knocked it and recovered. Written criteria fixed before the run, and a second rater on a subset, are how this stops being a source of drift toward the author's preferred result.
Everything is per-lab. Objects, table height, camera placement and lighting differ between institutions, so success rates from two labs are not comparable even for the same nominal task. This is why cross-embodiment training pools far better than cross-lab evaluation does.
Simulated evaluation, done carefully
The obvious response is to evaluate in simulation, where trials are free and resets are instant. The obvious objection is that a simulator that was not good enough to train the policy is not good enough to judge it.
SIMPLER is the serious attempt to make simulated evaluation of real-world manipulation policies trustworthy, explicitly targeting the control and visual disparities between real and simulated environments and validating through paired sim-and-real evaluations that policy rankings correlate, including reproducing behaviour modes such as sensitivity to distribution shift (Li et al., 2024, arXiv:2405.05941). The important design choice is that it is built to reproduce the real evaluation, matching camera poses and object sets, rather than being a generic benchmark suite.
Practical protocol
Report \(n\) alongside every success rate, and a confidence interval rather than a bare percentage. Interleave conditions. Fix the initial-state distribution and publish it. Separate failure taxonomies from success counts, since "failed to grasp" and "grasped then dropped in transit" point at completely different fixes and both show up as a zero. Where a policy is evaluated in simulation, state the measured correlation with hardware rather than asserting that the simulator is realistic.
None of this is difficult. It is skipped because it costs trials, and trials cost human hours, which is precisely why the field's reported numbers are noisier than they look.
12 flashcards for this concept
Click a card to reveal the answer.