Draft v0.2 · open for comment
The paper
MotionBench: a goal-referenced framework for evaluating robot performance from trajectory data. Design, open-source implementation and preliminary experiments. We’re publishing the method before the pilots so it can be criticised first.
Abstract
Robot performance is hard to compare across agents, environments and control methods: a student driving a robot, the team’s autonomous code and a commercial product are rarely measured the same way. MotionBench scores any attempt at a task as the share of the task’s stated goal it achieved, computed from the attempt’s recorded path alone, without access to the agent’s software.
We describe the scoring model, an open implementation (a deterministic scorer, a path format, phone-video capture tools, and a simulator that runs unmodified VEX AIM Python code), and an evaluation protocol with public and unseen layouts.
In a preliminary simulated experiment, the best of four routines on the public layout outscores a parameterized simulated-driver baseline by 1.10 points out of 6, but on ten unseen layouts its score falls by 1.20 points and it shows no detectable difference from that baseline, while a camera-based routine keeps its score. Tag-free capture is about 3 cm off on synthetic video; it hasn’t been validated on real footage, and the simulated drivers aren’t calibrated against real people. Those gaps, and a paired comparison of real drivers with their own code, are the next research objectives.
What changed since v0.1, after an external review: paper/CHANGES.md.
Where it stands
- A draft for arXiv, revised once after an external review, written to be commented on before it’s final.
- Every number in it can be regenerated with one script, with checksums for the exact outputs.
- Almost all results so far are simulated, and labelled that way. The paper lists what the first pilots with student teams must establish.
- Everything in it can be rerun: the source is in the paper folder of the toolkit, and the numbers come from published results.
What we’d most like comments on
- Is the share of the goal the right headline, and is chance shown fairly beside it?
- Are the protocol’s numbers enough: 5 runs to be ranked, unseen layouts, the verified checklist?
- Are the simulated drivers plausible? What would you measure to calibrate them?
- Which tasks should come next?
See also the protocol and how scoring works.