MotionBench was started by two middle-school VEX competitors. Read their story.
The question we want to answer
A student drives a robot across a 12-foot field. Her team’s autonomous code does the same run. An AI writes a program that does it too. A robot vacuum cleans a living room, and a person with a stick vacuum cleans the same room. Which did the job best, and by how much?
Today nobody can answer that with a number someone else can check. Competitions score matches, not methods. Product pages quote marketing figures. Research benchmarks mostly live in simulation. MotionBench exists to make the question answerable: openly, cheaply, and with people as the reference.
Why confined spaces, and why not cars
We measure machines that move inside a bounded space towards an explicit goal that a person could also achieve. That closed world is what makes honest measurement possible: the arena can be reset, the run can be repeated five times in ten minutes, and a human can do exactly the same task under the same camera.
Self-driving cars, sidewalk delivery robots and drones in open airspace are a different problem. They share uncontrolled space with the public, carry life-safety stakes, answer to regulators, and already have their own evaluation communities. Mixing them in would blur our method and borrow claims we have no right to make. MotionBench never scores them, and nothing on our leaderboard is a safety certification.
Why people are the yardstick
An absolute score means nothing outside its own task. “Six waypoints in 11 seconds” tells you little; “1.09 times the median human driver, with a worst run that still beat most of them” tells a student, a parent, a researcher and a product reviewer the same thing. So every result is reported as a human-normalized score, where 1.0 is the median person on that task and 0 is a random-walk floor.
That only works if we actually measure people. Collecting honest human baselines, from rookies to the best drivers, is as much a part of this project as the code.
Why VEX first
The VEX V5 programme puts thousands of teams on an identical 12 ft square field, and every match already splits into an autonomous period and a driver-controlled period. The human-versus-machine comparison is built into the sport, and teams now routinely ask AI tools to help write their routines. Our VEX tasks use game-agnostic primitives (navigate, collect, deliver) so they never reproduce any season’s game.
The larger story
The same scorer that measures a VEX auton measures a robot vacuum against a person with a vacuum, a warehouse robot against a picker with a cart, and a classroom arm against a pair of hands. One trace format, one set of rules, one question asked the same way everywhere a machine moves through a bounded space towards a goal. Over time that builds something that doesn’t exist yet: a public record of where machines in our rooms and workplaces match people, where they beat them, and where they still fall short.
What we aim to deliver in the first year
- Human baselines from at least 100 drivers across 20 teams on the two VEX tasks.
- A published comparison of human, team-written and AI-written routines, reviewed by the teams involved.
- A room-coverage study measuring at least three robot vacuums against people in the same rooms.
- A verified tier with hidden seeds, so leaderboard runs can’t be tuned in advance.
- Committers from at least three unaffiliated organizations, including a coach and a student alumnus.
How we promise to behave
Synthetic data stays labelled synthetic. Results ship with sample sizes and confidence intervals. Minors take part only with guardian consent, as traces rather than faces. We never imply an endorsement we haven’t been given, and we change scoring only through public RFCs. The full commitments are in the manifesto and the proposal is in RFC-0001.