Tasks, metrics and the rules of scoring
Spec v0.1. Every rule here is implemented once, in the Rust scorer, and called from every language.
Tasks
Waypoint navigation on a 12 ft field (MB-VEX-NAV@1.0)
Drive through six seeded waypoints in order, arriving within 5 cm and 5 degrees of each target pose, inside 20 seconds. Game-agnostic: uses only the standard 12 ft x 12 ft field and two generic barrier elements.
- Arena
- 3.66 × 3.66 m, 3 obstacles
- Goal
- Reach 6 waypoints in order within 5 cm and 5°
- Time budget
- 20 s
- Public seeds
- dev-01, dev-02, dev-03
- Domain
- vex
Collect and deliver on a 12 ft field (MB-VEX-PICK@1.0)
Collect six scattered objects and deliver them into the goal zone in the far corner within 60 seconds. Scored on the final world state: an object counts if its centre rests inside the zone. Game-agnostic: any small game object of the current season can stand in.
- Arena
- 3.66 × 3.66 m, 1 obstacle
- Goal
- Move objects into the "goal" zone; counted from the final positions
- Time budget
- 60 s
- Public seeds
- dev-01, dev-02
- Domain
- vex
Room floor coverage (MB-COVER@0.1)
Sweep the free floor of a 4 m x 3 m room with a sofa and a table, using a 30 cm cleaning tool, within 180 seconds. Scored on the percentage of free floor area swept. Applies equally to a robot vacuum and to a person with a stick vacuum tracked by the same overhead camera.
- Arena
- 4.00 × 3.00 m, 2 obstacles
- Goal
- Sweep free floor with a 30 cm tool; complete at 90%
- Time budget
- 180 s
- Public seeds
- dev-01, dev-02
- Domain
- indoor-floor
Six metric families for every run
| Family | What is measured | Why it matters |
|---|---|---|
| Success | Waypoints reached in order, objects delivered, % of floor covered | Did it do the job? This drives the headline score. |
| Efficiency | Time to goal, path length, path efficiency against an ideal path | The same result in half the time is a better run. |
| Precision | Final pose error, placement error, heading error | Separates controlled runs from lucky ones. |
| Coverage | Area covered, redundant sweeping, time to 90% | The robot-vacuum question, asked exactly. |
| Safety | Wall and obstacle contacts, collisions, rule flags | A fast run that hits things is not a good run. |
| Consistency | Spread over five or more runs, worst run, completion rate | Matches are lost on the worst run, not won on the best. |
Comparing with people
Each task’s human reference is the median of per-driver averages across everyone who drove it. The floor is a random-walk agent, because on some tasks wandering around already earns points (a random walk covers about half a room).
1.0 means “as good as the median person”; above 1.0 is better than most people. Every HNS comes with a 95% confidence interval, and agents share a rank when their intervals overlap.
For a single leaderboard number, the MotionBench Index blends four parts with weights fixed for this spec version:
HNS terms are clipped to 0–2. Consistency = 1 − (spread ÷ mean). Safety = 1 ÷ (1 + violations per run). Full definitions: RFC-0001, Part II.
Who is being measured
Every run is tagged with one agent class, and results are always shown by class.
- Human: a person driving or doing the task by hand. The reference.
- AI-written: a program or policy produced by an AI system.
- Team-written code: autonomous code written by people, such as a team's routine.
- Commercial robot: a closed commercial robot measured as it ships.
- Human + assist: a person driving with machine assistance.
- Random-walk floor: a random walk that sets the floor.
Verification tiers
| Tier | What it means | Where it appears |
|---|---|---|
| S, simulated | Simulator or synthetic generator | Its own board, always labelled |
| F, field | Real hardware, self-reported trace and video | Main board, marked unverified until reviewed |
| V, verified | Hidden seed issued at run time, uncut video with an on-screen code, reviewed | Main board |