MotionBench

Protocol v0.1 · open for comment

How a result is made, and checked

The rules every published MotionBench result follows. If a result breaks one, it says so. The full text is in docs/protocol.md.

Five principles

  1. 1

    Goal first

    Every task says what “done” means, and every attempt is scored as the share of that goal it achieved.

  2. 2

    Any agent

    A student driving, the team’s code, AI-written code, a driver with robot help and a commercial robot go on the same board, labelled by who was in control.

  3. 3

    Measured where robots work

    A phone, printed tags or the robot’s own log is enough. No special rig, and we never need the robot’s code.

  4. 4

    Re-scorable by anyone

    One open scorer gives the same answer for the same run every time. Every published number can be recomputed.

  5. 5

    Honest about uncertainty

    A range on every average, the worst run beside it, what luck alone achieves, and simulated results always labelled.

Three kinds of result, never mixed

S · Simulated

Made by a simulator

For testing the method and the tools. Always labelled simulated.

F · Field

A real robot, sent in by its owner

Reviewed by a person before it’s published, but not independently checked.

V · Verified

A real robot, checked end to end

Run under the checklist below.

The numbers

  • At least 5 runs per agent to be ranked; 10 recommended. Fewer runs are shown, unranked.
  • Consecutive attempts. Every attempt in a session counts. No picking the best ones.
  • Public and unseen layouts. Public layouts are for practice and the public boards. Unseen layouts are drawn from the task’s rules with a seed we don’t publish, to tell an agent that learned the task from one that learned the layout.
  • Simulated verdicts on games: every routine and driver plays the same 20 paired runs (same layouts, same robot imperfections), on the public layout and on unseen ones. Outcomes: better, worse, equivalent (within half a point), or no difference found.

What every board shows

  • the share of the goal achieved, with a 95% range;
  • the worst run;
  • how many runs achieved the whole goal;
  • time to the goal, only for agents that achieved it every time;
  • what a robot moving at random achieves, beside the score;
  • shared places while ranges overlap. Scores are never averaged across tasks.

Why unseen layouts matter

In our AIM sorting simulation, the best routine on the public layout scored 4.0 of 6, well ahead of a typical simulated driver. On layouts it had never seen it scored 2.8, with no detectable difference from that driver, because its plan knew where the barrels were. See the experiment.

Verified runs: the checklist

  1. We draw the layout and send it at the start of the session.
  2. One uncut video from a fixed camera shows the whole arena, a code word we send, and every run.
  3. The path comes from printed tags, or from our converter run by a reviewer.
  4. Every attempt in the session is reported, including the bad ones.
  5. Who or what was in control is stated; for code, the version of the routine (shared privately if asked).
  6. A second reviewer re-scores at least one run from the raw video.
  7. No video showing people is published without permission; for students, a parent’s or guardian’s.

Repeatability

  • The scorer gives the same report for the same run, every time. That’s tested on every change.
  • On test video with known positions, our tag-free converter is typically about 3 cm off (90% of frames within about 5 cm), and about 1 cm with printed tags. Checking this on real footage is a first pilot.
  • Every simulated run can be reproduced exactly from its seed.
  • How much real runs vary from session to session is part of the first pilots.

Disputes and versions

Think a result is wrong? Tell us within 30 days (a GitHub issue or the contact form). A reviewer who didn’t score it re-scores it from the raw path and video. Every change to a result is logged in the results changelog, and every board says when and how it was made.