Experiments · open dataset
Real people, scored: 120 walks across a room
Before measuring robots against people, we wanted to see what our scorer makes of real people. An open motion-capture dataset from Örebro University let us try it on 120 walks.
The dataset
THÖR-MAGNI is a recording of people moving through a 20 × 9 metre room at Örebro University in Sweden, tracked by a motion-capture system. Forty participants took part over five recording days, about three and a half hours in all. They walked between seven goal points around obstacles, alone or in groups, sometimes carrying a bucket, a box or a large object together, and sometimes sharing the room with a robot driven by remote control.
It was collected to study how people move around each other and around robots, and is used to train and test models that predict where people will walk next. The researchers published it openly: read the paper, download the full dataset, or use the processed walks from their prediction challenge, which is what we used.
Why we used it
MotionBench compares a robot with a person doing the same job in the same space. We looked for public data that already had both and found none. THÖR-MAGNI is the closest thing: precise, real human movement in a room with walls and clear goals. That makes it a good first check of the scoring on real people, before our own pilot sessions with students and robots.
What we did
- Joined the dataset’s 8-second pieces back into continuous walks.
- Cut out every walk from one goal point to another: 1,984 in total.
- Picked the most-walked route, goal 3 to goal 7: about 7 metres straight across the middle of the room, walked 120 times over five days.
- Wrote it as an experimental MotionBench task and turned each walk into a run, exactly as we would a robot’s.
- Added 25 runs of a simulated robot wandering at random, to set the zero point, and scored everything.
What the scorer saw
- Walks scored
- 120
- Reached the goal
- 120 of 120
- Median time
- 6.8 s
- Median speed
- 1.0 m/s
Most walks took 5 to 10 seconds and followed a nearly straight line. The slowest took 33.2 seconds: a pair of visitors who walked a 33-metre loop on the way, at a normal pace.
Replay every walk
The same explorer we use for robot runs. Pick a recording day and a walker, and watch the page re-check each walk from the raw positions. The random-walk runs are simulated and labelled as such.
What it taught us
Four changes to the benchmark, found by scoring real people.
Success alone can't tell people apart
Every walker reached the goal, so every one scores exactly 1.0 and they all share first place. On a simple point-to-point job, time, path efficiency and consistency do the real work. Our simulated VEX AIM board shows the same ceiling; now we've seen it on real people.
The time limit should follow the people
With a 60-second limit, a robot wandering at random reached the goal in 40% of its runs, which lifts the zero point of the scale. A limit of about three times the typical human time (roughly 20 s here) would push luck close to zero.
The ideal path needs to know the finish line
We measure the shortest path centre to centre, but a walk counts as finished on entering a 0.6 m circle around the goal. Many walks came out shorter than the 'ideal', and their efficiency was capped at 1.0. The ideal path should subtract the finish radius.
Rankings need enough tries
One walker with only two walks, one of which didn't count, ended up with a range so wide it overlapped first place. Shared ranks should only be given once an agent has a minimum number of runs.
One more, about our own work: the first time we joined the pieces, some walks jumped 14 metres in under half a second. Each recording day had several sessions that reused helmet numbers and restarted their clocks, so we had stitched different people together. A simple check for impossible speeds caught it before anything was published.
What this can and can’t tell you
- These are people, not robots. This is a test of the scoring method on real human movement, not a robot result and not a ranking of anyone.
- The task is experimental and built for this test: walk from one goal point to another, heading not scored. The room's obstacles aren't modelled yet, so bumps aren't counted.
- The published walks are sampled 2.5 times a second, coarser than our own captures.
- The goal points, room and instructions come from the original study, which was designed for a different purpose: studying how people move, not timing them.
Run it yourself: the converter and full instructions are in the open-source toolkit.
Credit
THÖR-MAGNI was recorded by researchers at Örebro University and is released under the CC BY 4.0 licence; the prediction-challenge files are MIT-licensed. If you use these results, cite the THÖR-MAGNI paper (Schreiter et al.) and the dataset. MotionBench is not affiliated with the THÖR-MAGNI team, and none of their data is stored in our repository.