Every task so far has one robot and one goal. A match has four robots, two alliances, two minutes and no tags. We’re building the tools to score it in the open, starting with a simulator that knows the right answer.
Watch the simulated match
Simulated, not real footage. One full match from our simulator: 15 s autonomous, a short pause, 105 s of driving, filmed from the same camera position as a real event broadcast.
Why matches are their own kind of task
On a single-robot task, the goal is clear and nobody else is on the field. In a match, the score depends on all four robots, the other alliance is in the way, and luck plays a part. That’s why MotionBench doesn’t try to score the game.
Instead, it measures each robot: where it went, how fast, how long it sat still, how much time it spent on the other side, how often it got into contact with another robot, all split into the 15-second autonomous period and driver control. Those numbers mean the same thing in every match, whoever the opponents were.
But first you need to know where every robot was, from ordinary event footage. That is the hard part, and it’s what this experiment is about.
What real match footage looks like
We studied a screen recording of an event livestream. After a few seconds of intro shots it cuts to one camera high above the field that doesn’t move at all for the rest of the match. From the four corners of the field alone, we can work out where that camera is.
Camera
Fixed, overhead
Height above the floor
≈ 3.7 m
Looking down
≈ 44°
Robots to follow
4
We don’t show that footage: it belongs to the event, and the people in it are students. Only the camera measurements are kept. That’s our rule for every match: we publish robot paths, never the video.
Partners look alike
The only thing that says which alliance a robot is on is a small red or blue plate, and the game blocks come in exactly the same red and blue.
Tall robots lean
Seen from above at an angle, a 45 cm robot's top sits well above where its wheels touch the floor. The footprint is what we measure, so the tracker has to see through the lean.
Robots touch
Pushing is part of the game. Two robots in contact look like one big shape, and that's where trackers swap them.
Things get in the way
Raised bars that robots park beside, loaders in the corners, people round the field reaching in, and a broadcast overlay on the edges.
A match that knows the answer
To test a tracker you need to know where the robots really were. So we built a match simulator and a renderer that films it from the same camera position as the real broadcast, with the same kinds of trouble: lookalike partners, robots pushing each other, blocks, people at the edges, an overlay. Every frame comes with the exact answer.
Scrub through the same match from above. Tick the box to see where the tracker thought each robot was: the dashed rings. Watch them swap numbers when robots push each other.
Autonomous0:15
0 s
What gets measured for each robot
The same match report works on simulated matches and on tracked real ones, because it only needs each robot’s path. Here it is for the match above.
Robot
Distance
Auton / driver
Sitting still
On the other side
Robot contacts
Wall contacts
Parked at the end
Red 1
36.4 m
4.7 m / 31.7 m
32%
29%
12
6
Yes
Red 2
25.3 m
5.8 m / 19.5 m
25%
40%
14
1
No
Blue 1
34.4 m
6.7 m / 27.8 m
38%
36%
13
9
Yes
Blue 2
44.5 m
5.8 m / 38.7 m
24%
47%
13
10
No
No robot goes onto the other side in autonomous, so “on the other side” is all driver control. Contacts are counted from distances: two robots whose centres come within half a metre are touching.
Can a tracker follow them? The first benchmark
A tracker watches the video and writes down where it thinks each robot is, frame by frame. We compare that with the simulator’s answer and ask four plain questions:
Found: in what share of frames did it find each robot?
How far off: how far from the real spot, typically and at its worst?
Swaps: how often did it mix two robots up?
Same robot all match: how much of the match did each robot keep one identity (IDF1)? And did it get the alliance colour right?
Simulated match
Found
Typically off by
90% within
Swaps
Same robot all match
Right alliance
m0001
97%
8 cm
18 cm
25
65%
68%
m0002
90%
9 cm
19 cm
33
66%
91%
m0003
90%
7 cm
16 cm
43
48%
62%
The baseline tracker
No training, no tags. It builds a picture of the empty field from the video itself, then asks, for every spot on the floor, whether a robot standing there would fill the picture the way the current frame does. Peaks become robots; tracks follow them from frame to frame; the plate colour decides the alliance.
It finds the robots and places them within a bumper’s width. Keeping each robot’s identity for the whole match is where it falls down, mostly when robots push each other. That’s the target for the next tracker.
On the real footage
On a minute of the real overhead shot, it kept a track on all four robots for the whole minute. One robot changed identity once, when a pause symbol from the screen recording appeared over the field and the tracker followed it. Four of its five tracks got the right alliance colour.
There’s no ground truth for real footage yet, so this is what it did, not how accurate it was.
What this can and can’t tell you
Every number on this page comes from simulated matches. They are for building and testing the tools, not a measurement of any team or robot.
The simulator is simple: robots plan around structures, pick up and deliver blocks, push and park, but nobody plays a real strategy. The field resembles a recent season's closely enough for tracking, but there's no rules model and no game score.
The real footage had no ground truth, so we can say what the tracker did on it but not how accurate it was. Hand-marked frames from real matches are the next step.
The baseline tracker is a reference point, not a finished tool. It finds robots well and keeps their identities badly, which is exactly what the benchmark is for.
Everything here is open source: the simulator, renderer, tracker and benchmark are in the toolkit, and the five simulated matches are in data/match-sim-v0.1.