AI writes the code, watches the run, tries again

How it works

1. 1

   The model writes a program

   It gets the robot's Python API, the game's rules and a photo of the field from above, and writes a complete program for the AIM robot.

2. 2

   The program runs, unchanged

   Exactly as on the real robot: through the robot's own Wi-Fi client library. Here a simulated AIM robot answers it instead of the real one, so a 60-second run takes a fraction of a second.

3. 3

   The camera watches

   A frame from above every 2 seconds, plus the program's log and the robot's telemetry. The next attempt is written after seeing all of it. Three attempts, three runs each.

4. 4

   MotionBench scores it

   Every run gets the game's own points and a MotionBench score against the task's goal, with rule breaks such as ramming a sign counted separately.

Checked against the real robot

Before trusting the simulator we gave it real work. The loop’s own recordings of the one-tag game hold four programs written by three AI models. We ran each program’s exact code in simulation 20 times, each time with different wheel slip and camera noise, and compared the outcome with what the real robot did. The pictures are the real camera’s last frame.

• 

  Model A · attempt 1

  Real robot

  Knocked the tag down with a kicked ball

  Simulation

  Same outcome in 18 of 20 runs

  Run time

  real 16.7 s · simulated 14.7 s

• 

  Model B · attempt 1

  Real robot

  Knocked the tag down with a kicked ball

  Simulation

  Same outcome in 20 of 20 runs

  Run time

  real 22.9 s · simulated 22.4 s

• 

  Model C · attempt 1

  Real robot

  Never found the tag

  Simulation

  Same outcome in 20 of 20 runs

  Run time

  real 60.0 s · simulated 60.0 s

• 

  Model C · attempt 2

  Real robot

  Drove into the tag carrying the ball

  Simulation

  Same outcome in 20 of 20 runs

  Run time

  real 60.1 s · simulated 60.0 s

The models are not named: they were run by the people who built the loop. The recordings also set the simulator’s robot size, camera geometry, how a held ball looks to the camera, and the Wi-Fi delay per command. Tag IDs, pixel sizes and end positions in the simulated logs match the real ones to within a few pixels and 2 to 4 cm.

Three models, three attempts, four games

Average game points per attempt over three runs, with the single runs in brackets. Attempt 2 is written after seeing attempt 1’s code, logs, telemetry and camera frames; attempt 3 after attempt 2’s. Every model call was made by a fresh Claude agent shown only what the loop sends the model.

“Failed”: a barrel was hit, which zeroes the run under the game’s rules. “Rammed”: the robot knocked a sign over itself, which doesn’t count. MotionBench boards for each game, with goal shares, ranges and the MotionBench Index, are in the replays.

What the loop caught

• Feedback works when the failure is visible

  In the barrel game, Opus went from 0 to 6 of 6 once it saw its barrels land in the wrong corners. In the one-tag game its first program rammed the sign on every run; after watching that, the next one kicked the ball instead and scored every time.

• Some mistakes never get fixed

  Sonnet and Haiku rammed the sign in the one-tag game on 9 and 8 of 9 runs and never scored: the same mistake one of the real models made on the real robot. Three attempts weren’t enough to unlearn it.

• The model never sees its score

  In the barrel-guard game Opus’s second program knocked both signs down on every run, but the ball grazed a barrel, so all 3 runs failed. Reading the frames, the model thought it had won. The loop only shows what happened, not how it was judged; adding the score to the feedback is the obvious fix.

• Minutes become seconds

  On the real field one attempt with three runs took about 10 minutes: the model writing (3 min), and for each run a field reset (94 s) and the run itself (43 s). Simulated runs take 0.27 s, so only the model’s time is left: about 3 minutes, and the 12 experiments here ran side by side, not one after another.

All 108 simulated runs (62 minutes of robot time, 29 seconds to compute), drawn on the field: pick a game, model, attempt and run, scrub through it next to the program’s own log, and read the program and the feedback it was written from. The page also has the MotionBench board for each game.

The games are MotionBench tasks you can run yourself: MB-AIM-TAG1@0.1, MB-AIM-TAG2@0.1, MB-AIM-TAG2B@0.1, MB-AIM-SORT18@0.1 (task details).