Pilot, 3 rollouts per cell, NCSA DeltaAI GH200, Oct 20, 2026
This is a small pilot run to check that the ablation pipeline works end to end before the full sweep. It is a sanity check, not evidence. The numbers below are 3 rollouts per cell. The full sweep is on the results page.
In LIBERO-Goal all 10 tasks share the same scene and the same objects, so the camera image alone cannot tell the model which task it is on. The instruction is the only real source of the goal. If the fine-tuned OpenVLA actually uses language, removing or swapping the instruction should change what it does, not just make it slightly worse.
Every condition uses the same image and the same model. Only the text prompt changes. In all four conditions the model appends token 29871 after "Out:" before generating actions.
| condition | prompt | what changes |
|---|---|---|
| original | In: What action should the robot take to {task}?\nOut: | nothing, the task's own instruction |
| blank | In: What action should the robot take to ?\nOut: | empty instruction in the same template |
| no_prompt | In: \nOut: | no template and no instruction |
| wrong | In: What action should the robot take to {task (i+1) mod 10}?\nOut: | the instruction of the next task |
| task | instruction | successes | steps to success |
|---|---|---|---|
| 0 | open the middle drawer of the cabinet | 2/3 | 160, 111 |
| 1 | put the bowl on the stove | 3/3 | 80, 95, 81 |
| 2 | put the wine bottle on top of the cabinet | 3/3 | 96, 87, 95 |
| 3 | open the top drawer and put the bowl inside | 1/3 | 221 |
| 4 | put the bowl on top of the cabinet | 3/3 | 88, 144, 107 |
| 5 | push the plate to the front of the stove | 2/3 | 135, 170 |
| 6 | put the cream cheese in the bowl | 2/3 | 128, 112 |
| 7 | turn on the stove | 3/3 | 84, 90, 121 |
| 8 | put the bowl on the plate | 3/3 | 82, 74, 73 |
| 9 | put the wine bottle on the rack | 3/3 | 118, 135, 149 |
| all | 10 tasks | 25/30 (83.3%) |
25 of 30 rollouts succeed (83.3%, Wilson 95% interval 66.4% to 92.7%). The paper's 79.2% sits inside that interval, so the GH200 setup reproduces the released checkpoint as far as 30 rollouts can tell. Steps count policy steps after the 10 wait steps.
The scene is set up for task 0, open the middle drawer of the cabinet. Under the wrong condition the prompt asks for task 1, put the bowl on the stove, and the patched evaluator also checks that goal at every step.
| condition | drawer opened (task 0) | steps run | bowl on stove (task 1) | env step it happened |
|---|---|---|---|---|
| original | 2/3 | 160, 300, 111 | not scored | |
| blank | 0/3 | 300, 300, 300 | not scored | |
| no_prompt | 0/3 | 300, 300, 300 | not scored | |
| wrong | 0/3 | 300, 300, 300 | 3/3 | 92, 87, 92 |
With the wrong instruction the robot never touches the drawer. Instead it puts the bowl on the stove in all three rollouts, at env steps 92, 87 and 92. That counter includes the 10 wait steps, so it is 82, 77 and 82 policy steps, about as fast as the original condition does task 1 when asked directly (80, 95 and 81). Blank and no_prompt also never open the drawer. So far the model does what the text says, not what the scene was set up for.
Mean inference latency is 0.258 s per step across all 39 episodes (episode means from 0.239 to 0.278 s). Wall time per episode ranges from 22.7 s to 91.8 s. Successful episodes stop early and take 22.7 to 66.4 s. Every failed episode runs the full 300 steps and takes 82.7 to 91.8 s, so an ablated episode costs about twice the original condition's mean of 43.4 s. The whole pilot took about 35 GPU minutes of episode time.
Task 0, episode 0 of each condition, agent-view camera at 224 by 224, 30 fps.
The full sweep, 10 tasks by 4 conditions by 20 rollouts, is done. See the results page: the original instruction succeeds on 170 of 200 episodes, and blank, no_prompt and wrong each go 0 of 200.