BEHAVIOR-1K 2026 challenge, Oct 15 to Oct 20
A planner that tells the policy which step comes next looks like the right fix for runs that loop because the robot does not know what to do next. Two things stand in the way. pt50 hardly listens to its prompt, so on the original tasks a planner cannot steer it, and on the new tasks the team's oracle test shows the follower, not the planner, is the weaker link.
Issue 35 ran pt50 on 400 held-out frames of 5 tasks under different prompts with identical noise. Swapping the task prompt for a subtask instruction changes the actions by 0.00082 in MSE, while just redrawing the noise changes them by 0.00341. Even a perfect planner would barely change what pt50 does, so it cannot fix the stuck runs on the original 50.
| prompt | MSE vs demo | MSE change vs task prompt |
|---|---|---|
| task prompt | 0.00239 | 0 |
| task prompt, fresh noise | 0.00331 | 0.00341 |
| subtask instruction | 0.00283 | 0.00082 |
| another task's prompt | 0.00353 | 0.00161 |
| empty | 0.00311 | 0.00113 |
| nonsense | 0.00282 | 0.00075 |
Gate 1 (PRs 67 and 68, Oct 15) asked each model, given the step list, the current step and recent head and wrist frames, which step the robot is on and whether that step is done. Opus 5.5 tracks the step best on real rollouts, and with two votes it is the only setup that gets false done under 10% (1 of 15) while keeping most of its recall (6 of 15 done moments). A wrong switch can sink an episode, so done detection is still the weak part.
| metric | Argon | Haiku 5.5 | Sonnet 5.5 | Opus 5.5 |
|---|---|---|---|---|
| demo step accuracy | 73.4% | 65.5% | 71.8% | 71.2% |
| rollout step accuracy | 53.3% | 50.0% | 46.7% | 70.0% |
| rollout false done | 4/15 | 3/15 | 2/15 | 2/15 |
| rollout done recall | 7/15 | 4/15 | 1/15 | 7/15 |
| latency median, p90 | 4.9 s, 16.2 s | 2.2 s, 3.5 s | 1.6 s, 2.0 s | 3.0 s, 4.7 s |
| cost per call | $0.0129 | $0.0002 | $0.0031 | $0.0085 |
On Oct 15 we checked the team's openpi-comet-b1k PR 14 (the task_id router) against the submitted pt50 warm-up image. Its warm-up matches the image exactly and the original-task path replays bit-identical on the first connection. With SESSIONS=1 a second connection gets a different noise stream, and the PR no longer accepts --task-name, which the submitted README tells the organizers to pass. Replaying its phase planner on recorded runs, the p90 step timers add up to more than the time limit for 24 of 50 tasks, and the hold rule fired 6 times on 8 recorded follower phase starts, 4 of them on pickups that never succeeded.
The team bucket already holds a full fine-tune on all 100 tasks and a line of recovery fine-tunes on top of it, so training on the new tasks is covered. The submission's new-task score of 0.031 mixes planner errors and follower errors together, which is what the oracle test below separates.
| checkpoint | what it is |
|---|---|
| instr100_35k_54999 | Phase follower on all 100 tasks, 55k steps at batch 1024 on 8 GH200 nodes for about 47 h, Oct 2 to 4 |
| r50_2500 to r50_9999 | Recovery fine-tune for the new 50 tasks, 10k steps, uploaded Oct 12 |
| final_A_1999 | Final candidate A, Oct 14, 2000 steps on 88% demos from all 100 tasks and 12% recoveries |
| phase_detector, phase_done, phase_models | Phase and done detectors, Oct 15 and 16 |
The team already ran the test that separates planner errors from follower errors. Their oracle sequencer advances the follower to the next annotated phase as soon as the simulator says the current phase is complete, so the planning is perfect. Each start is a demo phase start, scored by the q gain over 120 s.
| test | starts | pt50 q gain | follower q gain | follower minus pt50 (95% CI) |
|---|---|---|---|---|
| oracle sequencer, ten-task 35k follower, Sep 29 | 48 | +0.215 | +0.104 | -0.115 (-0.171 to -0.058) |
| same at 2x speed, Sep 30 | not stated | +0.333 | +0.146 | -0.191 (-0.291 to -0.091) |
With a perfect planner the follower gets about half of pt50's gain, and the paired gap excludes zero at both speeds. In a 2x2 on Oct 1 (44 starts) the sequencer reached +0.110 with a radius handover and +0.148 with a base-stop handover, a re-planner from privileged state reached +0.121, and pt50 reached +0.235. Re-issuing pick up after a drop does not rescue the follower, which completes 12 of 25 with a median of 33 s.
The follower does read its phase text, completing 28 of 41 phases with the right text against 7 of 41 with the wrong one, while pt50 does not read it. On untrained tasks (104 non-trivial starts, Oct 3) the corpus followers complete 37 and 41 phases against 45 for pt50, and pick up 8 and 9 times against 22. There pt50 minus the 20k follower is +0.020 (-0.011 to +0.052), not significant.
The team's conclusion is that the bottleneck is per-phase competence, picking up and then placing from its own grasp, not planning. A better planner alone cannot close the gap. The open gap is the 100-task 54,999-step follower with the oracle sequencer, which was queued on GCP on Oct 4 and has no recorded result in their notes.