Toy train, picked after a reposition
right armpickconfirmed at 33.2 sHand distance to the goal in cm base move descent and grasp plateau detector pass
0204060800 s10 s20 s30 s34.7 sdescend
BEHAVIOR-1K 2026 challenge, issue 74 recovery pilot, results as of 11:20 HST on Oct 22
Issue 74 tries to turn pt50's near misses into training data. For each of 44 failures on six original tasks, a sim labeller restores the state where pt50 got stuck and names the next unfinished phase, a pick or a place. A scripted recoverer with privileged object state then tries to finish that phase at three seeds, and the attempt only counts as a recording when the team's phase detector confirms the phase. As of 11:20 HST on Oct 22, 19 of 132 attempts are confirmed, 17 from the first pass and 2 from a rerun of the 27 pick timeouts with a stall fix. A fine-tune of pt50 with and without these recordings is queued on DeltaAI, and its results will be added here.
Each attempt is counted once. Where the stall-fix rerun has finished, its result replaces the first-pass timeout. "Finished, not confirmed" means the recoverer reported the phase done but the detector or its disturbance gate did not accept it.
| task | skill | cases | attempts | confirmed | pass rate | main failure reasons |
|---|---|---|---|---|---|---|
| assembling_gift_baskets | pick | 3 | 9 | 3 | 33% | finished, not confirmed 6 |
| assembling_gift_baskets | place | 1 | 3 | 2 | 67% | timed out, no reason 1 |
| collecting_childrens_toys | pick | 8 | 24 | 9 | 38% | plateau in approach 5, out of reach 3, not visible 3 |
| preparing_lunch_box | pick | 8 | 24 | 3 | 12% | not visible 6, plateau in approach 6, finished, not confirmed 3 |
| putting_shoes_on_rack | pick | 1 | 3 | 0 | 0% | plateau in approach 3 |
| putting_shoes_on_rack | place | 7 | 21 | 0 | 0% | finished, not confirmed 10, timed out carrying 6, timed out, no reason 3 |
| putting_up_Christmas_decorations_inside | pick | 7 | 21 | 2 | 10% | finished, not confirmed 8, not visible 6, too big to grasp 3 |
| putting_up_Christmas_decorations_inside | place | 1 | 3 | 0 | 0% | not visible 3 |
| sorting_vegetables | pick | 7 | 21 | 0 | 0% | not visible 12, too big to grasp 3, timed out moving the base 2 |
| sorting_vegetables | place | 1 | 3 | 0 | 0% | timed out carrying 3 |
| all six tasks | 44 | 132 | 19 | 14% |
The shoe-rack places above use the patched place routine. The 21 earlier shoe-rack place attempts with the team's original routine all failed and are not counted.
The timeouts page showed pick attempts running out the 40 second cap with the hand 10 to 35 cm short. The fix watches the goal error. When it has not dropped 1 cm in 3 seconds, the recoverer backs off and moves the base to a new spot, up to twice, and then fails with a named reason. While the error is still falling at the cap it gets 20 more seconds once. All 27 attempts were rerun with the same seeds, and the rerun finished at 11:17 HST.
| outcome | toys | lunch box | shoe rack | Christmas | all |
|---|---|---|---|---|---|
| before the fix, timed out with no reason | 11 | 12 | 3 | 1 | 27 |
| after, confirmed | 2 | 0 | 0 | 0 | 2 |
| after, finished, rejected by the disturbance gate | 0 | 0 | 0 | 1 | 1 |
| after, plateau in approach | 5 | 6 | 3 | 0 | 14 |
| after, plateau in descent | 2 | 3 | 0 | 0 | 5 |
| after, tipped | 2 | 0 | 0 | 0 | 2 |
| after, timed out in approach | 0 | 2 | 0 | 0 | 2 |
| after, timed out moving the base | 0 | 1 | 0 | 0 | 1 |
The fix mostly changes how the attempts fail, not whether they succeed. Only 2 more are confirmed, both on the same toy train case. The other 19 attempts that end in a named plateau stall with the hand 0.06 to 0.46 m from its goal even after two new base poses, and the failed attempts now end after a median of 30 s instead of all running to 40 s.
Each clip is the full attempt in real time. The panels are, from left to right, the left wrist camera, the right wrist camera and the head camera. The plot under each clip is the distance between the hand and its current goal. During a pick the goal is a pre-grasp pose next to the object until the descent, and then the grasp pose, so the line can jump there. Gaps in the line are base moves. The dashed green line is where the phase detector accepts the phase. While a clip plays, a thin line on the plot follows it.
Same layout as above. Dashed red lines mark where the stall rule fired.
The scripted recoverer can finish some picks. The confirmed picks are a dice, a toy train, a half apple and pillar candles, all small objects that are easy to grasp once the base is in the right place, and two more recoveries place a candle into a gift basket. It does not handle shoe-rack placement, and many picks end in a plateau a few centimetres short of the grasp.
The detector is not the main thing standing in the way. 27 attempts finished by the recoverer's own account but were not confirmed. In 17 of them the disturbance gate fired because the grasp moved something else, like the Christmas candle above. In putting_up_Christmas_decorations_inside 205 seed 0 the detector also had nothing to check, because after the restore the candle was on the floor rather than the sofa its contract was bound to. The other 10 are shoe-rack places. The labelled destination is the hall tree, the recoverer sets the shoe down on the bench seat, and the detector's Inside check on the hall tree is false there. The real rack is the lower shelf under the bench, which the recoverer does not aim for.
The labels hold up reasonably well. A check of 9 of the 44 labels against pt50's videos found 6 right, 2 wrong and 1 unclear.
So the remaining near-miss failures need either a better grasp strategy than a fixed reach from a few base poses, or human recoveries. The browser takeover on the sim page can now record those.
pt50 will be fine-tuned on the demos of nine tasks twice, once with the confirmed recoveries and once without, at batch 256 on 4 GH200 GPUs on DeltaAI, with a learning rate of 2.5e-6 for 1000 steps. The job is queued with an estimated start of about 12:44 HST on Oct 23, and a paired eval of the two models follows. Results will be added to this page.
Related pages are why the scripted recoverer times out on picks and issue 74. Counts come from the result.json of every attempt under recov50/runs/pilot and recov50/runs/rerun_plateau.