Kanta Saito
Activity

Why the scripted recoverer times out on picks

BEHAVIOR-1K 2026 challenge, issue 74 recovery pilot, privileged pick on pt50 failures, Oct 22

Issue 74 points the team's scripted recoverers at pt50's failures on six near-miss original tasks. A recoverer restores the sim state where pt50 got stuck and tries to finish the next phase with privileged object state. So far the phase detector confirms 12 of 86 attempts (14%). The largest group of failures is pick attempts that hit the 40 second cap with no failure reason recorded, 27 of them. Four of them are below. Three end with the hand parked 10 to 35 cm from where it needs to be, and the fourth was still closing in when time ran out.

86 attemptsscripted recoveries on pt50 failures from six near-miss tasks
12 confirmed (14%)finished phases the phase detector accepts
27 pick timeoutshit the 40 s pick cap with no failure reason

Four timeouts

Each clip is the full 40 second attempt in real time. The panels are, from left to right, the left wrist camera, the right wrist camera and the head camera. The plot under each clip is the distance between the hand and its current goal. During the approach the goal is a pre-grasp pose next to the object, and when the descent starts the goal switches to the grasp pose, so the line can jump there. Gaps in the line are the base moves, when the arm is not tracking a goal. While a clip plays, a thin line on the plot follows it.

Board game, still closing in at the cap

right armdescends at 36.5 stimed out at 40 s
collecting_childrens_toys instance 206, pick board_game_263 from the bed, the alternative case after the push label, real timeThe robot backs off the blocked bed, then works its right hand toward the board game for almost 27 seconds. The error keeps shrinking, from 16 cm at 22 s to 7.5 cm at 33 s, so it is converging, just slowly. It switches to the descent at 36.5 s and the 40 s cap ends the attempt 3.5 seconds later.

Club sandwich, in view and out of reach

left armdescends at 13.4 stimed out at 40 s
preparing_lunch_box instance 204, pick club_sandwich_209 from the chopping board, real timeThe descent starts early, at 13.4 s, and then the hand sits 11 to 20 cm from its goal for 26 seconds. The sandwich is on the chopping board in the head camera the whole time, while the left wrist camera is pressed against the red backsplash.

Dice, the approach never finishes

right armnever descendstimed out at 40 s
collecting_childrens_toys instance 202, pick dice_268 from the bed, real timeThe approach error falls from about 1 m to 35 cm by 22 s and then stops falling, staying between 30 and 41 cm until the cap. The robot never reaches the descent, and the head camera ends on empty shelves with the dice out of view.

What goes wrong

The recoverer places the base once, before it starts reaching. After that only the arm and torso move. When inverse kinematics cannot close the last 10 to 35 cm from where the base stopped, the recoverer keeps chasing the same target until the cap. Nothing in it notices that the error has stopped shrinking, so the attempt ends as a plain timeout with no reason attached.

The board game is the exception that shows the other half of the problem. Its error was still shrinking when the cap hit, so it might have finished with a few more seconds. The cookie, the sandwich and the dice had plateaued and would not have finished with any amount of extra time.

The fix under way has three parts.

The 27 pick timeouts will be rerun with this change.

The other failures

Timeouts are the largest group, but not the only one. At the first tally, taken a little after the 86-attempt count above, 75 attempts had failed and the reasons broke down as below. Timeouts were 39 at that point. A later pass narrowed the ones with no recorded reason to the 27 picks above.

Failure reasons at the first tally
reasonattempts
Timed out, no reason recorded39
Target not visible after looking around18
Recoverer finished, phase detector says not done10
Object too big for the gripper or out of reach6
Robot tipped over2