Activity

cook_hot_dogs before and after the warm-up fix

pt50 policy, BEHAVIOR-1K v3.9.3-post1 evaluator, 4 envs per sim, 2x action speed, instances 301-310, one rollout each, policy seed 0. Before is the baseline practice submission on the stock evaluator. After is the warm-up rerun, where the policy holds still for its first 2 requests so its first plan sees a fresh frame.

Result. Task Q fell from 0.30 to 0.05 (-0.25) and successes from 3 to 0. All of the drop comes from instances 305, 306 and 309. The other 7 instances score 0 in both runs. The baseline had no stale-frame flag on any cook_hot_dogs batch, so the warm-up had no stale frame to fix on this task. Its only effect here was to delay the first plan by 2 requests.

Per instance

Q counts the 2 BDDL goal predicates (each hot dog cooked), so it is 0, 0.5 or 1. An episode times out at 13718 steps (7:37 at 30 steps per second). Diverge is the first video frame where the head cameras of the two runs differ by more than 3 grey levels on average.

Instance	Q before	Q after	Success	Steps before	Steps after	Base m before	Base m after	Stale before	Diverge
301	0	0	no / no	13718	13718	17.65	18.54	no	frame 1
302	0	0	no / no	13718	13718	17.36	20.50	no	frame 12
303	0	0	no / no	13718	13718	7.95	19.05	no	frame 12
304	0	0	no / no	13718	13718	6.71	15.29	no	frame 6
305	1	0	yes / no	8843	13718	17.81	14.68	no	frame 11
306	1	0	yes / no	6755	13718	18.27	8.93	no	frame 3
307	0	0	no / no	13718	13718	28.72	15.57	no	frame 14
308	0	0	no / no	13718	13718	1.55	19.33	no	frame 14
309	1	0.5	yes / no	7281	13718	17.45	18.33	no	frame 4
310	0	0	no / no	13718	13718	17.63	25.86	no	frame 15
Mean	0.30	0.05	3 / 0			17.08	17.61	0 of 10

Frame 0 is identical in both runs for every instance, so the scenes and robot start poses match. The views split within half a second in all 10 instances, and the runs then follow unrelated paths.

Instances where the before run succeeded

Each instance has a whole-episode timelapse (before left, after right, about 33x, a finished side holds its last frame) and a real-time clip of the last 30 seconds of the before run, with the after run at the same moment. Head camera on the right of each half, the two wrist views on the left.

305: success at 4:54 before, Q 0 after

Before opened the fridge within about 40 s, carried both hot dogs to the microwave and started it. After spent its first 5.5 minutes at the oven next to the fridge, pawing at the fridge door edge. It only got the fridge open near 6:00, picked up one hot dog and timed out still at the fridge.

Timelapse, whole episode.

Real time, 4:24 to 4:54. Before closes the microwave with both hot dogs inside and starts it. After is still at the oven, fridge closed.

306: success at 3:45 before, Q 0 after

Before drove to the fridge, opened it by about 1:15, took both hot dogs and had them in the running microwave before 3:45. After never opened the fridge. It went to the wall oven instead, pulled the oven door down and spent the whole episode working the oven, which it finally opened near the end. It covered 8.9 m against 18.3 m before.

Timelapse, whole episode.

Real time, 3:15 to 3:45. Before shuts the microwave and starts it. After is still in front of the oven.

309: success at 4:02 before, Q 0.5 after

After was actually faster here. It had both hot dogs out of the fridge and the microwave running by about 3:20, but one hot dog ended up on the counter beside the microwave while it opened the door. It started the microwave with one hot dog inside, which counts as one of two predicates, then stood still with its arms down until the timeout and never retried the second one.

Timelapse, whole episode.

Real time, 3:32 to 4:02. Before loads the second hot dog and starts the microwave. After is idle, microwave already running, one hot dog left on the counter.

A case where both failed

308: Q 0 before and after

This one cuts the other way. Before turned to the nearby microwave, switched it on empty in the first 40 s and then stood in place for the rest of the episode (1.6 m of base travel). After went to the fridge, picked up a hot dog, let the other fall to the floor, and was still at the fridge holding one hot dog at the timeout. After made more progress here but scored the same.

Timelapse, whole episode.

Bottom line

The evidence fits seed noise better than a systematic effect of the warm-up.

No cook_hot_dogs batch was stale in the baseline, so the warm-up fixed nothing on this task. It only shifted the first plan by 2 hold requests, and the two runs split within 15 frames in every instance.
The three losses fail in three different ways: stuck at the oven and fridge door (305), working the wrong appliance (306), and one hot dog left behind after a faster start (309). A systematic cause would more likely show one shared failure, such as a bad first move.
The counts are small. 3 of 10 against 0 of 10 gives a two-sided Fisher exact p of about 0.21. Paired by instance there are 3 changes, all downward, so a sign test gives p = 0.25.
-0.25 is the largest drop among 50 tasks whose deltas range from -0.25 to +0.30. Across all 50 tasks the result is a tie (Q 0.1515 vs 0.1546, task bootstrap 95% CI -0.025 to +0.032, 20 tasks up, 16 down). With 10 rollouts per task, a task at the tail of that spread is expected by chance.
Instance 308 shows the reverse: the after run got further than the before run, which scored 0 while barely moving.

What the data cannot rule out is a small real effect on this one task. Two of the after runs (305, 306) spent most of the episode at the oven beside the fridge, and that pattern is worth one more look. To settle it, rerun cook_hot_dogs with policy seeds 1 and 2 on both evaluators. If the warm-up arm stays near 0 while the stock arm stays near 3 of 10, treat it as real.

Sources: results/sim/submission_pt50_baseline and submission_pt50_warmup, cook_hot_dogs json and videos. Stale flags from results/stale_flags/base_stale_i54.json (cook_hot_dogs list is empty). 50-task statistics from final50_summary.md. Divergence frames measured on 64 px greyscale head-camera crops over the first 60 s.