pt50 warm-up rerun: biggest gains and drops

BEHAVIOR-1K challenge, original 50 tasks, instances 301 to 310, one rollout each

Baseline practice submission versus the warm-up rerun, compared episode by episode. Four tasks moved up the most and four moved down the most. Each task has its per-instance scores and one or two side-by-side timelapses of the instances that moved the most.

Summary

run	episodes	mean Q	successes
pt50 baseline	500	0.1515	22
pt50 warm-up rerun	500	0.1546	18
delta (warm-up minus baseline)	500	+0.0032	-4
95% CI on delta (episode bootstrap)	500	-0.0225 to +0.0294	-

Overall it is a tie. Mean Q goes from 0.1515 to 0.1546 (delta +0.0032, 95% CI -0.0225 to +0.0294), and successes go from 22 to 18 out of 500.

The drops are all on tasks with partial credit, where one rollout per instance swings a lot from seed to seed. They look like seed noise rather than a regression.

The gains on turning_on_radio and make_microwave_popcorn are the ones that line up with the stale-frame bug, since the baseline is stale-flagged on 301 to 304 for both. The popcorn success is on 301, which is flagged. The radio successes in the warm-up run are on 307, 308 and 310 though, not on the flagged 301 to 304, so the radio gain is not explained by the flag alone.

Gains

task	base Q	warm-up Q	delta	succ warm-up	stale-flagged base ids	clips below
turning_on_radio	0.000	0.300	+0.300	3	301, 302, 303, 304	308, 310
hiding_Easter_eggs	0.167	0.378	+0.211	2	none	306, 302
storing_food	0.113	0.287	+0.175	0	301, 302, 303, 304	306, 305
make_microwave_popcorn	0.000	0.100	+0.100	1	301, 302, 303, 304	301

Clips play baseline on the left and warm-up on the right. Each side shows two wrist views and the head camera. The whole episode is squeezed to about 14 seconds and the side that finishes first holds its last frame.

turning_on_radio

0.000 to 0.300 (+0.300)successes 0 to 3baseline stale-flagged on 301, 302, 303, 304

Per-instance Q (S marks a success)

instance	301	302	303	304	305	306	307	308	309	310
baseline	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
warm-up	0.00	0.00	0.00	0.00	0.00	0.00	1.00 S	1.00 S	0.00	1.00 S

instance 308, baseline q 0.00, warm-up q 1.00 success, 8x speedThe baseline robot hovers by the side table and never touches the radio, while the warm-up robot picks it up and presses the button within 24 seconds.

instance 310, baseline q 0.00, warm-up q 1.00 success, 8x speedThe baseline robot stays by the table with the radio out of reach, while the warm-up robot lifts the radio and turns it on in about 20 seconds.

hiding_Easter_eggs

0.167 to 0.378 (+0.211)successes 0 to 2

Per-instance Q (S marks a success)

instance	301	302	303	304	305	306	307	308	309	310
baseline	0.78	0.67	0.00	0.00	0.00	0.00	0.00	0.00	0.22	0.00
warm-up	0.00	1.00 S	0.33	0.67	0.78	1.00 S	0.00	0.00	0.00	0.00

instance 306, baseline q 0.00, warm-up q 1.00 success, 27x speedThe baseline robot spends the episode wandering the yard and staring at the house and sky with no eggs in hand, while the warm-up robot takes the eggs out of the basket and sets them down around the yard.

instance 302, baseline q 0.67, warm-up q 1.00 success, 27x speedBoth runs start putting eggs out, but the baseline robot walks off toward the house after two of three, while the warm-up robot keeps placing eggs until the task completes at 2:39.

storing_food

0.113 to 0.287 (+0.175)successes 0 to 0baseline stale-flagged on 301, 302, 303, 304

Per-instance Q (S marks a success)

instance	301	302	303	304	305	306	307	308	309	310
baseline	0.25	0.00	0.00	0.62	0.00	0.00	0.00	0.00	0.25	0.00
warm-up	0.00	0.00	0.00	0.62	0.62	0.75	0.00	0.12	0.75	0.00

instance 306, baseline q 0.00, warm-up q 0.75, 71x speedThe baseline robot stands in front of the dark cabinets holding a jar for most of the episode, while the warm-up robot keeps working the counter by the window, moving the oatmeal boxes and chips.

instance 305, baseline q 0.00, warm-up q 0.62, 71x speedThe baseline robot holds a bag of chips in front of the oven for most of the episode with more chips dropped on the floor, while the warm-up robot moves around the kitchen and puts a bottle into a cabinet.

make_microwave_popcorn

0.000 to 0.100 (+0.100)successes 0 to 1baseline stale-flagged on 301, 302, 303, 304

Per-instance Q (S marks a success)

instance	301	302	303	304	305	306	307	308	309	310
baseline	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
warm-up	1.00 S	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

instance 301, baseline q 0.00, warm-up q 1.00 success, 12x speedThe baseline robot stands still in front of the stove facing the microwave for the whole episode, while the warm-up robot puts the popcorn in the microwave and closes it by 0:48.

Drops

task	base Q	warm-up Q	delta	succ base	succ warm-up	stale-flagged base ids	clips below
cook_hot_dogs	0.300	0.050	-0.250	3	0	none	305, 306
putting_shoes_on_rack	0.650	0.440	-0.210	0	0	none	308, 310
moving_boxes_to_storage	0.650	0.450	-0.200	5	4	none	305, 306
tidying_bedroom	0.500	0.300	-0.200	2	0	none	309, 308

cook_hot_dogs

0.300 to 0.050 (-0.250)successes 3 to 0

Per-instance Q (S marks a success)

instance	301	302	303	304	305	306	307	308	309	310
baseline	0.00	0.00	0.00	0.00	1.00 S	1.00 S	0.00	0.00	1.00 S	0.00
warm-up	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.50	0.00

instance 305, baseline q 1.00 success, warm-up q 0.00, 33x speedThe baseline robot takes a hot dog from the fridge to the microwave and switches it on, while the warm-up robot spends most of the episode at the oven and fridge doors and only has a hot dog in hand near the end.

instance 306, baseline q 1.00 success, warm-up q 0.00, 33x speedThe baseline robot carries the hot dog across the kitchen to the microwave, while the warm-up robot stays in front of the wall oven for the whole episode.

putting_shoes_on_rack

0.650 to 0.440 (-0.210)successes 0 to 0

Per-instance Q (S marks a success)

instance	301	302	303	304	305	306	307	308	309	310
baseline	0.60	0.70	0.40	0.80	0.90	0.20	0.50	0.90	0.80	0.70
warm-up	0.50	0.50	0.70	0.50	0.30	0.10	0.60	0.20	0.90	0.10

instance 308, baseline q 0.90, warm-up q 0.20, 28x speedThe baseline robot sets sneakers on the rack and goes back for the sandals, while the warm-up robot gets one sneaker near the rack and then chases a sandal across the floor.

instance 310, baseline q 0.70, warm-up q 0.10, 28x speedThe baseline robot carries the sneakers to the rack, while the warm-up robot holds one sandal for most of the episode and wanders the hallway with it.

moving_boxes_to_storage

0.650 to 0.450 (-0.200)successes 5 to 4

Per-instance Q (S marks a success)

instance	301	302	303	304	305	306	307	308	309	310
baseline	1.00 S	1.00 S	0.50	1.00 S	1.00 S	1.00 S	0.50	0.00	0.00	0.50
warm-up	1.00 S	0.50	1.00 S	1.00 S	0.00	0.00	0.00	0.00	1.00 S	0.00

instance 305, baseline q 1.00 success, warm-up q 0.00, 52x speedThe baseline robot carries the box into the storage room and sets it down, while the warm-up robot gets the box into the garage but keeps lifting it and carrying it back toward the doorway.

instance 306, baseline q 1.00 success, warm-up q 0.00, 52x speedThe baseline robot brings the box from the living room and sets it down in the storage room, while the warm-up robot keeps re-gripping and tilting the box in the garage without putting it down.

tidying_bedroom

0.500 to 0.300 (-0.200)successes 2 to 0

Per-instance Q (S marks a success)

instance	301	302	303	304	305	306	307	308	309	310
baseline	0.00	0.67	0.67	0.67	0.00	0.33	0.67	1.00 S	1.00 S	0.00
warm-up	0.33	0.33	0.00	0.33	0.00	0.33	0.67	0.67	0.33	0.00

instance 309, baseline q 1.00 success, warm-up q 0.33, 39x speedThe baseline robot moves the sandals and then puts the book on the nightstand, while the warm-up robot keeps handling the sandals and never gets to the book.

instance 308, baseline q 1.00 success, warm-up q 0.67, 39x speedThe baseline robot finishes the sandals and carries the book to the nightstand, while the warm-up robot juggles the sandals and then wanders off toward the door.