BEHAVIOR-1K 2026 challenge, public test instances 0 to 9, one run each, 99 of 100 tasks
The submission scores mean Q 0.092 with 18 successes. All 18 successes come from pt50 on the original 50 tasks. The step follower on the new tasks scores 0.031 with none, so half the benchmark contributes almost nothing.
| group | policy | tasks | mean Q | successes | tasks with any Q above 0 |
|---|---|---|---|---|---|
| original 50 | pt50 | 50 | 0.155 | 18 | 37 |
| new 49 | follower | 49 | 0.031 | 0 | 24 |
Several original tasks earn high partial credit but never finish. collecting_childrens_toys scores 0.586, setting_mousetraps 0.533, putting_up_Christmas_decorations_inside 0.456, putting_shoes_on_rack 0.440, picking_up_toys 0.383, assembling_gift_baskets 0.375 and sorting_vegetables 0.277, all with no success. pt50 does most of each task and then fails on one subgoal, so these are the best targets for recovery data. 13 original tasks score exactly zero, and those would need whole new skills rather than corrections.
| task | runs | mean Q | successes | runs with partial credit |
|---|---|---|---|---|
| collecting_childrens_toys | 10 | 0.586 | 0 | 9 |
| setting_mousetraps | 10 | 0.533 | 0 | 10 |
| preparing_lunch_box | 10 | 0.483 | 1 | 8 |
| putting_up_Christmas_decorations_inside | 10 | 0.456 | 0 | 10 |
| moving_boxes_to_storage | 10 | 0.450 | 4 | 1 |
| putting_shoes_on_rack | 10 | 0.440 | 0 | 10 |
| picking_up_trash | 10 | 0.400 | 3 | 2 |
| picking_up_toys | 10 | 0.383 | 0 | 8 |
| hiding_Easter_eggs | 10 | 0.378 | 2 | 3 |
| assembling_gift_baskets | 10 | 0.375 | 0 | 10 |
| putting_away_Halloween_decorations | 10 | 0.314 | 1 | 5 |
| turning_on_radio | 10 | 0.300 | 3 | 0 |
| tidying_bedroom | 10 | 0.300 | 0 | 7 |
| storing_food | 10 | 0.287 | 0 | 5 |
| sorting_vegetables | 10 | 0.277 | 0 | 9 |
| slicing_vegetables | 10 | 0.178 | 1 | 3 |
| chop_an_onion | 10 | 0.175 | 0 | 5 |
| loading_the_car | 10 | 0.167 | 1 | 2 |
| bringing_water | 10 | 0.167 | 0 | 3 |
| wash_a_baseball_cap | 10 | 0.100 | 1 | 0 |
| spraying_for_bugs | 10 | 0.100 | 0 | 2 |
| make_microwave_popcorn | 10 | 0.100 | 1 | 0 |
| freeze_pies | 10 | 0.086 | 0 | 4 |
| set_up_a_coffee_station_in_your_kitchen | 10 | 0.083 | 0 | 4 |
| clean_up_your_desk | 10 | 0.082 | 0 | 7 |
| sorting_household_items | 10 | 0.075 | 0 | 4 |
| boxing_books_up_for_storage | 10 | 0.067 | 0 | 3 |
| setting_the_fire | 10 | 0.062 | 0 | 3 |
| rearranging_kitchen_furniture | 10 | 0.050 | 0 | 2 |
| putting_dishes_away_after_cleaning | 10 | 0.050 | 0 | 2 |
| carrying_in_groceries | 10 | 0.050 | 0 | 2 |
| cook_hot_dogs | 10 | 0.050 | 0 | 1 |
| cleaning_up_plates_and_food | 10 | 0.043 | 0 | 2 |
| canning_food | 10 | 0.030 | 0 | 3 |
| chopping_wood | 10 | 0.025 | 0 | 1 |
| clearing_food_from_table_into_fridge | 10 | 0.020 | 0 | 1 |
| getting_organized_for_work | 10 | 0.010 | 0 | 1 |
| can_meat | 10 | 0.000 | 0 | 0 |
| bringing_in_wood | 10 | 0.000 | 0 | 0 |
| outfit_a_basic_toolbox | 10 | 0.000 | 0 | 0 |
| clean_boxing_gloves | 10 | 0.000 | 0 | 0 |
| wash_dog_toys | 10 | 0.000 | 0 | 0 |
| hanging_pictures | 10 | 0.000 | 0 | 0 |
| attach_a_camera_to_a_tripod | 10 | 0.000 | 0 | 0 |
| clean_a_patio | 10 | 0.000 | 0 | 0 |
| clean_a_trumpet | 10 | 0.000 | 0 | 0 |
| spraying_fruit_trees | 10 | 0.000 | 0 | 0 |
| cook_cabbage | 10 | 0.000 | 0 | 0 |
| cook_bacon | 10 | 0.000 | 0 | 0 |
| make_pizza | 10 | 0.000 | 0 | 0 |
The best new task is make_rose_centerpieces at 0.250. 24 of the 49 earn some partial credit, and none succeeds.
| task | runs | mean Q | successes | runs with partial credit |
|---|---|---|---|---|
| make_rose_centerpieces | 10 | 0.250 | 0 | 8 |
| thawing_frozen_food | 10 | 0.167 | 0 | 7 |
| putting_away_toys | 10 | 0.150 | 0 | 6 |
| tidying_bathroom | 10 | 0.125 | 0 | 5 |
| collecting_aluminum_cans | 10 | 0.100 | 0 | 5 |
| setup_a_bar_for_a_cocktail_party | 10 | 0.070 | 0 | 7 |
| rearrange_your_room | 10 | 0.067 | 0 | 1 |
| organizing_art_supplies | 10 | 0.060 | 0 | 3 |
| boxing_food_after_dinner | 10 | 0.050 | 0 | 3 |
| putting_dirty_dishes_in_sink | 10 | 0.050 | 0 | 1 |
| make_gift_bags_for_baby_showers | 10 | 0.050 | 0 | 3 |
| installing_a_fax_machine | 10 | 0.050 | 0 | 1 |
| composting_waste | 10 | 0.050 | 0 | 1 |
| dispose_of_batteries | 10 | 0.050 | 0 | 2 |
| organizing_school_stuff | 10 | 0.033 | 0 | 2 |
| clean_up_broken_glass | 10 | 0.033 | 0 | 1 |
| sorting_bottles_cans_and_paper | 10 | 0.031 | 0 | 5 |
| dispose_of_glass | 10 | 0.025 | 0 | 1 |
| installing_a_modem | 10 | 0.025 | 0 | 1 |
| halve_an_egg | 10 | 0.020 | 0 | 1 |
| polishing_shoes | 10 | 0.017 | 0 | 1 |
| packing_meal_for_delivery | 10 | 0.017 | 0 | 1 |
| laying_tile_floors | 10 | 0.013 | 0 | 1 |
| sorting_books_on_shelf | 10 | 0.009 | 0 | 1 |
| freeze_fruit | 10 | 0.000 | 0 | 0 |
| cook_a_brisket | 10 | 0.000 | 0 | 0 |
| tidying_living_room | 10 | 0.000 | 0 | 0 |
| re_shelving_library_books | 10 | 0.000 | 0 | 0 |
| sweeping_garage | 10 | 0.000 | 0 | 0 |
| scrubbing_bathroom_floor | 10 | 0.000 | 0 | 0 |
| bringing_paper_to_recycling | 10 | 0.000 | 0 | 0 |
| installing_smoke_detectors | 10 | 0.000 | 0 | 0 |
| setting_the_table | 10 | 0.000 | 0 | 0 |
| unloading_the_car | 10 | 0.000 | 0 | 0 |
| turning_out_all_lights_before_sleep | 10 | 0.000 | 0 | 0 |
| cleaning_up_branches_and_twigs | 10 | 0.000 | 0 | 0 |
| vacuuming_floors | 10 | 0.000 | 0 | 0 |
| clean_your_rusty_garden_tools | 10 | 0.000 | 0 | 0 |
| cook_a_frozen_pie | 10 | 0.000 | 0 | 0 |
| carrying_out_garden_furniture | 10 | 0.000 | 0 | 0 |
| put_together_a_basic_pruning_kit | 10 | 0.000 | 0 | 0 |
| make_cabinet_doors | 10 | 0.000 | 0 | 0 |
| store_batteries | 10 | 0.000 | 0 | 0 |
| store_honey | 10 | 0.000 | 0 | 0 |
| store_produce | 10 | 0.000 | 0 | 0 |
| installing_a_scanner | 10 | 0.000 | 0 | 0 |
| clean_a_keyboard | 10 | 0.000 | 0 | 0 |
| cook_brussels_sprouts | 10 | 0.000 | 0 | 0 |
| cook_broccolini | 10 | 0.000 | 0 | 0 |