SnapFlow one-step student in sim | Kanta Saito

SnapFlow distills pt50's 10-step flow sampler into a student that acts in 1 or 2 steps. Offline it works, but in sim every step-reduced arm loses success rate against pt50 at 10 steps, and it saves almost no time. Since we optimize for success, this is a negative result and pt50 at 10 steps stays the submission.

Arms

Public test instances on moving_boxes_to_storage, picking_up_trash and cook_hot_dogs, submission setting with the 2-step warm-up

arm	episodes	mean Q (95% CI)	successes	model ms per request (median, p90)
pt50, 10 steps	30	0.300 (0.167 to 0.439)	7	not logged
pt50, 1 step	20	0.017 (0.000 to 0.050)	0	2730, 2808, over 4629 requests
student, 2 steps	20	0.142 (0.025 to 0.267)	2	2730, 2758, over 7428 requests
student, 1 step	20	0.058 (0.000 to 0.142)	0	2717, 2829, over 5937 requests

pt50 at 10 steps is the issue 53 warm-up rerun, so its model time was not logged. Every arm spends about 2.7 s per request, because the VLM prefix dominates on the RTX 5000 and each flow step adds only 10 to 40 ms.

Paired differences

First arm minus second on the same task and instance, CIs resample instances within tasks

comparison	pairs	first Q	second Q	difference (95% CI)	successes
student, 1 step vs pt50, 1 step	20	0.058	0.017	0.042 (-0.033 to 0.133)	0 vs 0
student, 1 step vs pt50, 10 steps	20	0.058	0.292	-0.233 (-0.408 to -0.058)	0 vs 5
student, 2 steps vs pt50, 10 steps	20	0.142	0.292	-0.150 (-0.375 to 0.067)	2 vs 5
student, 1 step vs student, 2 steps	20	0.058	0.142	-0.083 (-0.242 to 0.067)	0 vs 2
pt50, 1 step vs pt50, 10 steps	20	0.017	0.292	-0.275 (-0.467 to -0.100)	0 vs 5

Per task

Mean Q (episodes, successes)

task	pt50, 10 steps	pt50, 1 step	student, 2 steps	student, 1 step
moving_boxes_to_storage	0.450 (10, 4)	0.000 (8, 0)	0.312 (8, 2)	0.062 (8, 0)
picking_up_trash	0.400 (10, 3)	0.042 (8, 0)	0.042 (8, 0)	0.083 (8, 0)
cook_hot_dogs	0.050 (10, 0)	0.000 (4, 0)	0.000 (4, 0)	0.000 (4, 0)

Takeaway

The 1-step student is clearly below pt50 at 10 steps, and the 2-step student is below it with a CI that still reaches zero. Against pt50 at 1 step the student is slightly better (+0.042, not significant), which matches the offline check where the 1-step student halves pt50's 1-step error against the 10-step teacher. So the distillation itself works, but fewer flow steps cost more success than it recovers.

The eval was stopped here. The last two pt50 1-step jobs (15114945 and 15116156) were cancelled and the pt50 2-step arm was never run, since no outcome of them would change the decision. Episode counts are small, with 20 or fewer per arm and 4 on cook_hot_dogs for the student. The full table is in distill/eval/snapflow/RESULTS.md on PR 61 and on issue 27.