← Activity

turning_on_radio before and after the evaluator fix

pt50 on BEHAVIOR-1K, RTX 5000, 2x action speed. The same instances and seeds run under three evaluators.

What the policy saw at step 0

Test instance 5, seed 0. In the stock run the policy received a frame of the living room with the radio on a coffee table right in front of it. The robot was actually facing a TV wall. All 4 envs in that batch got the same stale frame. With the fix, the policy gets the real view.

The first 30 seconds, side by side

Head camera only. The pre-v3.9.3 and fixed runs both drive to the radio and grab it. The stock run plans its first move for the scene it wrongly saw, closes a gripper on nothing, and never moves again.

Every test instance, all five runs

One clip per test instance, each episode sped up about 10x. Top row: before (pre-v3.9.3, 1 env), bug seed 0, bug seed 1. Bottom row: stock 1 env, fixed. Each tile is labelled stale or fresh and success or fail, and finished episodes hold their last frame. The chips under each clip give the same results.

The fix in evaluator.py

After loading a batch, the stock load_batch only re-renders and re-reads observations for tasks that sync lights, and turning_on_radio isn't one of them. The renderer runs a few frames behind, so with several envs the first observation still holds an old frame. The fix does the re-render and re-read for every task.

OmniGibson/omnigibson/eval/evaluator.py, load_batch, after env.reset@@ -554,10 +554,10 @@-        if self.should_sync_lights:-            for _ in range(3):-                og.sim.render()-            obs_list, _ = self.env.get_obs(env_indices=th.tensor(env_indices, dtype=th.long))+        # issue 49 stalecheck: re-render and re-read for every task so step 0 never carries a pre-load frame.+        for _ in range(int(os.environ.get("STALECHECK_RERENDERS", "3"))):+            og.sim.render()+        obs_list, _ = self.env.get_obs(env_indices=th.tensor(env_indices, dtype=th.long))

For the submission we can't change the organizers' evaluator, so the policy server holds still for its first 2 requests instead. That gets the same fresh image to the policy before it acts.

Pooled results

Public test instances 0 to 9. Stock and fixed are paired over 5 seeds (50 episodes each). Pre-v3.9.3 and stock 1-env ran seed 0 only.

Stale frames 28 to 0 (paired McNemar p = 7.5e-9) and driving 18 to 39 (p = 0.0001) are clear. Success 12 to 16 is not significant on its own (p = 0.42), because pt50 rarely solves most radio instances even with a good frame. Within the stock runs, though, success was 12 of 22 with a fresh first frame and 0 of 28 with a stale one (Fisher p = 5.3e-6).