Could not reproduce the Libero performance

[dragonlzm]

Hello! Thanks for the great work. I download the checkpoint from huggingface and run the inference using following command.

CUDA_VISIBLE_DEVICES=1 python experiments/robot/libero/run_libero_eval.py \
  --use_proprio True \
  --num_images_in_input 2 \
  --use_film False \
  --pretrained_checkpoint outputs/LIBERO-Spatial-Pro \
  --task_suite_name libero_spatial \
  --use_pro_version True \


# 12/17 [19:13:30] INFO     | >> Success: True                                                                                             run_libero_eval.py:222
#                  INFO     | >> # episodes completed so far: 500                                                                          run_libero_eval.py:222
#                  INFO     | >> # successes: 477 (95.4%)                                                                                  run_libero_eval.py:222
# 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [08:06<00:00,  9.72s/it]
#                  INFO     | >> Current task success rate: 1.0                                                                            run_libero_eval.py:222
#                  INFO     | >> Current total success rate: 0.954                                                                         run_libero_eval.py:222
# 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [1:11:27<00:00, 428.78s/it]
#                  INFO     | >> Final results:                                                                                            run_libero_eval.py:222
#                  INFO     | >> Total episodes: 500                                                                                       run_libero_eval.py:222
#                  INFO     | >> Total successes: 477                                                                                      run_libero_eval.py:222
#                  INFO     | >> Overall success rate: 0.9540 (95.4%)

However, it could not reproduce the number you guys provided 99.6% vs what I have 95.4%, any suggestion to reproduce the number? Thanks!

[WangYH-BUPT]

Hello! Thanks for the great work. I download the checkpoint from huggingface and run the inference using following command.

CUDA_VISIBLE_DEVICES=1 python experiments/robot/libero/run_libero_eval.py
--use_proprio True
--num_images_in_input 2
--use_film False
--pretrained_checkpoint outputs/LIBERO-Spatial-Pro
--task_suite_name libero_spatial
--use_pro_version True \

12/17 [19:13:30] INFO | >> Success: True run_libero_eval.py:222

INFO | >> # episodes completed so far: 500 run_libero_eval.py:222

INFO | >> # successes: 477 (95.4%) run_libero_eval.py:222

100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [08:06<00:00, 9.72s/it]

INFO | >> Current task success rate: 1.0 run_libero_eval.py:222

INFO | >> Current total success rate: 0.954 run_libero_eval.py:222

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [1:11:27<00:00, 428.78s/it]

INFO | >> Final results: run_libero_eval.py:222

INFO | >> Total episodes: 500 run_libero_eval.py:222

INFO | >> Total successes: 477 run_libero_eval.py:222

INFO | >> Overall success rate: 0.9540 (95.4%)

However, it could not reproduce the number you guys provided 99.6% vs what I have 95.4%, any suggestion to reproduce the number? Thanks!

This is due to differences in the GPUs used; this phenomenon also exists in OpenVLA-OFT, and it's a common issue with LLMs. The problem is actually quite easy to solve: train with your own GPU and then perform inference with the same GPU. The results are easily reproducible; many students in the discussion group have already reproduced results similar to or even surpassing those in the paper using their own GPUs.