Can training CogAct from PrismaticVLM achieve convergence?

[fx-hit]

How long does it take to achieve convergence when starting training from PrismaticVLM?
When loading the OpenVLA checkpoint and training with BridgeV2 for 5k iterations, some success rate is observed.
However, if loading the prism-dinosiglip-224px+7b model—first pretrained on the OpenX dataset with 16 A100 GPUs for 40k iterations, then trained on BridgeV2 for 25k iterations—the success rate remains zero.

[arnoldland]

That sounds a bit strange — it’s like there might be a bug somewhere. When we trained with batch_size=2048 from Prismatic-7B on OXE for 10K steps, we tested directly on Bridge and already saw around a 47.9% success rate, which then increased slowly over time.

Additionally, if we switch to a smaller model like paligemma-3B and fine-tune directly on BridgeV2 with batch_size=256, the success rate after 10K steps is still decent. This suggests that something in your setup might be not correct. You can also have a try to test directly with the checkpoints trained on OXE, without fine-tuning on BridgeV2.

[fx-hit]

Thank you for your reply. Perhaps something went wrong, and we still need to look for the bug. When we first tried to reproduce your model, we started by loading the VLM pretrained model and trained it on the bridgev2 dataset, but the success rate was still 0 after 20k iterations. Later, after reading the paper, we realized that the training actually started from the OpenVLA pretrained model. So we made the change, and the success rate reached 0.3 after just 5k iterations. We initially thought it might be due to the dataset size, so we later trained on OXE, but found that with the VLM-based checkpoint the success rate was still 0, while switching to the OpenVLA checkpoint worked fine.