Bullet over IG for training?

[samibouziri]

Hello,

Out of curiosity, I was wondering if there was a reason of training the model in Bullet instead of in IG? Is it due to the OMG planner?

[ychao-nvidia]

We went for Bullet first since historically the environment (handover-sim) and training pipelines (OMG-Planner and GA-DDPG) were both developed and tested in Bullet. That said, we don't see a major blocker for running everything in IG. There may be some minor hiccups:

• Bullet has better support for some necessary functions (e.g., a native IK solver) that you need to take care of separately if using IG.
• IG does not support "unloading" a model once the model is imported. This may introduce additional overhead for testing (for training it's fine), since you either need to (1) load all test environments at once (i.e., one env per scene) and do batch testing, (2) use one test environment but import all models and hide the ones not used in a test scene, or (3) use one test environment, load only the required models for a scene, but re-initialize IG for each test scene. We did (3) for our IG evaluation in the CVPR'23 paper.

This is assuming you want to stick with the current ray-based training pipeline. If you are thinking of leveraging GPU simulation, there will be one challenge---you can GPU-parallelize the env with IG, but the expert demonstration with OMG-Planner is not GPU parallelizable.