Create example for use of reasoning-gym with OpenRLHF

[andreaskoepf]

Create an example configuration for OpenRLHF to train a 1B or 3B model with REINFORCE++ and the simple arithmetic reasoning-gym dataset.

• locally running example with transformers generate inference
• ray based variant

[andreaskoepf]

PPO example running on local node with deepspeed can be found here: https://github.com/open-thought/reasoning-gym/tree/main/examples/OpenRLHF, wandb: https://wandb.ai/andreaskoepf/openrlhf_train_ppo

[joesharratt1229]

Happy to take this

[joesharratt1229]

Just realised the parent issue had already been assigned and the task is partly complete, apologies!

[andreaskoepf]

@joesharratt1229 an trl example is currently still open ... could you image to work on that? Just saw this with optuna .. https://github.com/s-smits/grpo-optuna/blob/main/main.py (but we could first do the baseline...)

[andreaskoepf]

btw alternatively for OpenRLHF the ray variant is also still missing, if someone wants to work on that.

[joesharratt1229]

Yes happy to work on a trl example

[andreaskoepf]

@joesharratt1229 fantastic, please feel free to assign yourself to the trl example issue #53 .