Question about the modification on Verl

[glgjss960]

It's really a good work.I wonder which components code files of Verl have the mem-alpha performed secondary development on?Thanks!This may help me understand the overall Implementation method.

[glgjss960]

Second, I figure out that the code flow of the training script is as below: verl/trainer/main_ppo->verl/trainer/ray_trainer->memalpha/llm_agent/generation->memory.py/agent.py->function.py, Is there anything missing in my understanding?Thanks！

[wangyu-ustc]

Thanks for the interests!
verl/trainer/main_ppo -> verl/trainer/ppo/ray_trainer.py -> memalpha/llm_agent/generation
Yes this is the training flow. As for agent.py and function.py, they are the agent formulation in generation.py. Your understanding is correct.

[glgjss960]

verl/workers/reward_manager/naive.py, verl/trainer/ppo/ray_trainer.py, except these two code files, is there any other code file under verl is modified?

[wangyu-ustc]

I cannot remember exactly, there are many commits here, in my impression, the above files are mainly the files that were modified.

[glgjss960]

Were the base verl repo you modified on verl==0.4.1.x?

[glgjss960]

I wonder what are verl/experimental/agent_loop/tool_agent_loop.py, verl/experimental/agent_loop/agent_loop.py and verl/experimental/agent_loop/single_agent_loop.py used for?

I noticed that these three codes didn't exist in verl<=0.4.1.x, but they also have some differences with those of verl>=0.5.x, it seemed that you had modified all .py codes under verl/experimental/agent_loop, however, I can't find any .py code that contains 'from verl.experimental.agent_loop.xxx import xxx' except .py codes in verl/experimental/agent_loop.

[glgjss960]

I wonder what are verl/experimental/agent_loop/tool_agent_loop.py, verl/experimental/agent_loop/agent_loop.py and verl/experimental/agent_loop/single_agent_loop.py used for?

I noticed that these three codes didn't exist in verl<=0.4.1.x, but they also have some differences with those of verl>=0.5.x, it seemed that you had modified all .py codes under verl/experimental/agent_loop, however, I can't find any .py code that contains 'from verl.experimental.agent_loop.xxx import xxx' except .py codes in verl/experimental/agent_loop.

I have figured out this problem.

[glgjss960]

Were the base verl repo you modified on verl==0.4.1.x?

I noticed that in /verl/version/version, It shows ' 0.4.0.dev ', I guess it is the version of the verl in mem-alpha repo, does it mean that the base verl repo you modified on is verl==0.4.0 ?

[wangyu-ustc]

Good question! In fact I was using the verl base that my colleagues were using (so that I can find them for help when I encounter issues), but I'm not quite sure what version they use. Can I kindly ask why you need this info? I think you could either fork my repo and start from here, or use the newest verl?