[bugfix] fix vit_lora tp#5
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements a monkey-patch for LoraModel to enable gradient averaging across tensor parallel domains for MegatronModule and adds peer-to-peer communication settings. It also introduces extensive test files for LLMs and MLLMs. Review feedback identifies a critical recursion bug in the monkey-patching implementation, suggests expanding support to other PEFT layers, and recommends removing hardcoded environment variables and dead code from the test scripts.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a monkey-patch for LoraModel to support gradient averaging in Megatron modules and adds comprehensive test suites for LLMs and MLLMs. Review feedback identifies several improvement opportunities, including removing hardcoded environment variables in tests to support varied environments, fixing a potential recursion bug in the patching logic by returning early, and cleaning up extensive commented-out code in the test files.
1 participant
No description provided.