update lora add#98
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request modifies the weight merging logic in src/mcore_bridge/tuners/lora.py to use orig_weight.data += delta_weight. The reviewer noted that accessing the .data attribute directly is discouraged in modern PyTorch because it bypasses autograd tracking and safety checks. They suggested using .detach() instead to safely perform the in-place addition.
| delta_weights = self.get_delta_weights(active_adapter) | ||
| for orig_weight, delta_weight in zip(orig_weights, delta_weights): | ||
| orig_weight += delta_weight | ||
| orig_weight.data += delta_weight |
There was a problem hiding this comment.
Using the .data attribute is discouraged in modern PyTorch because it bypasses autograd tracking and safety checks, which can lead to silent bugs. To safely perform this in-place addition without triggering autograd errors (since delta_weight may require gradients), you should detach delta_weight using .detach() instead.
| orig_weight.data += delta_weight | |
| orig_weight += delta_weight.detach() |
1 participant
No description provided.