-
Notifications
You must be signed in to change notification settings - Fork 629
Add worker interface "reload_weights" #4459
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Signed-off-by: ivyilike <[email protected]>
Signed-off-by: ivyilike <[email protected]>
|
Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request adds a reload_weights interface to the Ascend worker, allowing for in-place reloading of model weights without re-initializing the model. The implementation correctly adds the reload_weights method to NPUModelRunner and NPUWorker. However, there is a critical issue in NPUWorker.reload_weights where an assertion will incorrectly fail when sleep mode is enabled, preventing the feature from working as intended in that configuration.
Signed-off-by: ivyilike <[email protected]>
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
Signed-off-by: ivyilike <[email protected]>
What this PR does / why we need it?
Some worker interfaces called by vLLM are missing in vLLM Ascend. It's good to add the support.
this PR add support to "reload_weights" interface.
this will explicitly reload weights inplace, without re-initialize the model