[NPU]: support use-rollout-routing-replay#292
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new training script run-qwen3-30B-A3B-npu-use-routing-replay.sh for running Qwen3-30B-A3B training on NPUs. Feedback on the script highlights two main issues: first, there are duplicate arguments (--use-dynamic-batch-size and --max-tokens-per-gpu) with conflicting values (8192 vs 20480) that need to be cleaned up; second, a hardcoded user directory path is used for the prompt dataset, which should be replaced with a configurable environment variable to improve portability.
| --use-dynamic-batch-size \ | ||
| --max-tokens-per-gpu 8192 \ |
There was a problem hiding this comment.
The arguments --use-dynamic-batch-size and --max-tokens-per-gpu are duplicated in this script (see lines 90-91). Additionally, --max-tokens-per-gpu is specified with conflicting values (8192 vs 20480). Please remove the duplicate entries and keep only the correct configuration (likely 20480 to match the max model length of 20k).
| \ | ||
| --hf-checkpoint /home/data/Qwen3-30B-A3B/ \ | ||
| \ | ||
| --prompt-data /home/w00893744/dataset/dapo-math-17k/dapo-math-17k.jsonl \ |
There was a problem hiding this comment.
The path /home/w00893744/dataset/dapo-math-17k/dapo-math-17k.jsonl contains a hardcoded user directory (w00893744), which prevents the script from being portable across different environments or users. Consider using a configurable environment variable with a fallback default.
| --prompt-data /home/w00893744/dataset/dapo-math-17k/dapo-math-17k.jsonl \ | |
| --prompt-data "${PROMPT_DATA:-/home/data/dapo-math-17k/dapo-math-17k.jsonl}" \ |
|
I'd suggest putting the sh script in #270, and also using the ray submit submission format. 
|
done |
11740e1 to
f992c58
Compare
Signed-off-by: Windfeng8 <523758380@qq.com>
…replay.sh Signed-off-by: Windfeng8 <523758380@qq.com>
Signed-off-by: Windfeng8 <523758380@qq.com>
f992c58 to
5763331
Compare
floatlibai
left a comment
floatlibai
left a comment
There was a problem hiding this comment.
A few comments on the script you submitted.
| # will prevent ray from buffering stdout/stderr | ||
| export PYTHONUNBUFFERED=1 | ||
| export SLIME_SCRIPT_TRAIN_BACKEND=megatron | ||
| export PYTHONPATH="/workspace/issue205/Megatron-LM:/workspace/issue205/Megatron-Bridge/src:${PYTHONPATH}" |
There was a problem hiding this comment.
I suggest export PYTHONPATH="/root/Megatron-Bridge/src:/root/Megatron-LM/:$PYTHONPATH"
| ) | ||
|
|
||
| ROLLOUT_ARGS=( | ||
| --prompt-data /home/w00893744/dataset/dapo-math-17k.jsonl |
There was a problem hiding this comment.
please remove your Employee ID and use a general path
| ) | ||
|
|
||
| VLLM_ARGS=( | ||
| --rollout-backend vllm |
| --use-rollout-routing-replay | ||
| --vllm-weight-sync-mode native | ||
| --vllm-gpu-memory-utilization 0.7 | ||
| --vllm-enable-sleep-mode |
There was a problem hiding this comment.
this line is useless too
| --attention-backend flash | ||
| --use-flash-attn | ||
| --no-gradient-accumulation-fusion | ||
| --train-memory-margin-bytes 2147483648 |
There was a problem hiding this comment.
unessary I think, you can ref scripts/run-qwen3-30B-A3B-npu.sh on newest ascend branch
| }" | ||
|
|
||
| ray job submit --address="http://127.0.0.1:8265" \ | ||
| --runtime-env-json="${RUNTIME_ENV_JSON}" \ |
There was a problem hiding this comment.
These runtime-envs are all unessary, I suggest removing them
| ray job submit --address="http://127.0.0.1:8265" \ | ||
| --runtime-env-json="${RUNTIME_ENV_JSON}" \ | ||
| -- python3 "${REPO_ROOT}/train.py" \ | ||
| --train-backend megatron \ |
There was a problem hiding this comment.
Remove this line too, only one training backend now.
|
I wonder if this script/pr is necessary, since it only adds one parameter ( |
3 participants
[RFC] Steps and Test Result for Running Qwen3-30B on NPU(A3) （use-rollout-routing-replay）
#270