-
Notifications
You must be signed in to change notification settings - Fork 237
Add GLM-5.1 FP8 B300 SGLang configurations / 新增 GLM-5.1 FP8 B300 SGLang 配置 #2320
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
+264
−0
Closed
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # NOTE: At the time of submission, https://cookbook.sglang.io/autoregressive/GLM/GLM-5.1 | ||
| # does not have a B300-specific recipe, so this script reuses the GLM5 FP8 | ||
| # B200 SGLang recipe until B300-specific guidance is available. | ||
|
|
||
| source "$(dirname "$0")/../../benchmark_lib.sh" | ||
|
|
||
| check_env_vars \ | ||
| MODEL \ | ||
| TP \ | ||
| CONC \ | ||
| ISL \ | ||
| OSL \ | ||
| RANDOM_RANGE_RATIO \ | ||
| RESULT_FILENAME | ||
|
|
||
| # `hf download` creates the target directory if needed and is idempotent. | ||
| # When MODEL_PATH is unset for a stand-alone run, fall back to the HF cache. | ||
| if [[ -n "${MODEL_PATH:-}" ]]; then | ||
| if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then | ||
| hf download "$MODEL" --local-dir "$MODEL_PATH" | ||
| fi | ||
| else | ||
| hf download "$MODEL" | ||
| export MODEL_PATH="$MODEL" | ||
| fi | ||
|
|
||
| if [[ -n "$SLURM_JOB_ID" ]]; then | ||
| echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME" | ||
| fi | ||
|
|
||
| nvidia-smi | ||
|
|
||
| export SGLANG_ENABLE_JIT_DEEPGEMM=1 | ||
|
|
||
| SERVER_LOG=/workspace/server.log | ||
|
|
||
| echo "CONC: $CONC, ISL: $ISL, OSL: $OSL" | ||
|
|
||
| EVAL_CONTEXT_ARGS="" | ||
| if [[ "${EVAL_ONLY}" == "true" ]]; then | ||
| setup_eval_context | ||
| EVAL_CONTEXT_ARGS="--context-length $EVAL_MAX_MODEL_LEN" | ||
| fi | ||
|
|
||
| start_gpu_monitor | ||
|
|
||
| set -x | ||
| PYTHONNOUSERSITE=1 python3 -m sglang.launch_server \ | ||
| --model-path "$MODEL_PATH" \ | ||
| --served-model-name "$MODEL" \ | ||
| --host 0.0.0.0 \ | ||
| --port "$PORT" \ | ||
| --trust-remote-code \ | ||
| --tensor-parallel-size "$TP" \ | ||
| --data-parallel-size 1 \ | ||
| --expert-parallel-size 1 \ | ||
| --tool-call-parser glm47 \ | ||
| --reasoning-parser glm45 \ | ||
| --kv-cache-dtype fp8_e4m3 \ | ||
| --quantization fp8 \ | ||
| --attention-backend nsa \ | ||
| --nsa-decode-backend trtllm \ | ||
| --nsa-prefill-backend trtllm \ | ||
| --moe-runner-backend flashinfer_trtllm \ | ||
| --cuda-graph-max-bs "$CONC" \ | ||
| --max-running-requests "$CONC" \ | ||
| --mem-fraction-static 0.85 \ | ||
| --chunked-prefill-size 32768 \ | ||
| --max-prefill-tokens 32768 \ | ||
| --enable-flashinfer-allreduce-fusion \ | ||
| --disable-radix-cache \ | ||
| --stream-interval 30 \ | ||
| --model-loader-extra-config '{"enable_multithread_load": true}' \ | ||
| $EVAL_CONTEXT_ARGS > "$SERVER_LOG" 2>&1 & | ||
|
|
||
| SERVER_PID=$! | ||
|
|
||
| wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID" | ||
|
|
||
| pip install -q datasets pandas | ||
|
|
||
| run_benchmark_serving \ | ||
| --model "$MODEL" \ | ||
| --port "$PORT" \ | ||
| --backend vllm \ | ||
| --input-len "$ISL" \ | ||
| --output-len "$OSL" \ | ||
| --random-range-ratio "$RANDOM_RANGE_RATIO" \ | ||
| --num-prompts "$((CONC * 10))" \ | ||
| --max-concurrency "$CONC" \ | ||
| --result-filename "$RESULT_FILENAME" \ | ||
| --result-dir /workspace/ | ||
|
|
||
| if [[ "${RUN_EVAL}" == "true" ]]; then | ||
| run_eval --framework lm-eval --port "$PORT" | ||
| append_lm_eval_summary | ||
| fi | ||
|
|
||
| stop_gpu_monitor | ||
| set +x |
108 changes: 108 additions & 0 deletions
108
benchmarks/single_node/fixed_seq_len/glm5_fp8_b300_mtp.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,108 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # NOTE: At the time of submission, https://cookbook.sglang.io/autoregressive/GLM/GLM-5.1 | ||
| # does not have a B300-specific recipe, so this script reuses the GLM5 FP8 | ||
| # B200 SGLang recipe until B300-specific guidance is available. | ||
|
|
||
| source "$(dirname "$0")/../../benchmark_lib.sh" | ||
|
|
||
| check_env_vars \ | ||
| MODEL \ | ||
| TP \ | ||
| CONC \ | ||
| ISL \ | ||
| OSL \ | ||
| RANDOM_RANGE_RATIO \ | ||
| RESULT_FILENAME | ||
|
|
||
| # `hf download` creates the target directory if needed and is idempotent. | ||
| # When MODEL_PATH is unset for a stand-alone run, fall back to the HF cache. | ||
| if [[ -n "${MODEL_PATH:-}" ]]; then | ||
| if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then | ||
| hf download "$MODEL" --local-dir "$MODEL_PATH" | ||
| fi | ||
| else | ||
| hf download "$MODEL" | ||
| export MODEL_PATH="$MODEL" | ||
| fi | ||
|
|
||
| if [[ -n "$SLURM_JOB_ID" ]]; then | ||
| echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME" | ||
| fi | ||
|
|
||
| nvidia-smi | ||
|
|
||
| export SGLANG_ENABLE_JIT_DEEPGEMM=1 | ||
| export SGLANG_ENABLE_SPEC_V2=1 | ||
|
|
||
| SERVER_LOG=/workspace/server.log | ||
|
|
||
| echo "CONC: $CONC, ISL: $ISL, OSL: $OSL" | ||
|
|
||
| EVAL_CONTEXT_ARGS="" | ||
| if [[ "${EVAL_ONLY}" == "true" ]]; then | ||
| setup_eval_context | ||
| EVAL_CONTEXT_ARGS="--context-length $EVAL_MAX_MODEL_LEN" | ||
| fi | ||
|
|
||
| start_gpu_monitor | ||
|
|
||
| set -x | ||
| PYTHONNOUSERSITE=1 python3 -m sglang.launch_server \ | ||
| --model-path "$MODEL_PATH" \ | ||
| --served-model-name "$MODEL" \ | ||
| --host 0.0.0.0 \ | ||
| --port "$PORT" \ | ||
| --trust-remote-code \ | ||
| --tensor-parallel-size "$TP" \ | ||
| --data-parallel-size 1 \ | ||
| --expert-parallel-size 1 \ | ||
| --tool-call-parser glm47 \ | ||
| --reasoning-parser glm45 \ | ||
| --kv-cache-dtype fp8_e4m3 \ | ||
| --quantization fp8 \ | ||
| --attention-backend nsa \ | ||
| --nsa-decode-backend trtllm \ | ||
| --nsa-prefill-backend trtllm \ | ||
| --moe-runner-backend flashinfer_trtllm \ | ||
| --cuda-graph-max-bs "$CONC" \ | ||
| --max-running-requests "$CONC" \ | ||
| --mem-fraction-static 0.85 \ | ||
| --chunked-prefill-size 32768 \ | ||
| --max-prefill-tokens 32768 \ | ||
| --enable-flashinfer-allreduce-fusion \ | ||
| --disable-radix-cache \ | ||
| --stream-interval 30 \ | ||
| --speculative-algorithm EAGLE \ | ||
| --speculative-num-steps 3 \ | ||
| --speculative-eagle-topk 1 \ | ||
| --speculative-num-draft-tokens 4 \ | ||
| --model-loader-extra-config '{"enable_multithread_load": true}' \ | ||
| $EVAL_CONTEXT_ARGS > "$SERVER_LOG" 2>&1 & | ||
|
|
||
| SERVER_PID=$! | ||
|
|
||
| wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID" | ||
|
|
||
| pip install -q datasets pandas | ||
|
|
||
| run_benchmark_serving \ | ||
| --model "$MODEL" \ | ||
| --port "$PORT" \ | ||
| --backend vllm \ | ||
| --input-len "$ISL" \ | ||
| --output-len "$OSL" \ | ||
| --random-range-ratio "$RANDOM_RANGE_RATIO" \ | ||
| --num-prompts "$((CONC * 10))" \ | ||
| --max-concurrency "$CONC" \ | ||
| --result-filename "$RESULT_FILENAME" \ | ||
| --result-dir /workspace/ \ | ||
| --use-chat-template | ||
|
|
||
| if [[ "${RUN_EVAL}" == "true" ]]; then | ||
| run_eval --framework lm-eval --port "$PORT" | ||
| append_lm_eval_summary | ||
| fi | ||
|
|
||
| stop_gpu_monitor | ||
| set +x |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 The new config sets
model-prefix: glm5for modelzai-org/GLM-5.1-FP8, but every other GLM-5.1 config in the repo usesglm5.1— GLM-5 and GLM-5.1 are distinct models. Because the deprecatedglm5-fp8-b300-sglangconfig for the unrelatedzai-org/GLM-5-FP8model used the identical prefixglm5with the same runner/tp/ep/isl/osl/conc,utils/compare_results.py's baseline lookup (a prefixLIKEmatch) will silently pick historical GLM-5 rows as the regression baseline for this GLM-5.1 config. Should bemodel-prefix: glm5.1.Extended reasoning...
The bug:
configs/nvidia-master.yamlsetsmodel: zai-org/GLM-5.1-FP8butmodel-prefix: glm5(line ~1291). Every other GLM-5.1-FP8/GLM-5.1-NVFP4 entry in the repo (e.g.configs/deprecated/nvidia-glm5-glm5.1-master.yamllines 650, 957, 1261, 1382, 1525, 1782) usesmodel-prefix: glm5.1, while GLM-5 (non-5.1) entries useglm5. This new config breaks that convention, using the GLM-5 prefix for a GLM-5.1 model.Code path: The
model-prefixYAML field flows throughgenerate_sweep_configs.pyinto theMODEL_PREFIXenv var, then intoinfmax_model_prefixin the result JSON written bybenchmark_lib.sh.utils/compare_results.pyreads this value (model = result['infmax_model_prefix'].lower()) and uses it in its regression-baseline query:With
model = 'glm5', this becomesLIKE 'glm5%', which matches BOTH'glm5'and'glm5.1'rows in the DB, since'glm5'is a literal prefix of'glm5.1'.Why this causes contamination, concretely: The deprecated config key
glm5-fp8-b300-sglangpreviously existed inconfigs/deprecated/nvidia-glm5-glm5.1-master.yaml(lines 40-53) for the distinct modelzai-org/GLM-5-FP8, also withmodel-prefix: glm5, and identical hardware/framework/precision/tp/ep/isl/osl/conc (b300/sglang/fp8/tp8/ep1/isl8192/osl1024/conc 4-256). Every other column inBASELINE_QUERY's WHERE clause matches between the old GLM-5 rows and this new GLM-5.1 config. So:c.model = 'glm5'(frominfmax_model_prefix).compare_results.pyruns its baseline query withmodel = 'glm5', i.e.LIKE 'glm5%'.c.model = 'glm5'from the old GLM-5 (not 5.1) config, it satisfies every filter andORDER BY date DESC LIMIT 1picks it as 'the' baseline.If
model-prefixwere correctly set toglm5.1, the query would beLIKE 'glm5.1%', which does not match plain'glm5'rows (since'glm5'does not start with'glm5.1'), correctly isolating GLM-5.1 baselines from GLM-5 ones.Why nothing else catches this: The WHERE clause intentionally uses a prefix
LIKEmatch (per the comment aboveBASELINE_QUERY) to handle cases likemodel-prefix: gptossmatching DB modelgptoss120b— a legitimate use of prefix matching for suffix variation. That same permissiveness backfires here becauseglm5is also a valid, distinct model family name that is a literal string-prefix ofglm5.1. There's no additional disambiguation column (e.g. exact model name) in the baseline query to prevent this collision.Fix: change
model-prefix: glm5tomodel-prefix: glm5.1in this new config entry, consistent with every other GLM-5.1 config in the repository. Separately, this also means results from this config would otherwise be permanently mislabeled under theglm5model family in the results DB, independent of the baseline-lookup issue.