Skip to content

[NV] minimax-m3-b200/b300-mtp: update fp8 MTP to vLLM nightly / [NV] minimax-m3-b200/b300-mtp：将 FP8 MTP 更新至 vLLM nightly #2337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xinli-sw wants to merge 4 commits into
base: main
Choose a base branch
from
+24 −86
Open

[NV] minimax-m3-b200/b300-mtp: update fp8 MTP to vLLM nightly / [NV] minimax-m3-b200/b300-mtp：将 FP8 MTP 更新至 vLLM nightly #2337

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7148fb9
[NV] minimax-m3-b200/b300-mtp: vLLM nightly image, GQA draft head, FL…
xinli-sw Jul 26, 2026
1274cd2
[NV] minimax-m3-b200/b300-mtp: add perf-changelog pr-link
xinli-sw Jul 26, 2026
924e327
Merge remote-tracking branch 'origin/main' into minimaxm3-fp8-b200-b3…
xinli-sw Jul 26, 2026
315b9a0
[NV] minimax-m3-b200/b300-mtp: drop tp4-ep4-DEP (dp-attn) arm from th…
xinli-sw Jul 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 7 additions & 41 deletions benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b200_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@
# MiniMax-M3 MXFP8 B200 single-node vLLM recipe with EAGLE3 speculative
# decoding — the repo's spec-decoding=mtp variant of minimaxm3_fp8_b200.sh
# (https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3). Adds the
# Inferact/MiniMax-M3-EAGLE3 draft head via --speculative-config with 3
# Inferact/MiniMax-M3-EAGLE3-GQA draft head via --speculative-config with 3
# speculative tokens. Everything else keeps the non-MTP serve shape:
# --block-size 128 is mandatory (MSA sparse_block_size is 128; the default 16
# misaligns sparse indexing), and --language-model-only skips the vision
# encoder for the text-only benchmark. dp-attn=true maps to DP×EP (DEP);
# ep>1 maps to TP+EP (TEP).
#
# The target uses the FlashInfer TRT-LLM attention path. The EAGLE3 drafter is
# pinned separately to TRITON_ATTN.
# The target uses the FlashInfer TRT-LLM attention path. The EAGLE3-GQA drafter
# is pinned separately to FLASH_ATTN.

source "$(dirname "$0")/../../benchmark_lib.sh"

Expand All @@ -27,40 +27,7 @@ check_env_vars \
RANDOM_RANGE_RATIO \
RESULT_FILENAME

# The 0618 image keeps MiniMax M3 top-k indices in a persistent
# [head_kv, max_tokens, topK] buffer for CUDA graphs. Slicing that buffer to
# the actual prefill length is non-contiguous when TP leaves multiple local KV
# heads, and the MSA CSR builder rejects it. Materialize the slice until the
# image includes this fix.
python3 - <<'PYEOF' || { echo "MiniMax M3 MSA contiguity patch failed" >&2; exit 1; }
import importlib.util
import pathlib

spec = importlib.util.find_spec("vllm")
if spec is None or not spec.submodule_search_locations:
raise RuntimeError("Could not locate the installed vllm package")

target = (
pathlib.Path(next(iter(spec.submodule_search_locations)))
/ "models"
/ "minimax_m3"
/ "nvidia"
/ "sparse_attention_msa.py"
)
src = target.read_text()
old = " prefill_topk = topk[:, nd:num_tokens, :]\n"
new = " prefill_topk = topk[:, nd:num_tokens, :].contiguous()\n"

if new in src:
print(f"[minimax-m3-msa-patch] already applied: {target}")
elif src.count(old) == 1:
target.write_text(src.replace(old, new, 1))
print(f"[minimax-m3-msa-patch] patched: {target}")
else:
raise RuntimeError(f"Expected exactly one patch anchor in {target}")
PYEOF

DRAFT_MODEL="Inferact/MiniMax-M3-EAGLE3"
DRAFT_MODEL="Inferact/MiniMax-M3-EAGLE3-GQA"

if [[ -n "$SLURM_JOB_ID" ]]; then
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
Expand Down Expand Up @@ -116,14 +83,13 @@ $PARALLEL_ARGS \
--gpu-memory-utilization 0.90 \
--max-model-len $MAX_MODEL_LEN \
--block-size 128 \
--attention-config '{"backend": "FLASHINFER", "use_trtllm_attention": true}' \
--attention-config.indexer_kv_dtype "fp8" \
--attention-config '{"backend": "FLASHINFER", "use_trtllm_attention": true, "indexer_kv_dtype": "fp8"}' \
--kv-cache-dtype fp8 \
--language-model-only \
--max-cudagraph-capture-size 2048 \
--max-num-batched-tokens "$((ISL * 2 ))" \
--speculative-config "{\"method\": \"eagle3\", \"model\": \"$DRAFT_MODEL_PATH\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"attention_backend\": \"TRITON_ATTN\"}" \
--stream-interval 20 --no-enable-prefix-caching \
--speculative-config "{\"method\": \"eagle3\", \"model\": \"$DRAFT_MODEL_PATH\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"attention_backend\": \"FLASH_ATTN\"}" \
--stream-interval 32 --no-enable-prefix-caching \
--trust-remote-code > $SERVER_LOG 2>&1 &

SERVER_PID=$!
Expand Down
48 changes: 7 additions & 41 deletions benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b300_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@
# MiniMax-M3 MXFP8 B300 single-node vLLM recipe with EAGLE3 speculative
# decoding — the repo's spec-decoding=mtp variant of minimaxm3_fp8_b300.sh
# (https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3). Adds the
# Inferact/MiniMax-M3-EAGLE3 draft head via --speculative-config with 3
# Inferact/MiniMax-M3-EAGLE3-GQA draft head via --speculative-config with 3
# speculative tokens. Everything else keeps the non-MTP serve shape:
# --block-size 128 is mandatory (MSA sparse/index cache); the benchmark is
# text-only, so --language-model-only frees the vision encoder's VRAM.
#
# The target uses the FlashInfer TRT-LLM attention path. The EAGLE3 drafter is
# pinned separately to TRITON_ATTN.
# The target uses the FlashInfer TRT-LLM attention path. The EAGLE3-GQA drafter
# is pinned separately to FLASH_ATTN.

source "$(dirname "$0")/../../benchmark_lib.sh"

Expand All @@ -25,40 +25,7 @@ check_env_vars \
RANDOM_RANGE_RATIO \
RESULT_FILENAME

# The 0618 image keeps MiniMax M3 top-k indices in a persistent
# [head_kv, max_tokens, topK] buffer for CUDA graphs. Slicing that buffer to
# the actual prefill length is non-contiguous when TP leaves multiple local KV
# heads, and the MSA CSR builder rejects it. Materialize the slice until the
# image includes this fix.
python3 - <<'PYEOF' || { echo "MiniMax M3 MSA contiguity patch failed" >&2; exit 1; }
import importlib.util
import pathlib

spec = importlib.util.find_spec("vllm")
if spec is None or not spec.submodule_search_locations:
raise RuntimeError("Could not locate the installed vllm package")

target = (
pathlib.Path(next(iter(spec.submodule_search_locations)))
/ "models"
/ "minimax_m3"
/ "nvidia"
/ "sparse_attention_msa.py"
)
src = target.read_text()
old = " prefill_topk = topk[:, nd:num_tokens, :]\n"
new = " prefill_topk = topk[:, nd:num_tokens, :].contiguous()\n"

if new in src:
print(f"[minimax-m3-msa-patch] already applied: {target}")
elif src.count(old) == 1:
target.write_text(src.replace(old, new, 1))
print(f"[minimax-m3-msa-patch] patched: {target}")
else:
raise RuntimeError(f"Expected exactly one patch anchor in {target}")
PYEOF

DRAFT_MODEL="Inferact/MiniMax-M3-EAGLE3"
DRAFT_MODEL="Inferact/MiniMax-M3-EAGLE3-GQA"

# `hf download` creates the target dir if missing and is itself idempotent.
# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE.
Expand Down Expand Up @@ -118,14 +85,13 @@ $PARALLEL_ARGS \
--gpu-memory-utilization 0.90 \
--max-model-len $MAX_MODEL_LEN \
--block-size 128 \
--attention-config '{"backend": "FLASHINFER", "use_trtllm_attention": true}' \
--attention-config.indexer_kv_dtype "fp8" \
--attention-config '{"backend": "FLASHINFER", "use_trtllm_attention": true, "indexer_kv_dtype": "fp8"}' \
--kv-cache-dtype fp8 \
--language-model-only \
--max-cudagraph-capture-size 2048 \
--max-num-batched-tokens "$((ISL * 2 ))" \
--speculative-config "{\"method\": \"eagle3\", \"model\": \"$DRAFT_MODEL_PATH\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"attention_backend\": \"TRITON_ATTN\"}" \
--stream-interval 20 --no-enable-prefix-caching \
--speculative-config "{\"method\": \"eagle3\", \"model\": \"$DRAFT_MODEL_PATH\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"attention_backend\": \"FLASH_ATTN\"}" \
--stream-interval 32 --no-enable-prefix-caching \
--trust-remote-code > $SERVER_LOG 2>&1 &

SERVER_PID=$!
Expand Down
6 changes: 2 additions & 4 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7478,7 +7478,7 @@ minimaxm3-fp4-b200-vllm:
# precedent: spec decode pays off at low/mid concurrency while acceptance
# dilutes in big batches, and the draft weights + draft KV shave headroom.
minimaxm3-fp8-b200-vllm-mtp:
image: vllm/vllm-openai:minimax-m3-0618-x86_64-cu130
image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9
model: MiniMaxAI/MiniMax-M3-MXFP8
model-prefix: minimaxm3
runner: b200-dgxc
Comment on lines 7478 to 7484

@claude claude Bot Jul 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The config comments above minimaxm3-fp8-b200-vllm-mtp (~line 7458-7459) and minimaxm3-fp8-b300-vllm-mtp (~line 7513) still say the drafter is 'Inferact/MiniMax-M3-EAGLE3' pinned to 'TRITON_ATTN', but this PR switches both recipes to 'Inferact/MiniMax-M3-EAGLE3-GQA' with 'FLASH_ATTN'. Please update these two comment blocks to match, similar to how the h200/h100 mtp entries already describe the FLASH_ATTN/GQA rationale.

Extended reasoning...

configs/nvidia-master.yaml contains a hand-written comment block directly above each minimaxm3-fp8-*-vllm-mtp config entry explaining the EAGLE3 speculative-decoding setup for that recipe. For minimaxm3-fp8-b200-vllm-mtp the comment (around lines 7456-7459) reads: "...adds the Inferact/MiniMax-M3-EAGLE3 draft head... The target uses the FlashInfer TRT-LLM attention path. The EAGLE3 drafter is pinned separately to TRITON_ATTN." For minimaxm3-fp8-b300-vllm-mtp the comment (around lines 7511-7513) similarly references the plain Inferact/MiniMax-M3-EAGLE3 draft head with no attention-backend caveat.

This PR changes exactly what those comments describe: the draft head becomes Inferact/MiniMax-M3-EAGLE3-GQA and the speculative attention_backend moves from TRITON_ATTN to FLASH_ATTN (visible in the matching .sh recipe diffs, where the header comments were correctly updated: "Inferact/MiniMax-M3-EAGLE3-GQA draft head" and "The EAGLE3-GQA drafter is pinned separately to FLASH_ATTN."). However, the diff hunk touching configs/nvidia-master.yaml only replaces the image: line inside each block (lines 7461-7467 for b200, 7517-7523 for b300) — the descriptive comment lines sitting just above those keys were left untouched.

The result is that after this PR merges, the YAML comments for the b200 and b300 mtp entries misdescribe the actual recipe: they name the wrong draft-head repo (missing "-GQA") and, for b200, explicitly claim the drafter is "pinned to TRITON_ATTN" when it is now pinned to FLASH_ATTN. This is a real inconsistency the PR introduces/worsens, not a pre-existing one, since the PR's own script changes are what make the comments wrong. Contrast this with the sibling h200/h100 mtp entries further down in the same file (~lines 7539-7546 and 7565-7570), which already correctly explain the FLASH_ATTN/GQA rationale — those are the template this PR's two comment blocks should be brought in line with.

Step-by-step proof: (1) Before this PR, b200 mtp comment says draft head = Inferact/MiniMax-M3-EAGLE3, backend = TRITON_ATTN, matching the old script's DRAFT_MODEL="Inferact/MiniMax-M3-EAGLE3" and "attention_backend": "TRITON_ATTN". (2) This PR's diff to minimaxm3_fp8_b200_mtp.sh changes DRAFT_MODEL to "Inferact/MiniMax-M3-EAGLE3-GQA" and attention_backend to "FLASH_ATTN", and updates that script's own header comment to match. (3) The configs/nvidia-master.yaml diff for the same recipe only touches the image: line — the comment block above it is byte-for-byte identical to before. (4) Therefore post-merge, the YAML comment and the actual recipe behavior diverge: reading the config comment alone, a maintainer would believe the drafter still uses TRITON_ATTN with the non-GQA head, which is now false. The same divergence applies to the b300 comment (missing "-GQA" and no FLASH_ATTN mention).

This is comment-only and has zero effect on runtime behavior — the scripts themselves (the actual source of truth for how the benchmark runs) are correct. So while it's a genuine, PR-introduced inconsistency worth fixing to keep the config self-documenting, it doesn't block anything at merge. Fix: update the two comment blocks in configs/nvidia-master.yaml to say "Inferact/MiniMax-M3-EAGLE3-GQA draft head" and "pinned separately to FLASH_ATTN" (b200) / add the same GQA/FLASH_ATTN mention (b300), mirroring the phrasing already used in the h200/h100 mtp comments.

Expand All @@ -7494,7 +7494,6 @@ minimaxm3-fp8-b200-vllm-mtp:
- { tp: 8, ep: 8, conc-start: 1, conc-end: 256, spec-decoding: mtp }
- { tp: 4, conc-start: 1, conc-end: 64, spec-decoding: mtp }
- { tp: 4, ep: 4, conc-start: 64, conc-end: 256, spec-decoding: mtp }
- { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 128, spec-decoding: mtp }
- { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 256, spec-decoding: mtp }

# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of MiniMax-M3 NVFP4
Expand Down Expand Up @@ -7534,7 +7533,7 @@ minimaxm3-fp4-b200-vllm-mtp:
# big batches, and the draft weights + draft KV shave headroom — tp2-ep2 is
# dropped entirely since its KV headroom was already thin without a draft.
minimaxm3-fp8-b300-vllm-mtp:
image: vllm/vllm-openai:minimax-m3-0618-x86_64-cu130
image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9
model: MiniMaxAI/MiniMax-M3-MXFP8
model-prefix: minimaxm3
runner: b300
Expand All @@ -7550,7 +7549,6 @@ minimaxm3-fp8-b300-vllm-mtp:
- { tp: 8, ep: 8, conc-start: 1, conc-end: 256, spec-decoding: mtp }
- { tp: 4, conc-start: 1, conc-end: 64, spec-decoding: mtp }
- { tp: 4, ep: 4, conc-start: 64, conc-end: 256, spec-decoding: mtp }
- { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 128, spec-decoding: mtp }
- { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 256, spec-decoding: mtp }

# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of
Expand Down
8 changes: 8 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5085,3 +5085,11 @@
description:
- "Bump SGLang container image from lmsysorg/sglang:v0.5.12-cu130 to lmsysorg/sglang:v0.5.15.post1-cu130 (https://github.com/sgl-project/sglang/releases/tag/v0.5.15.post1)"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2313

- config-keys:
- minimaxm3-fp8-b200-vllm-mtp
- minimaxm3-fp8-b300-vllm-mtp
description:
- "Bump image to vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9; set --stream-interval 32; consolidate --attention-config with indexer_kv_dtype fp8; drop the sparse_attention_msa contiguity patch (shipped in the nightly image)"
Comment on lines +5088 to +5093

@claude claude Bot Jul 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 This PR's title and description are English-only, but AGENTS.md mandates bilingual PR titles/descriptions (title format <English title> / <中文标题> and a mirroring ## 中文说明 body section) for every PR. Please add the Chinese title suffix and a Chinese translation section before merging.

Extended reasoning...

AGENTS.md line 7 states, as a top-level mandatory rule (not subjective style): "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: <English title> / <中文标题>. In the PR/issue body, follow the English content with its Chinese translation (e.g. a ## 中文说明 section mirroring the summary...). This applies to every PR and every issue." CLAUDE.md forwards all repo guidance to AGENTS.md, so this rule governs review of this PR.

The violation: The PR title is [NV] minimax-m3-b200/b300-mtp: update fp8 MTP to vLLM nightly — English only, with no / <中文标题> suffix as the mandated format requires. The PR description's Summary section (bump image, EAGLE3-GQA draft head, FLASH_ATTN backend, consolidated --attention-config, dropped contiguity patch) has no mirroring ## 中文说明 section translating those same points into Simplified Chinese.

Why this isn't caught elsewhere: The only automated enforcement visible in the PR timeline is the codeowner-signoff-verify bot, which checks for the English sign-off phrase As a PR reviewer and CODEOWNER, I have reviewed this and have — it does not check PR title/body bilinguality. AGENTS.md's own exception list (CODEOWNER sign-off template stays English-verbatim; bot-generated comments follow their own templates) does not cover the human-authored title/description, so no carve-out applies here.

Step-by-step proof:

  1. Read AGENTS.md line 7: rule requires <English title> / <中文标题> format and a ## 中文说明 mirror in the body, applying to "every PR."
  2. Read the PR metadata: title = [NV] minimax-m3-b200/b300-mtp: update fp8 MTP to vLLM nightly — no / separator, no Chinese characters anywhere.
  3. Read the PR description: only a ## Summary section in English with five bullet points; no ## 中文说明 or any Chinese text follows it.
  4. Cross-check the exceptions in AGENTS.md (CODEOWNER sign-off verbatim text, bot templates) — neither applies to this human-authored title/body — so the rule is violated with no exemption.

Impact and fix: This is a metadata/process gap, not a code defect — nothing breaks at runtime, no benchmark or config behavior is affected, and the fix is simply editing the PR title and appending a ## 中文说明 section (following the translation quality bar in AGENTS.md, e.g. matching vllm-ascend's README.zh.md style) — no code changes or re-review of the diff are required. Given that, while it's a genuine, explicit repo requirement worth flagging, it doesn't rise to a blocking correctness issue.

- "EAGLE3 draft head Inferact/MiniMax-M3-EAGLE3 -> Inferact/MiniMax-M3-EAGLE3-GQA; speculative attention_backend TRITON_ATTN -> FLASH_ATTN; num_speculative_tokens kept at 3"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2337
Loading