feat: add MiniMax-M3 RTX 6000 Pro vLLM recipe / 新增 MiniMax-M3 RTX 6000 Pro vLLM 基准配置#2306
feat: add MiniMax-M3 RTX 6000 Pro vLLM recipe / 新增 MiniMax-M3 RTX 6000 Pro vLLM 基准配置#2306Oseltamivir wants to merge 12 commits into
Conversation
Register the Latitude RTX 6000 Pro runner and add a TP8 NVFP4 vLLM sweep config, Docker launcher, and benchmark recipe. 中文：注册 Latitude RTX 6000 Pro 运行器，并新增 TP8 NVFP4 vLLM 扫描配置、Docker 启动器和基准测试配方。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Record the final pull request URL for the RTX 6000 Pro MiniMax-M3 benchmark entry. 中文：为 RTX 6000 Pro MiniMax-M3 基准测试条目记录最终的拉取请求链接。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29982947239 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — straightforward new-hardware bring-up recipe (RTX PRO 6000) following the established single-node vLLM recipe pattern.
What was reviewed: the new benchmark script against its B200 NVFP4 counterpart (same vLLM flags, NCCL all-reduce swapped in for the PCIe topology as the comment explains), the new docker-based launch_rtx6000pro-lat.sh against the existing launch_h100-cr.sh docker-run launcher pattern (adds GPU_COUNT validation and a stale-container cleanup trap not present in the older script), and the runners.yaml/nvidia-master.yaml additions for consistency with existing cluster entries. The scenario is scoped to a single TP8/conc=1 canary, consistent with a first bring-up.
Extended reasoning...
This PR adds a new hardware target (8x RTX PRO 6000 Blackwell) for the MiniMax-M3 NVFP4 vLLM recipe: a new benchmark script, a new Docker-based runner launcher, and config/changelog entries wiring it into the sweep matrix.
Security risks
None identified. The new launch_rtx6000pro-lat.sh runs docker run --privileged with a broad --env allowlist, but this mirrors the existing launch_h100-cr.sh and launch_mi325x-tw.sh docker-based launchers already in the repo — same trust model (CI-controlled env vars, no user-controlled input reaching the container). The image is pinned by sha256 digest.
Level of scrutiny
Low-to-moderate. This is infra/config for an internal benchmark harness, not production-serving code, and it closely follows established patterns rather than introducing new mechanisms. I compared the new benchmark script line-by-line against minimaxm3_fp4_b200.sh and the new launcher against launch_h100-cr.sh; both are faithful adaptations with sensible topology-specific changes (NCCL all-reduce instead of FlashInfer/TRT-LLM, since RTX PRO 6000 lacks NVLink).
Other factors
The bug-hunting pass found nothing, and the two candidate issues it ruled out (placeholder-looking PR link, unverified image digest) are both non-issues on inspection — the PR link correctly points to #2306 and the digest pin is the same pattern used elsewhere in nvidia-master.yaml. The new runner launcher adds defensive touches (GPU_COUNT integer validation, stale-container cleanup trap) beyond what older docker launchers have, which is a plus. Scope is limited to a single canary scenario (TP8, concurrency=1), appropriate for initial hardware bring-up.
Avoid the NCCL 2.28.9 bnxt_re initialization segfault by defaulting this single-node runner to non-IB transport. 中文：默认禁用该单节点运行器的 NCCL IB 传输，避免 NCCL 2.28.9 初始化 bnxt_re 时发生段错误。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29984051893 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29984084429 |
Force the non-FlashInfer Marlin MoE backend, preserve MiniMax-M3 SwiGLU parameters with a documented compatibility patch, and use Triton attention for block size 128. 中文：在 SM120 上强制使用非 FlashInfer 的 Marlin 混合专家后端，通过已记录豁免的兼容补丁保留 MiniMax-M3 的 SwiGLU 参数，并为 128 块大小使用 Triton 注意力后端。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29987453043 |
Run the 8k/1k recipe at concurrency 1, 4, 16, and 64 for both execution modes.\n\n为两种执行模式添加 8k/1k 并发 1、4、16 和 64 的扫描。
Note that the Marlin DEP8 arm must keep the standard all-gather/reduce-scatter activation layout. 说明 Marlin DEP8 配置必须保留标准的全收集与归约分散激活布局。
| VLLM_SITE_PACKAGES="$( | ||
| python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])' | ||
| )" | ||
| VLLM_MARLIN_PATCH=/workspace/benchmarks/patches/vllm/minimax_m3_nvfp4_marlin.patch | ||
| if ! patch --batch --forward --fuzz=0 -p1 -d "$VLLM_SITE_PACKAGES" \ | ||
| < "$VLLM_MARLIN_PATCH"; then | ||
| echo "Failed to apply the pinned MiniMax-M3 Marlin compatibility patch" >&2 | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
🟡 The Marlin compatibility patch step (lines 61-69) hard-aborts if patch --forward returns nonzero, but --forward does not make re-applying an already-applied patch a no-op — GNU patch exits 1 with "Reversed (or previously applied) patch detected!" in that case too. This breaks re-running the script against an already-patched vLLM install (e.g. the repo's own debug-runs fast-iteration workflow of SSHing onto the runner and re-running in the same container), and will also bite if the pinned image ever upstreams just one of the two patch hunks. The sibling minimaxm3_fp8_b200.sh recipe avoids this by explicitly detecting the already-applied state before failing.
Extended reasoning...
The bug: minimaxm3_fp4_rtx6000pro.sh lines 65-69 run:
if ! patch --batch --forward --fuzz=0 -p1 -d "$VLLM_SITE_PACKAGES" < "$VLLM_MARLIN_PATCH"; then
echo "Failed to apply the pinned MiniMax-M3 Marlin compatibility patch" >&2
exit 1
fiThe intent of --forward (GNU patch's -N) is presumably to make the script idempotent — i.e., skip hunks that are already applied so re-running the script against a previously-patched vLLM install doesn't fail. That's not what it does. --forward only suppresses applying a patch that would go backwards relative to already-modified source; it does not exit 0 when a hunk is already present. On a second apply of the identical patch, GNU patch prints Reversed (or previously applied) patch detected! Skipping patch., writes a .rej file, and exits 1 — a status this script treats identically to a genuinely broken/mismatched patch.
Reachability: normal CI is unaffected, since runners/launch_rtx6000pro-lat.sh uses docker run --rm, giving a fresh, unpatched container on every invocation. The failure mode is triggered by this repo's own debug-runs skill, which instructs engineers to SSH onto the runner and re-run the benchmark script directly in the same live container (via docker exec/enroot) for fast iteration on things like CONC. The second invocation in that same container hits an already-patched vLLM tree and hard-fails at the patch step before vllm serve even starts, defeating the fast-iteration workflow the skill exists for. It would also resurface once the pinned image upstreams just one of the two patch hunks (per the removal plan in docs/waiver/2306.md) before the patch file is deleted — a partially-applied multi-hunk patch still returns nonzero overall even though the still-needed hunk would apply cleanly.
Step-by-step proof:
- First run: container has an unpatched vLLM install.
patch --batch --forward --fuzz=0 -p1 -d $VLLM_SITE_PACKAGES < minimax_m3_nvfp4_marlin.patchapplies both hunks cleanly, exits 0.vllm servestarts normally. - Engineer SSHes onto the runner mid-debug, re-runs the same script in the same container to tweak
CONC. patchis invoked again against the now-already-patchednvfp4.pyandmarlin_moe.py. GNU patch detects both hunks are already present, prints "Reversed (or previously applied) patch detected! Skipping patch.", writes.rej, and exits 1.- The script's
if ! patch ...; then echo ...; exit 1; fitreats this exit code as failure and aborts beforevllm serveis ever invoked — even though the vLLM install is in the exact state the script wants it in.
Why nothing else prevents this: --fuzz=0 and -p1 only control hunk-matching leniency and path stripping; neither affects idempotency. There's no pre-check for an already-patched state, unlike the sibling minimaxm3_fp8_b200.sh recipe (lines ~30-55), which runs a small Python idempotent-patch routine that explicitly prints [minimax-m3-msa-patch] already applied and treats that as success rather than failure — i.e., idempotency is a known, established concern for these runtime-patch recipes that this new recipe doesn't carry over.
Suggested fix: mirror the b200 recipe's approach (an idempotent Python-based patch application, or a pre-check such as patch --dry-run -R to detect 'already applied' and treat it as success rather than failure) so re-running the script against an already-patched install is a no-op instead of a hard failure.
Severity: all three independent verifiers empirically confirmed the behavior with GNU patch and agreed on nit — this is a real, reproducible bug, but normal CI (fresh container per run) is unaffected; it only degrades the manual/debug re-run path and the future partial-upstream scenario, not the sweep this PR is being merged for.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29991031047 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30005703519 |
| --- a/vllm/model_executor/layers/fused_moe/oracle/nvfp4.py | ||
| +++ b/vllm/model_executor/layers/fused_moe/oracle/nvfp4.py | ||
| @@ -176,2 +176,3 @@ | ||
| NvFp4MoeBackend.FLASHINFER_TRTLLM, |
There was a problem hiding this comment.
maybe upstream if sm120 then marlin else flashinfer?
Add the TP4 and TP4/EP4 MTP sweep with the external EAGLE3 draft, SM120-safe Triton attention, concurrency-sized decode graphs, and chat-formatted prompts. Keep the required Marlin compatibility patch and document its expanded scope. 中文：新增使用外部 EAGLE3 草稿模型的 TP4 与 TP4/EP4 MTP 扫描，采用兼容 SM120 的 Triton attention、按并发数调整的解码 CUDA Graph，以及聊天模板格式的提示词。保留必需的 Marlin 兼容补丁并扩展其豁免文档范围。
Warm the Hugging Face cache without capturing the formatted hf CLI output, then pass stable target and draft identifiers to vLLM. 中文：预热 Hugging Face 缓存时不再捕获 hf CLI 的格式化输出，并将稳定的目标模型与草稿模型标识传给 vLLM。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30066828667 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30066976722 |
中文：将 origin/main 合并到 MiniMax-M3 RTX 分支。
8ba9324 to
6ba91f8
Compare
2 participants
No description provided.