[do not merge][testing][chore] update minimaxm3-fp8-b{200,300}/gb{200,300}: bump to latest vLLM image - #2334
[do not merge][testing][chore] update minimaxm3-fp8-b{200,300}/gb{200,300}: bump to latest vLLM image#2334xinli-sw wants to merge 8 commits into
Conversation
xinli-sw
commented
Jul 26, 2026
xinli-sw
commented
- image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 for all seven fp8 NVIDIA (B200, B300, GB200, GB300) configs
- non-eagle configs (b200-vllm, b300-vllm, b300-dynamo-vllm, gb200-dynamo-vllm, gb300-dynamo-vllm): add stream-interval: 32
- eagle configs (b200-vllm-mtp, b300-vllm-mtp): add speculative-config (EAGLE3-GQA, 1 token, FLASH_ATTN) and attention-config (FLASHINFER+TRT-LLM, fp8 kv); H100/H200 configs untouched
… stream-interval/speculative-config - image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 for all seven fp8 NVIDIA (B200, B300, GB200, GB300) configs - non-eagle configs (b200-vllm, b300-vllm, b300-dynamo-vllm, gb200-dynamo-vllm, gb300-dynamo-vllm): add stream-interval: 32 - eagle configs (b200-vllm-mtp, b300-vllm-mtp): add speculative-config (EAGLE3-GQA, 1 token, FLASH_ATTN) and attention-config (FLASHINFER+TRT-LLM, fp8 kv); H100/H200 configs untouched Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Bump container in all 22 disagg recipe files (b300-fp8, gb300-fp8, gb200-fp8 8k1k) and append perf-changelog entry for PR #2334. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # baseline above. The image includes vLLM PR #46380, so no runtime patch is | ||
| # needed. | ||
| minimaxm3-fp8-b300-dynamo-vllm: | ||
| image: vllm/vllm-openai:minimax-m3-0618-x86_64-cu130 | ||
| image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 | ||
| model: MiniMaxAI/MiniMax-M3-MXFP8 | ||
| model-prefix: minimaxm3 | ||
| runner: b300 | ||
| precision: fp8 | ||
| framework: dynamo-vllm | ||
| stream-interval: 32 | ||
| router: { name: dynamo-router, version: "1.3.0.dev20260614" } | ||
| kv-p2p-transfer: nixl | ||
| multinode: true |
There was a problem hiding this comment.
🔴 Adding stream-interval, speculative-config, and attention-config as new top-level keys on these 7 minimaxm3 configs breaks config validation: SingleNodeMasterConfigEntry/MultiNodeMasterConfigEntry in utils/matrix_logic/validation.py use extra='forbid' and don't declare these fields. Running load_config_files(['configs/nvidia-master.yaml']) raises a pydantic extra_forbidden error that aborts generate_sweep_configs.py for the entire master config file, not just minimaxm3 — the schema needs to be extended (or these settings routed through an existing allowed mechanism like search-space/additional-settings) before this can merge.
Extended reasoning...
This PR adds three new top-level YAML keys to 7 minimaxm3 entries in configs/nvidia-master.yaml: stream-interval (5 configs — minimaxm3-fp8-b300-dynamo-vllm, -gb300-dynamo-vllm, -gb200-dynamo-vllm, -b300-vllm, -b200-vllm) and speculative-config + attention-config (the two -mtp eagle configs, -b200-vllm-mtp and -b300-vllm-mtp). None of these keys exist in the Pydantic schema that validates this file.
SingleNodeMasterConfigEntry and MultiNodeMasterConfigEntry (utils/matrix_logic/validation.py:736-787) both declare model_config = ConfigDict(extra='forbid', populate_by_name=True) and only enumerate image, model, model-prefix, precision, framework, runner, multinode, disagg, router, scenarios (plus kv-p2p-transfer for the multinode variant). There is no field for stream-interval, speculative-config, or attention-config anywhere in the schema, and a grep of utils/ confirms these three key names aren't consumed anywhere else in the codebase either — they aren't stripped or handled by some other preprocessing step before validation runs.
I reproduced the crash directly:
$ python3 -c "from utils.matrix_logic.validation import load_config_files; load_config_files(['configs/nvidia-master.yaml'])"
...
pydantic_core._pydantic_core.ValidationError: 1 validation error for MultiNodeMasterConfigEntry
stream-interval
Extra inputs are not permitted [type=extra_forbidden, input_value=32, input_type=int]
...
ValueError: Master config entry 'minimaxm3-fp8-b300-dynamo-vllm' failed validation:
1 validation error for MultiNodeMasterConfigEntry
stream-interval
Extra inputs are not permitted [type=extra_forbidden, input_value=32, input_type=int]
This isn't an opt-in check — load_config_files defaults validate=True and unconditionally calls validate_master_config on the full merged config dict at validation.py:955. generate_sweep_configs.py:1355 calls load_config_files(args.config_files) with no validate=False override, so this is on the hot path of every sweep-config generation run (run-sweep.yml, e2e-tests.yml dispatch) that includes nvidia-master.yaml.
Critically, because validate_master_config validates the entire merged dict up front, the very first offending entry (minimaxm3-fp8-b300-dynamo-vllm) raises and aborts processing for the whole file — every other model/framework/SKU combination defined in nvidia-master.yaml becomes uncomputable via generate_sweep_configs.py until this is fixed, not just the 7 minimaxm3 configs touched by this PR.
To fix: extend the Fields enum and the SingleNodeMasterConfigEntry/MultiNodeMasterConfigEntry models in utils/matrix_logic/validation.py to accept stream-interval, speculative-config, and attention-config (with appropriate types/validation), or route these settings through whatever existing sanctioned mechanism exists for framework-specific passthrough args (e.g. inside search-space/additional-settings if such an escape hatch exists) instead of adding them as bare top-level keys.
| # baseline above. The image includes vLLM PR #46380, so no runtime patch is | ||
| # needed. | ||
| minimaxm3-fp8-b300-dynamo-vllm: | ||
| image: vllm/vllm-openai:minimax-m3-0618-x86_64-cu130 | ||
| image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 | ||
| model: MiniMaxAI/MiniMax-M3-MXFP8 | ||
| model-prefix: minimaxm3 | ||
| runner: b300 | ||
| precision: fp8 | ||
| framework: dynamo-vllm | ||
| stream-interval: 32 | ||
| router: { name: dynamo-router, version: "1.3.0.dev20260614" } | ||
| kv-p2p-transfer: nixl | ||
| multinode: true |
There was a problem hiding this comment.
🔴 This PR bumps the Docker image on 7 minimaxm3-fp8 configs and adds new stream-interval/speculative-config/attention-config knobs, but doesn't append the required entry to perf-changelog.yaml. Per AGENTS.md's 'Updating Docker images' section this entry is 'required - triggers benchmarks', so please add a config-keys/description/pr-link entry for these 7 configs before merge.
Extended reasoning...
The bug: AGENTS.md under "Updating Docker images" (around line 152) states the workflow for bumping an image tag is a two-part checklist: update the tag in configs/*-master.yaml and "append a perf-changelog.yaml entry (required - triggers benchmarks)". This PR only does the first half — it changes the image: field (and adds stream-interval, speculative-config, attention-config) across 7 minimaxm3-fp8-* configs in configs/nvidia-master.yaml, but the PR's diff touches exactly one file. perf-changelog.yaml is untouched.
Why this matters mechanically, not just as a style nit: AGENTS.md also documents (line ~78) that perf-changelog.yaml "triggers which configs to benchmark and is read chronologically," and separately that "Changes to perf-changelog.yaml trigger benchmark runs." This isn't incidental documentation — the repo's own review automation encodes the same rule as a blocking check: .github/workflows/claude-pr-review.yml (~line 122-126) explicitly says if a master config file is edited without a matching perf-changelog.yaml edit in the same PR, the reviewer must comment that it needs to be added, and .github/workflows/claude.yml (line 163) repeats "Modify perf-changelog.yaml for any config changes affecting performance" as an important rule. So this is a documented, tooling-enforced convention, not a matter of taste.
Verification performed: I checked the tail of perf-changelog.yaml — the most recent entry is for PR #2313 (the dsr1-fp8-b300-sglang-mtp SGLang bump), and there's no entry referencing the new image hash 4080263bb2c5d10deac17aaeb88e0823bc35bca9, PR #2334, or any of the 7 minimaxm3-fp8-* config-keys touched here. Comparable recent image-bump PRs in the same file (#2313, #2301) each added a corresponding changelog block, confirming this is the consistently-applied convention for this exact kind of change (image tag bump + new inference flags).
Step-by-step proof:
- The diff shows edits only to
configs/nvidia-master.yaml(<changed-files count="1">in the PR metadata), bumpingimage:onminimaxm3-fp8-b300-dynamo-vllm,-gb300-dynamo-vllm,-gb200-dynamo-vllm,-b300-vllm,-b200-vllm,-b200-vllm-mtp,-b300-vllm-mtptovllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9, plus addingstream-interval: 32(5 configs) andspeculative-config/attention-config(2 MTP configs). - Grepping
perf-changelog.yamlfor the new image hash, for PR [do not merge][testing][chore] update minimaxm3-fp8-b{200,300}/gb{200,300}: bump to latest vLLM image #2334, and for any of these 7 config-keys returns nothing. - The file's chronological tail (newest entry) is the PR dsr1-fp8-b300-sglang-mtp: bump SGLang image to v0.5.15.post1-cu130 #2313 block for
dsr1-fp8-b300-sglang-mtp, confirming no entry was appended after this PR's changes were made. - Per the "read chronologically" / "triggers which configs to benchmark" semantics in AGENTS.md, and the changelog-driven benchmark-sweep tooling, these 7 config changes (image bump + new EAGLE3 spec-decode config on the MTP variants) will not be tracked or triggered via the changelog-driven path without this entry.
The fix: Append a perf-changelog.yaml block (in the existing style, appended at the end of the file) with config-keys listing the 7 affected minimaxm3-fp8-* entries (or a wildcard like minimaxm3-fp8-* if it cleanly covers only these), a description list summarizing the vLLM nightly bump and the new stream-interval/speculative-config/attention-config knobs, and pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2334.
All four independent verifiers confirmed this same finding with no refutations, and cross-referenced both the AGENTS.md documentation and the actual CI review-bot workflow logic that treats this as a blocking check — so I'm reporting it as normal severity: without the entry, the changelog-driven benchmark path won't record or trigger these image/config changes, undermining the stated purpose of the PR (validating the new nightly image and EAGLE3 spec-decode arms).
| # precedent: spec decode pays off at low/mid concurrency while acceptance | ||
| # dilutes in big batches, and the draft weights + draft KV shave headroom. | ||
| minimaxm3-fp8-b200-vllm-mtp: | ||
| image: vllm/vllm-openai:minimax-m3-0618-x86_64-cu130 | ||
| image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 | ||
| model: MiniMaxAI/MiniMax-M3-MXFP8 | ||
| model-prefix: minimaxm3 | ||
| runner: b200-dgxc | ||
| precision: fp8 | ||
| framework: vllm | ||
| multinode: false | ||
| speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":1,"attention_backend":"FLASH_ATTN"}' | ||
| attention-config: '{"backend": "FLASHINFER", "use_trtllm_attention": true, "indexer_kv_dtype": "fp8"}' | ||
| scenarios: | ||
| fixed-seq-len: | ||
| - isl: 8192 |
There was a problem hiding this comment.
🟡 The comments above minimaxm3-fp8-b200-vllm-mtp and minimaxm3-fp8-b300-vllm-mtp still describe the drafter as plain EAGLE3 with 3 speculative tokens (and TRITON_ATTN for b200), but the speculative-config this PR actually adds uses Inferact/MiniMax-M3-EAGLE3-GQA, num_speculative_tokens: 1, and FLASH_ATTN. Please update both comments to match the shipped values so they don't mislead future readers; this is a docs-only nit with no runtime effect.
Extended reasoning...
What's stale: The doc comments immediately above minimaxm3-fp8-b200-vllm-mtp (around lines 7461-7465, right before the diff hunk at 7466) and above minimaxm3-fp8-b300-vllm-mtp (around lines 7518-7523, right before the hunk at 7524) describe the drafter as "the Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens[, drafter pinned to TRITON_ATTN])". These comments predate this PR and were written when no explicit speculative-config existed on these entries — they were, at the time, the only source of truth for what drafter/config was intended.
What the PR ships that contradicts them: This PR adds a concrete speculative-config to both entries:
speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":1,"attention_backend":"FLASH_ATTN"}'
Comparing this to the adjacent prose, there are three separate factual mismatches: (1) the model is Inferact/MiniMax-M3-EAGLE3-GQA, not plain Inferact/MiniMax-M3-EAGLE3 as the comment states; (2) num_speculative_tokens is 1, not the 3 the comment claims; (3) on the b200 entry, attention_backend is FLASH_ATTN, not TRITON_ATTN as the comment says.
Why this wasn't caught: The comments are free-form prose, not validated against the YAML values, so nothing enforces consistency between them and the config block they describe. Since this PR is what actually materializes the concrete speculative-decoding config for the first time on these entries, it's the PR's diff that creates the contradiction (previously there was no numeric config to disagree with).
Impact: This is documentation-only — the YAML values themselves are what vLLM will actually use at runtime, so nothing breaks operationally. The risk is purely that future readers/maintainers (including whoever tunes speculative decoding next) will trust the stale comment over the shipped config and draw wrong conclusions about the drafter model, token count, or attention backend, e.g. assuming 3-token speculation when only 1 token is configured.
Proof walkthrough:
- Open
configs/nvidia-master.yamlat theminimaxm3-fp8-b200-vllm-mtpentry. - Read the comment directly above it: "...Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens, drafter pinned to TRITON_ATTN)."
- Read the
speculative-configline this PR adds a few lines below:model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":1,"attention_backend":"FLASH_ATTN". - Compare: EAGLE3 vs EAGLE3-GQA, 3 tokens vs 1, TRITON_ATTN vs FLASH_ATTN — all three disagree.
- The same token-count mismatch (3 vs 1) applies to the
minimaxm3-fp8-b300-vllm-mtpcomment/config pair.
Fix: Update both comments to say "Inferact/MiniMax-M3-EAGLE3-GQA draft head (1 speculative token, FLASH_ATTN)" (or otherwise match whatever final values ship) so the prose and the config agree.
…om nvidia-master.yaml These belong in the benchmark shell scripts and srt-slurm recipe files, not as top-level master config fields. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30188929534 |
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30188965211 |
…aft, 1 tok, FLASH_ATTN - b200/b300 non-eagle: --stream-interval 20 -> 32 - b200/b300 mtp (eagle): draft Inferact/MiniMax-M3-EAGLE3-GQA, num_speculative_tokens 1, speculative attention_backend FLASH_ATTN, consolidated attention-config with indexer_kv_dtype fp8, --stream-interval 20 -> 32 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30188998025 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30189203817 |
…mage ships the fixes The new nightly vLLM image includes the MSA contiguity, NIXL heterogeneous-TP, and flashinfer_all_reduce fixes upstream, so the workaround patches are dead: - remove the inline sparse_attention_msa contiguity patch from the four single-node scripts (b200, b300, b200_mtp, b300_mtp) - delete the three dynamo setup scripts (minimax-m3-vllm-fixes.sh, minimax-m3-gb200-vllm-fixes.sh, minimax-m3-gb300-vllm-fixes.sh) and drop the SRTCTL_SETUP_SCRIPT wiring from launch_b300-nv/gb200-nv/gb300-nv Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30190326220 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30190326220 |
xinli-sw
changed the title
1 participant
