Skip to content

feat: add MiniMax-M3 RTX 6000 Pro vLLM recipe / 新增 MiniMax-M3 RTX 6000 Pro vLLM 基准配置 #2306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Oseltamivir wants to merge 12 commits into
base: main
Choose a base branch
from
+570 −0
Open

feat: add MiniMax-M3 RTX 6000 Pro vLLM recipe / 新增 MiniMax-M3 RTX 6000 Pro vLLM 基准配置 #2306

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
26e81c7
feat: add MiniMax-M3 RTX 6000 Pro vLLM recipe
Oseltamivir Jul 23, 2026
53dd009
chore: link performance changelog to PR 2306
Oseltamivir Jul 23, 2026
3de17dd
fix: disable NCCL IB on RTX 6000 Pro runner
Oseltamivir Jul 23, 2026
63be10c
Merge branch 'main' into agent/minimaxm3-rtx6000pro-vllm
Oseltamivir Jul 23, 2026
8ba857b
fix: use Marlin for MiniMax-M3 on SM120
Oseltamivir Jul 23, 2026
9630a91
feat: sweep MiniMax-M3 TP8 and DEP8 / 扩展 MiniMax-M3 TP8 与 DEP8 扫描
Oseltamivir Jul 23, 2026
c57e7de
docs: document DEP8 exchange path / 记录 DEP8 通信路径
Oseltamivir Jul 23, 2026
702dfcc
feat: switch MiniMax-M3 RTX sweep to TP4 and EP4
Oseltamivir Jul 23, 2026
58a9728
Merge remote-tracking branch 'origin/main' into agent/minimaxm3-rtx60…
Oseltamivir Jul 24, 2026
1fcd2e1
feat: add MiniMax-M3 RTX EAGLE3 sweep
Oseltamivir Jul 24, 2026
e817b25
fix: pass stable model IDs to MiniMax MTP
Oseltamivir Jul 24, 2026
6ba91f8
Merge origin/main into MiniMax-M3 RTX branch
Oseltamivir Jul 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions benchmarks/patches/vllm/minimax_m3_nvfp4_marlin.patch
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
--- a/vllm/model_executor/layers/fused_moe/oracle/nvfp4.py
+++ b/vllm/model_executor/layers/fused_moe/oracle/nvfp4.py
@@ -176,2 +176,3 @@
NvFp4MoeBackend.FLASHINFER_TRTLLM,

@functionstackx functionstackx Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe upstream if sm120 then marlin else flashinfer?

+ NvFp4MoeBackend.MARLIN,
}
--- a/vllm/model_executor/layers/fused_moe/experts/marlin_moe.py
+++ b/vllm/model_executor/layers/fused_moe/experts/marlin_moe.py
@@ -583,9 +583,12 @@
- # Gated-activation params (used by SWIGLUOAI_UNINTERLEAVE on packed w13).
- # silu == swigluoai with alpha=1, beta=0; configs that don't set these
- # (plain silu) fall back to the silu identity.
- self.gemm1_alpha = (
- quant_config.gemm1_alpha if quant_config.gemm1_alpha is not None else 1.0
- )
- self.gemm1_beta = (
- quant_config.gemm1_beta if quant_config.gemm1_beta is not None else 0.0
- )
+ if self.gemm1_clamp_limit is None:
+ self.gemm1_clamp_limit = moe_config.swiglu_limit
+
+ gemm1_alpha = quant_config.gemm1_alpha
+ if gemm1_alpha is None:
+ gemm1_alpha = moe_config.swiglu_alpha
+ self.gemm1_alpha = 1.0 if gemm1_alpha is None else gemm1_alpha
+
+ gemm1_beta = quant_config.gemm1_beta
+ if gemm1_beta is None:
+ gemm1_beta = moe_config.swiglu_beta
+ self.gemm1_beta = 0.0 if gemm1_beta is None else gemm1_beta
112 changes: 112 additions & 0 deletions benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_rtx6000pro.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
#!/usr/bin/env bash

# MiniMax-M3 NVFP4 RTX PRO 6000 Blackwell single-node vLLM recipe.
# This is the PCIe/SM120 counterpart to minimaxm3_fp4_b200.sh. It keeps
# the ModelOpt NVFP4, FP8 KV-cache, and MSA block-size settings while using
# NCCL collectives instead of the B200-tuned FlashInfer/TRT-LLM all-reduce.
#
# The pinned vLLM image does not contain the MiniMax-M3 Marlin fixes tracked
# by vLLM PRs #45836 and #48929. Apply the narrow compatibility patch covered
# by docs/waiver/2306.md before importing vLLM.

source "$(dirname "$0")/../../benchmark_lib.sh"

check_env_vars \
MODEL \
TP \
EP_SIZE \
DP_ATTENTION \
CONC \
ISL \
OSL \
MAX_MODEL_LEN \
RANDOM_RANGE_RATIO \
RESULT_FILENAME

if [[ "$MODEL" != /* ]]; then hf download "$MODEL"; fi

if [[ -n "$SLURM_JOB_ID" ]]; then
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
fi

nvidia-smi

SERVER_LOG=/workspace/server.log
GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.90}"

export VLLM_ENGINE_READY_TIMEOUT_S=3600
export VLLM_FLOAT32_MATMUL_PRECISION=high

if [ "${DP_ATTENTION}" = "true" ]; then
PARALLEL_ARGS=(
--tensor-parallel-size 1
--data-parallel-size "$TP"
--enable-expert-parallel
)
elif [ "$EP_SIZE" -gt 1 ]; then
PARALLEL_ARGS=(
--tensor-parallel-size "$TP"
--enable-expert-parallel
)
else
PARALLEL_ARGS=(--tensor-parallel-size "$TP")
fi

if [ "${EVAL_ONLY}" = "true" ]; then
setup_eval_context
MAX_MODEL_LEN="$EVAL_MAX_MODEL_LEN"
fi
start_gpu_monitor

VLLM_SITE_PACKAGES="$(
python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])'
)"
VLLM_MARLIN_PATCH=/workspace/benchmarks/patches/vllm/minimax_m3_nvfp4_marlin.patch
if ! patch --batch --forward --fuzz=0 -p1 -d "$VLLM_SITE_PACKAGES" \
< "$VLLM_MARLIN_PATCH"; then
echo "Failed to apply the pinned MiniMax-M3 Marlin compatibility patch" >&2
exit 1
fi
Comment on lines +61 to +69

@claude claude Bot Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The Marlin compatibility patch step (lines 61-69) hard-aborts if patch --forward returns nonzero, but --forward does not make re-applying an already-applied patch a no-op — GNU patch exits 1 with "Reversed (or previously applied) patch detected!" in that case too. This breaks re-running the script against an already-patched vLLM install (e.g. the repo's own debug-runs fast-iteration workflow of SSHing onto the runner and re-running in the same container), and will also bite if the pinned image ever upstreams just one of the two patch hunks. The sibling minimaxm3_fp8_b200.sh recipe avoids this by explicitly detecting the already-applied state before failing.

Extended reasoning...

The bug: minimaxm3_fp4_rtx6000pro.sh lines 65-69 run:

if ! patch --batch --forward --fuzz=0 -p1 -d "$VLLM_SITE_PACKAGES" < "$VLLM_MARLIN_PATCH"; then
    echo "Failed to apply the pinned MiniMax-M3 Marlin compatibility patch" >&2
    exit 1
fi

The intent of --forward (GNU patch's -N) is presumably to make the script idempotent — i.e., skip hunks that are already applied so re-running the script against a previously-patched vLLM install doesn't fail. That's not what it does. --forward only suppresses applying a patch that would go backwards relative to already-modified source; it does not exit 0 when a hunk is already present. On a second apply of the identical patch, GNU patch prints Reversed (or previously applied) patch detected! Skipping patch., writes a .rej file, and exits 1 — a status this script treats identically to a genuinely broken/mismatched patch.

Reachability: normal CI is unaffected, since runners/launch_rtx6000pro-lat.sh uses docker run --rm, giving a fresh, unpatched container on every invocation. The failure mode is triggered by this repo's own debug-runs skill, which instructs engineers to SSH onto the runner and re-run the benchmark script directly in the same live container (via docker exec/enroot) for fast iteration on things like CONC. The second invocation in that same container hits an already-patched vLLM tree and hard-fails at the patch step before vllm serve even starts, defeating the fast-iteration workflow the skill exists for. It would also resurface once the pinned image upstreams just one of the two patch hunks (per the removal plan in docs/waiver/2306.md) before the patch file is deleted — a partially-applied multi-hunk patch still returns nonzero overall even though the still-needed hunk would apply cleanly.

Step-by-step proof:

  1. First run: container has an unpatched vLLM install. patch --batch --forward --fuzz=0 -p1 -d $VLLM_SITE_PACKAGES < minimax_m3_nvfp4_marlin.patch applies both hunks cleanly, exits 0. vllm serve starts normally.
  2. Engineer SSHes onto the runner mid-debug, re-runs the same script in the same container to tweak CONC.
  3. patch is invoked again against the now-already-patched nvfp4.py and marlin_moe.py. GNU patch detects both hunks are already present, prints "Reversed (or previously applied) patch detected! Skipping patch.", writes .rej, and exits 1.
  4. The script's if ! patch ...; then echo ...; exit 1; fi treats this exit code as failure and aborts before vllm serve is ever invoked — even though the vLLM install is in the exact state the script wants it in.

Why nothing else prevents this: --fuzz=0 and -p1 only control hunk-matching leniency and path stripping; neither affects idempotency. There's no pre-check for an already-patched state, unlike the sibling minimaxm3_fp8_b200.sh recipe (lines ~30-55), which runs a small Python idempotent-patch routine that explicitly prints [minimax-m3-msa-patch] already applied and treats that as success rather than failure — i.e., idempotency is a known, established concern for these runtime-patch recipes that this new recipe doesn't carry over.

Suggested fix: mirror the b200 recipe's approach (an idempotent Python-based patch application, or a pre-check such as patch --dry-run -R to detect 'already applied' and treat it as success rather than failure) so re-running the script against an already-patched install is a no-op instead of a hard failure.

Severity: all three independent verifiers empirically confirmed the behavior with GNU patch and agreed on nit — this is a real, reproducible bug, but normal CI (fresh container per run) is unaffected; it only degrades the manual/debug re-run path and the future partial-upstream scenario, not the sweep this PR is being merged for.


set -x
vllm serve "$MODEL" --port "$PORT" \
"${PARALLEL_ARGS[@]}" \
--disable-custom-all-reduce \
--gpu-memory-utilization "$GPU_MEM_UTIL" \
--max-model-len "$MAX_MODEL_LEN" \
--kv-cache-dtype fp8 \
--block-size 128 \
--language-model-only \
--attention-backend TRITON_ATTN \
--moe-backend marlin \
--max-cudagraph-capture-size 2048 \
--max-num-seqs "$CONC" \
--max-num-batched-tokens "$((ISL * 2))" \
--stream-interval 20 \
--no-enable-prefix-caching \
--trust-remote-code > "$SERVER_LOG" 2>&1 &

SERVER_PID=$!

wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"

run_benchmark_serving \
--model "$MODEL" \
--port "$PORT" \
--backend vllm \
--input-len "$ISL" \
--output-len "$OSL" \
--random-range-ratio "$RANDOM_RANGE_RATIO" \
--num-prompts "$((CONC * 10))" \
--max-concurrency "$CONC" \
--result-filename "$RESULT_FILENAME" \
--result-dir /workspace/ \
--trust-remote-code

if [ "${RUN_EVAL}" = "true" ]; then
run_eval --framework lm-eval --port "$PORT"
append_lm_eval_summary
fi

stop_gpu_monitor
set +x
140 changes: 140 additions & 0 deletions benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_rtx6000pro_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
#!/usr/bin/env bash

# MiniMax-M3 NVFP4 RTX PRO 6000 Blackwell single-node vLLM recipe with
# EAGLE3 speculative decoding. The SM120 target and draft both use Triton
# attention; the target's NVFP4 experts use Marlin.
#
# The pinned vLLM image does not contain the MiniMax-M3 Marlin fixes tracked
# by vLLM PRs #45836 and #48929. Apply the narrow compatibility patch covered
# by docs/waiver/2306.md before importing vLLM.

source "$(dirname "$0")/../../benchmark_lib.sh"

check_env_vars \
MODEL \
TP \
EP_SIZE \
DP_ATTENTION \
CONC \
ISL \
OSL \
MAX_MODEL_LEN \
RANDOM_RANGE_RATIO \
RESULT_FILENAME

SERVED_MODEL_NAME="$MODEL"
TARGET_MODEL_PATH="${MODEL_PATH:-$MODEL}"
if [[ "$TARGET_MODEL_PATH" != /* ]]; then
hf download "$TARGET_MODEL_PATH"
fi

DRAFT_MODEL="Inferact/MiniMax-M3-EAGLE3"
hf download "$DRAFT_MODEL"

if [[ -n "$SLURM_JOB_ID" ]]; then
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
fi

nvidia-smi

SERVER_LOG=/workspace/server.log
GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.90}"
NUM_SPEC_TOKENS="${NUM_SPEC_TOKENS:-3}"

if [[ ! "$NUM_SPEC_TOKENS" =~ ^[1-9][0-9]*$ ]]; then
echo "NUM_SPEC_TOKENS must be a positive integer, got: $NUM_SPEC_TOKENS" >&2
exit 1
fi

# vLLM's speculative decode capture size is measured in tokens. Each active
# sequence can contribute one accepted token plus NUM_SPEC_TOKENS draft tokens.
CUDAGRAPH_CAPTURE_SIZE="$((CONC * (NUM_SPEC_TOKENS + 1)))"

export VLLM_ENGINE_READY_TIMEOUT_S=3600
export VLLM_FLOAT32_MATMUL_PRECISION=high

if [ "${DP_ATTENTION}" = "true" ]; then
PARALLEL_ARGS=(
--tensor-parallel-size 1
--data-parallel-size "$TP"
--enable-expert-parallel
)
DRAFT_TP=1
elif [ "$EP_SIZE" -gt 1 ]; then
PARALLEL_ARGS=(
--tensor-parallel-size "$TP"
--enable-expert-parallel
)
DRAFT_TP="$TP"
else
PARALLEL_ARGS=(--tensor-parallel-size "$TP")
DRAFT_TP="$TP"
fi

if [ "${EVAL_ONLY}" = "true" ]; then
setup_eval_context
MAX_MODEL_LEN="$EVAL_MAX_MODEL_LEN"
fi
start_gpu_monitor

VLLM_SITE_PACKAGES="$(
python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])'
)"
VLLM_MARLIN_PATCH=/workspace/benchmarks/patches/vllm/minimax_m3_nvfp4_marlin.patch
if ! patch --batch --forward --fuzz=0 -p1 -d "$VLLM_SITE_PACKAGES" \
< "$VLLM_MARLIN_PATCH"; then
echo "Failed to apply the pinned MiniMax-M3 Marlin compatibility patch" >&2
exit 1
fi

SPECULATIVE_CONFIG="$(
printf '{"method":"eagle3","model":"%s","num_speculative_tokens":%d,"draft_tensor_parallel_size":%d,"attention_backend":"TRITON_ATTN"}' \
"$DRAFT_MODEL" "$NUM_SPEC_TOKENS" "$DRAFT_TP"
)"

set -x
vllm serve "$TARGET_MODEL_PATH" \
--served-model-name "$SERVED_MODEL_NAME" \
--port "$PORT" \
"${PARALLEL_ARGS[@]}" \
--disable-custom-all-reduce \
--gpu-memory-utilization "$GPU_MEM_UTIL" \
--max-model-len "$MAX_MODEL_LEN" \
--kv-cache-dtype fp8 \
--block-size 128 \
--language-model-only \
--attention-backend TRITON_ATTN \
--moe-backend marlin \
--max-cudagraph-capture-size "$CUDAGRAPH_CAPTURE_SIZE" \
--max-num-seqs "$CONC" \
--max-num-batched-tokens "$((ISL * 2))" \
--speculative-config "$SPECULATIVE_CONFIG" \
--stream-interval 20 \
--no-enable-prefix-caching \
--trust-remote-code > "$SERVER_LOG" 2>&1 &

SERVER_PID=$!

wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"

run_benchmark_serving \
--model "$SERVED_MODEL_NAME" \
--port "$PORT" \
--backend vllm \
--input-len "$ISL" \
--output-len "$OSL" \
--random-range-ratio "$RANDOM_RANGE_RATIO" \
--num-prompts "$((CONC * 10))" \
--max-concurrency "$CONC" \
--result-filename "$RESULT_FILENAME" \
--result-dir /workspace/ \
--trust-remote-code \
--use-chat-template

if [ "${RUN_EVAL}" = "true" ]; then
run_eval --framework lm-eval --port "$PORT"
append_lm_eval_summary
fi

stop_gpu_monitor
set +x
38 changes: 38 additions & 0 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7347,6 +7347,44 @@ minimaxm3-fp8-b300-vllm:
- { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 128 }
- { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 512 }

# MiniMax-M3 NVFP4 single-node vLLM sweep using 4 of 8 RTX PRO 6000 GPUs.
minimaxm3-fp4-rtx6000pro-vllm:
image: vllm/vllm-openai:vllm-minimax-m3-perf-x86_64-13.0.1-8b00f41@sha256:6af4be7ae69a5f424de85b3d514ac79778bdcf4a9d05f93ec908a4095e0f1253
model: nvidia/MiniMax-M3-NVFP4
model-prefix: minimaxm3
runner: rtx6000pro-lat
precision: fp4
framework: vllm
multinode: false
scenarios:
fixed-seq-len:
- isl: 8192
osl: 1024
search-space:
- { tp: 4, conc-list: [1, 4, 16, 64] }
# Marlin EP4 uses the standard NCCL expert exchange; DeepEP/NIXL
# batched activation layouts are not compatible.
- { tp: 4, ep: 4, conc-list: [1, 4, 16, 64] }

# EAGLE3 speculative-decoding twin of the RTX PRO 6000 MiniMax-M3 NVFP4
# sweep. The external Inferact/MiniMax-M3-EAGLE3 draft uses three speculative
# tokens and Triton attention on SM120; prompts use the model's chat template.
minimaxm3-fp4-rtx6000pro-vllm-mtp:
image: vllm/vllm-openai:vllm-minimax-m3-perf-x86_64-13.0.1-8b00f41@sha256:6af4be7ae69a5f424de85b3d514ac79778bdcf4a9d05f93ec908a4095e0f1253
model: nvidia/MiniMax-M3-NVFP4
model-prefix: minimaxm3
runner: rtx6000pro-lat
precision: fp4
framework: vllm
multinode: false
scenarios:
fixed-seq-len:
- isl: 8192
osl: 1024
search-space:
- { tp: 4, conc-list: [1, 4, 16, 64], spec-decoding: mtp }
- { tp: 4, ep: 4, conc-list: [1, 4, 16, 64], spec-decoding: mtp }

# MiniMax-M3 NVFP4 (nvidia/MiniMax-M3-NVFP4) B300 single-node vLLM — FP4 variant
# of minimaxm3-fp8-b300-vllm. MiniMax-M3 modelopt NVFP4 support (vllm-project/vllm
# PR #46380) is baked into the perf container image, so no runtime patch is
Expand Down
9 changes: 9 additions & 0 deletions configs/runners.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,10 @@ labels:
- gb300-nv_0
- gb300-nv_1
- gb300-nv_2
rtx6000pro:
- rtx6000pro-lat_00
rtx6000pro-lat:
- rtx6000pro-lat_00
cluster:h100-cw:
- h100-cw_00
- h100-cw_01
Expand Down Expand Up @@ -251,6 +255,8 @@ labels:
- gb300-nv_0
- gb300-nv_1
- gb300-nv_2
cluster:rtx6000pro-lat:
- rtx6000pro-lat_00
cluster:mi300x-amds:
- mi300x-amds_00
- mi300x-amds_01
Expand Down Expand Up @@ -315,6 +321,9 @@ hardware:
cluster:gb200-nv:
available-cpu-dram-mib: 860_160
gpus-per-node: 4
cluster:rtx6000pro-lat:
available-cpu-dram-mib: 1_500_000
gpus-per-node: 8
cluster:mi300x-amds:
available-cpu-dram-mib: 2_321_924
gpus-per-node: 8
Expand Down
Loading
Loading