Validation report: Qwen3.5-122B-A10B-NVFP4 on 1× RTX PRO 6000

[jpezzulli]

I just wanted to report a successful real-use validation of txn545/Qwen3.5-122B-A10B-NVFP4 using the vLLM-Moet v0.24.0 SM120 lane.

Hardware

• 1× NVIDIA RTX PRO 6000 Blackwell, 96 GB
• Model remained fully resident on the GPU
• Model loading consumed 77.31 GiB
• Running EngineCore process reported approximately 88,506 MiB

Runtime shape

• Maximum model length: 200,704 tokens
• FP8 KV cache
• MTP speculative decoding: K=2
• Maximum sequences: 6
• Maximum batched tokens: 8,640
• VLLM_COMPILE with PIECEWISE CUDA graphs
• Prefix caching enabled

Observed performance

On a real chat containing approximately 82,200 input tokens:

• Prompt processing: approximately 6,514 tokens/second
• Generation: approximately 100.7 tokens/second
• MTP draft acceptance: approximately 69%

A separate run reached:

• Prompt processing: approximately 8,352 tokens/second
• Generation: approximately 100.6 tokens/second
• MTP draft acceptance: approximately 78%

This was interactive use through an OpenAI-compatible chat client, not a formal benchmark or concurrency test. I did not run a needle-in-a-haystack retrieval test.

Compilation also required a small workaround for the upstream vLLM torch.accelerator.device_index Dynamo problem currently tracked in [vLLM PR #40921](vllm-project/vllm#40921). That appears separate from the vLLM-Moet kernels themselves.

The main result is that Qwen3.5-122B-A10B-NVFP4 loads successfully and sustains approximately 100 generation tokens/second on one RTX PRO 6000 using this engine.

[pohnean]

@jpezzulli, Qwen3.5-122B-A10B-NVFP4 already fully fits the RTX PRO 6000 right? What benefit did you get from running with this engine? Just curious

[davised]

I tried to get Qwen3.5-122B-A10B-NVFP4 running on a single RTX Pro 6000 today and I could not. I didn't dig deeper to figure out why but I'm not surprised more optimizations and/or workarounds like the one linked are required before it's usable.

[pohnean]

I'm running with this config on regular vllm:

[jpezzulli]

One useful clarification about why my test hit the compile failure while the regular-vLLM configuration above does not: My launch explicitly used: "--compilation-config '{"mode":"VLLM_COMPILE","cudagraph_mode":"PIECEWISE","custom_ops":["all"]}'" I enabled that intentionally to exercise the optimized compiled/custom-op path in the MoET image. It is not required merely to load this model, and it is not what activates MoET’s W2 expert kernels. Those are controlled separately and were not enabled in this test. The model still ran through the compressed-tensors NVFP4 and FlashInfer CUTLASS paths. However, "custom_ops:["all"]" also forces stock vLLM’s RMSNormGated CUDA custom op. That is the path where Dynamo reaches "torch.accelerator.device_index(...)" and fails on unpatched vLLM 0.24.0. The regular-vLLM configuration shown above omits "--compilation-config", so it avoids that exact custom-op path. Both results are therefore consistent: stock vLLM can load the model with its default path, while explicitly enabling the optimized custom op reproduces the bug addressed by vllm-project/vllm#40921.

…

On Tue, Jul 14, 2026, 5:34 AM Poh Nean ***@***.***> wrote: *pohnean* left a comment (kacper-daftcode/vLLM-Moet#15) <#15 (comment)> I'm running with this config on regular vllm: image.png (view on web) <https://github.com/user-attachments/assets/ef708a4b-9fba-4a19-9dbf-02659e53130f> — Reply to this email directly, view it on GitHub <#15?email_source=notifications&email_token=AJFK2GEQ2SPXBB2ZR33Y74D5EX5C3A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJWG42TKNBTG4Z2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4967554373>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJFK2GD6WSY2H3I3765BQ535EX5C3AVCNFSNUABGKJSXA33TNF2G64TZHMYTENZQGU3TMNJZGA5US43TOVSTWNBYG44DIOJQGQ4TNILWAI> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AJFK2GH6OGLEA4G4JQWFBET5EX5C3A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJWG42TKNBTG4Z2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/AJFK2GAWTPGPMF6RPMJORGD5EX5C3A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJWG42TKNBTG4Z2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[dicksondickson]

@jpezzulli and @davised I'm running Unsloth's Qwen3.5 122b NVFP4 with 256k kvcache with the official VLLM docker image on a single RTX Pro 6000. I'm getting 13000 PP and 130t/s TG @ 32k depth.
Here is my run params:

MODEL="unsloth/Qwen3.5-122B-A10B-NVFP4"
MODELPATH="/root/models"

docker run --rm --runtime nvidia --gpus all \
  -v $HOME/models:$MODELPATH/ \
  -v $HOME/.cache/vllm:/root/.cache/vllm \
  -v $HOME/.cache/torch:/root/.cache/torch \
  -v $HOME/.cache/flashinfer:/root/.cache/flashinfer \
  -p 40000:40000 \
  --ipc=host \
  vllm/vllm-openai:latest \
    --model "$MODELPATH/$MODEL" \
    --served-model-name qwen35-122b \
    --port 40000 --host 0.0.0.0 \
    --gpu-memory-utilization 0.95 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --max-num-seqs 4 \
    --max-num-batched-tokens 8192 \
    --max-model-len 262144 \
    --kv-cache-dtype fp8 \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_xml \
    --chat-template $MODELPATH/chat-template/allanchan339/qwen3.5-enhanced.jinja

Benchmark

| model                           |            test |               t/s |      peak t/s |      ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:--------------------------------|----------------:|------------------:|--------------:|---------------:|---------------:|----------------:|
| unsloth/Qwen3.5-122B-A10B-NVFP4 |          pp2048 | 47230.99 ± 450.78 |               |  119.57 ± 0.42 |   43.39 ± 0.42 |   119.57 ± 0.42 |
| unsloth/Qwen3.5-122B-A10B-NVFP4 |            tg32 |     128.63 ± 0.19 | 132.78 ± 0.20 |                |                |                 |
| unsloth/Qwen3.5-122B-A10B-NVFP4 |  pp2048 @ d4096 | 24106.53 ± 159.39 |               |  331.09 ± 1.68 |  254.91 ± 1.68 |   331.09 ± 1.68 |
| unsloth/Qwen3.5-122B-A10B-NVFP4 |    tg32 @ d4096 |     129.13 ± 0.10 | 133.29 ± 0.10 |                |                |                 |
| unsloth/Qwen3.5-122B-A10B-NVFP4 |  pp2048 @ d8192 |  20801.32 ± 33.72 |               |  568.51 ± 0.80 |  492.33 ± 0.80 |   568.51 ± 0.80 |
| unsloth/Qwen3.5-122B-A10B-NVFP4 |    tg32 @ d8192 |     128.89 ± 0.25 | 133.05 ± 0.26 |                |                |                 |
| unsloth/Qwen3.5-122B-A10B-NVFP4 | pp2048 @ d16384 |  17449.54 ± 17.34 |               | 1132.55 ± 1.05 | 1056.36 ± 1.05 |  1132.55 ± 1.05 |
| unsloth/Qwen3.5-122B-A10B-NVFP4 |   tg32 @ d16384 |     127.72 ± 0.38 | 131.84 ± 0.39 |                |                |                 |
| unsloth/Qwen3.5-122B-A10B-NVFP4 | pp2048 @ d32768 |  13758.20 ± 18.92 |               | 2606.82 ± 3.48 | 2530.64 ± 3.48 |  2606.82 ± 3.48 |
| unsloth/Qwen3.5-122B-A10B-NVFP4 |   tg32 @ d32768 |     122.78 ± 0.28 | 130.96 ± 0.30 |                |                |                 |

llama-benchy (0.4.0)
date: 2026-07-19 16:46:36 | latency mode: generation