DeepSeek V4 Flash 0731 on vLLM-MoET: Single-GPU Validation Report

[jpezzulli]

DeepSeek V4 Flash 0731 on vLLM-MoET: Single-GPU Validation Report

Test date: July 31, 2026
Tested configuration: W2 base plus a 6 GiB FP4 correction pool on one NVIDIA RTX PRO 6000 Blackwell 96 GB

Executive summary

DeepSeek V4 Flash 0731 completed a frozen eight-case reasoning-quality screen on vLLM-MoET without an API error, length exhaustion, hard reasoning loop, or runtime failure. Seven cases were answered very well. The model received a raw dimension-weighted score of 96.16/100.

One deployment-critical defect triggered the suite's frozen fatal cap. In the idempotent-transfer review, the model correctly found most of the concurrency and data-integrity problems, but its proposed repair did not persist request ownership and terminal results for invalid-input and missing-account outcomes. That violated the explicit requirement that every outcome be idempotent. A separately generated, blinded rerun produced a different answer but repeated the same defect. The final case-weighted score after the cap was 87.54/100.

The primary suite generated 28,452 tokens in 715.31 seconds, for an aggregate effective generation rate of 39.78 tok/s. Per-request speed ranged from 32.52 to 40.99 tok/s. The targeted C5 rerun generated another 7,501 tokens at 39.55 tok/s.

This is a validation result for the complete tested configuration. It does not establish that W2 quantization, the FP4 correction tier, vLLM-MoET, or the lack of speculative decoding caused the C5 defect because no alternate-runtime or alternate-quantization comparator was run.

System and runtime configuration

Host

Component	Configuration
Motherboard	Supermicro X12DAi-N6
CPUs	2 x Intel Xeon Platinum 8358, 32 cores per socket
System memory	256 GiB nominal DDR4-3200 ECC RDIMM; approximately 219 GiB Linux-visible at the documented checkpoint because two memory channels were unavailable
GPU	NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB, one GPU used
GPU connection	PCIe 4.0 x16
NVIDIA driver	610.43.03

Software and model

Item	Captured value
Model	DeepSeek V4 Flash 0731 checkpoint
Architecture	DeepseekV4ForCausalLM
vLLM-MoET revision	f521dc10bbcdc86dc45936d9b7fd4ac88582c907
Patched vLLM revision	ee0da84ab9e04ac7610e28580af62c365e898389
Installed vLLM build	0.24.1.dev0+gee0da84ab.d20260727
Model runner	V1
Tensor parallelism	1
Speculative decoding	Disabled
Request sampling	No request-level override; checkpoint defaults were temperature 1.0 and top-p 1.0
Reasoning mode	Thinking enabled, high reasoning effort

MoET and serving settings

Setting	Value
W2 base	Complete base forced resident
FP4 correction pool	6 GiB, 512 slots x 12 MiB
Delta policy	Frequency-managed
FP4 prefill policy	Enabled with policy 1
Confidence gate	Disabled
AFRAG	Enabled
Fused unpermute	Disabled
Maximum model length	524,288 tokens
Maximum sequences	4
Maximum batched tokens	1,024
KV cache	DeepSeek fp8_ds_mla, block size 256
GPU-memory utilization	0.988
Prefix caching	Enabled
CUDA graphs	Full and piecewise

The runtime reported a model-load footprint of 87.41 GiB and a model-load time of 252.26 seconds. It allocated 3.89 GiB to KV cache, reported capacity for 863,773 tokens, and estimated 1.65x concurrency at the configured 524,288-token request ceiling. The suite did not submit a 524K request, so these are startup capacity figures rather than a demonstrated end-to-end context result.

Test method

The suite was frozen before execution. Each case started with fresh conversation context except C8, whose correction turn included the first prompt and answer. The endpoint received the frozen messages, max_tokens=32768, streaming enabled, and usage reporting enabled. It did not receive request-level temperature, top-p, top-k, seed, reasoning-effort, tool, stop-sequence, or formatting overrides.

Qualitative grading was performed from anonymized response text before runtime identity, timing, token counts, finish state, and operational metadata were revealed. The rubric scored correctness, evidence discipline, contradiction detection, constraint preservation, technical reasoning, unsupported-assumption control, confidence calibration, revision quality, loop behavior, and final-answer usability.

The frozen fatal rules capped a case at 35 if the answer contained its case-specific deployment-critical error. A confirmed hard loop would cap a case at 20, and repetitive 32K exhaustion without a usable answer would cap it at 10.

Case weights for C1 through C8 were 14, 12, 14, 14, 16, 10, 10, and 10. Generated-token counts use the deployed checkpoint tokenizer and include hidden reasoning plus the visible answer. Effective speed is local generated tokens divided by complete request wall time.

Results by test case

Case	What it tested	Raw / final score	Generated tokens	Wall time	Effective speed	Result
C1, Tunnel latency diagnosis	PMTU black-hole diagnosis, packet-size arithmetic, capture reconciliation, alternative rejection, and remediation	96 / 96	2,320	63.23 s	36.69 tok/s	Pass
C2, Authoritative ordering	Whether signed controller sequence overrides misleading workstation timestamps, optimistic UI state, and SIEM receipt order	98 / 98	1,064	28.45 s	37.40 tok/s	Pass
C3, Production optimization	Integer optimization under quantity, grade-mix, energy, capacity, and sunk-cost constraints	100 / 100	3,870	99.53 s	38.88 tok/s	Pass
C4, Non-preemptive scheduling	Resource-constrained scheduling, maintenance, deadlines, optional work, and proof of optimal makespan	91 / 91	10,159	247.98 s	40.97 tok/s	Pass, but substantially overlong
C5, Idempotent transfer review	Transaction ownership, request fingerprinting, missing accounts, invalid input, locking order, and duplicate-request consistency	86 / 35	8,141	198.59 s	40.99 tok/s	Fatal idempotency defect
C6, Release decision	Exact lot identity, post-rework test validity, waiver limits, and resistance to operational pressure	100 / 100	405	12.45 s	32.52 tok/s	Pass
C7, Posterior without independence	Identification limits, Fréchet bounds, sharp posterior range, and labeling of an independence-only estimate	100 / 100	1,402	35.82 s	39.14 tok/s	Pass
C8, Multi-turn correction	Recalculation after corrected evidence, explicit retraction, preservation of unaffected facts, and revised action	100 / 100	1,091 across two turns	29.26 s	37.29 tok/s	Pass

All nine measured requests ended naturally with finish_reason=stop. There were no API or generation errors, hard-loop terminations, 32K length exhaustions, or thermal stops.

C1: Tunnel latency diagnosis

Scenario. An HTTPS service moved behind an IP-in-IP tunnel adding exactly 96 bytes per packet. Physical MTU remained 1500, inner MTU remained 1500, and the firewall discarded ICMP packet-too-big messages. Requests became slow only when the first server flight contained an inner packet from 1405 through 1500 bytes. Pre-encapsulation capture showed the original large packet and a smaller retransmission about one second later, while physical-egress capture showed only the smaller retry.

Required reasoning. Identify a PMTU black hole, derive the 1404-byte boundary from 1500 - 96, reconcile capture placement, connect the delay to the retransmission timeout and smaller retry, reject the unrelated fan warning and unchanged DNS/application/CPU/GC evidence, and propose an MTU/MSS or valid-PTB correction with verification.

Model result. The model correctly diagnosed the PMTU black hole, calculated the 1404-byte limit, reconciled both captures, connected the roughly one-second tail latency to retransmission behavior, rejected unsupported alternatives, and proposed a valid correction and boundary test. The small deduction was for treating the IP DF behavior as established rather than as the best inference from the supplied packet behavior.

Score and speed. 96/100. 2,320 generated tokens in 63.23 seconds, or 36.69 tok/s.

C2: Authoritative ordering versus client logs

Scenario. A signed controller sequence showed checksum X at sequence 441, the first pressure alarm at 442, and the X-to-Y configuration write at 443. A workstation with possible clock skew showed that the write was requested earlier and optimistically displayed Y. SIEM receipt order was explicitly not controller application order.

Required reasoning. Decide whether the write caused the first alarm through the stated mechanism, give precedence to validated controller application order, explain why workstation timestamps, optimistic UI state, witness observation, and SIEM receipt order do not override it, and leave the actual cause unknown.

Model result. The model correctly concluded that Y could not have caused the first alarm through the stated mechanism because the alarm was applied while the authoritative state was still X. It reconciled all apparent conflicts and did not invent an alternate cause.

Score and speed. 98/100. 1,064 generated tokens in 28.45 seconds, or 37.40 tok/s.

C3: Production optimization with grade mix

Scenario. The model had to choose whole 100-input batches from two sources while satisfying accepted-unit volume, Grade H percentage, energy, and source-capacity constraints. Accepted output could not be discarded, and a previously paid audit fee was sunk.

Required reasoning. Prove that 6A+5B is the minimum-cost feasible plan with 1,026 accepted units, 522 Grade H units, 405 energy units, and $4,048 incremental cost. Compare it with 7A+4B, reject 5A+6B because its Grade H share is only 45.9%, and prove infeasibility if A capacity falls to five batches.

Model result. The model produced the correct integer formulation, evaluated the relevant feasible frontier, selected 6A+5B, treated the audit fee correctly as sunk, and proved that no feasible plan remains when A is capped at five batches.

Score and speed. 100/100. 3,870 generated tokens in 99.53 seconds, or 38.88 tok/s.

C4: Constrained non-preemptive schedule

Scenario. One GPU, one specialist, non-preemptive GPU work, a fixed GPU-maintenance window, release times, deadlines, and dependency constraints had to be scheduled. An optional one-hour GPU job X could be accepted only if it did not delay mandatory completion.

Required reasoning. Produce the optimal schedule B 0-2, E 2-4, H 4-6, C 4-8, maintenance 6-8, A 8-11, D 11-15, and F 15-17; prove that makespan 17 is a lower bound; and reject X because it would push H after maintenance and delay F to at least 19.

Model result. The schedule, lower-bound proof, makespan, and rejection of X were correct. The response repeatedly revisited already eliminated schedules before converging. It continued to make progress and did not satisfy the frozen hard-loop criteria, but the excess reasoning reduced usability and the loop-behavior score.

Score and speed. 91/100. 10,159 generated tokens in 247.98 seconds, or 40.97 tok/s.

C5: Idempotent transfer implementation review

Scenario. The model reviewed a money-transfer transaction with an outside-transaction request-ID lookup, source-only FOR UPDATE, unverified account updates, an ok-only ledger insert, and a unique-conflict fallback. Inputs could include negative amounts, identical accounts, missing accounts, and request IDs reused with different parameters. The required behavior explicitly stated that every outcome, including insufficient funds, must be idempotent.

Required reasoning. Identify the unsafe pre-check and duplicate race; retain and compare the request fingerprint; validate positive integer amounts, distinct accounts, and both account rows; lock both accounts in canonical order; atomically claim the request ID; persist every terminal outcome; and address avoidable deadlocks.

What the model got right. It correctly diagnosed insufficient-funds non-persistence, request-fingerprint mismatch, missing-account money creation or destruction, negative and self transfers, opposite-direction deadlock, canonical account locking, loser rollback on unique conflict, and the need to persist successful and insufficient outcomes.

Fatal defect. Its replacement still returned invalid-input and missing-account results without atomically persisting the request ID, fingerprint, and terminal outcome. Examples from the proposed ordering were:

if :cents <= 0 or :src_id = :dst_id:
    return "rejected"

and:

if low_row not found:
    return "missing account"

The answer even described invalid-outcome persistence as optional: If you want these invalid outcomes idempotent too, record them in the ledger, although the prompt had already required every outcome to be idempotent. A later call could therefore reuse the same request ID with changed parameters or receive a different outcome after account state changed. The proposed repair retained the exact class of inconsistent duplicate behavior prohibited by the frozen fatal rule.

Score and speed. Raw score 86/100, capped to 35/100. 8,141 generated tokens in 198.59 seconds, or 40.99 tok/s.

Independent targeted rerun

The permitted rerun used the same frozen system and C5 messages, the same runtime configuration, no request-level sampling override, and a fresh stochastic trajectory. It generated materially different prose but repeated the defect. Its pseudocode again returned invalid inputs before claiming the request ID and returned a missing-account error without persisting request ownership or a terminal result.

The blinded rerun grade was again 86/100 raw and 35/100 after the same fatal cap. It generated 7,501 local tokens in 189.64 seconds, or 39.55 tok/s, and ended naturally with finish_reason=stop without an error or loop.

C6: Release decision with distractions

Scenario. Release required valid T1 and T2 results attached to the exact post-rework shipping-lot identity. The current lot had T2 but only the pre-rework lot had T1. A truncated handheld display, good supplier history, urgent customer need, and a generic manager note were distractions. The waiver rule could not replace T1.

Required reasoning. Block release because the exact shipping lot lacked T1, explain why the earlier result did not carry across rework, and reject the waiver, display truncation, history, urgency, and generic note as authorization.

Model result. The model correctly blocked release and applied every controlling rule without being distracted by operational pressure or misleading display evidence.

Score and speed. 100/100. 405 generated tokens in 12.45 seconds, or 32.52 tok/s.

C7: Posterior probability without independence

Scenario. Marginal positive-test rates conditional on disease and no disease were supplied for two tests, but their joint conditional behavior and conditional independence were not.

Required reasoning. State that the exact posterior after two positives is not identifiable; derive joint-positive bounds of [0.50, 0.70] given disease and [0, 0.10] given no disease; derive the sharp posterior range of approximately 35.7% through 100%; and label the approximately 75.7% independence estimate as assumption-dependent rather than determined.

Model result. The model correctly identified non-identifiability, calculated the sharp Fréchet and posterior bounds, and clearly separated the optional independence-only estimate from conclusions licensed by the packet.

Score and speed. 100/100. 1,402 generated tokens in 35.82 seconds, or 39.14 tok/s.

C8: Fixed multi-turn correction

Scenario. The first turn required score aggregation for three candidate causes, application of a uniqueness-versus-ambiguity decision rule, and selection of the lowest-cost containment safe for the declared cause set. The second turn corrected one evidence vector and required recalculation from the beginning, explicit retraction, preservation of unaffected facts, and a revised action.

Required reasoning. Initially calculate A=3, B=4, C=5, report B/C ambiguity, and choose containment X. After the correction, calculate A=7, B=4, C=-2, select A uniquely, choose W, retract B/C and X, and explicitly retain the unaffected inputs and rules.

Model result. Both turns were correct. The model made the initial ambiguous B/C decision and selected X, then fully incorporated the correction, retracted the prior conclusion and action, preserved unaffected evidence, selected A uniquely, and changed containment to W.

Score and speed. 100/100. The first turn generated 528 tokens in 14.52 seconds at 36.36 tok/s. The correction generated 563 tokens in 14.74 seconds at 38.20 tok/s. Combined, C8 generated 1,091 tokens in 29.26 seconds at 37.29 tok/s.

Aggregate quality and operational results

Measure	Result
Raw dimension-weighted quality score	96.16/100
Final case-weighted score after fatal cap	87.54/100
Cases without a fatal defect	7 of 8
Measured requests	9
Generated tokens	28,452
Sum of measured request wall times	715.31 s
Aggregate effective generation rate	39.78 tok/s
Per-request effective speed range	32.52 to 40.99 tok/s
Natural stop completions	9 of 9
API or generation errors	0
Confirmed hard loops	0
Length exhaustions	0

Dimension scores were: correctness 96, evidence discipline 98, contradiction detection 100, constraint preservation 94, technical reasoning 96, unsupported-assumption control 94, confidence calibration 95, revision quality 100, loop behavior 91, and final-answer usability 95.

Interpretation

The tested vLLM-MoET configuration was operationally stable and delivered consistent single-request generation near 40 tok/s. It produced excellent answers on network diagnosis, event ordering, integer optimization, constrained scheduling, release governance, probabilistic identification limits, and multi-turn correction.

The C5 failure is narrow but material. It was not a superficial wording error: both independently sampled answers proposed transaction orderings that failed the explicit every-outcome idempotency requirement. The repetition makes this a reproducible reasoning-quality observation for this configuration.

No comparator was run after the frozen stop condition fired. The evidence therefore supports these claims only:

• The tested W2 plus 6 GiB FP4-delta configuration was stable across the suite.
• It answered seven of eight cases very well.
• It independently repeated one deployment-critical constraint-preservation defect.
• The test does not isolate the cause of that defect.

It would be incorrect to conclude from this run alone that the failure was caused by W2, the FP4 correction pool, vLLM-MoET, checkpoint-native sampling, or the absence of speculative decoding.

Additional MoET observations

• The tested path used Model Runner V1 and no speculative decoder. DSpark requires Model Runner V2, while this MoET W2 implementation rejects V2 because its required replay and gate lifecycle hooks are not implemented there.
• A separate attempt to increase the FP4 correction pool from 6 GiB to 7 GiB failed during the transient model-loading peak by approximately 400 MiB. Memory would have fallen after loading, but startup could not pass the allocation peak. Delaying correction-pool allocation until after the transient load peak may permit a larger pool on a 96 GB card.
• The 524,288-token model limit and 863,773-token reported KV capacity were not exercised by a full-window request in this suite.

Evidence availability

The original run archive is retained privately and is not part of this public report because it contains local host identifiers and filesystem paths. If an upstream maintainer needs the complete prompts, full response text, launcher, logs, or raw timing records, a separate sanitized evidence package can be prepared with relative filenames and host-specific details removed.

[eblanshey]

@jpezzulli can you please share your full command, including environment variables? I am getting poor output with gibberish and extra </think> tags for some reason. Using the official deepseek-ai/DeepSeek-V4-Flash-0731 model with the latest commit, de4af53.

[jpezzulli]

probably made some minor changes as I am working on getting dspark running on it now via MoE runner v2, but something like this. i think the only change was the context length.

One potentially important difference is the exact code revision. My successful validation was pinned to vLLM-MoET commit f521dc1, with the patched vLLM tree at commit ee0da84. Your reported failure is on vLLM-MoET commit de4af53. I have not reproduced my configuration on de4af53, so this could be a configuration difference or a regression between the two revisions.

#!/bin/bash
set -euo pipefail

source /opt/vllm-moet/.venv/bin/activate

export VLLM_USE_BREAKABLE_CUDAGRAPH=0
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0
export CUDA_HOME=/usr/local/cuda
export CUDA_PATH=/usr/local/cuda
export CC=/usr/bin/gcc-15
export CXX=/usr/bin/g++-15
export NVCC_CCBIN=/usr/bin/g++-15
unset NVCC_PREPEND_FLAGS

export XDG_CACHE_HOME="/srv/cache/vllm-moet/root-cache"
export FLASHINFER_WORKSPACE_BASE="/srv/cache/vllm-moet/root-cache/flashinfer"
export TORCH_EXTENSIONS_DIR="/srv/cache/vllm-moet/root-cache/torch_extensions"
export TRITON_CACHE_DIR="/srv/cache/vllm-moet/triton"
export CUDA_CACHE_PATH="/srv/cache/vllm-moet/nvidia/ComputeCache"

export VLLM_MOE_W2_CUBIT_DIR="/opt/vllm-moet/kernels/cubins-sm120"
export VLLM_NVFP4_DS_MLA_EXT_DIR="/opt/vllm-moet/.build/nvfp4-ds-mla"

export VLLM_MOE_W2=1
unset VLLM_MOE_W2_BASE_CACHE_GB
unset VLLM_MOE_W2_FORCE_POOL

export VLLM_MOE_W2_FORCE_RESIDENT=1
export VLLM_MOE_W2_DELTA_GB=6
export VLLM_MOE_W2_DELTA_POLICY=freq
export VLLM_MOE_W2_PREFILL_FP4=1
export VLLM_MOE_W2_GATE=0
export VLLM_MOE_W2_AFRAG=1
export VLLM_MOE_W2_FUSED_UNPERMUTE=0
export VLLM_MOE_W2_STORE_DIR="/srv/models/moet-packs/DeepSeek-V4-Flash"

exec vllm serve
/srv/models/hf/ds4flash0731
--served-model-name pennyroyal
--host 0.0.0.0
--port 8001
--tensor-parallel-size 1
--max-model-len 393216
--max-num-seqs 4
--kv-cache-dtype fp8
--block-size 256
--gpu-memory-utilization 0.988
--max-num-batched-tokens 2048
--enable-prefix-caching
--trust-remote-code
--load-format safetensors
--tokenizer-mode deepseek_v4
--reasoning-parser deepseek_v4
--default-chat-template-kwargs '{"enable_thinking":true,"reasoning_effort":"high"}'
--enable-auto-tool-choice
--tool-call-parser deepseek_v4
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}'
2>&1
mrkaos@thegrid:~$

[btothec]

I have been trying to get this going all day long. I have it running, but it takes about an hour to fully start for some reason, and it is only doing about 10 tok/s.

[jpezzulli]

How much ram do you have? Use codex to start is the first time and monitor it. Sounds like swap pressure in loading. Also i build from source and dont use the container.

[btothec]

How much ram do you have? Use codex to start is the first time and monitor it. Sounds like swap pressure in loading. Also i build from source and dont use the container.

I build from source also. I have 192 GB of RAM. The original version started up slowly too, just not this slow. But once it was up and running it was fast. I am trying a few other things to get this running properly now, hopefully I get get it running around the same speeds you have been getting. Looking forward to DSpark if you get that going. Have you made much headway on it?

[jpezzulli]

192 is plenty. i have 224 ddr ram. I keep all w2 weights on the RTX Pro. make sure you arent remaking the w2 files every startup. It takes about 5 min to start for me.

Dspark is a bitch. it requires the backend MoE 2 runner instead of the MoE runner moet defaults to. I got it working with ~100ish tokens a second at mtp 3 (beyond that it acceptance rate was low). The issue, I had to sacrifice most of the FP4 Delta Gate. Reasoning was hit a bit with that but tool calling and agentic tasks craters. Structured output went down and failed my testing method there. Right now I am down a path that cant be upstreamed here just to figure out something about how the w2 expert base plane moves to RAM to potentially find a solution. I should probably just also test if the MoE runner v2 vs v1 gives a performance boost as maybe we dont need the dspark if that give is enough. I wont sacrifice quality for speed.

also - if you have enough CPU/RAM and PCIE 5.0 - you can run the full weights on LvLLM. I almost got full MiniMax M3 running at nvfp4 but i need 256 gig ram...which i have but my motherboard has one bad DIMM slot which takes out the matching one - thus 224 ram. It just barely cant load as swap gets stuck eventually during the loading phase. I'll update if i find anything and can make an upstream PR if it is something that might work.

[SpiritualAlchemyst]

Is Lvllm any faster for something like minimax m3 compared to llama cpp? There's like no performance metrics from that repo so I never bothered to install and test it.

[jpezzulli]

I havent tried llama in a while. I may use it when i need a single request lane for helper models but if i remember correctly, it doesnt have anything like paged attention or radix attention for kvcahce so it defines lanes. It has been a while though. I didnt get it running with m3 due to the ram issue. I did run full weight ds4flash 0731 on it. Prefill was slower at about 1200 t/s. Decode was about 43 t/s. I am 6k+ with vllm and slightly slower decode oddly.

[SpiritualAlchemyst]

Thanks man. And yeah theres no way to really keep dspark and maintain quality on 1 card. I did come across this repo today btw which has a lot of information about the quality and trade-offs related to the flags for vllm moet. It might be of interest to you. https://huggingface.co/anoane/DeepSeek-V4-Flash-0731-vllm-moet-sm120-cache

[jpezzulli]

Oh - i may have found a way... i might be up all night but this could be a game changer if I don't screw something up.

i'll take a closer look at that link - but I have some reservations already. it seems to solve problems that don't exist.

i;ll keep y'all posted.

[btothec]

Oh - i may have found a way... i might be up all night but this could be a game changer if I don't screw something up.

i'll take a closer look at that link - but I have some reservations already. it seems to solve problems that don't exist.

i;ll keep y'all posted.

GREAT!

This is driving me crazy. I got the server to start up in about 50 seconds now. But I cannot get this over 8 to 10 tokens a second. I have no idea what the issue can be but I will leave it in your hands and just expect a fully working 150 tok/s 512K context version ready in the morning! /s

[jpezzulli]

Well buddy - yall we counting on me and I am going to deliver hopefully. Successful implementation passed but speed isnt as much as i expected. It is an increase for sure and the small sample size does tell me anything yet but I was hoping for more.

i am still doing some work, will run my quality and tools tests against it. submit PRs here (i dont control merge or even know if he wants this type of work.) Ill make a container that can be downloaded or a container recipe etc and option to build from source from my fork.

Claim | Current status

Normal CUDA-graph startup | Proven
Three canonical mapped to ram W2 layers using cuda memory extension | Proven
Zero redundant full-layer VRAM copies | Proven
Correct arithmetic inference | Proven
Exact 1,024-token decode | Proven, 55.02 tok/s
DSpark-3 acceptance | Proven, 608/1,251 or 48.60%
393,216 configured context accepted at startup (600k+ pool) Proven
393,216-token request | Not tested
Frozen quality suites | Not run
Long-duration stability | Not tested
Container execution | Not tested

any tips on how you got it to start in 50 seconds? I still take several minutes even with not rebuilding the w2 weights.

[DikiyLifter]

Yesterday I also tested all day, while without MTP I am selecting a configuration for RTX Pro 6000 96 GB and RAM 96 GB. I'm launching from Windows 11 in WSL.

Benchy test results:

`

model	test	t/s	peak t/s	ttfr (ms)	est_ppt (ms)	e2e_ttft (ms)
deepseek-v4-flash-0731	pp1024	792.45 ± 0.00		1233.73 ± 0.00	1052.43 ± 0.00	1233.73 ± 0.00
deepseek-v4-flash-0731	tg32	48.76 ± 0.00	50.34 ± 0.00
deepseek-v4-flash-0731	pp1024	1601.87 ± 0.00		711.93 ± 0.00	530.63 ± 0.00	711.93 ± 0.00
deepseek-v4-flash-0731	tg100	52.02 ± 0.00	58.00 ± 0.00
deepseek-v4-flash-0731	pp4096	2778.86 ± 0.00		1445.13 ± 0.00	1263.83 ± 0.00	1445.13 ± 0.00
deepseek-v4-flash-0731	tg100	59.43 ± 0.00	62.00 ± 0.00
deepseek-v4-flash-0731	pp8192	2879.22 ± 0.00		2645.51 ± 0.00	2464.21 ± 0.00	2645.82 ± 0.00
deepseek-v4-flash-0731	tg32	64.30 ± 0.00	66.37 ± 0.00
deepseek-v4-flash-0731	pp8192	2900.66 ± 0.00		2639.36 ± 0.00	2458.06 ± 0.00	2639.36 ± 0.00
deepseek-v4-flash-0731	tg100	60.15 ± 0.00	61.00 ± 0.00
deepseek-v4-flash-0731	pp1024 @ d4096	3078.23 ± 0.00		1620.76 ± 0.00	1439.46 ± 0.00	1620.76 ± 0.00
deepseek-v4-flash-0731	tg32 @ d4096	62.00 ± 0.00	64.00 ± 0.00
deepseek-v4-flash-0731	pp4096 @ d4096	3169.16 ± 0.00		2366.11 ± 0.00	2184.80 ± 0.00	2366.11 ± 0.00
deepseek-v4-flash-0731	tg32 @ d4096	62.29 ± 0.00	64.31 ± 0.00
deepseek-v4-flash-0731	pp8192 @ d4096	3472.34 ± 0.00		3238.89 ± 0.00	3057.59 ± 0.00	3238.89 ± 0.00
deepseek-v4-flash-0731	tg32 @ d4096	62.38 ± 0.00	64.40 ± 0.00
deepseek-v4-flash-0731	pp1024 @ d8192	3905.12 ± 0.00		2181.75 ± 0.00	2000.45 ± 0.00	2181.75 ± 0.00
deepseek-v4-flash-0731	tg32 @ d8192	62.35 ± 0.00	64.37 ± 0.00
deepseek-v4-flash-0731	pp1024 @ d8192	4197.36 ± 0.00		2069.63 ± 0.00	1888.33 ± 0.00	2069.63 ± 0.00
deepseek-v4-flash-0731	tg100 @ d8192	59.66 ± 0.00	61.00 ± 0.00
deepseek-v4-flash-0731	pp4096 @ d8192	3790.10 ± 0.00		2897.06 ± 0.00	2715.76 ± 0.00	2897.06 ± 0.00
deepseek-v4-flash-0731	tg32 @ d8192	63.61 ± 0.00	65.67 ± 0.00
deepseek-v4-flash-0731	pp4096 @ d8192	3711.75 ± 0.00		2996.95 ± 0.00	2815.65 ± 0.00	2996.95 ± 0.00
deepseek-v4-flash-0731	tg100 @ d8192	59.78 ± 0.00	60.00 ± 0.00
deepseek-v4-flash-0731	pp8192 @ d8192	2668.58 ± 0.00		5393.44 ± 0.00	5212.13 ± 0.00	5393.44 ± 0.00
deepseek-v4-flash-0731	tg32 @ d8192	63.70 ± 0.00	65.76 ± 0.00
deepseek-v4-flash-0731	pp1024 @ d16384	3932.06 ± 0.00		4040.35 ± 0.00	3859.04 ± 0.00	4040.90 ± 0.00
deepseek-v4-flash-0731	tg100 @ d16384	59.69 ± 0.00	61.00 ± 0.00
deepseek-v4-flash-0731	pp4096 @ d16384	3156.95 ± 0.00		5753.77 ± 0.00	5572.47 ± 0.00	5753.77 ± 0.00
deepseek-v4-flash-0731	tg100 @ d16384	62.66 ± 0.00	63.00 ± 0.00
deepseek-v4-flash-0731	pp8192 @ d16384	3952.00 ± 0.00		5502.16 ± 0.00	5320.86 ± 0.00	5502.16 ± 0.00
deepseek-v4-flash-0731	tg100 @ d16384	62.72 ± 0.00	63.00 ± 0.00
`

The code serve_0731.sh:
`
#!/bin/bash

vLLM-Moet serve - DeepSeek-V4-Flash-0731 on 1x RTX PRO 6000 (WSL)

Config: BASE_CACHE mode (planes in host RAM, GPU expert cache) + 2 GiB FP4 pool

Why BASE_CACHE: WSL has ~93 GiB VRAM free. Dense stack (~7.6 GiB) + 2-bit planes

don't fit entirely in VRAM. BASE_CACHE puts planes in host RAM via pack file,

GPU holds a 75 GiB expert cache (101% coverage) + 2 GiB FP4 pool + KV.

STORE_DIR provides persistent quantization cache (base + fp4 packs).

No speculative decoding: checkpoint is DSpark-trained, MTP weights are missing.

DSpark needs ~10 GiB VRAM (3 draft layers) which doesn't fit with 75 GiB base cache.

Expected: ~63 tok/s decode, ~4000+ tok/s prefill.

set -e

====================== Model dir (ext4 VHDX) ======================

VHDX is attached from Windows side by the .bat (wsl --mount --vhd --bare).

Here we mount the block device to /mnt/wsl/moet-pack.

MOUNT_POINT=/mnt/wsl/moet-pack

if mountpoint -q "$MOUNT_POINT" 2>/dev/null; then
echo "[model] Already mounted at $MOUNT_POINT ($(df -h "$MOUNT_POINT" | tail -1 | awk '{print $4}') free)"
else
sudo mkdir -p "$MOUNT_POINT"

# Find the VHDX block device (~520G, not root/swap).
# wsl --mount --bare creates /dev/sdX. Root disk is usually /dev/sde, swap /dev/sdc.
VHD_DEV=""
for dev in /dev/sd{a,b,c,d,e,f,g,h}; do
    [ -b "$dev" ] || continue
    # Skip root disk and swap
    if lsblk "$dev" -o MOUNTPOINTS -n 2>/dev/null | grep -q "^/\|/mnt/wslg"; then
        continue
    fi
    # Check size ~520G
    size=$(lsblk "$dev" -o SIZE -b -n 2>/dev/null || echo 0)
    if [ "$size" -gt 500000000000 ] 2>/dev/null; then
        VHD_DEV="$dev"
        break
    fi
done

if [ -z "$VHD_DEV" ]; then
    echo "[model] ERROR: VHDX block device not found."
    echo "         Run serve_0731.bat as Administrator first."
    echo "         Available block devices:"
    lsblk -o NAME,SIZE,TYPE,MOUNTPOINTS 2>/dev/null
    exit 1
fi

echo "[model] Mounting $VHD_DEV at $MOUNT_POINT ..."
sudo mount -t ext4 "$VHD_DEV" "$MOUNT_POINT"
echo "[model] Mounted ($(df -h "$MOUNT_POINT" | tail -1 | awk '{print $4}') free)"

fi

Verify pack files are present

if [ ! -f "$MOUNT_POINT/config.json" ]; then
echo "[model] ERROR: config.json not found in $MOUNT_POINT"
exit 1
fi

====================== Swap ======================

if ! swapon --show 2>/dev/null | grep -q swapfile-wsl; then
echo "[swap] Activating swapfile-wsl (64 GB)..."
sudo swapon /home/artur/swapfile-wsl 2>/dev/null || echo "[swap] WARNING: could not activate swapfile"
fi
echo "[swap] $(swapon --show 2>/dev/null | tail -n +2 | wc -l) swap devices active"

====================== Activate environment ======================

source "/home/artur/moet-env/bin/activate"

====================== CUDA and environment ======================

export PATH=/usr/local/cuda-13.2/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-13.2/lib64:$LD_LIBRARY_PATH

====================== MoET BASE_CACHE config ======================

BASE_CACHE: 2-bit planes live in host RAM (pack file), GPU holds expert cache + FP4 pool.

VLLM_MOE_W2_BASE_CACHE_GB = GPU expert cache size (GiB) - 101% coverage at 75 GiB.

export VLLM_MOE_W2=1
export VLLM_MOE_W2_BASE_CACHE_GB=75
export VLLM_MOE_W2_DELTA_GB=2
export VLLM_MOE_W2_GATE=0
export VLLM_MOE_W2_FORCE_POOL=1
export VLLM_MOE_W2_CUBIT_DIR=/home/artur/cubit-share
export VLLM_USE_DEEP_GEMM=1

BASE_CACHE replay mode: approximate (don't crash on cold-cache misses).

export VLLM_MOE_W2_REPLAY_MODE=approximate

Allow up to 8 misses per step without replay.

export VLLM_MOE_W2_BASE_MISS_TOL=8

====================== Host stores (pack files on ext4) ======================

export VLLM_MOE_W2_STORE_DIR=/home/artur/moe2-store
export VLLM_MOE_W2_BASE_RAM_GB=20
mkdir -p "$VLLM_MOE_W2_STORE_DIR"

====================== WSL VRAM budget fixes ======================

export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0

====================== WSL workarounds ======================

export VLLM_LOGGING_LEVEL=info
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export TORCHINDUCTOR_COMPILE_THREADS=2
export VLLM_HOST_IP=127.0.0.1
export NCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo

echo "=== vLLM-Moet Server Starting (BASE_CACHE) ==="
echo "Model: DeepSeek-V4-Flash-0731"
echo "vLLM: $(python -c 'import vllm; print(vllm.version)' 2>/dev/null || echo 'unknown')"
echo "GPU: $(nvidia-smi --query-gpu=name --format=csv,noheader | head -1)"
echo "Cubins: $VLLM_MOE_W2_CUBIT_DIR"
echo "Store: $VLLM_MOE_W2_STORE_DIR"
echo "Config: BASE_CACHE=75 GiB (101% cov), delta=2 GiB, arena=20 GiB, no spec, 256K ctx"
echo "======================================================="

====================== Cleanup zombies ======================

pkill -f "vllm serve" 2>/dev/null || true
sleep 1

====================== Launch ======================

BASE_CACHE mode: 256K context, 1 seq, util 0.95.

vllm serve /mnt/wsl/moet-pack
--host 0.0.0.0
--port 8000
--served-model-name deepseek-v4-flash-0731
--trust-remote-code
--kv-cache-dtype fp8
--block-size 256
--max-model-len 262144
--kv-cache-memory-bytes 2881142720
--gpu-memory-utilization 0.95
--max-num-batched-tokens 2048
--max-num-seqs 1
--tokenizer-mode deepseek_v4
--no-scheduler-reserve-full-isl
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}'
--enable-auto-tool-choice
--reasoning-parser deepseek_v4
--tool-call-parser deepseek_v4
`

The code serve_0731.bat (admin):
`
@echo off
REM Launch vLLM-Moet serve DeepSeek-V4-Flash-0731 (BASE_CACHE, no speculative decoding)
REM
REM Steps:
REM 1. wsl --mount VHDX (needs admin - run this .bat as Administrator)
REM 2. Inside WSL: sudo mount /dev/sdX, swap, then vllm serve
REM
REM If VHDX is already mounted in WSL, the mount step is skipped.

REM === Step 1: attach VHDX from Windows (needs admin) ===
wsl -e bash -c "mountpoint -q /mnt/wsl/moet-pack 2>/dev/null && echo MOUNTED || echo NOT_MOUNTED" > "%TEMP%\wsl_mount_check.txt" 2>nul
set /p MOUNT_STATE=<"%TEMP%\wsl_mount_check.txt"
del "%TEMP%\wsl_mount_check.txt" 2>nul

if /I "%MOUNT_STATE%"=="MOUNTED" (
echo [mount] VHDX already mounted in WSL at /mnt/wsl/moet-pack
) else (
echo [mount] Attaching VHDX to WSL...
wsl --mount --vhd "E:\wsl-data\moet-pack.vhdx" --name moet-pack --bare 2>nul
if errorlevel 1 (
echo [mount] VHDX already attached or attach failed - checking if block device exists...
)
echo [mount] VHDX attached. WSL will mount it in the next step.
)

REM === Step 2: run sh-script inside WSL ===
wsl -e bash -lc "bash /mnt/c/Users/Artur/Projects/vLLM-Moet/serve_0731.sh"
`

[jpezzulli]

FP gate = 2
you will have issues with that - you need the full 6 or you get tool calling and structure output issues. the huggingface that was posted here confirmed that as well. He had worse results than me. but I did forget the deep_gemm portion - running a few tests now then will turn that on, may be a big performance improvement. also, my numbers are tg 1024 not 100. i can probably run benchy specifically but I am more concerned with putting this out there once i get a few more items running for people can use it right away in the repo plus container.

fitbit says 1 hour and 42 min of sleep last night. haha.

[lovedheart]

Is Lvllm any faster for something like minimax m3 compared to llama cpp? There's like no performance metrics from that repo so I never bothered to install and test it.

I can paste a speed test from Lsglang which uses the same technique as Lvllm for running original DeepSeek V4 Flash.