diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 8e2a97071..b846d7423 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -19,10 +19,12 @@ a practical unit that one agent can spike without silently dropping aliases. ## Architecture-support checklist At-a-glance view of which architectures we have actually engaged, and how far. -**355 architecture rows are inventoried at the pin, plus 14 rows that the pinned -registry does not contain = 369 architecture rows.** Those 14 are, by why they +**355 architecture rows are inventoried at the pin, plus 16 rows that the pinned +registry does not contain = 371 architecture rows.** Those 16 are, by why they are not at the pin: `KimiK3ForConditionalGeneration` and `MuseGlimmerForConditionalGeneration`, both released after the pin; +`Qwen3_5ForCausalLM` and `Qwen3_5MoeForCausalLM`, the text-only Qwen3.5 arms +upstream added after the pin in PR vllm#50210 (#490); `BailingMoeV3ForCausalLM`, a pin-lag row registered on vLLM `main` and absent only at the pin (#609); `MiniMaxH3DiTModel` and `LTX2VideoTransformer3DModel`, both DIFFUSION architectures released after the @@ -40,12 +42,12 @@ in neither core vLLM `main` nor `vllm-omni` — `VoxtralRealtimeForConditionalGeneration` and `BailingMMNativeForConditionalGeneration` (#610). -The rollup below counts **373 rows**, which is those 369 plus the four that are +The rollup below counts **375 rows**, which is those 371 plus the four that are not architectures at all and say so in their own sections: `MODEL-FACTORY-registry` (the cross-cutting registry contract), the two `MODEL-AUDIO` encoder-component rows, and the one `MODEL-HFDYNAMIC` row (dynamic Transformers compatibility is capability-driven and excluded from finite counts). -Of those 373, 50 are past `INVENTORIED` (engaged) and the remaining 323 are the +Of those 375, 52 are past `INVENTORIED` (engaged) and the remaining 323 are the known-but-not-started long tail — the same two numbers the rollup table states, and the way to re-derive every count here is to re-run [`scripts/check-agent-record.py`](../scripts/check-agent-record.py)'s parser over @@ -70,16 +72,16 @@ Rollup by lifecycle state (must equal the detailed per-state row counts): | State | Rows | |---|---| | INVENTORIED | 323 | -| PARTIAL | 20 | +| PARTIAL | 22 | | ACTIVE | 10 | | SPIKE | 8 | | BLOCKED | 5 | | DONE | 3 | | READY | 3 | | GATING | 1 | -| **Total** | **373** | +| **Total** | **375** | -Engaged architectures (the 50 non-`INVENTORIED` rows): +Engaged architectures (the 52 non-`INVENTORIED` rows): | Support | Architecture | Family / example | Status | Row | |---|---|---|---|---| @@ -87,6 +89,8 @@ Engaged architectures (the 50 non-`INVENTORIED` rows): | ✅ | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B (MoE) | STRICT token-exact 6/6 vs vLLM 0.25.0; 11/16 speed-grid cells at/above graphed vLLM, c1/c2 residual | `MODEL-TEXT-qwen3-moe-qwen3-moe-for-causal-lm` | | ✅ | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B (text path) | text-gen STRICT token-exact 235/235 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS; **M3-W0 landed** (vision-inclusive checkpoint `Qwen/Qwen3.6-27B` 51.7 GiB bf16 with 333 `visual.*` FOUND+fits+downloaded; 27B vision config resolved — depth 27/out 5120/**EMPTY deepstack**; MRoPE `[11,11,10]`/rot 64/theta 1e7; the bf16 GDN-hybrid loader ALREADY handles it). **M3-b LANDED 2026-07-25: image→text STRICT token-exact 32/32 vs vLLM 0.25.0** — Qwen3.6-27B image understanding works end-to-end (forked GDN-hybrid VL forward gated on mm input ⇒ text byte-identical; 27B/35B/Coder inertness re-passed 235/315/138). **M3d LANDED 2026-07-25: video→text STRICT token-exact 32/32 vs vLLM 0.25.0** — video works end-to-end too (`Qwen3_5VLGenerateGreedyVideo` reuses the M3c processor/windowed-tower/video-MRoPE on the GDN-hybrid backbone). **Qwen video modalities COMPLETE: image+video both work e2e** (audio N/A for Qwen). **VISION-FORWARD SPEED (2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`, multimodal-speed.md §16): the mm-forward tower BEATS vLLM** — per-image tower forward 142.3 ms (flash `AttentionDenseFlash`, hd-72) vs vLLM 0.25.0 ~250 ms eager encode = 0.57×; attribution-first nsys REFUTED a bigger lever (the t=784 vision attention is serial-latency-bound, flash only 1.04× over warp), STRICT 32/32 image/video HELD + goldens md5 unchanged. Row stays `PARTIAL` — vision-forward speed BEATS vLLM; **umbrella speed pending** on batched c2+/serving | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | | ✅ | `Qwen3_5MoeForConditionalGeneration` | Qwen3.6-35B-A3B (text path) | text-gen STRICT token-exact 315/315 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS, vision tower pending (M2/M3) so the row is `PARTIAL` (text-only) | `MODEL-MM-qwen3-5-qwen3-5-moe-for-conditional-generation` | +| 🚧 | `Qwen3_5ForCausalLM` | Qwen3.5 text-only dense arm (no checkpoint fits this box) | **REGISTERED, NOT RUN-GATED (2026-08-12, issue #490).** Ahead-of-pin forward port of upstream PR vllm#50210 (`ad5d29db7`): additive registration against the EXISTING dense factory + one backbone weight-namespace decision per checkpoint (`model.` vs `model.language_model.`, mixed index REFUSED). Dispatch, flat-config and namespace tests green; 27B/35B/Coder inert (VL prefix is still the seam default, goldens md5 unchanged). **NO token-exact oracle run and NO speed number — the run gate is OWED**, and no text-only Qwen3.5 checkpoint fits GB10. Unlike the MoE arm, this loader DOES route BF16/FP8/NVFP4 per projection by tensor presence, so a flat bf16 checkpoint is expected to load | `MODEL-TEXT-qwen3-5-qwen3-5-for-causal-lm` | +| 🚧 | `Qwen3_5MoeForCausalLM` | `Qwen/Qwen3.8-2.4T-A95B` (2.4T MoE — DOES NOT FIT this hardware) | **REGISTERED, NOT RUN-GATED (2026-08-12, issue #490).** Same ahead-of-pin port against the EXISTING MoE factory: 3.8 is the token-exact Qwen3.6-35B-A3B GDN-hybrid MoE backbone at larger scale, every knob config-driven, with the BACKBONE weight names identical modulo the backbone prefix. **CORRECTED 2026-08-12 — REGISTRATION IS NOT LOADABILITY.** This loader reads ONLY per-expert NVFP4 routed experts; the published `Qwen/Qwen3.8-2.4T-A95B` and `Qwen/Qwen3.6-35B-A3B` indices ship 3-D STACKED experts and ZERO `weight_scale`/`input_scale` (read live 2026-08-12), and our gated 35B row reads the REQUANTIZED `nvidia/Qwen3.6-35B-A3B-NVFP4` — so **the bf16/stacked MoE routed-expert arm is NOT implemented, is OWED, and such a checkpoint is REFUSED by a message naming it**. **NO token-exact oracle run and NO speed number — the run gate is OWED** (2.4T bf16 ≈ 4.8 TB, FP8 ≈ 2.4 TB vs 128 GB unified) and does NOT close on a fitting checkpoint alone: it needs one whose routed experts are per-expert NVFP4, or the owed arm implemented first. Also owed: MTP and GGUF arms for 3.8. The QUANTIZED (NVFP4) arm is the one that IS implemented | `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` | | 🚧 | `Qwen3VLForConditionalGeneration` | Qwen3-VL-4B-Instruct (image + video vehicle) | **IMAGE e2e WORKING (M2 CLOSED): STRICT image→text token-exact 32/32 vs vLLM 0.25.0** (full pipeline: C++ processor→M2a tower→merge→forked MRoPE/DeepStack greedy decode); correctness complete, **speed pending**. Underpinned by mm processor (M1), vision tower (M2a faithful), M2b/M2c numeric contracts (85/85). **VIDEO (M3c 2026-07-25): preprocessing + full wiring LANDED + unit-gated** — video-processor pixel_values_videos BIT-exact 41/41 (RED-first), video MRoPE positions BIT-exact vs vLLM, per-frame windowed tower faithful (rel-L2 0.072); video e2e **NEAR-TIE-ROBUST PASS** (gate form chosen BY MEASUREMENT 2026-07-25): teacher-forcing vLLM 0.25.0 on OUR exact tokens proves the sole divergence is ONE genuine bf16 near-tie at tok22 (gap **0.125 nats**, our token vLLM's 2nd of 4 tied choices) with every downstream token == vLLM's teacher-forced argmax at gap **0.0** — the tower already accumulates in f32 everywhere (matches vLLM), so the residual is the irreducible bf16 envelope, NOT a fixable numeric gap. VIDEO understanding now WORKS e2e (correctness complete, **speed pending**); image e2e STRICT 32/32 unchanged (the deterministic strict-pass proof, no regression). **VISION-FORWARD SPEED (2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`, multimodal-speed.md §16): the shared vision tower now defaults to the flash-tiled `AttentionDenseFlash` (byte-identical to warp; STRICT image 32/32 HELD, nsys default 4B e2e = 24 flash-kernel inst/zero warp, goldens md5 unchanged); the tower already BEATS vLLM (~250 ms eager encode) — the profile REFUTED a bigger lever.** **OpenAI SERVING (2026-07-28, `CLAIM-MM-SERVING-W1`, [mm-serving.md](specs/mm-serving.md)): first CPU brick landed — the chat request now parses OpenAI multimodal content parts (`image_url`/`input_audio`/`audio_url`) + decodes base64/data-URI + routes to the EXISTING processor → `MultiModalInputs` (gate `test_chat_mm` 5/5, CPU no-weights). This model is the named `MM-SERVE-E2E` closing gate: a real image+prompt OpenAI `/v1/chat/completions` request → token-correct output vs the mm oracle (needs the engine mm plumbing `MM-SERVE-ENGINE` + DGX). Not yet wired into the OpenAI server end-to-end.** **SERVING W2 (2026-07-28, `CLAIM-MM-SERVING-W2`): `MM-SERVE-ENGINE` DONE** — the engine now CARRIES the parsed `MultiModalInputs`: additive `LLMEngine`/`AsyncLLM` `add_request(MultiModalInputs)`+`generate` overloads via `InputProcessor::process_inputs_mm` (mm_features onto `EngineCoreRequest`/`Request`), chat-template placeholder-STRING helpers, serving_chat `MultiModalChatFn` seam (default unset ⇒ text byte-identical); CPU gate `test_input_processor` +2 + `test_chat_mm` +2 (full chain parse→route→engine request with the 196-slot expanded prompt + mm handles). **SERVING W3 (2026-07-28, `CLAIM-MM-SERVING-E2E`): the `MM-SERVE-E2E` CPU SEAM BODY DONE** — `MakeQwen3VLImageChatFn` (chat_mm.{h,cpp}) turns an image chat request into the placeholder-EXPANDED engine input (marker-inject → chat template → `EncodeWithSpecialTokens` single image_pad marker→one image_token_id → `RouteImageRgb` EXPAND to 196 image tokens + mm_features), wired in `examples/server/main.cpp` (guarded on `preprocessor_config.json`; text path byte-identical). Gate `test_chat_mm` 8/8 (RED text path = 0 image tokens) + `test_openai_serving` (production seam invoked + routed). **The GPU e2e forward is ARCHITECTURALLY BLOCKED (not box contention):** the engine model runner has no mm forward — `ModelForwardInput` has no vision field, `runner.cpp` ignores `Request.mm_features`, this arch is UNREGISTERED (`REGISTER_VLLM_MODEL` absent), and the M2c `Qwen3VLGenerateGreedy` is a standalone driver. Fold-into-engine recipe (add a vision-embed field → runner runs the tower via the encoder cache + merge → register the arch → dgx token-exact vs the M2c golden) in `specs/mm-serving.md`. **ENGINE MM-FORWARD LANDED (2026-07-28, `CLAIM-ENGINE-MM-FORWARD`): the block is RESOLVED — `ModelForwardInput` gains an additive default-nullopt `mm` field (merged inputs_embeds + 3-D MRoPE + DeepStack), Qwen3-VL is now `REGISTER_VLLM_MODEL`-registered (`qwen3_vl_registry.cpp`, `supports_multimodal=true`), and the registered forward FOLDS the M2c decode into `ModelRegistry::Forward` via the SHARED `Qwen3VLForwardStepLastLogits` (`Qwen3VLGenerateGreedyViaRegistry` drives every step through `ModelRegistry::Forward`). Text inertness (RED line, shared-path change): `test_runner` 16/16 + `test_scheduler` 36/36 + `test_model_registry` 24/24 + `test_chat_mm` 8/8 + `test_openai_serving` 41/41 all green (nullopt-for-text ⇒ byte-identical). Engine mm-forward token-exact gate `test_qwen3vl_registry_e2e` (image→text THROUGH `ModelRegistry::Forward` == M2c golden STRICT, dgx.casa GB10). Residual: the in-runner scheduler-fed tower run + the full server `/v1/chat/completions` GPU e2e.** | `MODEL-MM-qwen3-vl-qwen3-vlfor-conditional-generation` | | 🚧 | `VoxtralForConditionalGeneration` | Voxtral-Mini-3B-2507 (AUDIO→text, audio-track A3) | **FIRST e2e AUDIO UNDERSTANDING: audio→text vs vLLM 0.25.0** — A2 Whisper-large-v3 encoder at Voxtral config (128 mel/1280/32L/head_dim 64) + AudioLanguageAdapter projector + masked-scatter merge into the LANDED Mistral/Llama decoder (untied lm_head, mistral-consolidated q/k rope-permute) → forked greedy. Gate form BY MEASUREMENT: vLLM greedy K=5 deterministic ⇒ STRICT is the bar; STRICT prefix 33/48 exact vs vLLM greedy, then the ratified near-tie-robust gate PASSES (worst teacher-forced gap 0.0 nats, sole branch = a 4-way bf16 tie at -2.069 nats) — bit-exact infeasible (encoder uses different bf16 GEMM/attn kernels). Decoder proven token-exact (ref-audio→48/48). Additive driver/loader gated on audio ⇒ Mistral text byte-identical. **Speed pending** (decode now graph-captured via `VoxtralDecodeGraph`, lever #3 W1 2026-07-27: small real win 60.94 vs 61.71 ms/tok, still ~1.49× vs vLLM 40.8 ms). **Decode-kernel efficiency ATTRIBUTED + VALIDATED ceiling (2026-07-27, `CLAIM-MM-SPEED-DECODE-KERN`, multimodal-speed.md §11):** the whole ~20 ms/tok residual is the naive scalar `PagedAttentionKernel` decode attention (723 µs × 30 layers = 21.7 ms/step, ~120× the KV floor); the 1:1 vLLM lever (FA2 `flash_attn_varlen` decode) is already in-binary, gated off only because the driver's single KV block (444) isn't ÷16. `block_size÷16` → decode via FA2: TPOT **59.4→38.2 ms/tok (−21.2, ~36%) = 0.94× vLLM 40.8 ms — BEATS parity**, and the FA2 sequence is a VALID vLLM greedy branch (teacher-force PASS, 0 divergences, gap 0.0). But it flips the committed near-tie golden's exact-tie branch (`repro` 48→18) → blocked byte-exact (RED line). bf16 near-tie / golden-pinning ceiling; RECORDS-ONLY (14/14 held, golden unchanged); win reachable via `block_size÷16` + golden regen. **ADOPTED 2026-07-27 (USER-APPROVED, `CLAIM-MM-SPEED-DECODE-KERN-ADOPT`, multimodal-speed.md §12): FA2 decode SHIPS as the Voxtral default — audio DECODE now BEATS vLLM (0.97×), the LAST mm decode-speed gap CLOSED.** One-line `block_size÷16` routes decode via FA2 `LaunchDecodeVarlenFA2Bf16` (nsys: `flash_fwd_splitkv` 1410 @ 18.5 µs, ZERO `PagedAttentionKernel`). `test_voxtral_e2e` converted to the ratified near-tie DISTRIBUTIONAL gate (binding = teacher-force PASS, KERNEL-INDEPENDENT; strict prefix exact to the first bf16 tie — FA2 takes the other side of the pos-18 2-way EXACT tie ⇒ prefix 18, `>=18`; determinism anchor to the FA2 seq). `voxtral_neartie.json` md5 `3d199c2d…`→`937b9ad3…`; STRICT golden `voxtral_golden.json` `8ab87b7e…` UNCHANGED. **Gate PASS 16/16**; teacher-force vLLM 0.25.0 on the FA2 seq = **0 divergent, worst gap 0.0000 nats, PASS**. Capture-safe (graph S=1 + 46 replays valid; compute-sanitizer **0 errors** on the graphed-FA2-decode path; 3 runs byte-identical) ⇒ ships as the DEFAULT graph path. **A/B (same-binary, 6 reps rep0 dropped, steady-state):** scalar 60.50 → FA2 **39.50 ms/tok** (−21.0, ~35%, NON-OVERLAPPING) = **0.97× vLLM 40.8 ms — BEATS**. Audio DECODE now correctness- AND speed-DONE; row stays `PARTIAL`/`ACTIVE` — umbrella DONE bar has audio TTFT (32-layer Whisper encoder, UNMEASURED vs vLLM 43 ms) + c2+ batched / `audio_url` serving ingestion still open (same as image/video). **ENCODER TTFT MEASURED + warp-attention brick 2026-07-27 (`CLAIM-MM-SPEED-AUDIO-ENC`, multimodal-speed.md §13):** the Whisper encoder ran the naive `kAttention` (O(t²) per-key block-`__syncthreads`) — routed the encoder self-attention (hd-64, non-causal) to the warp-scoped `vt::AttentionDenseFast` (§7's vision-tower fix; `kAttention` untouched ⇒ text byte-identical). Encoder forward **8870→1890 ms (4.7×, NON-OVERLAPPING)**; `test_voxtral_e2e` **16/16** with ZERO token flips (naive arm also 16/16 same tokens; goldens md5 UNCHANGED), proof-of-run nsys `AttentionWarpKernel` 32 inst / zero naive. **NOT at parity:** ~1.89 s vs vLLM's 43 ms (~44×) — the warp kernel is STILL 31.8 ms/layer (O(t²), memory-bound on redundant K/V reads); closing needs a flash-TILED non-causal hd-64 attention (LARGE) + resident one-time encoder weights (MEDIUM, byte-exact). **ENCODER FLASH-TILED KERNEL LANDED 2026-07-28 (`CLAIM-MM-SPEED-AUDIO-ENC-KERNEL`, multimodal-speed.md §14):** §13 lever #1 done — new `vt::AttentionDenseFlash` (`OpId::kAttentionDenseFlash`, `cuda_ops.cu`) shared-memory-TILES K/V across a block of `kFlashBr=16` query-warps (FA2 K/V tiling, structure-ported from vendored `flash_fwd_kernel.h:52`), per-warp math copied verbatim from `AttentionWarpKernel` ⇒ BIT-IDENTICAL; encoder routed to it by default (`VT_WHISPER_ENC_WARP=1`/`EAGER=1` A/B); `kAttention`/`kAttentionDenseFast` untouched ⇒ text/vision byte-identical. **16/16** default-flash, flash/warp/eager token md5-IDENTICAL (`89923566…`) ⇒ ZERO flips, goldens md5 UNCHANGED, nsys proof `AttentionDenseFlashKernel` 32 inst / zero warp+naive, RED confirmed, compute-sanitizer 0, 3 runs identical. **A/B same-binary:** attention **35.11→19.29 ms/layer (1.82×)**, encoder forward **~1834→~1375 ms (1.33×)**. **NOT at parity:** ~1.37 s vs vLLM ~43 ms (~32×, was ~44×) — the scalar warp-per-query recurrence is now serial-latency-bound (L2 already served the redundant reads ⇒ 1.8× not 16×); gap-closer = tensor-core MMA hd-64 non-causal FA2 (LARGE) + resident encoder weights (MEDIUM). **ENC lever #2 LANDED 2026-07-28 (`CLAIM-MM-SPEED-AUDIO-ENC-RESIDENT`, multimodal-speed.md §15): device-resident one-time encoder weights** (mirror the Qwen `d_dev` residency seam; each of the 487 encoder weight tensors f32→bf16-converted + H2D-uploaded ONCE into a `mutable shared_ptr` handle, reused across forwards). BYTE-EXACT (moves data, not math): `test_voxtral_e2e` **16/16**, goldens md5 UNCHANGED (`8ab87b7e…`/`937b9ad3…`), compute-sanitizer 0 errors. **Proof-of-run/RED (`VT_WHISPER_ENC_REMARSHAL`):** nsys HtoD resident 740 ops/9.4 GB vs re-marshal 1714 ops/11.9 GB (−974 ops, −2.5 GB). **A/B same-binary (`flock`, 6 reps rep0 dropped):** encoder forward **~1377→~729 ms (−648 ms, 1.89×)** — removes the confirmed ~0.75 s per-call host marshalling. Trajectory 1834→1375→729 ms; vs vLLM ~43 ms now ~17× (was ~32×). NOT at parity — residual 729 ms is GPU-compute-bound (scalar warp attention); LARGE gap-closer remains tensor-core MMA hd-64 FA2. Audio TTFT/encoder stays speed-pending | `MODEL-MM-voxtral-voxtral-for-conditional-generation` **ENCODER TTFT lever #1 MEASURED 2026-08-12 (`CLAIM-MM-SPEED-AUDIO-ENC-FA2`, S17, #432):** the vendored FA-2 forward on tensor cores (new dense hd-64 non-split instantiation + `vt::AttentionDenseFa2`) takes the encoder attention 19,278 -> 166.5 us/layer (**115.8x**) and the encoder forward 731.7 -> 133.0 ms (**5.50x**). Restated against the PIN (`555967922`, TTFT median 46.02 ms, production/graphed, re-measured 2026-08-12): the ENCODER FORWARD goes from 15.90x to **2.89x** of vLLM's whole TTFT. That is NOT a TTFT ratio - our projector, merge and prefill are unmeasured and absent from the numerator (review finding F3); the earlier `~17.1x -> ~3.11x` used the carried-forward 0.25.0 denominator of 42.8 ms, so the published figure was CONSERVATIVE. It LANDS OPT-IN (`VT_WHISPER_ENC_FA2=1`), NOT default: the oracle teacher-force PASSES the ratified band (worst 0.125 nats, over-band 0) but with 3 divergent positions where the shipping kernel has 0 at gap 0.0. **WHY is a HYPOTHESIS:** the stated cause (bf16 P before the PV MMA) was REFUTED by mutation M4 - the same rounding forced into the scalar kernel left the tokens unchanged. Default path + goldens UNCHANGED, `test_voxtral_e2e` 16/16; the op now has unit tests (`tests/vt/test_ops_attention_dense_fa2.cpp`). ADOPTION = developer decision; audio TTFT stays speed-pending. | | 🚧 | `ParakeetForCTC` (encoder + CTC head) | Parakeet / FastConformer ASR: an audio COMPONENT, not a registry arch (vLLM wraps transformers' `ParakeetEncoder` inside `nano_nemotron_vl.py`) | P4 LANDED 2026-08-07 on CPU: encoder + CTC greedy + log-mel front end, gated stage-by-stage against a dumped HuggingFace `ParakeetForCTC` oracle (rel-L2 <= 3.5e-7) with greedy and collapsed token ids EXACT, plus an independent in-test reference and 20/20 mutants killed. Real transcript verified 2026-08-07 on `nvidia/parakeet-ctc-0.6b` AND `-ctc-1.1b`; no CUDA provider, no GPU suite, no aarch64, no speed number. **ONE-SURFACE ROW 1 (2026-08-07): NOW a registered arch** — `ParakeetForCTC` self-registers (BEYOND-PIN: pinned vLLM has Parakeet only as the NemotronH audio component, registry.py:511-513) with the SupportsTranscription-only ModelInfo; text paths refuse by task; served via `vllm_transcribe` (ABI v11) + `/v1/audio/transcriptions`; example = thin `vllm.h` client, transcript byte-identical to pre-fold | `MODEL-AUDIO-PARAKEET-ENCODER` | @@ -306,6 +310,8 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-TEXT-telechat2-tele-chat2-for-causal-lm` | `TeleChatForCausalLM`, `TeleChat2ForCausalLM` | `registry.py:203-204`; `vllm/model_executor/models/telechat2.py::TeleChat2ForCausalLM` | causal generation / text | model loader/forward | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-teleflm-tele-flmfor-causal-lm` | `TeleFLMForCausalLM` | `registry.py:206`; `vllm/model_executor/models/teleflm.py::TeleFLMForCausalLM` | causal generation / text | model loader/forward | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-zamba2-zamba2-for-causal-lm` | `Zamba2ForCausalLM` | `registry.py:207`; `vllm/model_executor/models/zamba2.py::Zamba2ForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; Mamba/SSM state | ☐ required | `INVENTORIED` | none | unassigned | +| `MODEL-TEXT-qwen3-5-qwen3-5-for-causal-lm` | `Qwen3_5ForCausalLM` | **BEYOND-PIN — NOT in `555967922`** (the text-only Qwen3.5 arms landed upstream AFTER our parity pin, in PR [vllm#50210](https://github.com/vllm-project/vllm/pull/50210) at `ad5d29db7`, which registers both against the SAME `qwen3_5` module: upstream `registry.py:202-203` @ `ad5d29db7`). Deliberately NOT written as a pinned module/class target — the pin-derived static invariants below describe `555967922`, and a post-pin arm must not silently inflate them, the same convention `MuseGlimmerForConditionalGeneration` and `KimiK3ForConditionalGeneration` follow. Upstream class: `qwen3_5.py:439-440` @ `ad5d29db7` (`class Qwen3_5ForCausalLM(Qwen3_5ForCausalLMBase): pass` — the shared base UNCHANGED) | causal generation / text | model loader/forward; Mamba/SSM state; GDN/linear-attention state; paged attention | ✅ [Qwen3.5/3.8 text-only arms](specs/qwen38-text-only.md) | `PARTIAL` | **LANDED 2026-08-12 (`MODEL-QWEN38-TEXT-ONLY`, issue #490): the arch loads; NO token gate exists and one is OWED.** Additive registration against the EXISTING dense factory — no forward, no KV-cache spec, no loader fork: `src/vllm/model_executor/models/qwen3_5_dense.cpp:256` over the text-only `_ModelInfo` `src/vllm/model_executor/models/qwen3_5_common.h:42` (hybrid YES, multimodal NO, mirroring `Qwen3_5ForCausalLMBase`'s IsHybrid-without-SupportsMultiModal). The backbone weight namespace is resolved ONCE per checkpoint — `include/vllm/model_executor/models/qwen3_5_weights.h:524`, `src/vllm/model_executor/models/qwen3_5_weights.cpp:597` — and threaded through the dense loader at `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp:675`, mirroring upstream's single `WeightsMapper(orig_to_new_prefix={"model.language_model.": "model."})` (`qwen3_5.py:296-300` @ `ad5d29db7`) rather than a per-lookup fallback; a MIXED index is REFUSED. Tests `tests/vllm/models/test_qwen3_8_text_only.cpp:586` (dispatch), `:633` (published flat 3.8 config), `:718` (namespace resolution + mixed refusal), `:769` (the dense loader reads byte-identical weights through either namespace), `:1008` (the VL seam DEFAULT driven with the prefix argument omitted). Registry contract re-pinned at `tests/vllm/models/test_model_registry.cpp:150` (40 archs). **RUN GATE OWED:** no text-only Qwen3.5 checkpoint fits GB10 (2.4T bf16 ≈ 4.8 TB, FP8 ≈ 2.4 TB), so there is NO token-exact oracle run and NO speed number; the row may not advance on dispatch and mapping tests alone. The DENSE gate closes when a fitting `Qwen3_5ForCausalLM` checkpoint appears — this loader routes BF16/FP8/NVFP4 per projection by tensor presence, so a flat bf16 checkpoint is expected to load. MTP and GGUF arms for 3.8 are NOT implemented and are owed; the sibling MoE row additionally owes the bf16/stacked expert arm (matrix row below, porting-inventory §9 deviation 17(e)). | unassigned | +| `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` | `Qwen3_5MoeForCausalLM` | **BEYOND-PIN — NOT in `555967922`** (the text-only Qwen3.5 arms landed upstream AFTER our parity pin, in PR [vllm#50210](https://github.com/vllm-project/vllm/pull/50210) at `ad5d29db7`, which registers both against the SAME `qwen3_5` module: upstream `registry.py:202-203` @ `ad5d29db7`). Deliberately NOT written as a pinned module/class target — the pin-derived static invariants below describe `555967922`, and a post-pin arm must not silently inflate them, the same convention `MuseGlimmerForConditionalGeneration` and `KimiK3ForConditionalGeneration` follow. Upstream class: `qwen3_5.py:439-440` @ `ad5d29db7` (`Qwen3_5MoeForCausalLM(Qwen3_5ForCausalLMBase, QwenNextMixtureOfExperts)` = that same base plus `set_moe_parameters()`, `qwen3_5.py:443-449` @ `ad5d29db7`) | causal generation / text | model loader/forward; FusedMoE/grouped GEMM; Mamba/SSM state; GDN/linear-attention state; paged attention | ✅ [Qwen3.5/3.8 text-only arms](specs/qwen38-text-only.md) | `PARTIAL` | **LANDED 2026-08-12 (`MODEL-QWEN38-TEXT-ONLY`, issue #490): the arch loads; NO token gate exists and one is OWED.** Additive registration against the EXISTING MoE factory: `src/vllm/model_executor/models/qwen3_5_moe.cpp:231` over `src/vllm/model_executor/models/qwen3_5_common.h:42`. Motivating checkpoint `Qwen/Qwen3.8-2.4T-A95B` declares `Qwen3_5MoeForCausalLM` / `qwen3_5_moe_text` and is the token-exact Qwen3.6-35B-A3B GDN-hybrid MoE backbone at larger scale — hidden 2048→8192, layers 40→92, experts 256→512, top-k 8→10 — every knob config-driven, and the published safetensors names are IDENTICAL modulo the backbone prefix (3D-stacked `mlp.experts.gate_up_proj`, `mlp.shared_expert_gate.weight`, top-level `lm_head`). One namespace decision per checkpoint threaded through the MoE loader AND its deferred per-layer expert closure at `src/vllm/model_executor/models/qwen3_5_weights.cpp`. Tests `tests/vllm/models/test_qwen3_8_text_only.cpp:586` (dispatch), `:633` (published flat config), `:718` (namespace resolution + mixed refusal), `:836` (byte-identical MoE loads on BOTH expert-residency paths), `:922` (the unimplemented-layout REFUSAL below), `:1008` (the VL seam DEFAULT driven with the prefix argument omitted); 27B/35B/Coder inertness by construction (the VL prefix stays the seam default, `include/vllm/model_executor/models/qwen3_5_weights.h`) with parity goldens md5-unchanged. **CORRECTED 2026-08-12 (review FAIL on records honesty): REGISTRATION IS NOT LOADABILITY, and the debt was recorded INVERTED.** The earlier text said the quantized arm was owed; it is the ONLY one implemented. `LoadQwen3_5Moe` reads exclusively per-expert NVFP4 (`LoadMoeExpertsInto` -> `LoadNvfp4Raw`: `U8` weight + `F8_E4M3` `.weight_scale` + `.weight_scale_2`), with no stacked and no bf16 branch. Read live 2026-08-12, `Qwen/Qwen3.8-2.4T-A95B` publishes 93x `mlp.experts.gate_up_proj` + 93x `.down_proj` (3-D STACKED) and ZERO `weight_scale`/`input_scale`; `Qwen/Qwen3.6-35B-A3B` is the same under the VL prefix; our gated 35B row reads the requantized `nvidia/Qwen3.6-35B-A3B-NVFP4`. **The bf16/stacked MoE routed-expert arm is NOT implemented and is OWED**, and such a checkpoint is now REFUSED by a message naming it (`CheckMoeExpertLayoutSupported`) instead of dying on `expected U8 for lm_head.weight`. **RUN GATE OWED:** the 2.4T checkpoint cannot be executed on this hardware, so NO token-exact oracle run and NO speed number exist — and the MoE gate does NOT close on a fitting checkpoint alone, since a fitting PUBLISHED MoE checkpoint would still be refused; it needs one whose routed experts are per-expert NVFP4, or the owed arm implemented first. Also owed: MTP and GGUF arms for 3.8. See porting-inventory §9 deviation 17(e). | unassigned | ## MODEL-EMBED - Embedding diff --git a/.agents/porting-inventory.md b/.agents/porting-inventory.md index 7f5c928e4..f5a31c931 100644 --- a/.agents/porting-inventory.md +++ b/.agents/porting-inventory.md @@ -142,7 +142,10 @@ implementation targets, 261 modules) plus a dynamic Transformers-compatible path. The generic ordered architecture-to-type-erased- factory contract is now implemented for the two architectures whose text paths exist locally: `Qwen3_5ForConditionalGeneration` and -`Qwen3_5MoeForConditionalGeneration`. Live loading consumes the full +`Qwen3_5MoeForConditionalGeneration` — plus, since 2026-08-12, their TEXT-ONLY +arms `Qwen3_5ForCausalLM` / `Qwen3_5MoeForCausalLM`, which are two further +registrations over those SAME two factories rather than new implementations (§9 +deviation 17: ahead-of-pin, run gate OWED). Live loading consumes the full `config.architectures` list in order and mirrors pinned unknown, previously-supported, and out-of-tree rejection; unimplemented IDs remain unsupported rather than being misclassified by `num_experts`. The execution row @@ -153,7 +156,7 @@ Python lazy-import/subprocess caching remain explicitly deferred. | Family | Marquee members | Needs | Tier | |---|---|---|---| -| **Qwen3.5/3.6 hybrid (incl. MoE)** | `Qwen3_5ForConditionalGeneration` (27B dense-hybrid, **VL multimodal** wrapper — vision_config present), `Qwen3_5MoeForConditionalGeneration` / `qwen35moe` (35B-A3B) | GDN layers ×3 : 1 gated full-attn (qk-norm, partial RoPE 64d, output gate), MoE 256e top-8 + shared expert (35B) / **dense SwiGLU MLP** (27B), GemmaRMSNorm-style `(1+w)` — ✅ `25326fc` (35B forward correctness-grade, **safetensors**; 16/16 greedy on GB10 = M0 exit; GGUF k-quant load M0.10). **27B (co-equal gate):** CPU-first scaffolding started — arch/quant surveyed (`.agents/specs/qwen27b-w4a4-notes.md`: dense hybrid, W4A4, **and a VL wrapper — text path first, ViT deferred**), CPU W4A4 emulation reference + skipping greedy-parity gate landed; dense loader + single-seq `ForwardDense` + batched PAGED `Qwen3_5DenseModel::Forward` (paged==dense CPU-anchored) + `GPUModelRunner` dense route all landed CPU-green; W4A4 GPU GEMM + oracle golden pending (GPU-gated). **Ordinary plain-BF16 leaf (`LOAD-SAFETENSORS-DIRECT-DENSE`):** raw-NK stacked/tied 4B loading plus layer-bounded discrete-CUDA staging is implemented; real-weight CPU load passes 1656/1656 and local AOT-CUDA retained-host/direct-device tokens match at 1664/1664. The corrected AOT series completes 18/18 legs: ON/OFF/vLLM total **6155.10/6064.06/6730.46 tok/s**, ON=OFF 128/128, peak PSS **2.405/8.571/7.569 GiB**; current ON is 0.9316x the previous AOT result. Current-v0.25 oracle/sanitizer, strict VRAM and external 27B/35B regressions remain open, so this is not a support claim. serving M1–M3 | **T0 (the gate)** | +| **Qwen3.5/3.6 hybrid (incl. MoE)** | `Qwen3_5ForConditionalGeneration` (27B dense-hybrid, **VL multimodal** wrapper — vision_config present), `Qwen3_5MoeForConditionalGeneration` / `qwen35moe` (35B-A3B), plus the TEXT-ONLY arms `Qwen3_5ForCausalLM` and `Qwen3_5MoeForCausalLM` (`qwen3_5_text` / `qwen3_5_moe_text`; motivating checkpoint `Qwen/Qwen3.8-2.4T-A95B`) | GDN layers ×3 : 1 gated full-attn (qk-norm, partial RoPE 64d, output gate), MoE 256e top-8 + shared expert (35B) / **dense SwiGLU MLP** (27B), GemmaRMSNorm-style `(1+w)` — ✅ `25326fc` (35B forward correctness-grade, **safetensors**; 16/16 greedy on GB10 = M0 exit; GGUF k-quant load M0.10). **27B (co-equal gate):** CPU-first scaffolding started — arch/quant surveyed (`.agents/specs/qwen27b-w4a4-notes.md`: dense hybrid, W4A4, **and a VL wrapper — text path first, ViT deferred**), CPU W4A4 emulation reference + skipping greedy-parity gate landed; dense loader + single-seq `ForwardDense` + batched PAGED `Qwen3_5DenseModel::Forward` (paged==dense CPU-anchored) + `GPUModelRunner` dense route all landed CPU-green; W4A4 GPU GEMM + oracle golden pending (GPU-gated). **Ordinary plain-BF16 leaf (`LOAD-SAFETENSORS-DIRECT-DENSE`):** raw-NK stacked/tied 4B loading plus layer-bounded discrete-CUDA staging is implemented; real-weight CPU load passes 1656/1656 and local AOT-CUDA retained-host/direct-device tokens match at 1664/1664. The corrected AOT series completes 18/18 legs: ON/OFF/vLLM total **6155.10/6064.06/6730.46 tok/s**, ON=OFF 128/128, peak PSS **2.405/8.571/7.569 GiB**; current ON is 0.9316x the previous AOT result. Current-v0.25 oracle/sanitizer, strict VRAM and external 27B/35B regressions remain open, so this is not a support claim. **TEXT-ONLY arms (`MODEL-TEXT-qwen3-5-qwen3-5-for-causal-lm`, `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm`, issue #490, both `PARTIAL`):** additive registration against the existing dense/MoE factories plus a once-per-checkpoint backbone-namespace resolution (`model.language_model.` or flat `model.`, mixed REFUSED); AHEAD OF THE PIN at `ad5d29db7` / vllm#50210 and the **run gate is OWED** — no Qwen3.8 checkpoint fits GB10, so nothing about generated tokens, memory or speed is claimed. **The bf16 / 3-D-stacked MoE routed-expert arm is NOT implemented and is OWED** (the published MoE repos ship exactly that layout; only an NVFP4 requant loads), so the MoE gate does not close on a fitting checkpoint alone; such a checkpoint is refused by name (§9 deviation 17(e); [spec](specs/qwen38-text-only.md)). serving M1–M3 | **T0 (the gate)** | | Dense decoders | Llama 3.x, Qwen2/3, Mistral, Gemma 2/3, Phi | GQA + RoPE + SwiGLU + RMSNorm (subset of T0 layer set) | T1 | | MoE decoders | Mixtral, Qwen3-MoE (30B-A3B), GLM-4-MoE, OLMoE | FusedMoE 🚧 `65788b3` (correctness-grade eager; grouped-GEMM perf M2.2) | T1 | | Qwen3-Next | `Qwen3NextForCausalLM` | same stack, interleaved-GQA weight layout | T1 | @@ -1605,6 +1608,88 @@ Examples: `examples/cli` ✅ (C-API client), `examples/server` ✅ (OpenAI serve module; f32, an annotated escape — it runs once per load and its output is narrowed to the stream dtype on upload). Owner: the LTX-2.5 row. +17. **Ahead-of-pin forward port: the Qwen3.5 TEXT-ONLY arms are anchored on a + POST-PIN upstream commit (2026-08-12, + `MODEL-TEXT-qwen3-5-qwen3-5-for-causal-lm` + + `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm`, issue + [#490](https://github.com/mudler/vllm.cpp/issues/490)).** Our parity pin is + `555967922` (2026-07-26), whose `vllm/model_executor/models/registry.py` + carries only `Qwen3_5ForConditionalGeneration` and + `Qwen3_5MoeForConditionalGeneration`. The text-only arms + `Qwen3_5ForCausalLM` / `Qwen3_5MoeForCausalLM` arrived upstream AFTERWARDS, + in [vllm#50210](https://github.com/vllm-project/vllm/pull/50210) at + `ad5d29db7` (a descendant of the pin). Every `file:line` this row cites — + `registry.py:202-203`, `qwen3_5.py:296-300`, `qwen3_5.py:439-449` — therefore + points at a **post-pin head, not at the pin**, which is a deliberate + exception to "port from the pinned oracle". Same shape as deviation 16 above + and taken for the same reason: at the pin the feature does not exist at all, + so there is nothing there to mirror. It is recorded here, and argued for in + the commit that introduced it, because no checker enforces the anchor rule. + Consequences, all binding while this stands: + (a) this row does **not** advance the pin and reconciles nothing else in the + `555967922..ad5d29db7` range — the next [upstream-sync](upstream-sync.md) + cycle reconciles it deliberately, and until then a Qwen3.5 change must check + both anchors; (b) **no token or speed axis is claimable for + `Qwen/Qwen3.8-2.4T-A95B`** — 2.4T bf16 is ~4.8 TB and the released FP8 + variant ~2.4 TB against GB10's 128 GB unified, with no smaller Qwen3.8 + sibling, so the run gate is OWED (both rows are `PARTIAL`, never `DONE`, and + the gap is recorded in [BENCHMARKS](../docs/BENCHMARKS.md) §Open gaps). The + DENSE gate closes when a `Qwen3_5ForCausalLM` checkpoint that fits GB10 + appears; **the MoE one does not** — per (e) below, a fitting PUBLISHED MoE + checkpoint would still be refused at load, so the MoE gate needs a fitting + checkpoint whose routed experts are per-expert NVFP4, or (e) implemented + first; (c) **one behavior deliberately diverges from upstream, in the + strict direction.** Upstream normalizes the two weight namespaces with + `WeightsMapper(orig_to_new_prefix={"model.language_model.": "model."})`, so a + checkpoint carrying backbone tensors under BOTH spellings LOADS upstream, its + names silently rewritten; `ResolveQwen3_5BackbonePrefix` + (`src/vllm/model_executor/models/qwen3_5_weights.cpp:597-615`) REFUSES it. + The reason is that our loaders resolve the namespace ONCE and a mixed index + is the one input where a per-lookup mapper binds half a model from each + namespace and still appears to succeed — a wrong model that loads, which no + name-mapping test can see. A refusal is strictly safer than upstream's + rewrite and cannot make a checkpoint upstream rejects load here, so it is + tracked rather than reconciled; it is revisited if upstream ever publishes a + genuinely mixed index; (d) **the published config's dtype key is NOT + consumed, tree-wide.** `Qwen/Qwen3.8-2.4T-A95B` is transformers 4.57.3 and + spells the model dtype `dtype`; `hf_config.cpp:520-522` reads only the legacy + `torch_dtype`, so `HfConfig::torch_dtype` is EMPTY on this document. Inert + today — nothing in the tree reads that field — and deliberately NOT fixed on + this row, because consuming `dtype` is a behavior change on every model and + owes its own row, RED-first test and inertness proof. Pinned by an assertion + in `tests/vllm/models/test_qwen3_8_text_only.cpp` so it cannot drift + silently; (e) **the bf16 / 3-D-STACKED MoE ROUTED-EXPERT ARM IS NOT + IMPLEMENTED, and this was recorded INVERTED until 2026-08-12.** The earlier + text here, and on every other surface, said "MTP, quantized and GGUF arms for + 3.8 are NOT implemented and are recorded as owed". The QUANTIZED arm is the + only one that IS implemented: `LoadQwen3_5Moe` routes every routed expert + through `LoadMoeExpertsInto` + (`src/vllm/model_executor/models/qwen3_5_weights.cpp:519-530`) into + `LoadNvfp4Raw` (`:433-462`), which hard-requires per-expert + `experts...weight` = `U8`, `.weight_scale` = `F8_E4M3` and + `.weight_scale_2`. There is **no stacked branch and no bf16 branch** — + unlike `gemma4_weights.cpp:326`, which dispatches between layouts. Read live + 2026-08-12, the published indices have neither shape: + `Qwen/Qwen3.8-2.4T-A95B` has 93x `mlp.experts.gate_up_proj` + 93x + `.down_proj` (3-D stacked) and ZERO names matching `weight_scale` or + `input_scale`, with `lm_head.weight` alone; `Qwen/Qwen3.6-35B-A3B` is the + same under the VL prefix. Our gated 35B row reads the REQUANTIZED + `nvidia/Qwen3.6-35B-A3B-NVFP4`, so this loader **has never read a published + Qwen bf16 MoE repo**. What is owed is therefore the stacked/bf16 MoE expert + arm (with the bf16 shared expert, the FP8-less attention tower and the bf16 + `lm_head` on that path), which needs its own row, spec, RED-first test and + NVFP4 inertness proof. Until it exists such a checkpoint is REFUSED by name + (`CheckMoeExpertLayoutSupported`, same file), per AGENTS.md §Shared seams. + **The DENSE arm is NOT affected and the asymmetry is deliberate record:** + `LoadQwen3_5Dense` routes BF16 vs FP8 vs NVFP4 per projection by tensor + presence (`qwen3_5_dense_weights.cpp:354-360,472-503`) and routes the head by + dtype (`LoadDenseLmHead` / `LoadLmHeadAnyDtype`, `:215-233,515-547`), so it + may genuinely load a flat bf16 checkpoint. The whole deviation is discharged + by a pin advance that includes `ad5d29db7`, at which point (a) and the + anchors become ordinary mirrored behavior and only (c), (d) and (e) survive + as tracked items. Scope and gates: + [qwen3.8 text-only spec](specs/qwen38-text-only.md). + ## 10. E2E test suites (T0 deliverable) 1. **Op parity**: golden dumps from upstream vLLM (Python, test-time only) → diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index c7d09a647..7aca46f69 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -127,6 +127,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#412](https://github.com/mudler/vllm.cpp/issues/412) | `MODEL-MM-muse-glimmer-muse-glimmer-for-conditional-generation` | Muse Glimmer config defaults are neutral values, not the architecture's constants: the released GGUF's sandwich post-norms ran at 1e-5 instead of 1e-8 | bug | | [#489](https://github.com/mudler/vllm.cpp/issues/489) | `MODEL-MM-qwen3-5-qwen3-5-moe-for-conditional-generation` | GDN `output_gate_type` is never parsed: a "sigmoid" checkpoint silently computes silu gating, and no token gate can see it | bug | | [#609](https://github.com/mudler/vllm.cpp/issues/609) | `MODEL-TEXT-bailing-moe-v3-bailing-moe-v3-for-causal-lm` | `BailingMoeV3ForCausalLM` (recipe `inclusionAI/Ling-3.0-flash`) is registered on vLLM `main` but absent at the pin and had no model-matrix row. The issue's other architecture, `Qwen3_5MoeForCausalLM`, is owned by [#490](https://github.com/mudler/vllm.cpp/issues/490) / PR #601, which registers it rather than only rowing it | bug | +| [#490](https://github.com/mudler/vllm.cpp/issues/490) | `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` | Qwen3.5/3.8 text-only checkpoints cannot load: `Qwen3_5MoeForCausalLM` unregistered and the loader hardcodes the VL `model.language_model.` prefix. Registration + one-per-checkpoint namespace resolution LANDED 2026-08-12 (sibling row `MODEL-TEXT-qwen3-5-qwen3-5-for-causal-lm`); the issue stays OPEN because the RUN GATE is owed — no text-only Qwen3.5 checkpoint fits this hardware, AND (corrected 2026-08-12) the published MoE repos ship 3-D stacked, unquantized experts that this loader does not implement, so the MoE gate does not close on a fitting checkpoint alone; that arm is owed and is refused by name | feature | | [#298](https://github.com/mudler/vllm.cpp/issues/298) | `PERF-27B-LMHEAD-DSR` | `main` is RED on the `device-leakage` DSR ratchet: `PrepareLmHeadResident` added a 33rd `vt_ifdef` to the shared layer | bug | | [#213](https://github.com/mudler/vllm.cpp/issues/213) | `PERF-27B-LMHEAD-FP4` | Qwen3.6 NVFP4 baselines (27B and 35B-A3B) must reach vLLM speed parity | perf | | [#339](https://github.com/mudler/vllm.cpp/issues/339) | `PERF-27B-LMHEAD-FP4` | 27B c1: every fp8 input projection asks for an f32 output, selecting the slower nvjet template family where vLLM emits bf16 (48 f32-out projections 18.51 ms vs 48 bf16-out 7.05 ms). The merged GDN `in_proj` arm is built DEFAULT OFF as `VT_GDN_FP8_IN_BF16`, spec [`perf-fp8-alpha-fold.md`](specs/perf-fp8-alpha-fold.md) §Attempt 4 — UNMEASURED: no committed gate loads the fp8 tower (`row/GATE-27B-FP8-TOWER-GOLDEN` builds that arm) | perf | diff --git a/.agents/specs/qwen38-text-only.md b/.agents/specs/qwen38-text-only.md new file mode 100644 index 000000000..0fa9baa0f --- /dev/null +++ b/.agents/specs/qwen38-text-only.md @@ -0,0 +1,436 @@ +# Qwen3.5/3.8 text-only arms: `Qwen3_5MoeForCausalLM`, `Qwen3_5ForCausalLM` + +**Rows:** `MODEL-TEXT-qwen3-5-qwen3-5-for-causal-lm`, +`MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` (both NEW beyond-pin rows; this row +adds them to [`model-matrix.md`](../model-matrix.md) with the inventory counts +bumped, mirroring the `MuseGlimmer`/`KimiK3` beyond-pin precedent) +**Issue:** [#490](https://github.com/mudler/vllm.cpp/issues/490) +**Also closes:** [#627](https://github.com/mudler/vllm.cpp/issues/627) — the +pre-existing misaligned-load UB this row's new test was the first gate to reach +(see `## Outcome` → "The safetensors alignment class"). +**Lifecycle:** `READY` +**Owner:** unassigned + +## Scope + +Load text-only checkpoints of the Qwen3.5-family GDN-hybrid backbone we already +run — the arms upstream calls `Qwen3_5ForCausalLM` and `Qwen3_5MoeForCausalLM`. +`Qwen/Qwen3.8-2.4T-A95B` is the motivating checkpoint; it is the same +architecture at larger scale, not a new one. + +In scope: + +- register the two text-only architecture strings against the existing dense and + MoE factories; +- accept both the VL-prefixed (`model.language_model.`) and clean (`model.`) + weight namespaces in the Qwen3.5 dense and MoE loaders; +- resolve a **flat** (non-nested, no `vision_config`) text config through the + existing path; +- prove the 27B / 35B / Coder gates stay byte-identical. + +Out of scope: any speed claim, any GGUF arm for 3.8, the vision tower +(a text-only checkpoint has none), MTP weights for 3.8, advancing the parity pin, +the **bf16 / 3-D-stacked MoE routed-expert arm** (owed, and refused by name +here — see [What this row does NOT make loadable](#what-this-row-does-not-make-loadable)), +and **any support claim for the 2.4T checkpoint itself**, which this hardware +cannot execute (see Gates). + +## Why this is not a new port + +`config.json` for `Qwen/Qwen3.8-2.4T-A95B` declares `Qwen3_5MoeForCausalLM` / +`model_type: qwen3_5_moe_text`. Against Qwen3.6-35B-A3B, which we run token-exact +315/315, every structural knob is identical — `head_dim` 256, +`linear_key/value_head_dim` 128, `linear_num_key_heads` 16, +`full_attention_interval` 4, `attn_output_gate` true, `partial_rotary_factor` +0.25, `rope_theta` 1e7, `mtp_num_hidden_layers` 1, `linear_conv_kernel_dim` 4, +and `vocab_size` 248320 (the same tokenizer). The differences are scale only: +hidden 2048->8192, layers 40->92, attention heads 16->64, KV heads 2->4, linear +V-heads 32->128, experts 256->512, top-k 8->10, moe/shared intermediate +512->2048. All of these are read from config, not hardcoded +(`qwen3_5_common.cpp:40-47`; the only expert constraint is `num_experts > 0` at +`qwen3_5_weights.cpp:618`). + +The 3.8 config also carries `output_gate_type: "swish"`, which normalizes to +silu. That key is handled by its own row (issue #489) and is not re-litigated +here. + +## Upstream chain + +| Upstream anchor | Contract to mirror | +|---|---| +| upstream `vllm/model_executor/models/registry.py:202-203` @ `ad5d29db7` | `Qwen3_5ForCausalLM` and `Qwen3_5MoeForCausalLM` are registered text-only arms of the same `qwen3_5` module. | +| upstream `vllm/model_executor/models/qwen3_5.py:439-449` @ `ad5d29db7` | `Qwen3_5ForCausalLM` is the shared base unchanged; `Qwen3_5MoeForCausalLM` is that base plus the MoE hyperparameters — no separate backbone. | +| upstream `vllm/model_executor/models/qwen3_5.py:296-300` @ `ad5d29db7` | `WeightsMapper(orig_to_new_prefix={"model.language_model.": "model."})` — the canonical namespace is `model.`, and the VL-prefixed form is accepted and rewritten. | + +**Ahead-of-pin, stated as such.** Our parity pin is `555967922`, whose registry +carries only the `ForConditionalGeneration` entries. The text-only arms arrived +upstream in **PR #50210 / `ad5d29db7`**, which is post-pin. This row is a +deliberate forward port of one upstream PR, not a mirror of the pin, and it does +not advance the pin or reconcile anything else in that range. That is visible +debt argued here and in the commit, not a silent divergence. + +## Design + +Weight-name evidence, read from the published indices of both checkpoints: + +| | Qwen3.6-35B-A3B (published) | Qwen3.8-2.4T-A95B (published) | +|---|---|---| +| embed | `model.language_model.embed_tokens.weight` | `model.embed_tokens.weight` | +| layer | `model.language_model.layers.0.linear_attn.*` | `model.layers.0.linear_attn.*` | +| experts | `...mlp.experts.gate_up_proj` + `.down_proj` (3D STACKED, 41x) | `...mlp.experts.gate_up_proj` + `.down_proj` (3D STACKED, 93x) | +| shared | `...mlp.shared_expert_gate.weight` | `...mlp.shared_expert_gate.weight` | +| head | `lm_head.weight` | `lm_head.weight` | +| quant scales | **NONE** — 0 `weight_scale`, 0 `input_scale` | **NONE** — 0 `weight_scale`, 0 `input_scale` | + +The **backbone** names are identical modulo the prefix, so the namespace +decision is the whole of what this row changes in the loader. **It is not the +whole of what either checkpoint needs**, and an earlier revision of this spec — +and of the two commits below it — wrongly said it was. See +[What this row does NOT make loadable](#what-this-row-does-not-make-loadable). + +1. **One prefix decision, resolved once.** The Qwen3.5 loaders currently + concatenate the literal `model.language_model.` in 4 places + (`qwen3_5_weights.cpp:560,632,633,659`) and 3 more in + `qwen3_5_dense_weights.cpp`. Replace the literal with a single resolved + backbone prefix, chosen once per checkpoint by probing which namespace the + shard index actually contains, then used everywhere. Mirrors upstream's single + `WeightsMapper` rather than scattering a fallback into each lookup — a + per-lookup fallback would let a checkpoint load half from one namespace and + half from the other and still appear to succeed. +2. **Registration is additive.** Two `REGISTER_VLLM_MODEL` entries pointing at + the existing dense and MoE factories. No factory, forward, or KV-cache change: + `ModelRegistry::Resolve` is exact-match with no aliasing + (`model_registry.cpp:217-231`), so the strings must be present literally. +3. **Config resolution already works.** `ResolveTextConfig` falls through to the + top-level document when there is no `text_config` + (`hf_config.cpp:113-122`), and `qwen3_5_moe_text` is already in + `IsQwen35Family` (`:128-132`), so the `partial_rotary_factor` 0.25 default + applies to a flat config. MRoPE is mm-path-only and every text caller passes + `nullptr` (`qwen3_5.cpp:7540`), so a config without `mrope_section` is + unaffected. Both facts get a test rather than an assumption. + +## What this row does NOT make loadable + +Corrected 2026-08-12 after an independent review returned FAIL on records +honesty. The registration and the namespace resolution are sound; the claim +built on top of them was not. + +**The MoE arm cannot read a published Qwen3.5-family MoE checkpoint, in either +namespace.** `LoadQwen3_5Moe` routes every routed expert through +`LoadMoeExpertsInto` (`qwen3_5_weights.cpp:519-530`) into `LoadNvfp4Raw` +(`:433-462`), which hard-requires per-expert `experts...weight` = `U8`, +`.weight_scale` = `F8_E4M3` and `.weight_scale_2`. There is **no stacked branch +and no bf16 branch** — unlike `gemma4_weights.cpp:326`, which dispatches between +layouts. Against that, the published indices (read live 2026-08-12): + +- `Qwen/Qwen3.8-2.4T-A95B`: 1609 tensors, 93x `mlp.experts.gate_up_proj` + + 93x `.down_proj` (3-D stacked, 92 backbone layers + 1 MTP), **zero** names + matching `weight_scale` or `input_scale`, `lm_head.weight` alone. +- `Qwen/Qwen3.6-35B-A3B`: 1045 tensors, the same stacked spelling under + `model.language_model.`, **zero** `weight_scale`. + +So the 2.4T load would die at `w.lm_head_fp4 = LoadNvfp4Raw(get, "lm_head")` +(`:679`) before the experts are even reached, and would die again at the FP8 +attention, the routed experts and the shared expert. Our gated 35B row reads the +REQUANTIZED `nvidia/Qwen3.6-35B-A3B-NVFP4`; this loader **has never read a +published Qwen bf16 MoE repo**. + +**The dense/MoE asymmetry is real and must not be flattened.** +`LoadQwen3_5Dense` DOES route BF16 vs FP8 vs NVFP4 per projection by tensor +presence (`qwen3_5_dense_weights.cpp:354-360,472-503`, and +`LoadDenseLmHead`/`LoadLmHeadAnyDtype` at `:215-233,515-547`), so the DENSE +text-only arm may genuinely load a flat bf16 checkpoint. Only the MoE arm +cannot. Any statement about "the text-only arms" that does not make that +distinction is wrong. + +**What is therefore OWED, named:** the **bf16 / 3-D-stacked MoE routed-expert +arm** (plus the bf16 shared expert, the bf16 FP8-less attention tower, and the +bf16 `lm_head`, all on the MoE path). That is a real port with its own spec, +RED-first test and NVFP4 inertness proof — it is not this row. Until it exists, +this row ships a **REFUSAL that names the missing piece** +(`CheckMoeExpertLayoutSupported`, `qwen3_5_weights.cpp`), because AGENTS.md +requires an unimplemented arm be refused by name rather than discovered later, +and this spec's own stop conditions said the same. + +**Consequence for the run gate.** "It closes when a text-only +`Qwen3_5[Moe]ForCausalLM` checkpoint that fits GB10 appears" is FALSE for the +MoE arm: a fitting *published* (bf16/stacked) MoE checkpoint would still be +refused at load. The MoE run gate needs a fitting checkpoint **whose routed +experts are per-expert NVFP4**, or the owed stacked/bf16 arm implemented first. +For the DENSE arm a fitting bf16 checkpoint is sufficient. + +## Risks + +- **Regression on gated rows.** These loaders serve 27B/35B/Coder. A prefix bug + breaks checkpoints we currently gate. Mitigated by byte-identical golden md5, + not by a green suite. +- **Half-resolved namespace.** Probing per lookup instead of once could load a + mixture. Mitigated by design point 1 and a test with a deliberately mixed + index, which must be refused. Refusing where upstream's `WeightsMapper` + NORMALIZES is a deliberate divergence in the strict direction and is recorded + as such in [porting-inventory](../porting-inventory.md) §9 deviation 17(c). +- **Untestable scale.** 92 layers / 512 experts is far past anything we can + instantiate. Mitigated by testing config resolution and name mapping directly, + and by *not* claiming the checkpoint runs. +- **Ahead-of-pin drift.** The forward-ported arm could diverge if upstream + changes it before our next sync. Recorded in + [porting-inventory](../porting-inventory.md) §9 deviation 17 as ahead-of-pin, + with the two arms added to its §5 Qwen3.5 row, so the next sync cycle + reconciles it deliberately. + +## Tests + +1. Architecture dispatch: a flat Qwen3.8-shaped config resolves + `Qwen3_5MoeForCausalLM` to the MoE registration, and `Qwen3_5ForCausalLM` to + the dense one. RED first — today both raise unsupported. +2. Config resolution on the real 3.8 shape: flat doc, no `vision_config`, no + `mrope_section`; assert the scale fields and the 0.25 rotary default. +3. Weight-name mapping: a clean (`model.`) index and a VL-prefixed + (`model.language_model.`) index both resolve every expected backbone tensor + name; a mixed index is refused. +4. Loader byte-equality, DENSE and MoE. Two synthetic one-layer checkpoints + with byte-identical payloads and only the namespace differing must load to + byte-identical weights through the production `LoadQwen3_5Dense` and + `LoadQwen3_5Moe`. The MoE case runs on BOTH expert-residency paths — + `shards_owner == nullptr` (eager) and non-null (deferred), with + `load_layer_experts` actually driven — because the deferred closure captures + the resolved prefix by value and executes after the resolving frame returns, + which is a third prefix site the dense loader has no analogue of. +4c. Refusal (added 2026-08-12): a synthetic checkpoint in the PUBLISHED shape — + 3-D stacked `mlp.experts.gate_up_proj` / `.down_proj`, bf16, no scale tensors + — must be refused by `LoadQwen3_5Moe` with a message that NAMES the offending + tensor and the required per-expert NVFP4 layout, in both namespaces; likewise + a per-expert-but-unquantized index and an NVFP4 index with a bf16 `lm_head`. + The supported per-expert NVFP4 layout must still load unchanged, asserted in + the same case so the gate cannot be satisfied by refusing everything. +5. Inertness: 27B/35B/Coder suites unchanged, golden md5 unchanged. The + per-layer seam DEFAULT is pinned by DRIVING `LoadQwen3_5MoeLayer` / + `LoadQwen3_5DenseLayer` with the prefix argument OMITTED — asserting the two + named constants does not pin it, and flipping both defaults VL->flat left the + original case green (review finding F7). +6. Record count (added 2026-08-13): the `MODEL` ratchet bump 362 -> 364 in + `scripts/check-agent-record.py` is tied to the two rows behind it — + `MODEL-TEXT-qwen3-5-qwen3-5-for-causal-lm` and + `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` each appear exactly once in + `.agents/model-matrix.md`, and the pin equals the MODEL rows that file + carries. The existing ratchet test moves the pin by one, which holds for ANY + pin value and so cannot say whether THIS value is right; these do. + +## Gates + +- Focused: the targets above plus the Qwen3.5 dense/MoE suites. +- Full gate on the row before push. +- **The run gate is OWED and must be recorded as owed.** 2.4T bf16 is ~4.8 TB; + the only other released variant is `Qwen/Qwen3.8-2.4T-A95B-FP8` at ~2.4 TB; + GB10 has 128 GB unified and no smaller Qwen3.8 sibling exists. There is + therefore **no token-exact oracle run for this checkpoint**, and the row may + not reach `DONE` on the strength of dispatch and mapping tests. What this row + can honestly claim is that the architecture is registered and the weight + namespace resolves — nothing about generated tokens. + +If a `Qwen3_5ForCausalLM` checkpoint small enough to execute appears, that +becomes the DENSE run gate and closes that axis. **It does not close the MoE +axis**: a fitting *published* (bf16/stacked) MoE checkpoint would still be +refused at load, so the MoE gate needs one whose routed experts are per-expert +NVFP4, or the owed stacked/bf16 arm implemented first (see +[What this row does NOT make loadable](#what-this-row-does-not-make-loadable)). + +## Evidence required + +- RED capture of the dispatch test before registration. +- A mutation capture per prefix site — dense and MoE, including the deferred + expert closure — showing the hardcoded VL literal makes the flat load throw. +- Green focused + full gate after. +- RED capture of the refusal case before `CheckMoeExpertLayoutSupported` exists + (the literal `qwen3_5 weights: expected U8 for lm_head.weight`), plus a + mutation per refusal branch and one that refuses unconditionally, which must + turn the SUPPORTED-layout assertions red. +- Golden md5 before/after for 27B/35B/Coder showing no drift. +- The owed run gate recorded explicitly in the row and in `docs/STATUS.md`. +- Executable mutation evidence for the checker change, in + `tests/scripts/test_agent_record.py`, which is what `scripts/check-pr-size.py` + runs BASE-checker-against-HEAD-tree: the literal RED + `AssertionError: 364 != 362 : the MODEL pin must equal the MODEL rows + model-matrix.md carries` with the pin at its BASE value, a second RED from + deleting one of the two new matrix rows, and green after both restorations. + +## Stop conditions + +- If the prefix cannot be resolved once per checkpoint without touching the + per-tensor lookup contract, stop and return `NEEDS_DECISION` rather than + scattering fallbacks through the loader. +- If any 27B/35B/Coder golden md5 moves, stop — that is a regression on a gated + row, and this row carries no evidence that could justify it. +- Do not implement an MTP arm, a stacked/bf16 MoE expert arm, or a GGUF arm for + 3.8 on speculation; refuse them with a message naming the missing piece and + record them as owed. (The QUANTIZED arm is the one that IS implemented — the + earlier wording here had this inverted.) + +## Now + +Both rows are `PARTIAL` (2026-08-12). Registration, the once-per-checkpoint +backbone-namespace resolution and the tests above are landed on +`row/MODEL-QWEN38-TEXT-ONLY`; full CPU gate green (396/396, 1 skipped: +`test_voxtral_e2e`, fixture absent) and `tests/parity/goldens` md5-unchanged. + +**Next step is the OWED run gate, and nothing else advances these rows.** The +DENSE one needs a `Qwen3_5ForCausalLM` checkpoint that fits GB10; the MoE one +needs a fitting checkpoint whose routed experts are PER-EXPERT NVFP4, because a +published (stacked/bf16) MoE checkpoint is refused at load. Neither exists +today. Until they do, the honest claim stays "the architecture is registered, +the weight namespace resolves, and an unimplemented expert layout is refused by +name". + +Also owed, and deliberately NOT implemented on speculation: the **bf16 / +3-D-stacked MoE routed-expert arm** (this was recorded INVERTED as "the +quantized arm is owed" until 2026-08-12 — the quantized arm is the only one +implemented), and the MTP and GGUF arms for 3.8. + +## Outcome + +**Measured.** Architecture dispatch for both strings, config resolution on the +PUBLISHED `Qwen/Qwen3.8-2.4T-A95B` `config.json` (committed verbatim as +`tests/vllm/models/fixtures/qwen3_8_2_4t_a95b/config.json`, md5 +`303dc59227f1d03afc941646e8df3132`) — the scale fields, the 92-entry +`layer_types` list and its `[linear,linear,linear,full] x 23` pattern, the NESTED +`rope_parameters` block both loaders' rope actually reads, and the absence of +`text_config` / `vision_config` / `mrope_section` — and weight-namespace +resolution on a clean index, a VL-prefixed index, a vision-inclusive VL index, an +index carrying `mtp.*`, a mixed index and an empty one. The strongest of these is +not a name-mapping assertion: two synthetic one-layer checkpoints with +byte-identical payloads and only the namespace differing load to byte-identical +weights through the production `LoadQwen3_5Dense` — and, on the MoE arm this row +exists for, through the production `LoadQwen3_5Moe` on BOTH expert-residency +paths, the deferred `load_layer_experts` closure included. Each of the three MoE +prefix sites was reverted to the hardcoded VL literal in turn and each RED is the +flat checkpoint failing to bind: `layers.0.input_layernorm.weight` (per-layer +base), `embed_tokens.weight` (top level) and `layers.0.mlp.experts.0.gate_proj +.weight` (the deferred closure). + +**Rejected.** A per-lookup namespace fallback — it would let a checkpoint bind +half its tensors from each namespace and still appear to load, which is exactly +the failure a name-mapping test cannot see. Also rejected: a blanket +"starts with `model.`" probe, because `model.visual.*` on a vision-inclusive 27B +checkpoint would have made it look like a flat text checkpoint and turned a +checkpoint we gate today into a refusal. Only the three structural backbone +spellings vote. + +**Why the defaults are what they are.** The per-layer public seams +(`LoadQwen3_5MoeLayer`, `LoadQwen3_5DenseLayer`) default `backbone_prefix` to the +VL spelling, so every 27B/35B/Coder caller is byte-identical by construction +rather than by re-measurement. The text-only arms register with +`kQwen3_5TextInfo` (hybrid YES, multimodal NO) because upstream's +`Qwen3_5ForCausalLMBase` inherits `IsHybrid` but not `SupportsMultiModal`; the +`ForConditionalGeneration` wrappers remain the multimodal registrations. + +**What was NOT established, and what an earlier revision wrongly claimed.** Any +claim about generated tokens, memory or speed for `Qwen/Qwen3.8-2.4T-A95B` — +that checkpoint cannot be executed on this hardware and was never run. And, +corrected 2026-08-12 after a review FAIL, **any claim that hardware size is the +only thing between this code and that checkpoint**: it is not, because the MoE +loader reads only per-expert NVFP4 experts and both published Qwen MoE repos +ship 3-D stacked, unquantized ones. That arm is OWED and is now refused by a +message naming it (`CheckMoeExpertLayoutSupported`), with the fixture and the +literal RED in `tests/vllm/models/test_qwen3_8_text_only.cpp`. The DENSE loader +routes BF16/FP8/NVFP4 by tensor presence and is not subject to that gap — the +asymmetry is deliberate record, not an oversight. + +**The refusal was verified against the REAL gated checkpoint, and its `mtp.` +exclusion is LOAD-BEARING.** Added 2026-08-12 after an independent review. The +published `nvidia/Qwen3.6-35B-A3B-NVFP4` safetensors index was fetched and read +directly (`model.safetensors.index.json`, **124,468 tensors**): it **does** +contain the exact 3-D stacked spelling `CheckMoeExpertLayoutSupported` refuses, +as `mtp.layers.0.mlp.experts.gate_up_proj` and `mtp.layers.0.mlp.experts +.down_proj` — and only there. Under the resolved backbone +(`model.language_model.`) there are **zero** stacked expert names, **zero** +expert `.weight` without a `_scale` sibling, and both `lm_head.weight_scale` and +`lm_head.weight_scale_2` are present. So the checkpoint we gate today is **not** +refused — but only because the scan is anchored at `layers.` +(`qwen3_5_weights.cpp:633,638`) and `mtp.` is under neither backbone spelling. + +That exclusion was pinned by nothing in-tree. Broadening the scan to every +`.mlp.experts.` name would refuse the one checkpoint this arm is gated on, on a +**CUDA-only load path**, with the entire CPU suite still green. The supported +fixture (`MoeOneLayerSpecs`) therefore now carries those two `mtp.` names, so +the shape of the real index is what the inertness assertions run against, and +case 4c gains a subcase that both re-asserts the fixture still carries them +(count `== 1`, and zero under either backbone prefix) and that the load stays +clean. RED-first: with the `layers.` filter dropped, the new subcase +fails with `3-D stacked routed experts are not implemented ... found +"mtp.layers.0.mlp.experts.gate_up_proj"` (2 cases failed, `Status: FAILURE!`, +assertion count 747 → 277 as the thrown cases abort); `src/` restored +byte-for-byte afterwards and back to 7/7, 747/747. + +**What the "refused by name" guarantee does and does not cover.** It covers the +routed experts and `lm_head` only: the stacked spelling, a per-expert `.weight` +with no `.weight_scale` beside it, and an unquantized `lm_head` that is present +(a checkpoint with no `lm_head.weight` at all is the tied-head case and is +deliberately not this refusal). Everything else on the MoE path still surfaces +its raw loader error — a bf16 **shared** expert, a bf16 (FP8-less) attention +tower, the compressed-tensors `weight_packed` spelling, and a tied-head MoE +checkpoint. No surface claims otherwise, so this is a clarification of scope +rather than a gap; widening the refusal belongs with the owed stacked/bf16 arm, +which has to read those layouts anyway. + +**Recorded as tracked debt** in [porting-inventory](../porting-inventory.md) §9 +deviation 17, with the two arms carried on its §5 Qwen3.5 row and the owed run +gate on [BENCHMARKS](../../docs/BENCHMARKS.md) §Open gaps: the ahead-of-pin +anchor `ad5d29db7` (17a/b), the deliberate REFUSAL of a mixed namespace where +upstream's `WeightsMapper` would normalize it (17c), the published config's +transformers-4.57.3 `dtype` key, which `hf_config.cpp:520-522` does not consume +(17d — inert, no reader, and a fix would touch every model, so it is pinned by +an assertion rather than smuggled in here), and the **unimplemented bf16 / +3-D-stacked MoE routed-expert arm** (17e, added 2026-08-12 — it was previously +recorded inverted, as the quantized arm being the owed one). + +**The safetensors alignment class** ([#627](https://github.com/mudler/vllm.cpp/issues/627), +fixed 2026-08-13). This row's `test_qwen3_8_text_only` was the first gate ever to +run a safetensors weight loader under UBSan, and it went RED on +`qwen3_5_weights.cpp:298` — `load of misaligned address ... for type +'const uint16_t', which requires 2 byte alignment`. **The defect is pre-existing +(`8ee2c0766`), not this row's**; what this row supplied is the first synthetic +checkpoint whose tensor offsets are not all even, which is a legitimate shape a +real file can have because a safetensors offset is just the running byte total of +everything ahead of it. + +The observed site was one instance of a class. A sweep of every +`reinterpret_cast(.data)` in `src/vllm/model_executor/models/` +found **fifteen** across nine `*_weights.cpp` loaders, including two with +stricter-than-2-byte requirements: `olmo2_weights.cpp` forms a `const float*` +(4-byte) and `qwen3_dspark_weights.cpp` a `const int64_t*` (8-byte). All fifteen +now go through `vt::LoadUnaligned` — the seam `ea4deb203` introduced and that +`dense_loaders::TransposeBf16` and `minimax_h3_vae_loader.cpp:87` already used; +the local `qwen3_5_weights.cpp` copy of `TransposeBf16` had simply never been +migrated to it. Two sites could not take a byte pointer and were handled in kind: +`internlm2_weights.cpp` only ever bulk-`memcpy`s from its source, so it keeps a +`const uint8_t*` and scales its offsets, and `gemma4_weights.cpp` feeds a typed +scale pointer to `DequantFp8ChannelToBf16` (whose header is outside this row's +authority), so it copies the N-element scale row into an aligned buffer first. + +**Inertness is proven three ways, not asserted.** (1) `vt::LoadUnaligned` is +`memcpy`, which is bit-identical to `*(const T*)p` on every input the old code +was *allowed* to read — a scratch harness ran the original and rewritten form of +all six loops over one payload at an aligned base and got byte-identical output, +then reproduced those same bytes from the rewritten form at misaligned bases 1 +through 8. (2) `tests/parity/goldens` is untouched and every golden-comparing +suite passes. (3) Full CPU gate 404/404 (2 skipped: `test_voxtral_e2e` and +`test_modelopt_mixed_precision_checkpoint`, fixtures absent) and — the gate that +was red — full ASan+UBSan 404/404, `test_qwen3_8_text_only` back to 7/7 and +747/747 with zero runtime errors. + +**No load-time regression, checked rather than assumed.** `TransposeBf16` is a +hot load-time loop and a naive per-element `memcpy` is exactly the kind of change +that can turn one load into a call. It does not here: at `-O2` the old and new +inner loops are instruction-for-instruction identical — the same six +instructions, the same `movzx REG, WORD PTR [rax]`, no call emitted. The whole +delta is five prologue instructions (one callee-saved push/pop pair and two +address setups), paid once per call, not per element. + +**Still owed, and deliberately not touched here** because they fall outside this +row's authority (`*_weights.cpp`): the same cast survives at +`voxtral.cpp:51,347`, `qwen3_vl.cpp:78` and `qwen3_5_mtp.cpp:71`. The first three +are genuine misaligned *loads* of the same severity as the one UBSan caught; the +`qwen3_5_mtp.cpp` one only forms the pointer and then `memcpy`s through it, so it +is UB but will not fire the `alignment` check. None is reached by any suite that +runs under sanitizers today, which is precisely why they need a follow-up rather +than a grep. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index b24c5da5d..206cfc704 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -424,6 +424,7 @@ built on it rather than keeping the flattering one. | CPU keep-quant MoE decode | **No number owed**: correctness-only P0. The grouped keep-quant GEMM read activations as f32 whatever their dtype, so CPU MoE decode emitted token-0 garbage from `b4f5610a` (2026-07-31) | Speed unmeasured and unclaimed; `test_ops_quant_dot` GREEN (150224 assertions) | | Accepted-and-inert serve args (`SERVE-RECIPE-ARGS`, #606) | **No number owed**: argument parsing only, so nothing to time and no oracle leg. Correctness gate 4 cases / 58 asserts GREEN, RED-first, mutation-proven | None. A speed axis would be fabricated; closes on review plus the operator gate rerun | | DeepSeek-V2-Lite MLA | Attributed miss, `ACTIVE` | Throughput at every concurrency | +| Qwen3.5 text-only arms (#490) | **No number; gates OWED**, both `PARTIAL` | No fitting ckpt: no denominator; MoE needs NVFP4 experts: published REFUSED (§9 17e) | | Qwen3.5 upstream throughput levers (roadmap C10) | NOT MEASURED. vLLM's 2026-08-06 25K tok/s/GPU is a GB200/NVLink72 disaggregated cluster result, not comparable to one GB10, and is NOT adopted as our bar | Advance the parity pin past `555967922` so the referenced PRs exist, re-capture goldens at zero drift, then port the GDN prefill kernel | | DeepSeek-V4-Flash | **Parity with ds4 (0.997x)** | Optional beat-path: f16 tensor-core DSA/router (near-tie class) | | DeepSeek-V4-Flash vs vLLM | Infeasible on one Spark | 2x GB10 with TP2 over the NCCL seam | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 35a8eefa0..b857ce1ec 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -80,7 +80,7 @@ are our reading of their documented behavior, not measurements. | Merged fp8 projection folds per-column alpha in the GEMM epilogue | ◐ `VT_FP8_ALPHA_VEC_EPILOGUE`, CUDA only, default off, ungated; refuses split-K under a bf16-D equivalence claim (`claims_splitk1_premise`, default off) | n/a | n/a | n/a | | `vt::MulColVecF32` carries a bf16 store width | ✅ f32 arm byte-identical; bf16 arm rounds once; CPU + CUDA | n/a | ☐ | ☐ | | bf16 / fp16 | ✅ | ✅ | ✅ | ✅ | -| Safetensors direct load, no conversion | ✅ | ✅ | ✅ | ☐ | +| Safetensors direct load, no conversion | ✅ at ANY tensor byte offset: the format aligns nothing, so the `*_weights.cpp` loaders never form a typed pointer into the mapping (#627). `voxtral.cpp`, `qwen3_vl.cpp`, `qwen3_5_mtp.cpp` still do and are OWED | ✅ | ✅ | ☐ | | Weights uploaded straight from the file mapping (no host copy first) | ◐ verbatim tensors only (37.8% of 27B BF16); arbitrary-offset reads are defined, including Laguna graph staging. Merged/transposed and merged FP4 weights still copy | ✅ | ✅ | ✅ mmap | ## Model coverage @@ -111,6 +111,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. |---|---|---|---| | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B NVFP4 (`unsloth` @`890bdef7`, `nvidia` @`0893e160`); Qwen3.5-4B BF16 | 27B strict 235/235 text + 32/32 image/video; 4B cached 3/3 | `unsloth` 27B at/above vLLM, `nvidia` ModelOpt 0.85x; 4B throughput 1.021x. Loads BF16, FP8 and NVFP4 (CT + ModelOpt naming); a `modelopt_mixed` FP8 tower stays NATIVE (#164), GDN `in_proj_qkvz` merged. CUDA/CPU only | | `Qwen3_5MoeForConditionalGeneration` | Qwen3.6-35B-A3B (NVFP4, GDN MoE) | strict 315/315 text vs vLLM 0.25.0 | gate model: 0.93x to 1.03x grid | +| `Qwen3_5ForCausalLM`, `Qwen3_5MoeForCausalLM` | none: no text-only Qwen3.5 checkpoint fits this hardware | **NO RUN GATE, OWED.** Dispatch/config/namespace gated on `test_qwen3_8_text_only.cpp`; NO token claim. MoE reads ONLY per-expert NVFP4: the published stacked/bf16 layout is unimplemented, OWED, refused by name | not measured | | `Qwen3ForCausalLM` | Qwen3 dense 0.6B/1.7B/4B/32B, NVFP4A16 | near-tie strict 16/16 vs vLLM 0.25.0 | c1 every-axis parity, c8 decode residual | | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B | strict 6/6 vs vLLM 0.25.0 | 11/16 grid cells at or above graphed vLLM | | `Qwen3VLForConditionalGeneration` | Qwen3-VL-4B-Instruct (image + video) | image strict 32/32, video near-tie vs vLLM 0.25.0 | vision tower 0.57x vs vLLM encode; umbrella pending | diff --git a/docs/STATUS.md b/docs/STATUS.md index ce2d935cd..3038b35f9 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -699,6 +699,52 @@ oracle-blocked for a gate (see the capability table above). ### Frontier and hardware-blocked families +**Qwen3.5 text-only arms (`Qwen3_5ForCausalLM`, `Qwen3_5MoeForCausalLM`) — +REGISTERED 2026-08-12, RUN GATE OWED (#490).** An ahead-of-pin forward port of +upstream PR vllm#50210 (`ad5d29db7`), which registers both arms against the same +`qwen3_5` module our gated `ForConditionalGeneration` wrappers already use. Two +additive `REGISTER_VLLM_MODEL` lines against the EXISTING dense and MoE +factories: no forward, no KV-cache spec, no loader fork. + +The other half is ONE backbone weight-namespace decision per checkpoint: +`model.` for a text-only arm, `model.language_model.` for the wrappers, and a +MIXED index refused rather than half-bound. `Qwen/Qwen3.8-2.4T-A95B` declares +`Qwen3_5MoeForCausalLM` / `qwen3_5_moe_text` and is the token-exact +Qwen3.6-35B-A3B GDN-hybrid MoE backbone at larger scale, every knob +config-driven, with the BACKBONE weight names identical modulo that prefix. + +**CORRECTED 2026-08-12: the prefix is NOT the only thing between this code and +`Qwen/Qwen3.8-2.4T-A95B`, and the first two commits of this row said it was.** +The MoE loader reads only PER-EXPERT NVFP4 routed experts (`LoadMoeExpertsInto` +-> `LoadNvfp4Raw`: `U8` weight + `F8_E4M3` `.weight_scale` + `.weight_scale_2`), +with no stacked and no bf16 branch. The published indices (read live 2026-08-12) +have neither: `Qwen/Qwen3.8-2.4T-A95B` carries 93x `mlp.experts.gate_up_proj` + +93x `.down_proj` (3-D STACKED) and **zero** `weight_scale` / `input_scale` +tensors, and `Qwen/Qwen3.6-35B-A3B` is the same under the VL prefix — our gated +35B row reads the REQUANTIZED `nvidia/Qwen3.6-35B-A3B-NVFP4`. + +So the **bf16 / 3-D-stacked MoE routed-expert arm is NOT implemented and is +OWED**, and a published MoE checkpoint is now REFUSED by a message naming it +rather than dying on `expected U8 for lm_head.weight`. The DENSE arm is not +affected: `LoadQwen3_5Dense` routes BF16 vs FP8 vs NVFP4 per projection by +tensor presence, so it may genuinely load a flat bf16 checkpoint. That asymmetry +is the record. + +What is claimed is dispatch, flat-config resolution, namespace resolution and +the refusal, gated by `tests/vllm/models/test_qwen3_8_text_only.cpp`, with +27B/35B/Coder inert and parity goldens md5-unchanged. **What is NOT claimed is a +single generated token.** 2.4T bf16 is ~4.8 TB and the released FP8 variant +~2.4 TB against 128 GB of unified memory, and no smaller Qwen3.8 sibling exists, +so there is no token-exact oracle run and no speed number. + +Both rows therefore stay `PARTIAL`. The owed **DENSE** run gate closes when a +`Qwen3_5ForCausalLM` checkpoint that fits GB10 appears. The **MoE** run gate +needs more: a fitting *published* (bf16/stacked) MoE checkpoint would still be +refused at load, so it needs a fitting checkpoint whose routed experts are +per-expert NVFP4, or the owed stacked/bf16 arm implemented first. Also NOT +implemented and recorded as owed: that stacked/bf16 MoE expert arm, and the MTP +and GGUF arms for 3.8. This does not advance the parity pin. + Larger DeepSeek / GLM / MiniMax / Gemma-4 variants are recorded as **hardware-blocked** (they do not fit 119 GiB of unified memory on this box) or **spiked-only**, per the [model matrix](../.agents/model-matrix.md). diff --git a/docs/USAGE.md b/docs/USAGE.md index ec0cb298c..6d174430a 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -198,6 +198,26 @@ build/examples/vllm-cli \ | `--repeat N` | `1` | Load once, then run N blocking completions. Use it to read a warm decode tok/s without paying model load each time. Not supported with `--stream`, which falls back to 1 | | `-h`, `--help` | | Print usage and exit | +`--model` resolves a Qwen3.5-family checkpoint's backbone under EITHER weight +namespace. The multimodal wrappers (`Qwen3_5ForConditionalGeneration`, +`Qwen3_5MoeForConditionalGeneration`) publish the text backbone nested under +`model.language_model.`; the text-only arms (`Qwen3_5ForCausalLM`, +`Qwen3_5MoeForCausalLM`) publish it flat under `model.`. The loader decides which +ONCE per checkpoint from the shard index, and REFUSES a checkpoint that carries +backbone tensors under both rather than binding half the model from each. + +**Resolving the namespace is not the same as loading the checkpoint, and the +MoE and dense arms differ.** The dense loader routes each projection to BF16, +FP8 or NVFP4 by tensor presence, so a flat bf16 `Qwen3_5ForCausalLM` checkpoint +is expected to load. The **MoE** loader reads only PER-EXPERT NVFP4 routed +experts, while the published MoE repos (`Qwen/Qwen3.8-2.4T-A95B`, +`Qwen/Qwen3.6-35B-A3B`) ship 3-D stacked, unquantized experts — that arm is +**not implemented**, and such a checkpoint is refused at load with a message +naming what is missing. Use an NVFP4 requant (e.g. +`nvidia/Qwen3.6-35B-A3B-NVFP4`) for the MoE path. No text-only Qwen3.5 +checkpoint has been RUN here at all — see [STATUS.md](STATUS.md) for the owed +run gates. + GGUF and safetensors mapped-payload paths, plus safetensors index paths, use the host's native filesystem encoding, including Unicode paths on Windows. Native Windows release artifacts are not published yet; they will remain unavailable diff --git a/include/vllm/model_executor/models/qwen3_5_dense.h b/include/vllm/model_executor/models/qwen3_5_dense.h index d2a44f71f..888a91dbd 100644 --- a/include/vllm/model_executor/models/qwen3_5_dense.h +++ b/include/vllm/model_executor/models/qwen3_5_dense.h @@ -184,11 +184,15 @@ GdnLayerWeights LoadQwen3_5DenseGdn(const TensorResolver& get, const std::string& layer_base); // Load one dense decoder layer. `layer_type` is "linear_attention" or -// "full_attention". Prefix is "model.language_model.layers.{layer_idx}.". Routes -// each Linear to ordinary BF16 or compressed NVFP4 based on tensor presence. -Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer(const TensorResolver& get, - const std::string& layer_type, - int64_t layer_idx); +// "full_attention". Prefix is "{backbone_prefix}layers.{layer_idx}.", defaulting +// to the VL spelling every checkpoint we gate today uses, so this seam is +// byte-identical for the 27B/35B/Coder callers; `LoadQwen3_5Dense` passes the +// prefix it resolved ONCE from the shard index +// (`ResolveQwen3_5BackbonePrefix`). Routes each Linear to ordinary BF16 or +// compressed NVFP4 based on tensor presence. +Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer( + const TensorResolver& get, const std::string& layer_type, int64_t layer_idx, + const std::string& backbone_prefix = std::string(kQwen3_5VlBackbonePrefix)); // The same load with an EXPLICIT presence probe — what `LoadQwen3_5Dense` calls // per layer. The resolver-only overload above answers `has` with a constant @@ -199,7 +203,8 @@ Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer(const TensorResolver& get, // production takes. Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer( const TensorResolver& get, const std::function& has, - const std::string& layer_type, int64_t layer_idx); + const std::string& layer_type, int64_t layer_idx, + const std::string& backbone_prefix = std::string(kQwen3_5VlBackbonePrefix)); // Full dense-model load across the given shards. Uses config.num_hidden_layers // and config.layer_types. Text path only — the vision tower (model.visual.*) diff --git a/include/vllm/model_executor/models/qwen3_5_weights.h b/include/vllm/model_executor/models/qwen3_5_weights.h index 512e470b2..82cdff607 100644 --- a/include/vllm/model_executor/models/qwen3_5_weights.h +++ b/include/vllm/model_executor/models/qwen3_5_weights.h @@ -25,6 +25,7 @@ #include #include #include +#include #include #include "vllm/model_executor/model_loader/safetensors_reader.h" @@ -488,6 +489,52 @@ struct Qwen3_5MoeWeights { // Resolves a tensor name to its StTensor (across shards). Throws if absent. using TensorResolver = std::function; +// --- Backbone weight namespace (MODEL-TEXT-qwen3-5-*-for-causal-lm) ----------- +// +// A Qwen3.5-family checkpoint publishes its text backbone under ONE of two +// spellings. The multimodal wrappers we already gate (Qwen3.6-27B / 35B-A3B / +// Coder) nest it under `model.language_model.`; the TEXT-ONLY arms +// (`Qwen3_5ForCausalLM` / `Qwen3_5MoeForCausalLM`, e.g. +// `Qwen/Qwen3.8-2.4T-A95B`) publish it flat under `model.`. The BACKBONE names +// are otherwise identical — same `mlp.shared_expert_gate.weight`, same +// top-level `lm_head`. +// +// THE PREFIX IS NOT THE ONLY THING BETWEEN THIS LOADER AND A PUBLISHED +// CHECKPOINT, and an earlier revision of this comment wrongly implied it was. +// The published Qwen3.5-family MoE repos ship 3-D STACKED routed experts +// (`...mlp.experts.gate_up_proj` / `.down_proj`) and carry no quantization +// scales at all, while `LoadQwen3_5Moe` reads ONLY per-expert NVFP4. That arm +// is OWED and is refused by name (`CheckMoeExpertLayoutSupported`, +// `qwen3_5_weights.cpp`). The DENSE loader is different: it routes BF16 vs FP8 +// vs NVFP4 per projection by tensor presence, so it may genuinely read a flat +// bf16 checkpoint. Resolving the namespace is what THIS seam does; it is not a +// support claim for either published checkpoint. +// +// Upstream normalizes the two with ONE mapper — +// WeightsMapper(orig_to_new_prefix={"model.language_model.": "model."}) +// (vllm/model_executor/models/qwen3_5.py:296-300 @ `ad5d29db7`, PR #50210, +// which is AHEAD OF our `555967922` parity pin and recorded as such) — +// so `model.` is canonical and the VL spelling is its accepted alias. +inline constexpr std::string_view kQwen3_5VlBackbonePrefix = + "model.language_model."; +inline constexpr std::string_view kQwen3_5TextBackbonePrefix = "model."; + +// Decides which of the two the checkpoint uses, ONCE, from the shard index, so +// every subsequent lookup in a load uses one namespace. Deliberately NOT a +// per-lookup fallback: a fallback would let a checkpoint bind half its tensors +// from one namespace and half from the other and still appear to load. +// +// Only BACKBONE spellings vote — `embed_tokens.weight`, +// `norm.weight` and `layers.`. `model.visual.*` (the +// vision-inclusive 27B/35B towers) and the top-level `lm_head.*` / `mtp.*` +// therefore cast no vote, which is what keeps a vision checkpoint from looking +// like a flat text one. +// +// Throws std::runtime_error when BOTH namespaces carry backbone tensors (a +// mixed index is refused, never half-loaded) and when NEITHER does. +std::string ResolveQwen3_5BackbonePrefix( + const std::vector& tensor_names); + // --- ENG-LOAD-DIRECT-UPLOAD (issue #150) ------------------------------------- // // THE DEFECT THIS CLOSES. Loading a checkpoint copies the weights TWICE: the @@ -532,11 +579,14 @@ void SetLoadDirectUploadOverrideForTesting(std::optional value); // Load one decoder layer's weights from real tensors. `layer_type` is // "linear_attention" or "full_attention"; `num_experts` drives the expert loop. // Exercised on real data by the Task 3 unit test (both layer types live in -// shard 1). Prefix is "model.language_model.layers.{layer_idx}.". -Qwen3_5MoeLayerWeights LoadQwen3_5MoeLayer(const TensorResolver& get, - const std::string& layer_type, - int64_t layer_idx, - int64_t num_experts); +// shard 1). Prefix is "{backbone_prefix}layers.{layer_idx}.", and the default +// is the VL spelling every checkpoint we gate today uses, so this seam is +// byte-identical for the 27B/35B/Coder callers. `LoadQwen3_5Moe` passes the +// prefix it resolved once from the shard index. +Qwen3_5MoeLayerWeights LoadQwen3_5MoeLayer( + const TensorResolver& get, const std::string& layer_type, int64_t layer_idx, + int64_t num_experts, + const std::string& backbone_prefix = std::string(kQwen3_5VlBackbonePrefix)); // Full-model load: resolves every param across the given shards (name -> shard // looked up from each file's own header), dequantizes/transposes, and returns diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index b14570750..07be63db6 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -78,7 +78,26 @@ # architecture. Re-derived, which is the only way this pin is ever allowed to # move. test_music3_and_indextts_rows_both_survive_their_collision names all # three rows, because a count alone cannot see that failure. - "MODEL": (AGENTS / "model-matrix.md", 373), + # 375 since 2026-08-14: +`MODEL-TEXT-qwen3-5-qwen3-5-for-causal-lm` and + # +`MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` (issue #490), the TEXT-ONLY + # arms of the Qwen3.5 backbone — the eighth architecture the #609/#610 audit + # found and deliberately left to its owner, plus its dense sibling. Both are + # beyond-pin: they are not among the 355 registry architectures at + # `555967922` because they landed upstream afterwards (PR vllm#50210 @ + # `ad5d29db7`), exactly like the Muse Glimmer row above. Their Upstream cells + # deliberately carry no pinned module/class target, so the pin-derived static + # invariants in check_model_invariants are UNCHANGED (324/373/356/310/261) — + # this is the row-EXISTS count only, bumped because two new rows exist, never + # to make a transition pass. This row was authored against 362 -> 364, then + # re-derived to 370 -> 372, and is now RE-DERIVED AGAIN to 373 -> 375: the + # #609/#610 backfill, LTX-2.5, IndexTTS-2.5 and MiniMax-Music3 all landed + # while it was in review, and every one of them moved this pin. The number is + # counted off the matrix as it stands after the merge, never carried forward + # from the branch — a justification framed against a number this file no + # longer carries would be false about the file it sits in, and + # `Qwen35TextOnlyRowsAreCounted` is what ties this value to the two rows the + # matrix actually holds. + "MODEL": (AGENTS / "model-matrix.md", 375), # 82 since 2026-07-21: +`QUANT-NVFP4-CT-W4A16` (compressed-tensors NVFP4A16 / # W4A16 — NVFP4 weights with BF16 activations, distinct from the existing # `QUANT-NVFP4-CT-W4A4` and `QUANT-NVFP4-MO-W4A16` rows in both scheme diff --git a/src/vllm/model_executor/models/gemma4_weights.cpp b/src/vllm/model_executor/models/gemma4_weights.cpp index 4b406fc18..dea8052bb 100644 --- a/src/vllm/model_executor/models/gemma4_weights.cpp +++ b/src/vllm/model_executor/models/gemma4_weights.cpp @@ -135,7 +135,16 @@ OwnedTensor LoadFp8ChannelToBf16RawNk(const TensorResolver& get, const std::stri VT_CHECK(s.shape[0] == N, "gemma4 fp8: scale N"); OwnedTensor o = MakeOwned(vt::DType::kBF16, {N, K}); o.nk = true; - DequantFp8ChannelToBf16(w.data, reinterpret_cast(s.data), N, K, + // `s.data` points into the mmap'd safetensors payload at an arbitrary byte + // offset, so a `const uint16_t*` onto it is undefined to form or load through + // (issue #627), and DequantFp8ChannelToBf16 takes a typed scale pointer. Copy + // the N-element scale row into an aligned buffer first — N is the output + // channel count, negligible next to the N*K dequant it feeds. + std::vector scale(static_cast(N)); + VT_CHECK(s.nbytes >= scale.size() * sizeof(uint16_t), + "gemma4 fp8: scale tensor too small for " + base); + std::memcpy(scale.data(), s.data, scale.size() * sizeof(uint16_t)); + DequantFp8ChannelToBf16(w.data, scale.data(), N, K, reinterpret_cast(o.bytes.data())); MaybeReleaseSourcePages(w.data, w.nbytes); MaybeReleaseSourcePages(s.data, s.nbytes); diff --git a/src/vllm/model_executor/models/internlm2_weights.cpp b/src/vllm/model_executor/models/internlm2_weights.cpp index ab60dee10..3e8fdd5c1 100644 --- a/src/vllm/model_executor/models/internlm2_weights.cpp +++ b/src/vllm/model_executor/models/internlm2_weights.cpp @@ -98,13 +98,19 @@ OwnedTensor DeinterleaveWqkv(const StTensor& wqkv, const HfConfig& cfg, VT_CHECK(wqkv.nbytes == expected, "internlm2: byte-size mismatch for " + name); OwnedTensor merged = MakeOwned(vt::DType::kBF16, {out_rows, H}); - const auto* src = reinterpret_cast(wqkv.data); + // `src` stays BYTE-typed: it points into the mmap'd safetensors payload, whose + // per-tensor offset is the running total of everything ahead of it and so need + // not be even, and a `const uint16_t*` onto an odd byte is undefined to form + // (issue #627). Every read below is a bulk memcpy, so byte offsets — the same + // arithmetic scaled by sizeof(uint16_t) — copy the identical bytes. + const uint8_t* src = wqkv.data; auto* dst = reinterpret_cast(merged.bytes.data()); - const size_t row = static_cast(H); // elems per output row + const size_t row = static_cast(H); // elems per output row + const size_t row_bytes = row * sizeof(uint16_t); // bytes per output row if (WrongSplitRed()) { // RED path: copy wqkv straight through (naive [q|k|v] concat, NO // de-interleave) — the WRONG split; heads end up scrambled. - std::memcpy(dst, src, static_cast(out_rows) * row * sizeof(uint16_t)); + std::memcpy(dst, src, static_cast(out_rows) * row_bytes); MaybeReleaseSourcePages(wqkv.data, wqkv.nbytes); merged.nk = true; return merged; @@ -113,16 +119,16 @@ OwnedTensor DeinterleaveWqkv(const StTensor& wqkv, const HfConfig& cfg, const int64_t block = g * (groups + 2) * Dh; // first wqkv row of kv-group g // q: groups*Dh rows -> q section at g*groups*Dh std::memcpy(dst + static_cast(g * groups * Dh) * row, - src + static_cast(block) * row, - static_cast(groups * Dh) * row * sizeof(uint16_t)); + src + static_cast(block) * row_bytes, + static_cast(groups * Dh) * row_bytes); // k: Dh rows -> k section (after q_rows) at g*Dh std::memcpy(dst + static_cast(q_rows + g * Dh) * row, - src + static_cast(block + groups * Dh) * row, - static_cast(Dh) * row * sizeof(uint16_t)); + src + static_cast(block + groups * Dh) * row_bytes, + static_cast(Dh) * row_bytes); // v: Dh rows -> v section (after q_rows+kv_rows) at g*Dh std::memcpy(dst + static_cast(q_rows + kv_rows + g * Dh) * row, - src + static_cast(block + (groups + 1) * Dh) * row, - static_cast(Dh) * row * sizeof(uint16_t)); + src + static_cast(block + (groups + 1) * Dh) * row_bytes, + static_cast(Dh) * row_bytes); } MaybeReleaseSourcePages(wqkv.data, wqkv.nbytes); merged.nk = true; // raw [N=out_rows, K=H] for vt::MatmulBT diff --git a/src/vllm/model_executor/models/laguna_weights.cpp b/src/vllm/model_executor/models/laguna_weights.cpp index f845351c9..03dd69267 100644 --- a/src/vllm/model_executor/models/laguna_weights.cpp +++ b/src/vllm/model_executor/models/laguna_weights.cpp @@ -37,6 +37,7 @@ #include "vllm/model_executor/layers/quantization/compressed_tensors/nvfp4_emulation.h" // DequantCtNvfp4WeightToF32 #include "vllm/model_executor/models/qwen3_5_gguf_weights.h" // OwnGgufQuantBlocks #include "vt/dtype.h" +#include "vt/unaligned.h" namespace vllm { namespace { @@ -264,10 +265,12 @@ OwnedTensor LnLoadF32Direct(const TensorResolver& get, const std::string& name) } else if (t.dtype == "BF16") { const size_t n = t.nbytes / 2; // bf16 = 2 bytes/elem VT_CHECK(o.bytes.size() == n * 4, "laguna nvfp4: BF16->F32 size mismatch " + name); - const auto* src = reinterpret_cast(t.data); + // Unaligned: `t.data` is an arbitrary byte offset into the mmap (#627). auto* dst = reinterpret_cast(o.bytes.data()); for (size_t i = 0; i < n; ++i) { - const uint32_t bits = static_cast(src[i]) << 16; // bf16 -> high 16 bits of f32 + const uint32_t bits = static_cast( + vt::LoadUnaligned(t.data + i * 2)) + << 16; // bf16 -> high 16 bits of f32 std::memcpy(&dst[i], &bits, 4); } } else { diff --git a/src/vllm/model_executor/models/olmo2_weights.cpp b/src/vllm/model_executor/models/olmo2_weights.cpp index 268e2d543..d21758663 100644 --- a/src/vllm/model_executor/models/olmo2_weights.cpp +++ b/src/vllm/model_executor/models/olmo2_weights.cpp @@ -48,6 +48,7 @@ #include "vllm/model_executor/model_loader/safetensors_reader.h" #include "vllm/model_executor/models/dense_weight_loaders.h" // MakeOwned + bf16 helpers #include "vt/dtype.h" +#include "vt/unaligned.h" namespace vllm { namespace { @@ -65,8 +66,16 @@ bool RawBool(const nlohmann::json& doc, const char* key, bool fallback) { // Downcast a contiguous F32 buffer to bf16 (round-to-nearest-even), the same // rounding torch's .to(bfloat16) and vt::F32ToBF16 use. -void F32ToBf16Into(const float* src, int64_t n, uint16_t* dst) { - for (int64_t i = 0; i < n; ++i) dst[i] = vt::F32ToBF16(src[i]); +// `src` is `const void*`: every caller below hands it a pointer INTO the mmap'd +// safetensors payload, whose per-tensor byte offset is the running total of +// everything ahead of it and so need not be a multiple of 4. Forming or loading +// through a `const float*` there is undefined (issue #627); vt::LoadUnaligned is +// the project's seam for it. +void F32ToBf16Into(const void* src, int64_t n, uint16_t* dst) { + const auto* bytes = static_cast(src); + for (int64_t i = 0; i < n; ++i) { + dst[i] = vt::F32ToBF16(vt::LoadUnaligned(bytes + i * 4)); + } } // F32 tensor -> owned bf16, copied verbatim (optionally reshaped). Mirrors @@ -80,8 +89,7 @@ OwnedTensor LoadF32ToBf16Direct(const TensorResolver& get, const std::string& na const int64_t n = o.Numel(); VT_CHECK(t.nbytes == static_cast(n) * sizeof(float), "olmo2: byte-size mismatch for " + name); - F32ToBf16Into(reinterpret_cast(t.data), n, - reinterpret_cast(o.bytes.data())); + F32ToBf16Into(t.data, n, reinterpret_cast(o.bytes.data())); MaybeReleaseSourcePages(t.data, t.nbytes); return o; } @@ -96,11 +104,13 @@ OwnedTensor LoadF32ToBf16Transposed(const TensorResolver& get, const int64_t out_dim = t.shape[0]; const int64_t in_dim = t.shape[1]; OwnedTensor o = MakeOwned(vt::DType::kBF16, {in_dim, out_dim}); - const auto* src = reinterpret_cast(t.data); + // Unaligned: `t.data` is an arbitrary byte offset into the mmap (#627). + const uint8_t* src = t.data; auto* dst = reinterpret_cast(o.bytes.data()); for (int64_t r = 0; r < out_dim; ++r) for (int64_t c = 0; c < in_dim; ++c) - dst[c * out_dim + r] = vt::F32ToBF16(src[r * in_dim + c]); + dst[c * out_dim + r] = + vt::F32ToBF16(vt::LoadUnaligned(src + (r * in_dim + c) * 4)); MaybeReleaseSourcePages(t.data, t.nbytes); return o; } @@ -136,7 +146,7 @@ OwnedTensor LoadMergedF32ToBf16RawNK(const TensorResolver& get, const int64_t n = shard.shape[0] * in_dim; VT_CHECK(shard.nbytes == static_cast(n) * sizeof(float), "olmo2: byte-size mismatch for " + names[i]); - F32ToBf16Into(reinterpret_cast(shard.data), n, dst + off); + F32ToBf16Into(shard.data, n, dst + off); off += static_cast(n); MaybeReleaseSourcePages(shard.data, shard.nbytes); } diff --git a/src/vllm/model_executor/models/parakeet_weights.cpp b/src/vllm/model_executor/models/parakeet_weights.cpp index 3b913329e..4e3ed4ad9 100644 --- a/src/vllm/model_executor/models/parakeet_weights.cpp +++ b/src/vllm/model_executor/models/parakeet_weights.cpp @@ -56,6 +56,7 @@ #include "vllm/model_executor/models/parakeet_encoder.h" #include "vllm/model_executor/models/parakeet_transducer.h" #include "vt/dtype.h" +#include "vt/unaligned.h" namespace vllm::multimodal { namespace { @@ -100,9 +101,11 @@ std::vector ToF32(const std::string& name, const StTensor& t) { if (t.nbytes != out.size() * sizeof(uint16_t)) { throw std::runtime_error("parakeet: byte count mismatch for " + name); } - const uint16_t* src = reinterpret_cast(t.data); + // Unaligned: `t.data` is an arbitrary byte offset into the mmap (#627). + const bool f16 = (t.dtype == "F16"); for (size_t i = 0; i < out.size(); ++i) { - out[i] = (t.dtype == "F16") ? vt::F16ToF32(src[i]) : vt::BF16ToF32(src[i]); + const uint16_t bits = vt::LoadUnaligned(t.data + i * 2); + out[i] = f16 ? vt::F16ToF32(bits) : vt::BF16ToF32(bits); } } else { throw std::runtime_error("parakeet: unsupported dtype " + t.dtype + " for " + name); diff --git a/src/vllm/model_executor/models/phi_weights.cpp b/src/vllm/model_executor/models/phi_weights.cpp index f4ff4360a..d24233fbf 100644 --- a/src/vllm/model_executor/models/phi_weights.cpp +++ b/src/vllm/model_executor/models/phi_weights.cpp @@ -32,6 +32,7 @@ #include "vllm/model_executor/model_loader/safetensors_reader.h" #include "vllm/model_executor/models/dense_weight_loaders.h" #include "vt/dtype.h" +#include "vt/unaligned.h" namespace vllm { namespace { @@ -46,8 +47,16 @@ using dense_loaders::MakeOwned; // `.to(bfloat16)` uses (F16 is a subset of F32, so F16ToF32 is exact). inline uint16_t F16ToBf16(uint16_t h) { return vt::F32ToBF16(vt::F16ToF32(h)); } -void F16ToBf16Into(const uint16_t* src, int64_t n, uint16_t* dst) { - for (int64_t i = 0; i < n; ++i) dst[i] = F16ToBf16(src[i]); +// `src` is `const void*`: every caller below hands it a pointer INTO the mmap'd +// safetensors payload, whose per-tensor byte offset is the running total of +// everything ahead of it and so need not be even. Forming or loading through a +// `const uint16_t*` there is undefined (issue #627); vt::LoadUnaligned is the +// project's seam for it. +void F16ToBf16Into(const void* src, int64_t n, uint16_t* dst) { + const auto* bytes = static_cast(src); + for (int64_t i = 0; i < n; ++i) { + dst[i] = F16ToBf16(vt::LoadUnaligned(bytes + i * 2)); + } } // F16 tensor -> owned bf16, copied verbatim (optionally reshaped). Mirrors @@ -61,8 +70,7 @@ OwnedTensor LoadF16ToBf16Direct(const TensorResolver& get, const std::string& na const int64_t n = o.Numel(); VT_CHECK(t.nbytes == static_cast(n) * sizeof(uint16_t), "phi: byte-size mismatch for " + name); - F16ToBf16Into(reinterpret_cast(t.data), n, - reinterpret_cast(o.bytes.data())); + F16ToBf16Into(t.data, n, reinterpret_cast(o.bytes.data())); MaybeReleaseSourcePages(t.data, t.nbytes); return o; } @@ -76,11 +84,13 @@ OwnedTensor LoadF16ToBf16Transposed(const TensorResolver& get, const int64_t out_dim = t.shape[0]; const int64_t in_dim = t.shape[1]; OwnedTensor o = MakeOwned(vt::DType::kBF16, {in_dim, out_dim}); - const auto* src = reinterpret_cast(t.data); + // Unaligned: `t.data` is an arbitrary byte offset into the mmap (#627). + const uint8_t* src = t.data; auto* dst = reinterpret_cast(o.bytes.data()); for (int64_t r = 0; r < out_dim; ++r) for (int64_t c = 0; c < in_dim; ++c) - dst[c * out_dim + r] = F16ToBf16(src[r * in_dim + c]); + dst[c * out_dim + r] = + F16ToBf16(vt::LoadUnaligned(src + (r * in_dim + c) * 2)); MaybeReleaseSourcePages(t.data, t.nbytes); return o; } @@ -115,7 +125,7 @@ OwnedTensor LoadMergedF16ToBf16RawNK(const TensorResolver& get, const int64_t n = shard.shape[0] * in_dim; VT_CHECK(shard.nbytes == static_cast(n) * sizeof(uint16_t), "phi: byte-size mismatch for " + names[i]); - F16ToBf16Into(reinterpret_cast(shard.data), n, dst + off); + F16ToBf16Into(shard.data, n, dst + off); off += static_cast(n); MaybeReleaseSourcePages(shard.data, shard.nbytes); } @@ -148,7 +158,7 @@ OwnedTensor LoadMergedF16ToBf16Vector(const TensorResolver& get, const int64_t n = shard.shape[0]; VT_CHECK(shard.nbytes == static_cast(n) * sizeof(uint16_t), "phi: byte-size mismatch for " + names[i]); - F16ToBf16Into(reinterpret_cast(shard.data), n, dst + off); + F16ToBf16Into(shard.data, n, dst + off); off += static_cast(n); MaybeReleaseSourcePages(shard.data, shard.nbytes); } diff --git a/src/vllm/model_executor/models/qwen3_5_common.h b/src/vllm/model_executor/models/qwen3_5_common.h index d22297d3a..1e4ca96c7 100644 --- a/src/vllm/model_executor/models/qwen3_5_common.h +++ b/src/vllm/model_executor/models/qwen3_5_common.h @@ -30,6 +30,24 @@ inline constexpr ModelInfo kQwen3_5Info{ .score_type = "bi-encoder", }; +// The same record for the TEXT-ONLY arms `Qwen3_5ForCausalLM` / +// `Qwen3_5MoeForCausalLM`, whose upstream base `Qwen3_5ForCausalLMBase` +// inherits HasInnerState + IsHybrid but NOT SupportsMultiModal +// (vllm/model_executor/models/qwen3_5.py:287-296 @ `ad5d29db7`, PR #50210 — +// AHEAD of our `555967922` parity pin and recorded as such). Their multimodal +// wrappers are the separate `ForConditionalGeneration` registrations, so the +// ONLY difference is supports_multimodal. `has_inner_state` stays false for the +// same reason `KimiLinearForCausalLM` leaves it false: our ModelInfo is a +// consumed subset whose only reader short-circuits on is_hybrid. +inline constexpr ModelInfo kQwen3_5TextInfo{ + .is_text_generation_model = true, + .is_pooling_model = false, + .is_hybrid = true, + .has_inner_state = false, + .supports_multimodal = false, + .score_type = "bi-encoder", +}; + // Per-family config hook. LoadHfConfig/HfConfigFromGguf already materialize the // consumed Qwen fields; this explicit hook is where a family adds normalization // or validation without changing the registry/runner contract. diff --git a/src/vllm/model_executor/models/qwen3_5_dense.cpp b/src/vllm/model_executor/models/qwen3_5_dense.cpp index 74229a26a..4914e4733 100644 --- a/src/vllm/model_executor/models/qwen3_5_dense.cpp +++ b/src/vllm/model_executor/models/qwen3_5_dense.cpp @@ -243,4 +243,17 @@ std::unique_ptr BorrowQwen3_5DenseLoadedModel( REGISTER_VLLM_MODEL(qwen3_5_dense, "Qwen3_5ForConditionalGeneration", kQwen3_5DenseFactory, kQwen3_5Info) +// TEXT-ONLY arm of the SAME backbone. Upstream's `Qwen3_5ForCausalLM` IS +// `Qwen3_5ForCausalLMBase` unchanged (`class Qwen3_5ForCausalLM(...): pass`, +// qwen3_5.py:439-440 @ `ad5d29db7`) and is registered against the same `qwen3_5` +// module (registry.py:202 @ `ad5d29db7`, PR #50210), so this is the SAME +// factory, additively: no forward, no KV-cache spec and no loader fork. +// +// AHEAD OF THE PIN, DELIBERATELY. `555967922` (.agents/upstream-sync.md) carries +// only the ForConditionalGeneration entries; the text-only arms landed upstream +// after it. This is a forward port of ONE upstream PR and does not advance the +// pin. See .agents/specs/qwen38-text-only.md §Gates for the owed run gate. +REGISTER_VLLM_MODEL(qwen3_5_dense_text, "Qwen3_5ForCausalLM", + kQwen3_5DenseFactory, kQwen3_5TextInfo) + } // namespace vllm diff --git a/src/vllm/model_executor/models/qwen3_5_dense_weights.cpp b/src/vllm/model_executor/models/qwen3_5_dense_weights.cpp index dc904bf3d..6be8446ac 100644 --- a/src/vllm/model_executor/models/qwen3_5_dense_weights.cpp +++ b/src/vllm/model_executor/models/qwen3_5_dense_weights.cpp @@ -18,6 +18,7 @@ #include "vllm/platforms/interface.h" #include "vt/backend.h" #include "vt/dtype.h" +#include "vt/unaligned.h" namespace vllm { @@ -114,8 +115,10 @@ OwnedTensor LoadToF32(const TensorResolver& get, const std::string& name) { "qwen3_5 dense: byte-size mismatch for " + name); std::memcpy(dst, t.data, t.nbytes); } else { - const auto* src = reinterpret_cast(t.data); - for (int64_t i = 0; i < n; ++i) dst[i] = vt::BF16ToF32(src[i]); + // Unaligned: `t.data` is an arbitrary byte offset into the mmap (#627). + for (int64_t i = 0; i < n; ++i) { + dst[i] = vt::BF16ToF32(vt::LoadUnaligned(t.data + i * 2)); + } } MaybeReleaseSourcePages(t.data, t.nbytes); return o; @@ -624,12 +627,12 @@ OwnedTensor MaterializeCtNvfp4Bf16Transposed(const TensorResolver& get, return o; } -Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer(const TensorResolver& get, - const TensorExists& has, - const std::string& layer_type, - int64_t layer_idx) { +Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer( + const TensorResolver& get, const TensorExists& has, + const std::string& layer_type, int64_t layer_idx, + const std::string& backbone_prefix) { const std::string base = - "model.language_model.layers." + std::to_string(layer_idx) + "."; + backbone_prefix + "layers." + std::to_string(layer_idx) + "."; Qwen3_5DenseLayerWeights layer; layer.input_layernorm = LoadModelBf16Direct(get, base + "input_layernorm.weight"); @@ -648,21 +651,31 @@ Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer(const TensorResolver& get, return layer; } -Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer(const TensorResolver& get, - const std::string& layer_type, - int64_t layer_idx) { +Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer( + const TensorResolver& get, const std::string& layer_type, int64_t layer_idx, + const std::string& backbone_prefix) { // The public resolver-only seam is used by the compressed-tensors parity // fixture, where every routed projection is NVFP4. const TensorExists has = [](const std::string&) { return true; }; - return LoadQwen3_5DenseLayer(get, has, layer_type, layer_idx); + return LoadQwen3_5DenseLayer(get, has, layer_type, layer_idx, + backbone_prefix); } Qwen3_5DenseWeights LoadQwen3_5Dense(const std::vector& shards, const HfConfig& config, vt::Queue* load_queue) { std::unordered_map where; - for (const SafetensorsFile& shard : shards) - for (const std::string& name : shard.Names()) where[name] = &shard; + std::vector all_names; + for (const SafetensorsFile& shard : shards) { + for (const std::string& name : shard.Names()) { + where[name] = &shard; + all_names.push_back(name); + } + } + // ONE namespace decision for the whole checkpoint (qwen3_5_weights.h): the + // VL-nested spelling for the wrappers we gate, the flat `model.` spelling for + // a text-only arm, and a refusal for a mixed index. + const std::string backbone = ResolveQwen3_5BackbonePrefix(all_names); const TensorResolver get = [&where](const std::string& name) -> const StTensor& { auto it = where.find(name); @@ -679,10 +692,8 @@ Qwen3_5DenseWeights LoadQwen3_5Dense(const std::vector& shards, "qwen3_5 dense: layer_types size must equal num_hidden_layers"); Qwen3_5DenseWeights w; - w.embed_tokens = - LoadBf16Direct(get, "model.language_model.embed_tokens.weight"); - w.final_norm = - LoadModelBf16Direct(get, "model.language_model.norm.weight"); + w.embed_tokens = LoadBf16Direct(get, backbone + "embed_tokens.weight"); + w.final_norm = LoadModelBf16Direct(get, backbone + "norm.weight"); // The 27B owns an explicit head; smaller Qwen3.5 checkpoints tie logits to // the embedding table and omit lm_head.weight. if (DenseCheckpointHasLmHead(has, "lm_head")) { @@ -695,7 +706,7 @@ Qwen3_5DenseWeights LoadQwen3_5Dense(const std::vector& shards, bool direct_device = DirectDeviceLoadEligible(load_queue); for (int64_t l = 0; l < config.num_hidden_layers; ++l) { w.layers.push_back(LoadQwen3_5DenseLayer( - get, has, config.layer_types[static_cast(l)], l)); + get, has, config.layer_types[static_cast(l)], l, backbone)); if (direct_device) { direct_device = IsPlainBf16Qwen3_5Dense(w); if (direct_device) StageAndReleaseLoadedDense(w, *load_queue); diff --git a/src/vllm/model_executor/models/qwen3_5_moe.cpp b/src/vllm/model_executor/models/qwen3_5_moe.cpp index 1e33afb76..5c87a464d 100644 --- a/src/vllm/model_executor/models/qwen3_5_moe.cpp +++ b/src/vllm/model_executor/models/qwen3_5_moe.cpp @@ -215,4 +215,20 @@ std::unique_ptr BorrowQwen3_5MoeLoadedModel( REGISTER_VLLM_MODEL(qwen3_5_moe, "Qwen3_5MoeForConditionalGeneration", kQwen3_5MoeFactory, kQwen3_5Info) +// TEXT-ONLY arm of the SAME backbone. Upstream registers it against the same +// `qwen3_5` module (registry.py:202-203 @ `ad5d29db7`, PR #50210) and its class +// is `Qwen3_5ForCausalLMBase` plus `set_moe_parameters()` — not a separate model +// (qwen3_5.py:443-449). So this is the SAME factory, additively: no forward, no +// KV-cache spec and no loader fork. `Qwen/Qwen3.8-2.4T-A95B` is the motivating +// checkpoint; it declares `Qwen3_5MoeForCausalLM` / `qwen3_5_moe_text` and is +// the 35B-A3B architecture at larger scale, all of it config-driven. +// +// AHEAD OF THE PIN, DELIBERATELY. `555967922` (.agents/upstream-sync.md) carries +// only the ForConditionalGeneration entries; the text-only arms landed upstream +// after it. This is a forward port of ONE upstream PR and does not advance the +// pin. There is NO run gate for the 2.4T checkpoint on this hardware — see +// .agents/specs/qwen38-text-only.md §Gates, which records that gate as OWED. +REGISTER_VLLM_MODEL(qwen3_5_moe_text, "Qwen3_5MoeForCausalLM", + kQwen3_5MoeFactory, kQwen3_5TextInfo) + } // namespace vllm diff --git a/src/vllm/model_executor/models/qwen3_5_weights.cpp b/src/vllm/model_executor/models/qwen3_5_weights.cpp index 0f2472138..d87977228 100644 --- a/src/vllm/model_executor/models/qwen3_5_weights.cpp +++ b/src/vllm/model_executor/models/qwen3_5_weights.cpp @@ -3,6 +3,7 @@ // (.agents/specs/qwen36-forward-notes.md §6). #include "vllm/model_executor/models/qwen3_5_weights.h" +#include #include #include #include @@ -10,6 +11,7 @@ #include #include #include +#include #include #if defined(__unix__) || defined(__APPLE__) @@ -20,6 +22,7 @@ #include "vllm/model_executor/model_loader/nvfp4_dequant.h" #include "vt/backend.h" #include "vt/dtype.h" +#include "vt/unaligned.h" namespace vllm { @@ -288,12 +291,20 @@ float ReadF32Scalar(const StTensor& t) { } // src bf16 [rows, cols] -> dst bf16 [cols, rows]. -void TransposeBf16(const uint16_t* src, int64_t rows, int64_t cols, - uint16_t* dst) { +// +// `src` is `const void*`, not `const uint16_t*`, because the caller below hands +// it a pointer INTO the mmap'd safetensors payload. A tensor's offset there is +// the running byte total of everything ahead of it, so a bf16 tensor that +// follows an odd-length one starts on an odd byte and the typed pointer is +// undefined to form or load through (issue #627). `vt::LoadUnaligned` is the +// project's seam for that — the same one `ReadF32Scalar` above open-codes with +// memcpy and `dense_loaders::TransposeBf16` already uses for this exact loop. +void TransposeBf16(const void* src, int64_t rows, int64_t cols, uint16_t* dst) { + const auto* bytes = static_cast(src); for (int64_t r = 0; r < rows; ++r) { - const uint16_t* src_row = src + r * cols; + const uint8_t* src_row = bytes + r * cols * 2; for (int64_t c = 0; c < cols; ++c) { - dst[c * rows + r] = src_row[c]; + dst[c * rows + r] = vt::LoadUnaligned(src_row + c * 2); } } } @@ -329,7 +340,7 @@ OwnedTensor LoadBf16Transposed(const TensorResolver& get, const int64_t out_dim = t.shape[0]; const int64_t in_dim = t.shape[1]; OwnedTensor o = MakeOwned(vt::DType::kBF16, {in_dim, out_dim}); - TransposeBf16(reinterpret_cast(t.data), out_dim, in_dim, + TransposeBf16(t.data, out_dim, in_dim, reinterpret_cast(o.bytes.data())); MaybeReleaseSourcePages(t.data, t.nbytes); return o; @@ -343,9 +354,12 @@ OwnedTensor LoadBf16ToF32(const TensorResolver& get, const std::string& name) { "qwen3_5 weights: expected 1-D tensor for " + name); const int64_t n = t.shape[0]; OwnedTensor o = MakeOwned(vt::DType::kF32, {n}); - const auto* src = reinterpret_cast(t.data); + // Unaligned: `t.data` is an arbitrary byte offset into the mmap (#627). + const uint8_t* src = t.data; auto* dst = reinterpret_cast(o.bytes.data()); - for (int64_t i = 0; i < n; ++i) dst[i] = vt::BF16ToF32(src[i]); + for (int64_t i = 0; i < n; ++i) { + dst[i] = vt::BF16ToF32(vt::LoadUnaligned(src + i * 2)); + } MaybeReleaseSourcePages(t.data, t.nbytes); return o; } @@ -555,9 +569,10 @@ MoeBlockWeights LoadMoe(const TensorResolver& get, const std::string& base, Qwen3_5MoeLayerWeights LoadLayerImpl(const TensorResolver& get, const std::string& layer_type, int64_t layer_idx, int64_t num_experts, - bool with_experts) { + bool with_experts, + const std::string& backbone_prefix) { const std::string base = - "model.language_model.layers." + std::to_string(layer_idx) + "."; + backbone_prefix + "layers." + std::to_string(layer_idx) + "."; Qwen3_5MoeLayerWeights layer; layer.input_layernorm = LoadBf16Direct(get, base + "input_layernorm.weight"); layer.post_attention_layernorm = @@ -575,8 +590,122 @@ Qwen3_5MoeLayerWeights LoadLayerImpl(const TensorResolver& get, return layer; } +// True iff any name in `names` is a backbone tensor under `prefix`. Only the +// three structural backbone spellings vote (see qwen3_5_weights.h): the +// vision tower (`model.visual.*`) and the top-level `lm_head.*` / `mtp.*` are +// deliberately NOT backbone names, so they cannot decide the namespace. +bool HasBackboneUnder(const std::vector& names, + std::string_view prefix) { + const std::string embed = std::string(prefix) + "embed_tokens.weight"; + const std::string norm = std::string(prefix) + "norm.weight"; + const std::string layers = std::string(prefix) + "layers."; + for (const std::string& name : names) { + if (name == embed || name == norm) return true; + if (name.compare(0, layers.size(), layers) == 0) return true; + } + return false; +} + +// --- Unimplemented MoE expert arms, REFUSED BY NAME (issue #490) ------------- +// +// `LoadMoeExpertsInto` above reads exactly ONE routed-expert layout: per-expert +// NVFP4 (`...mlp.experts..{gate,up,down}_proj` through `LoadNvfp4Raw`, which +// hard-requires a `U8` `.weight`, an `F8_E4M3` `.weight_scale` and a +// `.weight_scale_2`). There is no stacked branch and no bf16 branch — unlike +// `gemma4_weights.cpp:326`, which dispatches between layouts. +// +// The PUBLISHED Qwen3.5-family MoE repos do not have that layout. Read off the +// live safetensors indices 2026-08-12: `Qwen/Qwen3.8-2.4T-A95B` has 93x +// `mlp.experts.gate_up_proj` + 93x `.down_proj` (3-D STACKED) and ZERO +// `weight_scale` / `input_scale` tensors; `Qwen/Qwen3.6-35B-A3B` is the same +// under the VL prefix. Our gated 35B row reads the REQUANTIZED +// `nvidia/Qwen3.6-35B-A3B-NVFP4`, so this loader has never read a published +// Qwen bf16 MoE checkpoint. Left alone, such a load dies at +// `LoadNvfp4Raw(get, "lm_head")` with "expected U8 for lm_head.weight" — which +// reads as a corrupt checkpoint, not as an unimplemented arm. +// +// AGENTS.md: an arm that is not implemented "is refused with a message naming +// the missing piece ... never left to be discovered later", and the row's spec +// (.agents/specs/qwen38-text-only.md) says the same in its stop conditions. +// This is that refusal and ONLY that: the stacked/bf16 MoE expert arm is OWED, +// and implementing it needs its own spec, RED-first test and NVFP4 inertness +// proof. Inert on the supported layout — every name it inspects already has to +// exist for the load to succeed at all. +void CheckMoeExpertLayoutSupported(const std::vector& names, + const std::string& backbone) { + static const std::string kRequired = + " This loader implements only the per-expert NVFP4 layout: " + ".mlp.experts..{gate,up,down}_proj.weight (U8 packed) + " + ".weight_scale (F8_E4M3) + .weight_scale_2, and lm_head the same way. The " + "published bf16 repos (Qwen/Qwen3.8-2.4T-A95B, Qwen/Qwen3.6-35B-A3B) ship " + "the 3-D stacked, unquantized layout; an NVFP4 requant (e.g. " + "nvidia/Qwen3.6-35B-A3B-NVFP4) ships the supported one. The stacked and " + "unquantized MoE expert arms are OWED, not silently unsupported: see " + ".agents/specs/qwen38-text-only.md."; + const std::string layers = backbone + "layers."; + const std::string experts = ".mlp.experts."; + const std::string weight = ".weight"; + const std::unordered_set present(names.begin(), names.end()); + for (const std::string& name : names) { + if (name.compare(0, layers.size(), layers) != 0) continue; + const size_t at = name.find(experts); + if (at == std::string::npos) continue; + const size_t rest = at + experts.size(); + if (rest >= name.size()) continue; + // `experts.` is the per-expert spelling; anything else — the + // published `experts.gate_up_proj` / `experts.down_proj` — is the stacked + // one, where a single 3-D tensor holds every expert. + if (std::isdigit(static_cast(name[rest])) == 0) { + VT_CHECK(false, + "qwen3_5 weights: 3-D stacked routed experts are not implemented " + "for the safetensors MoE arm -- found \"" + + name + "\"." + kRequired); + } + if (name.size() > weight.size() && + name.compare(name.size() - weight.size(), weight.size(), weight) == 0 && + present.count(name + "_scale") == 0) { + VT_CHECK(false, + "qwen3_5 weights: unquantized routed experts are not implemented " + "for the safetensors MoE arm -- \"" + + name + "\" has no \"" + name + "_scale\" beside it." + + kRequired); + } + } + // The MoE head is likewise NVFP4-only here, where the DENSE loader routes a + // head by dtype (`LoadDenseLmHead` / `LoadLmHeadAnyDtype`). A checkpoint with + // no `lm_head.weight` at all is the tied-head case and is not this refusal. + if (present.count("lm_head.weight") != 0 && + present.count("lm_head.weight_scale") == 0) { + VT_CHECK(false, + "qwen3_5 weights: an unquantized lm_head is not implemented for " + "the safetensors MoE arm -- \"lm_head.weight\" has no " + "\"lm_head.weight_scale\" beside it." + + kRequired); + } +} + } // namespace +std::string ResolveQwen3_5BackbonePrefix( + const std::vector& tensor_names) { + // `model.language_model.` is tested FIRST because it is also a `model.` + // name: a plain "starts with model." test would match both spellings. + const bool vl = HasBackboneUnder(tensor_names, kQwen3_5VlBackbonePrefix); + // ...so the canonical probe must EXCLUDE the VL-prefixed names, which the + // backbone spellings above already do (`model.language_model.` is neither + // `model.embed_tokens.weight`, nor `model.norm.weight`, nor `model.layers.`). + const bool flat = HasBackboneUnder(tensor_names, kQwen3_5TextBackbonePrefix); + VT_CHECK(!(vl && flat), + "qwen3_5 weights: checkpoint carries backbone tensors under BOTH " + "\"model.language_model.\" and \"model.\"; refusing a mixed weight " + "namespace rather than binding half the model from each"); + VT_CHECK(vl || flat, + "qwen3_5 weights: no Qwen3.5 backbone tensors found under either " + "\"model.language_model.\" or \"model.\""); + return std::string(vl ? kQwen3_5VlBackbonePrefix + : kQwen3_5TextBackbonePrefix); +} + // External-linkage seam so the DENSE loader can keep an FP8 GDN tower native. Fp8Weight LoadFp8RawShared(const TensorResolver& get, const std::string& proj) { return LoadFp8Raw(get, proj); @@ -585,9 +714,10 @@ Fp8Weight LoadFp8RawShared(const TensorResolver& get, const std::string& proj) { Qwen3_5MoeLayerWeights LoadQwen3_5MoeLayer(const TensorResolver& get, const std::string& layer_type, int64_t layer_idx, - int64_t num_experts) { + int64_t num_experts, + const std::string& backbone_prefix) { return LoadLayerImpl(get, layer_type, layer_idx, num_experts, - /*with_experts=*/true); + /*with_experts=*/true, backbone_prefix); } Qwen3_5MoeWeights LoadQwen3_5Moe( @@ -601,9 +731,21 @@ Qwen3_5MoeWeights LoadQwen3_5Moe( // which `shards_owner` keeps mmap'd. auto where = std::make_shared>(); + std::vector all_names; for (const SafetensorsFile& shard : shards) { - for (const std::string& name : shard.Names()) (*where)[name] = &shard; + for (const std::string& name : shard.Names()) { + (*where)[name] = &shard; + all_names.push_back(name); + } } + // ONE namespace decision for the whole checkpoint (qwen3_5_weights.h): the + // VL-nested spelling for the wrappers we gate, the flat `model.` spelling for + // a text-only arm, and a refusal for a mixed index. + const std::string backbone = ResolveQwen3_5BackbonePrefix(all_names); + // ...and ONE decision about the routed-expert layout, before any tensor is + // touched, so an arm we do not implement is refused by name rather than + // discovered as a dtype complaint about `lm_head` (issue #490). + CheckMoeExpertLayoutSupported(all_names, backbone); const TensorResolver get = [where](const std::string& name) -> const StTensor& { auto it = where->find(name); @@ -628,16 +770,16 @@ Qwen3_5MoeWeights LoadQwen3_5Moe( const bool defer_experts = shards_owner != nullptr; Qwen3_5MoeWeights w; - w.embed_tokens = - LoadBf16Direct(get, "model.language_model.embed_tokens.weight"); - w.final_norm = LoadBf16Direct(get, "model.language_model.norm.weight"); + w.embed_tokens = LoadBf16Direct(get, backbone + "embed_tokens.weight"); + w.final_norm = LoadBf16Direct(get, backbone + "norm.weight"); w.lm_head_fp4 = LoadNvfp4Raw(get, "lm_head"); // M2.2b fp4-resident w.layers.reserve(static_cast(config.num_hidden_layers)); for (int64_t l = 0; l < config.num_hidden_layers; ++l) { w.layers.push_back(LoadLayerImpl(get, config.layer_types[static_cast(l)], l, config.num_experts, - /*with_experts=*/!defer_experts)); + /*with_experts=*/!defer_experts, + backbone)); } if (defer_experts) { @@ -647,7 +789,9 @@ Qwen3_5MoeWeights LoadQwen3_5Moe( // last layer is built. Does NOT capture the (movable) Qwen3_5MoeWeights — the // target MoE block is passed in by reference, so the closure survives the // model's move into the LoadedModel. - w.load_layer_experts = [where, shards_owner, num_experts]( + // `backbone` is captured BY VALUE: the closure outlives this frame, and it + // must keep using the ONE namespace resolved above rather than re-deciding. + w.load_layer_experts = [where, shards_owner, num_experts, backbone]( int64_t layer, MoeBlockWeights& moe) { const TensorResolver g = [where](const std::string& name) -> const StTensor& { @@ -656,8 +800,8 @@ Qwen3_5MoeWeights LoadQwen3_5Moe( "qwen3_5 weights: tensor not found: " + name); return it->second->Get(name); }; - const std::string mlp = "model.language_model.layers." + - std::to_string(layer) + ".mlp."; + const std::string mlp = + backbone + "layers." + std::to_string(layer) + ".mlp."; LoadMoeExpertsInto(g, mlp, num_experts, moe); }; } diff --git a/src/vllm/model_executor/models/qwen3_dspark_weights.cpp b/src/vllm/model_executor/models/qwen3_dspark_weights.cpp index 85e2fa811..b5ec931d0 100644 --- a/src/vllm/model_executor/models/qwen3_dspark_weights.cpp +++ b/src/vllm/model_executor/models/qwen3_dspark_weights.cpp @@ -14,6 +14,7 @@ #include "vllm/model_executor/model_loader/safetensors_reader.h" #include "vllm/model_executor/models/qwen3_dspark.h" #include "vt/dtype.h" +#include "vt/unaligned.h" namespace vllm { namespace { @@ -96,10 +97,13 @@ Qwen3DSparkWeights LoadQwen3DSpark(const TensorResolver& get, const HfConfig& co VT_CHECK(d2t->dtype == "I64", "qwen3_dspark: expected I64 for d2t"); VT_CHECK(d2t->shape.size() == 1 && d2t->shape[0] == w.draft_vocab_size, "qwen3_dspark: d2t must be [draft_vocab_size]"); - const auto* src = reinterpret_cast(d2t->data); + // Unaligned: `d2t->data` is an arbitrary byte offset into the mmap and I64 + // wants 8-byte alignment, so a `const int64_t*` onto it is undefined to form + // or load through (issue #627). + const uint8_t* src = d2t->data; w.draft_id_to_target_id.resize(static_cast(w.draft_vocab_size)); for (int64_t i = 0; i < w.draft_vocab_size; ++i) { - const int64_t off = src[i]; + const int64_t off = vt::LoadUnaligned(src + i * 8); const int64_t target = i + off; VT_CHECK(target >= 0 && target < w.vocab_size, "qwen3_dspark: d2t maps a draft id outside the target vocab"); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 8467c02f7..d90527b62 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -247,6 +247,12 @@ foreach(_gateup_lever VT_DENSE_MARLIN_GATEUP VT_NVFP4_MARLIN VT_MOE_FUSED_W13) PROPERTIES ENVIRONMENT "${_gateup_lever}=0") endforeach() vllm_cpp_add_test(test_qwen35_plain_weights vllm/models/test_qwen35_plain_weights.cpp) +vllm_cpp_add_test(test_qwen3_8_text_only vllm/models/test_qwen3_8_text_only.cpp) +# The PUBLISHED `Qwen/Qwen3.8-2.4T-A95B` config.json, committed VERBATIM, so the +# config-resolution case reads the real document (nested `rope_parameters`, 92 +# `layer_types`, transformers-4.57.3 `dtype`) instead of a paraphrase of it. +target_compile_definitions(test_qwen3_8_text_only PRIVATE + QWEN3_8_CONFIG_FIXTURE="${CMAKE_CURRENT_SOURCE_DIR}/vllm/models/fixtures/qwen3_8_2_4t_a95b/config.json") # POOL-DEVICE-KEY (#516, .agents/specs/pool-device-key.md): the shared scratch # pool's free list must be keyed by DEVICE as well as size class, or a block # allocated through one backend is handed to a DBuf running on another — diff --git a/tests/scripts/test_agent_record.py b/tests/scripts/test_agent_record.py index 8cb71bf53..5922f5d29 100644 --- a/tests/scripts/test_agent_record.py +++ b/tests/scripts/test_agent_record.py @@ -329,7 +329,8 @@ def test_model_row_ratchet_is_load_bearing(self) -> None: (`c8fc24a50`); the seven recipe architectures that had no row at all took it 362 -> 369 (#609, #610, `eba6ab7c7`); LTX-2.5 took it 369 -> 370 (#435, `cefacd2d0`); IndexTTS-2.5 took it 370 -> 372, being two - architectures (#634); MiniMax-Music3 took it to 373 (#672). Without this, + architectures (#634); MiniMax-Music3 took it to 373 (#672); and the two + text-only Qwen3.5 arms took it 373 -> 375 (#490). Without this, bumping the number to silence a failure is indistinguishable from bumping it because a row really landed. """ @@ -887,5 +888,52 @@ def test_the_backend_pin_is_load_bearing(self) -> None: ) +class Qwen35TextOnlyRowsAreCounted(unittest.TestCase): + """The MODEL ratchet bump 373 -> 375 is backed by two real rows (#490). + + Same shape, and the same reason, as the BACKEND class above: the count is + re-pinned by hand, so a bump with nothing behind it is indistinguishable + from a bump for rows that really landed. `test_model_row_ratchet_is_ + load_bearing` proves the pin BINDS by moving it, which holds for any value + of the pin; it cannot say whether THIS value is the right one. These two + tests do, by tying the pin to the rows the matrix actually carries. + """ + + ROWS = ( + "MODEL-TEXT-qwen3-5-qwen3-5-for-causal-lm", + "MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm", + ) + + def test_both_text_only_rows_exist_in_the_model_matrix(self) -> None: + lines = ( + (ROOT / ".agents/model-matrix.md") + .read_text(encoding="utf-8") + .splitlines() + ) + for row in self.ROWS: + matching = [line for line in lines if line.startswith(f"| `{row}` |")] + self.assertEqual(len(matching), 1, f"{row} must appear exactly once") + + def test_the_model_pin_equals_the_rows_the_matrix_carries(self) -> None: + """MUTATION: the pin and the tree disagreeing by one row must be RED. + + Counted the way `check_matrices` counts, so a pin left behind by a + landing row -- or moved ahead of one -- fails here and not only inside + the checker's own error list. + """ + path, expected = agent_record.MATRICES["MODEL"] + errors: list[str] = [] + rows, _ = agent_record.check_matrices(errors) + actual = sum( + row.item_id.startswith("MODEL-") for row in rows if row.path == path + ) + self.assertEqual( + actual, + expected, + "the MODEL pin must equal the MODEL rows model-matrix.md carries", + ) + self.assertEqual([error for error in errors if "MODEL rows" in error], []) + + if __name__ == "__main__": unittest.main() diff --git a/tests/vllm/models/fixtures/qwen3_8_2_4t_a95b/config.json b/tests/vllm/models/fixtures/qwen3_8_2_4t_a95b/config.json new file mode 100644 index 000000000..fb3dc968b --- /dev/null +++ b/tests/vllm/models/fixtures/qwen3_8_2_4t_a95b/config.json @@ -0,0 +1,142 @@ +{ + "architectures": [ + "Qwen3_5MoeForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "attn_output_gate": true, + "bos_token_id": 248044, + "dtype": "bfloat16", + "eos_token_id": 248044, + "full_attention_interval": 4, + "head_dim": 256, + "hidden_act": "silu", + "hidden_size": 8192, + "initializer_range": 0.02, + "layer_types": [ + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention" + ], + "linear_conv_kernel_dim": 4, + "linear_key_head_dim": 128, + "linear_num_key_heads": 16, + "linear_num_value_heads": 128, + "linear_value_head_dim": 128, + "mamba_ssm_dtype": "float32", + "max_position_embeddings": 262144, + "model_type": "qwen3_5_moe_text", + "moe_intermediate_size": 2048, + "mtp_num_hidden_layers": 1, + "mtp_use_dedicated_embeddings": false, + "num_attention_heads": 64, + "num_experts": 512, + "num_experts_per_tok": 10, + "num_hidden_layers": 92, + "num_key_value_heads": 4, + "output_gate_type": "swish", + "output_router_logits": false, + "pad_token_id": null, + "partial_rotary_factor": 0.25, + "rms_norm_eps": 1e-06, + "rope_parameters": { + "partial_rotary_factor": 0.25, + "rope_theta": 10000000, + "rope_type": "default" + }, + "router_aux_loss_coef": 0.001, + "shared_expert_intermediate_size": 2048, + "tie_word_embeddings": false, + "transformers_version": "4.57.3", + "use_cache": true, + "vocab_size": 248320 +} \ No newline at end of file diff --git a/tests/vllm/models/test_model_registry.cpp b/tests/vllm/models/test_model_registry.cpp index 47505fdb3..948618011 100644 --- a/tests/vllm/models/test_model_registry.cpp +++ b/tests/vllm/models/test_model_registry.cpp @@ -44,10 +44,14 @@ HfConfig Config(std::vector architectures) { TEST_CASE("registry_imports: every registered architecture has a complete factory") { const auto registrations = ModelRegistry::Registrations(); - // 34 text archs + the 3 Parakeet transcription-only archs (ARCH-ONE-SURFACE + // 36 text archs + the 3 Parakeet transcription-only archs (ARCH-ONE-SURFACE // ROW 1: ParakeetForCTC/ForRNNT/ForTDT, SupportsTranscription mirror) + the // LlamaModel embedding arch (ARCH-ONE-SURFACE ROW 6, is_pooling_model). - REQUIRE(registrations.size() == 38); + // 38 -> 40 on MODEL-QWEN38-TEXT-ONLY: `Qwen3_5ForCausalLM` and + // `Qwen3_5MoeForCausalLM`, the TEXT-ONLY arms of the Qwen3.5 backbone + // (upstream registry.py:202-203 @ `ad5d29db7`), registered against the + // EXISTING dense and MoE factories. + REQUIRE(registrations.size() == 40); for (const ModelRegistration& registration : registrations) { CAPTURE(registration.architecture); @@ -83,6 +87,10 @@ TEST_CASE("self_registration: every arch self-registers from its own TU") { CHECK(has_arch("OPTForCausalLM")); CHECK(has_arch("Qwen3_5ForConditionalGeneration")); CHECK(has_arch("Qwen3_5MoeForConditionalGeneration")); + // The text-only arms of that same backbone: two more REGISTER_VLLM_MODEL + // lines against the SAME factories, ZERO shared-array edit. + CHECK(has_arch("Qwen3_5ForCausalLM")); + CHECK(has_arch("Qwen3_5MoeForCausalLM")); // MLA campaign W7: the DeepSeek-V2 MLA model, likewise one new TU + one // REGISTER_VLLM_MODEL line and ZERO edit to a shared array. CHECK(has_arch("DeepseekV2ForCausalLM")); @@ -139,12 +147,14 @@ TEST_CASE("self_registration: every arch self-registers from its own TU") { // with the kExampleConfigArchitectures ledger; adding a model appends its two // entries here. const std::vector supported = ModelRegistry::SupportedArchs(); - REQUIRE(supported.size() == 38); + REQUIRE(supported.size() == 40); CHECK(std::is_sorted(supported.begin(), supported.end())); // The full byte-order sequence. Note "MiniCPM3" < "MiniCPMF" and "Phi3" < // "PhiF" ('3' 0x33 < 'F' 0x46); "OPT" < "Olmo" ('P' 0x50 < 'l' 0x6C); and among // the Qwens "Qwen3F" < "Qwen3M" < "Qwen3V" < "Qwen3_" - // ('F' 0x46 < 'M' 0x4D < 'V' 0x56 < '_' 0x5F). + // ('F' 0x46 < 'M' 0x4D < 'V' 0x56 < '_' 0x5F). Within "Qwen3_5[Moe]For", + // "Ca" < "Co" ('a' 0x61 < 'o' 0x6F), so each text-only ForCausalLM arm sorts + // immediately BEFORE its ForConditionalGeneration wrapper. const std::vector kSortedArchs{ "CohereForCausalLM", "DeepseekV2ForCausalLM", @@ -181,7 +191,9 @@ TEST_CASE("self_registration: every arch self-registers from its own TU") { "Qwen3ForCausalLM", "Qwen3MoeForCausalLM", "Qwen3VLForConditionalGeneration", + "Qwen3_5ForCausalLM", "Qwen3_5ForConditionalGeneration", + "Qwen3_5MoeForCausalLM", "Qwen3_5MoeForConditionalGeneration", "StableLmForCausalLM", }; @@ -242,6 +254,16 @@ TEST_CASE("registry_model_property: Qwen registrations match pinned _ModelInfo") // registrations (Kimi-K3 = KDA+MLA hybrid backbone + MoonViT-V2 vision). CHECK(registration.info.is_hybrid); CHECK(registration.info.supports_multimodal); + } else if (registration.architecture == "Qwen3_5ForCausalLM" || + registration.architecture == "Qwen3_5MoeForCausalLM") { + // MODEL-QWEN38-TEXT-ONLY: upstream's `Qwen3_5ForCausalLMBase` inherits + // HasInnerState + IsHybrid but NOT SupportsMultiModal + // (qwen3_5.py:287-296 @ `ad5d29db7`) — these are the TEXT arms, and the + // ConditionalGeneration wrappers above are the multimodal registrations. + // has_inner_state stays false by the blanket assertion above, the same + // kQwen3_5Info convention KimiLinearForCausalLM follows. + CHECK(registration.info.is_hybrid); + CHECK_FALSE(registration.info.supports_multimodal); } else if (registration.architecture == "NemotronHForCausalLM") { // Nemotron-H (MODEL-NEMOTRON-H W3, #517): text-only HYBRID -- 23 Mamba2 // layers carry a recurrent-state KV group alongside 6 GQA full-attention @@ -590,7 +612,7 @@ TEST_CASE("Qwen3.5 SSM cache dtype accepts upstream torch aliases exactly") { TEST_CASE("hf_registry_coverage: every registration has an example config fixture") { // C++ fixture registry for the currently implemented subset. Keep this list // alias-for-alias with the central ordered table, mirroring HF_EXAMPLE_MODELS. - constexpr std::array kExampleConfigArchitectures{ + constexpr std::array kExampleConfigArchitectures{ "CohereForCausalLM", "DeepseekV2ForCausalLM", "DeepseekV4ForCausalLM", @@ -626,7 +648,9 @@ TEST_CASE("hf_registry_coverage: every registration has an example config fixtur "Qwen3ForCausalLM", "Qwen3MoeForCausalLM", "Qwen3VLForConditionalGeneration", + "Qwen3_5ForCausalLM", "Qwen3_5ForConditionalGeneration", + "Qwen3_5MoeForCausalLM", "Qwen3_5MoeForConditionalGeneration", "StableLmForCausalLM", }; @@ -713,7 +737,8 @@ TEST_CASE("raise_for_unsupported: subset default message and order match oracle" "'ParakeetForCTC', 'ParakeetForRNNT', 'ParakeetForTDT', " "'Phi3ForCausalLM', 'PhiForCausalLM', 'Qwen3ForCausalLM', " "'Qwen3MoeForCausalLM', 'Qwen3VLForConditionalGeneration', " - "'Qwen3_5ForConditionalGeneration', " + "'Qwen3_5ForCausalLM', 'Qwen3_5ForConditionalGeneration', " + "'Qwen3_5MoeForCausalLM', " "'Qwen3_5MoeForConditionalGeneration', 'StableLmForCausalLM'])", std::runtime_error); @@ -736,7 +761,8 @@ TEST_CASE("raise_for_unsupported: subset default message and order match oracle" "'ParakeetForCTC', 'ParakeetForRNNT', 'ParakeetForTDT', " "'Phi3ForCausalLM', 'PhiForCausalLM', 'Qwen3ForCausalLM', " "'Qwen3MoeForCausalLM', 'Qwen3VLForConditionalGeneration', " - "'Qwen3_5ForConditionalGeneration', " + "'Qwen3_5ForCausalLM', 'Qwen3_5ForConditionalGeneration', " + "'Qwen3_5MoeForCausalLM', " "'Qwen3_5MoeForConditionalGeneration', 'StableLmForCausalLM'])", std::runtime_error); } diff --git a/tests/vllm/models/test_qwen3_8_text_only.cpp b/tests/vllm/models/test_qwen3_8_text_only.cpp new file mode 100644 index 000000000..a279a7846 --- /dev/null +++ b/tests/vllm/models/test_qwen3_8_text_only.cpp @@ -0,0 +1,1123 @@ +// Qwen3.5-family TEXT-ONLY arms: `Qwen3_5ForCausalLM` / `Qwen3_5MoeForCausalLM`. +// +// AHEAD-OF-PIN FORWARD PORT of upstream PR #50210 (`ad5d29db7`). Our parity pin +// is `555967922`, whose registry.py carries only the `ForConditionalGeneration` +// strings, so this file is deliberately anchored on a POST-PIN upstream head and +// says so; it does not advance the pin. +// +// upstream vllm/model_executor/models/registry.py:202-203 @ `ad5d29db7` +// "Qwen3_5ForCausalLM": ("qwen3_5", "Qwen3_5ForCausalLM"), +// "Qwen3_5MoeForCausalLM": ("qwen3_5", "Qwen3_5MoeForCausalLM"), +// upstream vllm/model_executor/models/qwen3_5.py:439-449 @ `ad5d29db7` +// `Qwen3_5ForCausalLM` IS `Qwen3_5ForCausalLMBase` unchanged; the MoE arm is +// that same base plus `set_moe_parameters()` — one backbone, two arms. +// upstream vllm/model_executor/models/qwen3_5.py:296-300 @ `ad5d29db7` +// WeightsMapper(orig_to_new_prefix={"model.language_model.": "model."}) +// — `model.` is CANONICAL and the VL-prefixed spelling is its accepted alias. +// +// WHAT THIS FILE DOES NOT CLAIM. `Qwen/Qwen3.8-2.4T-A95B` cannot be executed on +// this hardware (2.4T bf16 ≈ 4.8 TB, the FP8 variant ≈ 2.4 TB, GB10 has 128 GB +// unified and no smaller Qwen3.8 sibling exists), so there is NO token gate for +// that checkpoint and none is implied here. These cases pin architecture +// dispatch, flat-config resolution and weight-namespace resolution — nothing +// about generated tokens. See .agents/specs/qwen38-text-only.md §Gates. +// +// WHAT IT DOES PIN, and why each shape was chosen: +// 1. dispatch — both architecture strings resolve to the EXISTING +// dense/MoE factories with the text `_ModelInfo`. +// 2. config — the PUBLISHED config.json, committed verbatim as +// fixtures/qwen3_8_2_4t_a95b/config.json. +// 3. namespace probe — clean / VL / vision-inclusive / mtp / mixed / empty. +// 4. dense LOADER — two synthetic checkpoints, byte-identical payloads, +// only the namespace differing, loaded through the +// production `LoadQwen3_5Dense`. +// 4b. MoE LOADER — the same proof through `LoadQwen3_5Moe`, on BOTH +// expert-residency paths (eager, and the DEFERRED +// `load_layer_experts` closure actually driven). The +// MoE arm is the one this row exists for and it +// threads the prefix through three sites the dense +// loader has no analogue of. +// 5. inertness — the VL spelling stays the per-layer seam DEFAULT. +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/model_executor/models/qwen3_5_dense.h" +#include "vllm/model_executor/models/qwen3_5_weights.h" +#include "vllm/transformers_utils/hf_config.h" + +using vllm::HfConfig; +using vllm::ModelRegistration; +using vllm::ModelRegistry; + +namespace { + +HfConfig ArchConfig(std::vector architectures) { + HfConfig config; + config.architectures = std::move(architectures); + return config; +} + +// --------------------------------------------------------------------------- +// Synthetic checkpoint plumbing (same shape as tests/vllm/test_load_direct_upload +// .cpp: a real safetensors file on disk, opened through the production reader). +// --------------------------------------------------------------------------- + +std::string U64Le(uint64_t v) { + std::string s(8, '\0'); + for (int i = 0; i < 8; ++i) s[i] = static_cast((v >> (8 * i)) & 0xff); + return s; +} + +class TempFile { + public: + explicit TempFile(const std::string& bytes, const char* tag) { + static int counter = 0; + path_ = (std::filesystem::temp_directory_path() / + ("vllm_qwen3_8_" + std::string(tag) + "_" + + std::to_string(counter++) + ".safetensors")) + .string(); + std::ofstream out(path_, std::ios::binary); + out.write(bytes.data(), static_cast(bytes.size())); + } + ~TempFile() { std::remove(path_.c_str()); } + TempFile(const TempFile&) = delete; + TempFile& operator=(const TempFile&) = delete; + const std::string& path() const { return path_; } + + private: + std::string path_; +}; + +// One tiny synthetic tensor description: name + shape + safetensors dtype +// string, filled with a deterministic per-tensor pattern so a wrong binding +// shows up in the VALUES, not only in a name. The dense arm is all-BF16 (the +// default); the MoE arm needs the ONE per-class quantization `LoadQwen3_5Moe` +// can read — per-tensor FP8 attention and PER-EXPERT NVFP4 projections/head — +// because that loader hard-requires each of those dtypes +// (qwen3_5_weights.cpp:385-463) and carries no bf16 and no stacked branch. +// +// That is NOT what the published Qwen3.5-family MoE repos ship. Both +// `Qwen/Qwen3.8-2.4T-A95B` and `Qwen/Qwen3.6-35B-A3B` publish 3D-STACKED, +// UNQUANTIZED experts and zero `weight_scale` tensors; the layout below is what +// an NVFP4 requant (e.g. `nvidia/Qwen3.6-35B-A3B-NVFP4`, which is what our gated +// 35B row actually reads) ships. The published layout is a NOT-IMPLEMENTED arm +// and is pinned as a REFUSAL further down, not as a load. +struct Spec { + std::string name; + std::vector shape; + std::string dtype = "BF16"; +}; + +int64_t Numel(const std::vector& shape) { + int64_t n = 1; + for (const int64_t d : shape) n *= d; + return n; +} + +// Byte width of the safetensors dtypes this file emits. Must agree with the +// reader's own table (safetensors_reader.cpp:41-48), which REJECTS a header +// whose shape times dtype width does not equal its data_offsets span. +size_t ElemSize(const std::string& dtype) { + if (dtype == "BF16") return 2; + if (dtype == "F32") return 4; + return 1; // U8 (packed fp4 codes) / F8_E4M3 +} + +// Builds a whole safetensors file from `specs`. The bytes of a tensor depend +// ONLY on its position in `specs`, so two files built from the same specs with +// different NAMES carry byte-identical payloads — which is what lets the +// namespace test compare two loads for byte equality. +std::string BuildSafetensors(const std::vector& specs) { + std::string header = "{"; + std::string body; + uint64_t offset = 0; + for (size_t i = 0; i < specs.size(); ++i) { + const int64_t n = Numel(specs[i].shape); + const std::string& dtype = specs[i].dtype; + const size_t elem = ElemSize(dtype); + const auto nbytes = static_cast(n) * elem; + if (i != 0) header += ","; + header += "\"" + specs[i].name + "\":{\"dtype\":\"" + dtype + "\",\"shape\":["; + for (size_t d = 0; d < specs[i].shape.size(); ++d) { + if (d != 0) header += ","; + header += std::to_string(specs[i].shape[d]); + } + header += "],\"data_offsets\":[" + std::to_string(offset) + "," + + std::to_string(offset + nbytes) + "]}"; + offset += nbytes; + + const size_t at = body.size(); + body.resize(at + static_cast(nbytes)); + char* dst = body.data() + at; + if (dtype == "BF16") { + for (size_t e = 0; e < static_cast(n); ++e) { + // A finite, distinct bf16 per (tensor index, element index). + const auto v = static_cast(0x3d00 + ((i * 37 + e * 7) & 0x1ff)); + std::memcpy(dst + e * 2, &v, 2); + } + } else if (dtype == "F32") { + for (size_t e = 0; e < static_cast(n); ++e) { + // A finite, POSITIVE, distinct f32 — these are the per-tensor + // weight_scale / input_scale / weight_scale_2 scalars, and the loader + // multiplies them into `alpha`, so a NaN would make an equality + // assertion on the derived scalar meaningless. + const float v = 0.125F * static_cast((i * 5 + e * 3) % 7 + 1); + std::memcpy(dst + e * 4, &v, 4); + } + } else { + for (size_t e = 0; e < static_cast(n); ++e) { + dst[e] = static_cast((i * 37 + e * 7) & 0xff); + } + } + } + header += "}"; + return U64Le(header.size()) + header + body; +} + +// The full backbone tensor list of a ONE-layer full-attention Qwen3.5 dense +// checkpoint under prefix `p`, plus the top-level tied-head case (no lm_head). +// Names verified against the published `Qwen/Qwen3.8-2.4T-A95B` and +// `Qwen/Qwen3.6-35B-A3B` safetensors indices: identical modulo the prefix. +std::vector DenseOneLayerSpecs(const std::string& p) { + const std::string l = p + "layers.0."; + const std::string sa = l + "self_attn."; + const std::string mlp = l + "mlp."; + constexpr int64_t kHidden = 8; + constexpr int64_t kFfn = 16; + constexpr int64_t kHeadDim = 4; + constexpr int64_t kQ = 8; // 2 heads x 4 + constexpr int64_t kKv = 4; // 1 head x 4 + return { + {p + "embed_tokens.weight", {6, kHidden}}, + {p + "norm.weight", {kHidden}}, + {l + "input_layernorm.weight", {kHidden}}, + {l + "post_attention_layernorm.weight", {kHidden}}, + {sa + "q_proj.weight", {kQ, kHidden}}, + {sa + "k_proj.weight", {kKv, kHidden}}, + {sa + "v_proj.weight", {kKv, kHidden}}, + {sa + "o_proj.weight", {kHidden, kQ}}, + {sa + "q_norm.weight", {kHeadDim}}, + {sa + "k_norm.weight", {kHeadDim}}, + {mlp + "gate_proj.weight", {kFfn, kHidden}}, + {mlp + "up_proj.weight", {kFfn, kHidden}}, + {mlp + "down_proj.weight", {kHidden, kFfn}}, + }; +} + +// The tensor payload of a safetensors blob: everything after the 8-byte header +// length and the JSON header itself. +std::string Payload(const std::string& file) { + uint64_t header_len = 0; + for (int i = 7; i >= 0; --i) { + header_len = (header_len << 8) | static_cast(file[static_cast(i)]); + } + return file.substr(8 + static_cast(header_len)); +} + +std::vector NamesOf(const std::vector& specs) { + std::vector names; + names.reserve(specs.size()); + for (const Spec& s : specs) names.push_back(s.name); + return names; +} + +HfConfig OneLayerDenseConfig() { + HfConfig config; + config.model_type = "qwen3_5_text"; + config.hidden_size = 8; + config.num_hidden_layers = 1; + config.layer_types = {"full_attention"}; + return config; +} + +// Byte-for-byte comparison of a loaded tensor pair. +void CheckSameBytes(const vllm::OwnedTensor& a, const vllm::OwnedTensor& b, + const char* what) { + CAPTURE(what); + REQUIRE(a.rank == b.rank); + for (int i = 0; i < a.rank; ++i) CHECK(a.shape[i] == b.shape[i]); + REQUIRE(a.bytes.size() == b.bytes.size()); + REQUIRE(a.bytes.size() > 0); + CHECK(std::memcmp(a.bytes.data(), b.bytes.data(), a.bytes.size()) == 0); +} + +// --------------------------------------------------------------------------- +// MoE arm. `Qwen3_5MoeForCausalLM` is the architecture this row exists for, and +// the MoE loader threads the resolved prefix through THREE further sites the +// dense loader has no analogue of: the per-layer base (LoadLayerImpl, +// qwen3_5_weights.cpp:560-561), the top-level embed/norm pair +// (LoadQwen3_5Moe:677-678), and — only on the shards-owner path — the DEFERRED +// per-layer routed-expert closure (:698-709), which captures the prefix BY +// VALUE and runs long after the resolution frame is gone. +// +// The quantization per weight class is the ONLY scheme `LoadQwen3_5Moe` reads +// (qwen3_5_weights.h:1-13): bf16 embeds/norms/router/shared-gate, per-tensor +// FP8 attention, PER-EXPERT NVFP4 routed experts + shared expert + lm_head. +// NVFP4 requires K % 16 == 0, which is what fixes the toy hidden size at 16. +// +// This is the NVFP4-requant layout (`nvidia/Qwen3.6-35B-A3B-NVFP4` and the +// like), NOT the layout of the published `Qwen/Qwen3.6-35B-A3B` or +// `Qwen/Qwen3.8-2.4T-A95B` repos, which are stacked and unquantized. What +// follows therefore proves the prefix is threaded correctly through the arm we +// implement; the published arm is a refusal, pinned separately. +// --------------------------------------------------------------------------- +constexpr int64_t kMoeHidden = 16; +constexpr int64_t kMoeInter = 16; +constexpr int64_t kMoeVocab = 6; +constexpr int64_t kMoeExperts = 2; +constexpr int64_t kMoeHeadDim = 4; +constexpr int64_t kMoeQ = 8; // 2 heads x 4 +constexpr int64_t kMoeKv = 4; // 1 head x 4 + +// Per-tensor FP8 (W8A8) projection: weight + weight_scale + input_scale. Both +// scalars are emitted so the fixture loads on EITHER arm of DenseNativeEnabled() +// (fp8-resident on a CUDA+cutlass build, dequant-to-bf16 otherwise). +void AppendFp8(std::vector& out, const std::string& proj, int64_t out_dim, + int64_t in_dim) { + out.push_back({proj + ".weight", {out_dim, in_dim}, "F8_E4M3"}); + out.push_back({proj + ".weight_scale", {1}, "F32"}); + out.push_back({proj + ".input_scale", {1}, "F32"}); +} + +// NVFP4 W4A16 projection: packed [N, K/2] U8 codes + [N, K/16] fp8-e4m3 group +// scales + the f32 per-tensor global. +void AppendNvfp4(std::vector& out, const std::string& proj, + int64_t out_dim, int64_t in_dim) { + out.push_back({proj + ".weight", {out_dim, in_dim / 2}, "U8"}); + out.push_back({proj + ".weight_scale", {out_dim, in_dim / 16}, "F8_E4M3"}); + out.push_back({proj + ".weight_scale_2", {1}, "F32"}); +} + +// The full tensor list of a ONE-layer, full-attention, two-expert Qwen3.5 MoE +// checkpoint under backbone prefix `p`. `lm_head` is deliberately TOP-LEVEL +// (unprefixed) in both spellings, exactly as both published indices have it. +std::vector MoeOneLayerSpecs(const std::string& p) { + const std::string l = p + "layers.0."; + const std::string sa = l + "self_attn."; + const std::string mlp = l + "mlp."; + std::vector s{ + {p + "embed_tokens.weight", {kMoeVocab, kMoeHidden}}, + {p + "norm.weight", {kMoeHidden}}, + {l + "input_layernorm.weight", {kMoeHidden}}, + {l + "post_attention_layernorm.weight", {kMoeHidden}}, + }; + AppendFp8(s, sa + "q_proj", kMoeQ, kMoeHidden); + AppendFp8(s, sa + "k_proj", kMoeKv, kMoeHidden); + AppendFp8(s, sa + "v_proj", kMoeKv, kMoeHidden); + AppendFp8(s, sa + "o_proj", kMoeHidden, kMoeQ); + s.push_back({sa + "q_norm.weight", {kMoeHeadDim}}); + s.push_back({sa + "k_norm.weight", {kMoeHeadDim}}); + // Router + shared-expert gate (bf16, transposed at load). + s.push_back({mlp + "gate.weight", {kMoeExperts, kMoeHidden}}); + s.push_back({mlp + "shared_expert_gate.weight", {1, kMoeHidden}}); + for (int64_t e = 0; e < kMoeExperts; ++e) { + const std::string ex = mlp + "experts." + std::to_string(e) + "."; + AppendNvfp4(s, ex + "gate_proj", kMoeInter, kMoeHidden); + AppendNvfp4(s, ex + "up_proj", kMoeInter, kMoeHidden); + AppendNvfp4(s, ex + "down_proj", kMoeHidden, kMoeInter); + } + const std::string se = mlp + "shared_expert."; + AppendNvfp4(s, se + "gate_proj", kMoeInter, kMoeHidden); + AppendNvfp4(s, se + "up_proj", kMoeInter, kMoeHidden); + AppendNvfp4(s, se + "down_proj", kMoeHidden, kMoeInter); + AppendNvfp4(s, "lm_head", kMoeVocab, kMoeHidden); + // THE MTP DRAFT HEAD, AND WHY IT IS IN THE *SUPPORTED* FIXTURE. Read off the + // REAL gated index `nvidia/Qwen3.6-35B-A3B-NVFP4/model.safetensors.index.json` + // (124,468 tensors, fetched 2026-08-12): the checkpoint this loader is built + // for DOES carry the 3-D STACKED expert spelling — but only under the + // top-level `mtp.` prefix, as exactly these two names. Its 92 backbone layers + // are per-expert NVFP4 throughout (zero stacked names under + // `model.language_model.layers.`, zero expert `.weight` without a `_scale` + // sibling), and `lm_head.weight_scale` / `lm_head.weight_scale_2` are both + // present, so the checkpoint loads. + // + // It loads ONLY because `CheckMoeExpertLayoutSupported` scans names under + // `layers.` (qwen3_5_weights.cpp:633,638) and `mtp.` is neither + // `model.layers.` nor `model.language_model.layers.`. That exclusion is + // LOAD-BEARING, not incidental: broadening the scan to every `.mlp.experts.` + // name would refuse the one checkpoint we gate today, on a CUDA-only load + // path, with the whole CPU suite still green. These two entries put the real + // index's shape into the fixture so that regression is CPU-visible — see the + // dedicated subcase in case 4c. `LoadQwen3_5Moe` never requests `mtp.*` + // (LoadQwen3_5MTP loads the draft head separately, only under speculative + // decoding), so they are inert to every other assertion here. + const std::string mtp = "mtp.layers.0.mlp."; + s.push_back({mtp + "experts.gate_up_proj", + {kMoeExperts, 2 * kMoeInter, kMoeHidden}}); + s.push_back({mtp + "experts.down_proj", {kMoeExperts, kMoeHidden, kMoeInter}}); + return s; +} + +// --------------------------------------------------------------------------- +// The PUBLISHED Qwen3.5-family MoE layouts, which `LoadQwen3_5Moe` does NOT +// implement. Read off the live safetensors indices on 2026-08-12: +// +// Qwen/Qwen3.8-2.4T-A95B 1609 tensors, 92x `model.layers.N.mlp.experts +// .gate_up_proj` + 92x `.down_proj` (3-D STACKED), +// ZERO `weight_scale`, ZERO `input_scale`, +// `lm_head.weight` alone. +// Qwen/Qwen3.6-35B-A3B 1045 tensors, same stacked spelling under the VL +// prefix, ZERO `weight_scale`. +// +// `LoadMoeExpertsInto` (qwen3_5_weights.cpp:519-530) reads only per-expert +// `experts..` through `LoadNvfp4Raw` (:433-462), which hard-requires +// U8 `.weight` + F8_E4M3 `.weight_scale` + `.weight_scale_2`. There is no +// stacked branch and no bf16 branch, unlike `gemma4_weights.cpp:326`, which +// dispatches between layouts. Our gated 35B row reads the REQUANTIZED +// `nvidia/Qwen3.6-35B-A3B-NVFP4`; this loader has never read a published Qwen +// bf16 MoE repo. AGENTS.md requires an unimplemented arm be "refused with a +// message naming the missing piece" — these fixtures are that gate. +// --------------------------------------------------------------------------- + +// Everything a published Qwen3.5 MoE checkpoint carries EXCEPT the routed +// experts: all bf16, no scale tensors anywhere, `lm_head.weight` top-level. +std::vector PublishedMoeCoreSpecs(const std::string& p) { + const std::string l = p + "layers.0."; + const std::string sa = l + "self_attn."; + const std::string mlp = l + "mlp."; + return { + {p + "embed_tokens.weight", {kMoeVocab, kMoeHidden}}, + {p + "norm.weight", {kMoeHidden}}, + {l + "input_layernorm.weight", {kMoeHidden}}, + {l + "post_attention_layernorm.weight", {kMoeHidden}}, + {sa + "q_proj.weight", {kMoeQ, kMoeHidden}}, + {sa + "k_proj.weight", {kMoeKv, kMoeHidden}}, + {sa + "v_proj.weight", {kMoeKv, kMoeHidden}}, + {sa + "o_proj.weight", {kMoeHidden, kMoeQ}}, + {sa + "q_norm.weight", {kMoeHeadDim}}, + {sa + "k_norm.weight", {kMoeHeadDim}}, + {mlp + "gate.weight", {kMoeExperts, kMoeHidden}}, + {mlp + "shared_expert.gate_proj.weight", {kMoeInter, kMoeHidden}}, + {mlp + "shared_expert.up_proj.weight", {kMoeInter, kMoeHidden}}, + {mlp + "shared_expert.down_proj.weight", {kMoeHidden, kMoeInter}}, + {mlp + "shared_expert_gate.weight", {1, kMoeHidden}}, + {"lm_head.weight", {kMoeVocab, kMoeHidden}}, + }; +} + +// The published shape: ONE 3-D tensor per layer holding ALL experts. +std::vector PublishedStackedMoeSpecs(const std::string& p) { + std::vector s = PublishedMoeCoreSpecs(p); + const std::string mlp = p + "layers.0.mlp."; + s.push_back({mlp + "experts.gate_up_proj", + {kMoeExperts, 2 * kMoeInter, kMoeHidden}}); + s.push_back({mlp + "experts.down_proj", {kMoeExperts, kMoeHidden, kMoeInter}}); + return s; +} + +// The per-expert spelling this loader DOES resolve, but unquantized: plain bf16 +// `..weight` with no `.weight_scale` beside it. +std::vector UnquantizedPerExpertMoeSpecs(const std::string& p) { + std::vector s = PublishedMoeCoreSpecs(p); + const std::string mlp = p + "layers.0.mlp."; + for (int64_t e = 0; e < kMoeExperts; ++e) { + const std::string ex = mlp + "experts." + std::to_string(e) + "."; + s.push_back({ex + "gate_proj.weight", {kMoeInter, kMoeHidden}}); + s.push_back({ex + "up_proj.weight", {kMoeInter, kMoeHidden}}); + s.push_back({ex + "down_proj.weight", {kMoeHidden, kMoeInter}}); + } + return s; +} + +// Per-expert NVFP4 everywhere — the supported arm — but a plain bf16 head, the +// spelling the DENSE loader accepts (`LoadLmHeadAnyDtype`) and the MoE loader +// does not. +std::vector MoeSpecsWithBf16LmHead(const std::string& p) { + std::vector s; + for (const Spec& x : MoeOneLayerSpecs(p)) { + if (x.name.rfind("lm_head.", 0) != 0) s.push_back(x); + } + s.push_back({"lm_head.weight", {kMoeVocab, kMoeHidden}}); + return s; +} + +// Runs `fn` and returns the `what()` of whatever it threw, or "" if it returned +// normally. A refusal is only useful if it NAMES the missing piece, which +// CHECK_THROWS_AS cannot see. +std::string CaptureThrow(const std::function& fn) { + try { + fn(); + } catch (const std::exception& e) { + return e.what(); + } + return ""; +} + +bool Mentions(const std::string& haystack, const char* needle) { + return haystack.find(needle) != std::string::npos; +} + +// A TensorResolver + presence probe over ONE synthetic safetensors file, so a +// per-layer public seam can be driven exactly as the full loaders drive it — +// with the backbone-prefix argument OMITTED, which is the only way its DEFAULT +// is observable at all. +class ShardBag { + public: + ShardBag(const std::vector& specs, const char* tag) + : file_(BuildSafetensors(specs), tag), + shard_(vllm::SafetensorsFile::Open(file_.path())) {} + ShardBag(const ShardBag&) = delete; + ShardBag& operator=(const ShardBag&) = delete; + + vllm::TensorResolver Resolver() const { + const vllm::SafetensorsFile* shard = &shard_; + return [shard](const std::string& name) -> const vllm::StTensor& { + // Same message shape `LoadQwen3_5Moe`'s own resolver produces, so a miss + // is reported by the NAME that was looked for. + for (const std::string& have : shard->Names()) { + if (have == name) return shard->Get(name); + } + throw std::runtime_error("qwen3_5 weights: tensor not found: " + name); + }; + } + + std::function Has() const { + const vllm::SafetensorsFile* shard = &shard_; + return [shard](const std::string& name) { + for (const std::string& have : shard->Names()) { + if (have == name) return true; + } + return false; + }; + } + + private: + TempFile file_; + vllm::SafetensorsFile shard_; +}; + +HfConfig OneLayerMoeConfig() { + HfConfig config; + config.model_type = "qwen3_5_moe_text"; + config.hidden_size = kMoeHidden; + config.num_hidden_layers = 1; + config.layer_types = {"full_attention"}; + config.num_experts = kMoeExperts; + return config; +} + +void CheckSameNvfp4(const vllm::Nvfp4Weight& a, const vllm::Nvfp4Weight& b, + const char* what) { + CAPTURE(what); + CHECK(a.n == b.n); + CHECK(a.k == b.k); + CHECK(a.scale2 == b.scale2); + CheckSameBytes(a.packed, b.packed, what); + CheckSameBytes(a.scale, b.scale, what); +} + +void CheckSameFp8(const vllm::Fp8Weight& a, const vllm::Fp8Weight& b, + const char* what) { + CAPTURE(what); + CHECK(a.n == b.n); + CHECK(a.k == b.k); + CHECK(a.weight_scale == b.weight_scale); + CHECK(a.input_scale == b.input_scale); + CHECK(a.alpha == b.alpha); + CheckSameBytes(a.packed, b.packed, what); +} + +// Which arm of the attention projections the loader fills depends on the BUILD, +// not on this test: DenseNativeEnabled() keeps them fp8-resident on a +// CUDA+cutlass build and dequants them to bf16 otherwise. Assert on whichever +// this build populated, and REQUIRE that exactly one of the two is populated. +void CheckSameAttn(const vllm::FullAttnLayerWeights& a, + const vllm::FullAttnLayerWeights& b) { + const bool fp8 = !a.q_proj_fp8.Empty(); + REQUIRE(fp8 == !b.q_proj_fp8.Empty()); + REQUIRE(fp8 == a.q_proj.Empty()); + REQUIRE(fp8 == b.q_proj.Empty()); + if (fp8) { + CheckSameFp8(a.q_proj_fp8, b.q_proj_fp8, "q_proj fp8"); + CheckSameFp8(a.k_proj_fp8, b.k_proj_fp8, "k_proj fp8"); + CheckSameFp8(a.v_proj_fp8, b.v_proj_fp8, "v_proj fp8"); + CheckSameFp8(a.o_proj_fp8, b.o_proj_fp8, "o_proj fp8"); + } else { + CheckSameBytes(a.q_proj, b.q_proj, "q_proj"); + CheckSameBytes(a.k_proj, b.k_proj, "k_proj"); + CheckSameBytes(a.v_proj, b.v_proj, "v_proj"); + CheckSameBytes(a.o_proj, b.o_proj, "o_proj"); + } + CheckSameBytes(a.q_norm, b.q_norm, "q_norm"); + CheckSameBytes(a.k_norm, b.k_norm, "k_norm"); +} + +void CheckSameMoeBlock(const vllm::MoeBlockWeights& a, + const vllm::MoeBlockWeights& b) { + CheckSameBytes(a.router_gate, b.router_gate, "mlp.gate"); + CheckSameBytes(a.shared_gate, b.shared_gate, "mlp.shared_expert_gate"); + CheckSameNvfp4(a.shared_gate_proj_fp4, b.shared_gate_proj_fp4, + "shared_expert.gate_proj"); + CheckSameNvfp4(a.shared_up_proj_fp4, b.shared_up_proj_fp4, + "shared_expert.up_proj"); + CheckSameNvfp4(a.shared_down_proj_fp4, b.shared_down_proj_fp4, + "shared_expert.down_proj"); + REQUIRE(a.expert_gate_fp4.size() == static_cast(kMoeExperts)); + REQUIRE(b.expert_gate_fp4.size() == a.expert_gate_fp4.size()); + REQUIRE(a.expert_up_fp4.size() == a.expert_gate_fp4.size()); + REQUIRE(a.expert_down_fp4.size() == a.expert_gate_fp4.size()); + for (size_t e = 0; e < a.expert_gate_fp4.size(); ++e) { + CAPTURE(e); + CheckSameNvfp4(a.expert_gate_fp4[e], b.expert_gate_fp4[e], "expert gate"); + CheckSameNvfp4(a.expert_up_fp4[e], b.expert_up_fp4[e], "expert up"); + CheckSameNvfp4(a.expert_down_fp4[e], b.expert_down_fp4[e], "expert down"); + } +} + +void CheckSameMoeModel(const vllm::Qwen3_5MoeWeights& a, + const vllm::Qwen3_5MoeWeights& b) { + REQUIRE(a.layers.size() == b.layers.size()); + REQUIRE(a.layers.size() == 1u); + CheckSameBytes(a.embed_tokens, b.embed_tokens, "embed_tokens"); + CheckSameBytes(a.final_norm, b.final_norm, "final_norm"); + CheckSameNvfp4(a.lm_head_fp4, b.lm_head_fp4, "lm_head"); + for (size_t l = 0; l < a.layers.size(); ++l) { + CAPTURE(l); + const vllm::Qwen3_5MoeLayerWeights& x = a.layers[l]; + const vllm::Qwen3_5MoeLayerWeights& y = b.layers[l]; + CHECK_FALSE(x.is_linear_attention); + CHECK_FALSE(y.is_linear_attention); + CheckSameBytes(x.input_layernorm, y.input_layernorm, "input_layernorm"); + CheckSameBytes(x.post_attention_layernorm, y.post_attention_layernorm, + "post_attention_layernorm"); + CheckSameAttn(x.attn, y.attn); + CheckSameMoeBlock(x.moe, y.moe); + } +} + +} // namespace + +// =========================================================================== +// 1. Architecture dispatch. Upstream registers both text-only arms against the +// SAME `qwen3_5` module (registry.py:202-203 @ `ad5d29db7`), so ours must +// resolve to the SAME factories the ForConditionalGeneration wrappers use — +// a second factory would be a fork of a backbone we already gate. +// =========================================================================== +TEST_CASE("qwen3_8: both text-only architecture strings resolve to the Qwen3.5 factories") { + const HfConfig moe_config = ArchConfig({"Qwen3_5MoeForCausalLM"}); + const ModelRegistration& moe = ModelRegistry::Resolve(moe_config); + CHECK(moe.architecture == "Qwen3_5MoeForCausalLM"); + CHECK(moe.factory == + vllm::RegistrationFor("Qwen3_5MoeForConditionalGeneration").factory); + CHECK_FALSE(moe.factory->is_dense_model); + + const HfConfig dense_config = ArchConfig({"Qwen3_5ForCausalLM"}); + const ModelRegistration& dense = ModelRegistry::Resolve(dense_config); + CHECK(dense.architecture == "Qwen3_5ForCausalLM"); + CHECK(dense.factory == + vllm::RegistrationFor("Qwen3_5ForConditionalGeneration").factory); + CHECK(dense.factory->is_dense_model); + + // Upstream's `Qwen3_5ForCausalLMBase` inherits IsHybrid + HasInnerState but + // NOT SupportsMultiModal (qwen3_5.py:287-296 @ `ad5d29db7`): these are the + // TEXT arms, and their multimodal wrappers are separate registrations. Same + // convention as `KimiLinearForCausalLM` — hybrid yes, multimodal no. + for (const ModelRegistration* registration : {&moe, &dense}) { + CAPTURE(registration->architecture); + CHECK(registration->info.is_text_generation_model); + CHECK(registration->info.is_hybrid); + CHECK_FALSE(registration->info.supports_multimodal); + CHECK_FALSE(registration->info.is_pooling_model); + } +} + +// =========================================================================== +// 2. Config resolution on the REAL flat 3.8 config — the PUBLISHED DOCUMENT, +// not a hand-written approximation of it. +// +// `tests/vllm/models/fixtures/qwen3_8_2_4t_a95b/config.json` is +// https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B/raw/main/config.json, +// committed VERBATIM (md5 303dc59227f1d03afc941646e8df3132, fetched +// 2026-08-12). It declares `model_type: qwen3_5_moe_text` at the TOP level +// with no `text_config` wrapper, no `vision_config` and no `mrope_section`, +// so the composite-wrapper path our 27B/35B checkpoints take does not apply. +// +// Three things a paraphrase got wrong and this document does not: the rope +// knobs live in a NESTED `rope_parameters` block (so the parse actually +// reaches hf_config.cpp's rope_parameters branch, LooksLikeNestedRopeParameters +// included, rather than the no-block early return); `layer_types` is present +// and 92 entries long, which BOTH loaders hard-require to equal +// num_hidden_layers; and the dtype key is transformers 4.57.3's `dtype`, not +// the legacy `torch_dtype`. +// =========================================================================== +TEST_CASE("qwen3_8: the PUBLISHED 2.4T text config resolves through the shared Qwen3.5 path") { + const HfConfig config = vllm::LoadHfConfig(QWEN3_8_CONFIG_FIXTURE); + + // The architecture the registry will be asked for, straight off a flat doc. + REQUIRE(config.architectures.size() == 1); + CHECK(config.architectures[0] == "Qwen3_5MoeForCausalLM"); + CHECK(config.model_type == "qwen3_5_moe_text"); + CHECK_NOTHROW(ModelRegistry::Resolve(config)); + + // Scale: every knob that differs from the 35B is CONFIG, never a constant. + CHECK(config.hidden_size == 8192); + CHECK(config.num_hidden_layers == 92); + CHECK(config.num_attention_heads == 64); + CHECK(config.num_key_value_heads == 4); + CHECK(config.head_dim == 256); + CHECK(config.vocab_size == 248320); + CHECK(config.num_experts == 512); + CHECK(config.num_experts_per_tok == 10); + CHECK(config.moe_intermediate_size == 2048); + CHECK(config.shared_expert_intermediate_size == 2048); + CHECK(config.linear_num_key_heads == 16); + CHECK(config.linear_num_value_heads == 128); + CHECK(config.linear_key_head_dim == 128); + CHECK(config.linear_value_head_dim == 128); + CHECK(config.linear_conv_kernel_dim == 4); + CHECK(config.rope_theta == doctest::Approx(1e7).scale(0.0)); + + // `layer_types` is a HARD requirement of both Qwen3.5 loaders + // (qwen3_5_weights.cpp:660-663, qwen3_5_dense_weights.cpp) — they refuse a + // checkpoint whose list does not equal num_hidden_layers. The published + // pattern is [linear, linear, linear, full] x 23, i.e. + // full_attention_interval 4, the same interleave as the 35B. + REQUIRE(static_cast(config.layer_types.size()) == + config.num_hidden_layers); + REQUIRE(config.layer_types.size() == 92u); + for (size_t i = 0; i < config.layer_types.size(); ++i) { + CAPTURE(i); + CHECK(config.layer_types[i] == + ((i % 4 == 3) ? "full_attention" : "linear_attention")); + } + + // The rope block is NESTED and flat-valued, so ParseRopeParameters takes the + // rope_parameters branch, LooksLikeNestedRopeParameters is false (its values + // are scalars, not per-layer-type objects), and the block's own + // partial_rotary_factor/rope_theta win. The 0.25 also appears at top level, + // and IsQwen35Family would default it to 0.25 in any case — all three agree, + // which is why rotary_dim is 0.25 * 256. + CHECK(config.has_rope_parameters); + CHECK(config.rope_parameters.rope_type == "default"); + CHECK(config.rope_parameters.rope_theta == doctest::Approx(1e7).scale(0.0)); + CHECK(config.rope_parameters.partial_rotary_factor == + doctest::Approx(0.25).scale(0.0)); + CHECK(config.rotary_dim == 64); + + // A text-only checkpoint has no vision tower and no MRoPE sections. Both are + // ABSENT rather than empty-but-present, and neither may be synthesized. + CHECK(config.raw.find("vision_config") == config.raw.end()); + CHECK(config.raw.find("text_config") == config.raw.end()); + CHECK(config.rope_parameters.mrope_section.empty()); + + // DELIBERATELY UNCONSUMED, PINNED SO IT CANNOT DRIFT SILENTLY. transformers + // 4.57.3 writes the model dtype as `dtype`; hf_config.cpp:520-522 reads only + // the legacy `torch_dtype`, so `cfg.torch_dtype` is EMPTY on this document. + // That is inert today — nothing in the tree reads `HfConfig::torch_dtype` — + // and consuming `dtype` is a behavior change on EVERY model, so it is + // recorded as a tracked deviation (porting-inventory.md §9 deviation 17) + // rather than smuggled in on this row. This assertion is the tripwire: it + // fails the day someone teaches hf_config the new spelling, forcing the + // deviation to be discharged rather than forgotten. + CHECK(config.raw.at("dtype") == "bfloat16"); + CHECK(config.raw.find("torch_dtype") == config.raw.end()); + CHECK(config.torch_dtype.empty()); + + // The published document has no top-level `intermediate_size` — the MoE + // widths are `moe_intermediate_size` / `shared_expert_intermediate_size`. + CHECK(config.raw.find("intermediate_size") == config.raw.end()); +} + +// =========================================================================== +// 3. Weight-namespace resolution. Upstream normalizes with ONE WeightsMapper +// (qwen3_5.py:296-300 @ `ad5d29db7`); we mirror that with ONE resolution per +// checkpoint rather than a per-lookup fallback, because a per-lookup fallback +// would let a checkpoint bind half its tensors from each namespace and still +// appear to load. +// =========================================================================== +TEST_CASE("qwen3_8: the backbone weight namespace is resolved ONCE per checkpoint") { + SUBCASE("a clean `model.` index resolves to the canonical namespace") { + CHECK(vllm::ResolveQwen3_5BackbonePrefix( + NamesOf(DenseOneLayerSpecs("model."))) == "model."); + } + + SUBCASE("a VL-prefixed index resolves to `model.language_model.`") { + CHECK(vllm::ResolveQwen3_5BackbonePrefix( + NamesOf(DenseOneLayerSpecs("model.language_model."))) == + "model.language_model."); + } + + SUBCASE("a VISION-INCLUSIVE VL checkpoint is still the VL namespace") { + // The 27B/35B vision-inclusive checkpoints carry `model.visual.*` NEXT TO + // `model.language_model.*`. `model.visual.` is not a backbone spelling, so + // it must never be mistaken for the canonical `model.` namespace and turn a + // checkpoint we gate today into a refusal. + std::vector names = NamesOf(DenseOneLayerSpecs("model.language_model.")); + names.push_back("model.visual.patch_embed.proj.weight"); + names.push_back("model.visual.blocks.0.attn.qkv.weight"); + names.push_back("lm_head.weight"); + CHECK(vllm::ResolveQwen3_5BackbonePrefix(names) == "model.language_model."); + } + + SUBCASE("the optional `mtp.*` draft head does not decide the namespace") { + std::vector names = NamesOf(DenseOneLayerSpecs("model.")); + names.push_back("mtp.fc.weight"); + names.push_back("mtp.layers.0.input_layernorm.weight"); + CHECK(vllm::ResolveQwen3_5BackbonePrefix(names) == "model."); + } + + SUBCASE("a MIXED index is REFUSED, not silently half-loaded") { + std::vector names = NamesOf(DenseOneLayerSpecs("model.")); + names.push_back("model.language_model.layers.1.input_layernorm.weight"); + CHECK_THROWS_AS(vllm::ResolveQwen3_5BackbonePrefix(names), + std::runtime_error); + } + + SUBCASE("an index with NEITHER namespace is refused") { + const std::vector names{"lm_head.weight", "mtp.fc.weight"}; + CHECK_THROWS_AS(vllm::ResolveQwen3_5BackbonePrefix(names), + std::runtime_error); + } +} + +// =========================================================================== +// 4. The loader must USE the resolved prefix. Two synthetic checkpoints with +// byte-identical payloads and only the namespace differing must produce +// byte-identical weights — which no amount of name-mapping unit testing can +// show on its own. +// =========================================================================== +TEST_CASE("qwen3_8: the dense loader reads the SAME weights through either namespace") { + const std::vector vl = DenseOneLayerSpecs("model.language_model."); + const std::vector flat = DenseOneLayerSpecs("model."); + REQUIRE(vl.size() == flat.size()); + + const std::string vl_bytes = BuildSafetensors(vl); + const std::string flat_bytes = BuildSafetensors(flat); + // Same specs in the same order => IDENTICAL payloads, only the names differ. + // Assert that here, so a later byte-equality of the two loads cannot be + // satisfied by two identically-WRONG reads of two different payloads. + REQUIRE(Payload(vl_bytes) == Payload(flat_bytes)); + REQUIRE(vl_bytes != flat_bytes); + const TempFile vl_file(vl_bytes, "vl"); + const TempFile flat_file(flat_bytes, "flat"); + + std::vector vl_shards; + vl_shards.push_back(vllm::SafetensorsFile::Open(vl_file.path())); + std::vector flat_shards; + flat_shards.push_back(vllm::SafetensorsFile::Open(flat_file.path())); + + const HfConfig config = OneLayerDenseConfig(); + const vllm::Qwen3_5DenseWeights from_vl = + vllm::LoadQwen3_5Dense(vl_shards, config); + const vllm::Qwen3_5DenseWeights from_flat = + vllm::LoadQwen3_5Dense(flat_shards, config); + + REQUIRE(from_vl.layers.size() == 1); + REQUIRE(from_flat.layers.size() == 1); + // No lm_head in either index => both tie the head to the embedding table. + CHECK(from_vl.tied_lm_head); + CHECK(from_flat.tied_lm_head); + + CheckSameBytes(from_vl.embed_tokens, from_flat.embed_tokens, "embed_tokens"); + CheckSameBytes(from_vl.final_norm, from_flat.final_norm, "final_norm"); + const vllm::Qwen3_5DenseLayerWeights& a = from_vl.layers[0]; + const vllm::Qwen3_5DenseLayerWeights& b = from_flat.layers[0]; + CHECK_FALSE(a.is_linear_attention); + CHECK_FALSE(b.is_linear_attention); + CheckSameBytes(a.input_layernorm, b.input_layernorm, "input_layernorm"); + CheckSameBytes(a.post_attention_layernorm, b.post_attention_layernorm, + "post_attention_layernorm"); + CheckSameBytes(a.attn.q_proj, b.attn.q_proj, "q_proj"); + CheckSameBytes(a.attn.k_proj, b.attn.k_proj, "k_proj"); + CheckSameBytes(a.attn.v_proj, b.attn.v_proj, "v_proj"); + CheckSameBytes(a.attn.o_proj, b.attn.o_proj, "o_proj"); + CheckSameBytes(a.attn.q_norm, b.attn.q_norm, "q_norm"); + CheckSameBytes(a.attn.k_norm, b.attn.k_norm, "k_norm"); + CheckSameBytes(a.mlp.gate_up_proj, b.mlp.gate_up_proj, "gate_up_proj"); + CheckSameBytes(a.mlp.down_proj, b.mlp.down_proj, "down_proj"); +} + +// =========================================================================== +// 4b. THE SAME PROOF FOR THE MoE ARM — the architecture this row exists for. +// `LoadQwen3_5Moe` threads the resolved prefix through three sites the +// dense loader has no analogue of, and the third of them lives in a +// closure that runs AFTER the resolving frame has returned. Two synthetic +// one-layer MoE checkpoints with byte-identical payloads and only the +// namespace differing must load to byte-identical weights on BOTH expert +// residency paths: +// * shards_owner == nullptr -> routed experts loaded EAGERLY at load; +// * shards_owner != nullptr -> routed experts DEFERRED behind +// `load_layer_experts`, which is then driven explicitly so the closure +// body actually executes (an installed-but-never-called closure pins +// nothing). +// Reverting any of the three sites to the hardcoded `model.language_model.` +// literal makes the flat load throw `tensor not found`. +// =========================================================================== +TEST_CASE("qwen3_8: the MoE loader reads the SAME weights through either namespace") { + const std::vector vl = MoeOneLayerSpecs("model.language_model."); + const std::vector flat = MoeOneLayerSpecs("model."); + REQUIRE(vl.size() == flat.size()); + + const std::string vl_bytes = BuildSafetensors(vl); + const std::string flat_bytes = BuildSafetensors(flat); + // Same specs in the same order => IDENTICAL payloads, only the names differ, + // so a later byte-equality of the two loads cannot be satisfied by two + // identically-WRONG reads of two different payloads. + REQUIRE(Payload(vl_bytes) == Payload(flat_bytes)); + REQUIRE(vl_bytes != flat_bytes); + const TempFile vl_file(vl_bytes, "moe_vl"); + const TempFile flat_file(flat_bytes, "moe_flat"); + const HfConfig config = OneLayerMoeConfig(); + + SUBCASE("EAGER experts (no shards owner)") { + std::vector vl_shards; + vl_shards.push_back(vllm::SafetensorsFile::Open(vl_file.path())); + std::vector flat_shards; + flat_shards.push_back(vllm::SafetensorsFile::Open(flat_file.path())); + + const vllm::Qwen3_5MoeWeights from_vl = + vllm::LoadQwen3_5Moe(vl_shards, config); + const vllm::Qwen3_5MoeWeights from_flat = + vllm::LoadQwen3_5Moe(flat_shards, config); + + // No owner => no streaming closure, and the routed experts are already here. + CHECK_FALSE(static_cast(from_vl.load_layer_experts)); + CHECK_FALSE(static_cast(from_flat.load_layer_experts)); + REQUIRE(from_flat.layers.size() == 1u); + REQUIRE(from_flat.layers[0].moe.expert_gate_fp4.size() == + static_cast(kMoeExperts)); + CheckSameMoeModel(from_vl, from_flat); + } + + SUBCASE("DEFERRED experts (shards owner) — the closure runs") { + auto vl_owner = std::make_shared>(); + vl_owner->push_back(vllm::SafetensorsFile::Open(vl_file.path())); + auto flat_owner = std::make_shared>(); + flat_owner->push_back(vllm::SafetensorsFile::Open(flat_file.path())); + + vllm::Qwen3_5MoeWeights from_vl = + vllm::LoadQwen3_5Moe(*vl_owner, config, vl_owner); + vllm::Qwen3_5MoeWeights from_flat = + vllm::LoadQwen3_5Moe(*flat_owner, config, flat_owner); + + // Deferred precondition: the closure is installed and NOTHING routed is + // resident yet, so what the next four lines compare is what it produced. + REQUIRE(static_cast(from_vl.load_layer_experts)); + REQUIRE(static_cast(from_flat.load_layer_experts)); + REQUIRE(from_vl.layers.size() == 1u); + REQUIRE(from_flat.layers.size() == 1u); + CHECK(from_vl.layers[0].moe.expert_gate_fp4.empty()); + CHECK(from_flat.layers[0].moe.expert_gate_fp4.empty()); + + // Move first, exactly as the real load moves the weights into the + // LoadedModel before PrepareMarlinResident drives the closure. + vllm::Qwen3_5MoeWeights vl_moved = std::move(from_vl); + vllm::Qwen3_5MoeWeights flat_moved = std::move(from_flat); + vl_moved.load_layer_experts(0, vl_moved.layers[0].moe); + flat_moved.load_layer_experts(0, flat_moved.layers[0].moe); + + REQUIRE(flat_moved.layers[0].moe.expert_gate_fp4.size() == + static_cast(kMoeExperts)); + CheckSameMoeModel(vl_moved, flat_moved); + } +} + +// =========================================================================== +// 4c. THE PUBLISHED MoE EXPERT LAYOUT IS AN UNIMPLEMENTED ARM, AND IS REFUSED +// BY NAME. Registering the architecture and resolving the namespace does +// NOT make `Qwen/Qwen3.8-2.4T-A95B` loadable: its routed experts are 3-D +// STACKED and it carries no quantization scales at all, while +// `LoadQwen3_5Moe` reads only per-expert NVFP4. Before this gate the load +// died at `LoadNvfp4Raw(get, "lm_head")` with +// +// qwen3_5 weights: expected U8 for lm_head.weight +// +// which is indistinguishable from a corrupt or truncated checkpoint. +// AGENTS.md: an arm that is not implemented "is refused with a message +// naming the missing piece ... never left to be discovered later", and this +// row's spec §Stop conditions says the same. NOTE this adds a REFUSAL only: +// stacked/bf16 MoE expert loading is OWED and needs its own spec, RED-first +// test and NVFP4 inertness proof. +// =========================================================================== +TEST_CASE("qwen3_8: the published stacked/unquantized MoE arm is REFUSED, and the message names it") { + auto load = [](const std::vector& specs, const char* tag) { + return CaptureThrow([&specs, tag] { + const TempFile file(BuildSafetensors(specs), tag); + std::vector shards; + shards.push_back(vllm::SafetensorsFile::Open(file.path())); + vllm::LoadQwen3_5Moe(shards, OneLayerMoeConfig()); + }); + }; + + // Every refusal must say WHAT is missing and WHAT would be required — a bare + // "unsupported" is the failure mode this case exists to prevent. + auto names_the_requirement = [](const std::string& message) { + CAPTURE(message); + CHECK(Mentions(message, "qwen3_5 weights")); + CHECK(Mentions(message, "not implemented")); + CHECK(Mentions(message, "per-expert NVFP4")); + CHECK(Mentions(message, "weight_scale")); + // The two failures it must NOT degrade into: a raw dtype complaint about a + // tensor the reader never reached, or a bare lookup miss. + CHECK_FALSE(Mentions(message, "expected U8 for lm_head.weight")); + CHECK_FALSE(Mentions(message, "tensor not found")); + }; + + SUBCASE("3-D stacked experts, flat namespace — the Qwen3.8-2.4T-A95B shape") { + const std::string message = load(PublishedStackedMoeSpecs("model."), "stacked_flat"); + CAPTURE(message); + CHECK(Mentions(message, "model.layers.0.mlp.experts.gate_up_proj")); + CHECK(Mentions(message, "stacked")); + names_the_requirement(message); + } + + SUBCASE("3-D stacked experts, VL namespace — the Qwen3.6-35B-A3B shape") { + // The published 35B repo is stacked too; only the NVFP4 REQUANT loads. So + // this refusal is not a text-only-arm quirk — the MoE loader has never read + // a published Qwen bf16 MoE checkpoint under EITHER spelling. + const std::string message = + load(PublishedStackedMoeSpecs("model.language_model."), "stacked_vl"); + CAPTURE(message); + CHECK(Mentions(message, + "model.language_model.layers.0.mlp.experts.gate_up_proj")); + CHECK(Mentions(message, "stacked")); + names_the_requirement(message); + } + + SUBCASE("per-expert but UNQUANTIZED experts") { + const std::string message = + load(UnquantizedPerExpertMoeSpecs("model."), "unquant_experts"); + CAPTURE(message); + CHECK(Mentions(message, "model.layers.0.mlp.experts.0.gate_proj.weight")); + CHECK(Mentions(message, "unquantized")); + names_the_requirement(message); + } + + SUBCASE("NVFP4 experts but an UNQUANTIZED lm_head") { + // The dense arm accepts a bf16 head (`LoadLmHeadAnyDtype`, + // qwen3_5_dense_weights.cpp:215-233); the MoE arm hard-requires NVFP4. + const std::string message = + load(MoeSpecsWithBf16LmHead("model."), "bf16_lmhead"); + CAPTURE(message); + CHECK(Mentions(message, "lm_head.weight_scale")); + CHECK(Mentions(message, "unquantized")); + names_the_requirement(message); + } + + SUBCASE("the SUPPORTED per-expert NVFP4 layout is untouched by the check") { + // Inertness in the same case as the refusals, so the gate cannot be made + // green by refusing everything: the arm we DO implement still loads, under + // both namespaces. + CHECK(load(MoeOneLayerSpecs("model."), "inert_flat").empty()); + CHECK(load(MoeOneLayerSpecs("model.language_model."), "inert_vl").empty()); + } + + SUBCASE("the gated checkpoint's `mtp.` STACKED experts do not trip it either") { + // The one checkpoint this arm is actually gated on — + // `nvidia/Qwen3.6-35B-A3B-NVFP4`, 124,468 tensors in its published + // safetensors index — DOES contain the exact stacked spelling the first + // refusal rejects, as `mtp.layers.0.mlp.experts.{gate_up_proj,down_proj}`. + // It loads only because the scan is anchored at `layers.` + // (qwen3_5_weights.cpp:633,638) and `mtp.` is under neither backbone + // spelling. Nothing else in this suite pins that, so a later edit + // broadening the scan to every `.mlp.experts.` name would refuse the + // checkpoint we gate — on a CUDA-only load path, with the whole CPU suite + // green. This subcase is that CPU-visible pin. + auto carries = [](const std::vector& specs, const std::string& name) { + int hits = 0; + for (const Spec& x : specs) { + if (x.name == name) ++hits; + } + return hits; + }; + // The fixture must still BE the real index's shape; a fixture that quietly + // lost these names would leave the two loads below proving nothing. + for (const char* p : {"model.", "model.language_model."}) { + CAPTURE(p); + const std::vector specs = MoeOneLayerSpecs(p); + REQUIRE(carries(specs, "mtp.layers.0.mlp.experts.gate_up_proj") == 1); + REQUIRE(carries(specs, "mtp.layers.0.mlp.experts.down_proj") == 1); + // ...and they must be OUTSIDE the scanned namespace, which is the whole + // reason the refusal stays silent on them. + REQUIRE(carries(specs, std::string(p) + + "layers.0.mlp.experts.gate_up_proj") == 0); + } + CHECK(load(MoeOneLayerSpecs("model."), "mtp_stacked_flat").empty()); + CHECK(load(MoeOneLayerSpecs("model.language_model."), "mtp_stacked_vl") + .empty()); + } +} + +// =========================================================================== +// 5. INERTNESS of the gated rows. 27B / 35B / Coder are VL-prefixed +// checkpoints; the per-layer public seams keep the VL prefix as their +// DEFAULT, so every existing caller is unchanged by construction. +// +// ASSERTING THE TWO CONSTANTS IS NOT THAT PROOF. Flipping both header +// defaults from `kQwen3_5VlBackbonePrefix` to `kQwen3_5TextBackbonePrefix` +// left this case entirely green (review finding F7), because the default +// ARGUMENT is a third fact neither constant pins. So the seams are driven +// below with the prefix argument OMITTED, exactly as every 27B/35B/Coder +// caller drives them. +// =========================================================================== +TEST_CASE("qwen3_8: the VL prefix stays the default for the gated 27B/35B checkpoints") { + // The two spellings are named constants, not literals scattered per lookup. + CHECK(std::string(vllm::kQwen3_5VlBackbonePrefix) == "model.language_model."); + CHECK(std::string(vllm::kQwen3_5TextBackbonePrefix) == "model."); + + // A 35B-shaped VL index. `...mlp.experts.gate_up_proj` is the PUBLISHED + // stacked spelling, which the loader refuses (case 4c) — it appears here + // because the namespace probe must ignore it either way: only + // `embed_tokens.weight`, `norm.weight` and `layers.` + // vote, so what the expert tensors are called cannot change the answer. + std::vector names{ + "model.language_model.embed_tokens.weight", + "model.language_model.norm.weight", + "model.language_model.layers.0.input_layernorm.weight", + "model.language_model.layers.0.linear_attn.in_proj_qkv.weight", + "model.language_model.layers.0.mlp.experts.gate_up_proj", + "model.language_model.layers.0.mlp.shared_expert_gate.weight", + "model.language_model.layers.3.self_attn.q_proj.weight", + "lm_head.weight", + }; + CHECK(vllm::ResolveQwen3_5BackbonePrefix(names) == "model.language_model."); + + SUBCASE("LoadQwen3_5MoeLayer defaults to the VL spelling") { + const ShardBag vl(MoeOneLayerSpecs("model.language_model."), "seam_moe_vl"); + const ShardBag flat(MoeOneLayerSpecs("model."), "seam_moe_flat"); + // Prefix argument OMITTED on both calls. The VL layer must bind... + CHECK(CaptureThrow([&vl] { + (void)vllm::LoadQwen3_5MoeLayer(vl.Resolver(), "full_attention", 0, + kMoeExperts); + }).empty()); + // ...and the flat one must fail looking for the VL name, which is what the + // default actually being the VL spelling MEANS. + const std::string message = CaptureThrow([&flat] { + (void)vllm::LoadQwen3_5MoeLayer(flat.Resolver(), "full_attention", 0, + kMoeExperts); + }); + CAPTURE(message); + CHECK(Mentions(message, + "model.language_model.layers.0.input_layernorm.weight")); + } + + SUBCASE("LoadQwen3_5DenseLayer defaults to the VL spelling") { + const ShardBag vl(DenseOneLayerSpecs("model.language_model."), "seam_dn_vl"); + const ShardBag flat(DenseOneLayerSpecs("model."), "seam_dn_flat"); + CHECK(CaptureThrow([&vl] { + (void)vllm::LoadQwen3_5DenseLayer(vl.Resolver(), vl.Has(), + "full_attention", 0); + }).empty()); + const std::string message = CaptureThrow([&flat] { + (void)vllm::LoadQwen3_5DenseLayer(flat.Resolver(), flat.Has(), + "full_attention", 0); + }); + CAPTURE(message); + CHECK(Mentions(message, + "model.language_model.layers.0.input_layernorm.weight")); + } +} diff --git a/tests/vllm/test_model_loader_gguf.cpp b/tests/vllm/test_model_loader_gguf.cpp index becac1d30..9ae212704 100644 --- a/tests/vllm/test_model_loader_gguf.cpp +++ b/tests/vllm/test_model_loader_gguf.cpp @@ -89,7 +89,8 @@ TEST_CASE("FromModelDir rejects an unknown dense architecture before loading") { "'ParakeetForCTC', 'ParakeetForRNNT', 'ParakeetForTDT', " "'Phi3ForCausalLM', 'PhiForCausalLM', 'Qwen3ForCausalLM', " "'Qwen3MoeForCausalLM', 'Qwen3VLForConditionalGeneration', " - "'Qwen3_5ForConditionalGeneration', " + "'Qwen3_5ForCausalLM', 'Qwen3_5ForConditionalGeneration', " + "'Qwen3_5MoeForCausalLM', " "'Qwen3_5MoeForConditionalGeneration', 'StableLmForCausalLM'])", std::runtime_error); }