fix(qwen3_5): load quantized lm_head, not just BF16 (#164)

docs/BENCHMARKS.md

@@ -30,7 +30,7 @@ The binding comparison. vLLM runs its **production graphed config**, never
 
 | Model | Quant | vLLM pin | Axes passing | Disposition |
 |---|---|---|---:|---|
-| Qwen3.6-27B | NVFP4 | 0.25.0 | **115/124** | Effective parity-or-better, two-grid totality |
+| Qwen3.6-27B | NVFP4 | 0.25.0 | **115/124** | Effective parity-or-better, two-grid totality. Measured on `unsloth/Qwen3.6-27B-NVFP4` @`890bdef7` (BF16 head); @`ccdaab7e` re-quantized the head to FP8 |
 | Qwen3.6-35B-A3B | NVFP4 `modelopt_mixed` | 0.25.0 | 2/18 | 3-rep grid 2026-08-05 @`1ea26427`: 0.93-1.03x (c4 wins), c16 0.93x. Both c16 levers A/B'd NEG: drain event -1.9%, mirror 0.999x. ★ probe found a prod async batch-1 greedy DEGENERATION bug the mirror fixes |
 | DeepSeek-V2-Lite | bf16 MLA | 0.25.0 | 4/25 | Attributed miss, row stays `ACTIVE` |
 | Qwen3.5-4B | bf16 direct-load | 0.26.0.dev0 | throughput + host PSS | Exact chunks ON: total **1.021x PASS**; TTFT **1.086x**, TPOT **1.025x**, VRAM **1.018x OPEN**; local A/B **+2.152%** ([evidence](bench-evidence/qwen35-4b-sm120-main-20260807.md)) |

docs/FEATURES.md

@@ -94,7 +94,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks.
 <!-- supported-arch-table:begin -->
 | Architecture | Tested checkpoint(s) | Correctness gate | Speed vs reference |
 |---|---|---|---|
-| `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B NVFP4; Qwen3.5-4B BF16 | 27B strict 235/235 text + 32/32 image/video; 4B cached 3/3 | 27B at/above vLLM; 4B throughput 1.021x, latency/VRAM pending. CUDA/CPU only; the off-CUDA host-pointer bug (#125) is fixed but unrun |
+| `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B NVFP4; Qwen3.5-4B BF16 | 27B strict 235/235 text + 32/32 image/video; 4B cached 3/3 | 27B at/above vLLM; 4B throughput 1.021x, latency/VRAM pending. `lm_head` loads BF16, FP8 or NVFP4 (#164). CUDA/CPU only; the off-CUDA host-pointer bug (#125) is fixed but unrun |
 | `Qwen3_5MoeForConditionalGeneration` | Qwen3.6-35B-A3B (NVFP4, GDN MoE) | strict 315/315 text vs vLLM 0.25.0 | gate model: 0.93x to 1.03x grid |
 | `Qwen3ForCausalLM` | Qwen3 dense 0.6B/1.7B/4B/32B, NVFP4A16 | near-tie strict 16/16 vs vLLM 0.25.0 | c1 every-axis parity, c8 decode residual |
 | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B | strict 6/6 vs vLLM 0.25.0 | 11/16 grid cells at or above graphed vLLM |

docs/STATUS.md

@@ -69,7 +69,7 @@ token-for-token correctness against the pinned oracle.
 
 | Capability | State | Notes |
 |---|---|---|
-| Qwen3.6-27B (NVFP4) text generation | Correctness-complete, at/above vLLM speed | Token-exact greedy on GB10; beats vLLM 0.25.0 total throughput at every concurrency (1.007-1.045x), effective parity 115/124 axes |
+| Qwen3.6-27B (NVFP4) text generation | Correctness-complete, at/above vLLM speed | Token-exact greedy on GB10; beats vLLM 0.25.0 tput at every concurrency (1.007-1.045x), parity 115/124 axes. FP8/NVFP4 heads load (#164) |
 | Qwen3.6-35B-A3B (NVFP4, GDN MoE) | Correctness-complete; 3-rep grid 0.93-1.03x. Async batch-1 token-0 degeneration FIXED: `VT_ASYNC_DEVICE_MIRROR` default ON | Token-exact SYNC+ASYNC (RED→GREEN); c16 0.93x; `VT_ASYNC_EXECUTOR` Option A (H2D out of capture) GREEN+RED but A/B NEUTRAL → OFF; c16 residual is prefill glue |
 | Qwen3 / Qwen2 dense (BF16) | Correctness-complete, speed-pending. Async-serving P0 FIXED (`ROW-SERVE-ASYNC-DENSE-MIRROR`): classic-dense `Qwen3ForCausalLM` now honors the async device token-ids mirror; CPU-only -Werror test-guard fixes x2 | Near-tie-robust token-exact vs vLLM (Qwen3-0.6B, Qwen3-4B); c1 effective parity, c8 decode residual. **Async device-mirror (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`): the #31 fix ported to the classic dense family, dgx-VERIFIED.** The shared dense `EmbedInto` (qwen3.cpp) raced the async combine's device input-ids write against a stale host upload → token-0 degeneration on the depth-2 AsyncLLM serving path (quant-independent). `EmbedInto` now consumes the device override published by `ForwardQwen3ForCausalLM`'s `DeviceTokenIdsScope` (27B-dense template); gate `test_qwen3_dense_async_serving` RED on `VT_ASYNC_DEVICE_MIRROR=0`, GREEN default, byte-identical mirror-off. dgx GB10: async gate RED→GREEN 0.6B+4B, SACRED 0.6B+4B 184/184 unchanged (byte-neutral sync path), memcheck 0 errors; Yi30/Qwen3-8B-MXFP4 default-config e2e coherent + 3/4 token-exact (p2 = oracle-ratified near-tie, gap 0.0000), closing the QUANT-CT-MXFP4 async-default residual. RESIDUAL: sibling InternLM2/Mistral/Llama scope one-liner; W4 bench RAN; FA2 GQA-swap default-ON, c2-c8 <1.0x. `FLASH-PTXAS` #82: codegen at PARITY (no ptxas lever); gap=engine context. **D1 (2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`): the bf16 merged-QKV path (`Qwen3QkvMergeEnabled`/`VT_QWEN3_QKV_MERGE`) is now default-ON** — one `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (OLMo-2 exemplar), replacing three per-shard GEMMs. Bit-exact GEMM math (A/B unit `test_ops_qkv_merge` byte-identical, RED-first); the wider-N cuBLASLt K-reduction flips the 0.6B genuine bf16 near-tie so the SACRED 0.6B golden was regenerated (all tokens within the near-tie band, max 0.125 nats), while Qwen3-4B is byte-neutral (0 diffs, stays STRICT). Re-gated 0.6B 16/16 + 4B 16/16; consistency/launch-count fold (measured NEUTRAL on 4B decode), no new throughput owed |
 | Qwen3.5-4B plain BF16 direct loading on discrete CUDA | Correctness-complete; throughput passes, latency/VRAM open | Exact GDN chunks default ON and byte-identical to rollback. Local A/B: total/output +2.152%, TTFT -2.945%, TPOT/ITL -1.920%; sealed-vLLM comparison 1.021x throughput, 1.086x TTFT, 1.025x TPOT, +233 MiB VRAM ([evidence](bench-evidence/qwen35-4b-sm120-main-20260807.md)) |

docs/USAGE.md

@@ -50,6 +50,21 @@ Two more example binaries ship alongside it:
 - `tokenize` ([`examples/tokenize/main.cpp`](../examples/tokenize/main.cpp)), a
   tokenizer smoke tool taking `<tokenizer.json | model.gguf> <corpus.txt>`.
 
+### Quantized checkpoints: which `lm_head` forms load
+
+Publishers do not agree on how the output head is stored, and a single repo can
+change it between revisions. For the Qwen3.6 dense family we accept all three
+forms in use, so pick a checkpoint by its quality, not by its head:
+
+| `lm_head.weight` | Companion tensors | Seen in |
+|---|---|---|
+| `BF16` | none | `unsloth/Qwen3.6-27B-NVFP4` @`890bdef7` |
+| `F8_E4M3` | `lm_head.weight_scale` (per-output-channel or per-tensor) | `unsloth/Qwen3.6-27B-NVFP4` @`ccdaab7e` |
+| `U8` NVFP4 | `lm_head.weight_scale` + `weight_scale_2` (ModelOpt) or `weight_global_scale` (compressed-tensors) | `nvidia/Qwen3.6-27B-NVFP4` |
+
+The head is dequantized to BF16 at load, so all three cost the same memory once
+running. Any other dtype fails at load with a message naming what it saw.
+
 ## OpenAI-compatible server
 
 `server` is a small HTTP server speaking the OpenAI API. Source:

include/vllm/model_executor/models/qwen3_5_dense.h

@@ -31,6 +31,8 @@
 #include <vector>
 
 #include "vllm/model_executor/models/qwen3_5.h"  // PagedKvCache, GdnStateCache + v1 attention metadata
+#include <functional>
+
 #include "vllm/model_executor/models/qwen3_5_weights.h"  // OwnedTensor, Gdn/FullAttn weights, TensorResolver
 #include "vllm/transformers_utils/hf_config.h"
 #include "vt/device.h"
@@ -95,11 +97,14 @@ struct Qwen3_5DenseLayerWeights {
   DenseMlpWeights mlp;                   // every layer has a dense MLP
 };
 
-// Whole dense-model text weights. lm_head is bf16 (unquantized in the 27B).
+// Whole dense-model text weights. `lm_head` is always materialized bf16 here, but
+// the CHECKPOINT may store it BF16, FP8 (per-channel scale) or ModelOpt NVFP4 —
+// the 27B NVFP4 publishers disagree, and revisions of one repo disagree with each
+// other (issue #164). LoadLmHeadAnyDtype dequantizes all three to this operand.
 struct Qwen3_5DenseWeights {
   OwnedTensor embed_tokens;  // bf16 [vocab, H]  (NOT transposed; embed lookup)
   OwnedTensor final_norm;    // bf16 [H]
-  OwnedTensor lm_head;       // bf16 [H, vocab]  (unquantized -> Matmul-B layout)
+  OwnedTensor lm_head;       // bf16 [H, vocab]  (dequantized -> Matmul-B layout)
   // Mirrors tie_word_embeddings: logits reuse embed_tokens as raw [V,H]
   // torch-Linear storage, so no second host/device owner is created.
   bool tied_lm_head = false;
@@ -121,6 +126,15 @@ bool IsQwen27QuantizedLinear(const std::string& name);
 // reciprocates the global scale), rounds to bf16, and transposes. Exposed for
 // unit testing. The `<proj>.input_global_scale` (activation divisor) is ignored
 // on this bf16-activation correctness path (notes §3.4 / §5 step-6a).
+// `lm_head` across the three storage forms the 27B NVFP4 publishers actually ship
+// (issue #164): BF16, FP8 `+_scale` (per-output-channel or per-tensor), and NVFP4
+// `+_scale` `+_scale_2`/`+_global_scale`. Always returns bf16 [in, out] Matmul-B,
+// so a BF16 head is byte-identical to the previous LoadBf16Transposed call.
+// `has` probes optional companion tensors. Exported for the loader gate.
+OwnedTensor LoadLmHeadAnyDtype(const TensorResolver& get,
+                               const std::function<bool(const std::string&)>& has,
+                               const std::string& name);
+
 OwnedTensor MaterializeCtNvfp4Bf16Transposed(const TensorResolver& get,
                                              const std::string& proj);
 

src/vllm/model_executor/models/qwen3_5_dense_weights.cpp

@@ -13,6 +13,7 @@
 #include <vector>
 
 #include "vllm/model_executor/layers/quantization/compressed_tensors/nvfp4_emulation.h"
+#include "vllm/model_executor/model_loader/nvfp4_dequant.h"
 #include "vllm/model_executor/models/dense_weight_loaders.h"
 #include "vllm/platforms/interface.h"
 #include "vt/backend.h"
@@ -187,6 +188,132 @@ Nvfp4Weight LoadCtNvfp4Raw(const TensorResolver& get, const std::string& proj) {
   return r;
 }
 
+// --- lm_head dtype dispatch (issue #164) --------------------------------------
+// The 27B NVFP4 publishers do NOT agree on the OUTPUT HEAD, and the head is not
+// a compressed-tensors Linear, so none of the scheme probes above cover it:
+//
+//   BF16     `lm_head.weight` [V,H]                                (transpose)
+//   F8_E4M3  `lm_head.weight` [V,H] + `.weight_scale` [V,1] or []  (per-row/scalar)
+//   U8       `lm_head.weight` [V,H/2] + `.weight_scale` F8 [V,H/16]
+//                                     + `.weight_scale_2` f32       (ModelOpt NVFP4)
+//
+// This loader was written against `unsloth/Qwen3.6-27B-NVFP4` @890bdef7, which
+// ships a BF16 head — the snapshot every recorded 27B-NVFP4 benchmark ran on, so
+// those numbers are unaffected by this change. @ccdaab7e later re-quantized the
+// head to FP8 with a PER-OUTPUT-CHANNEL scale, and nvidia/Qwen3.6-27B-NVFP4 ships
+// a ModelOpt NVFP4 head; both hit the old unconditional BF16 assert.
+//
+// All three land on the SAME bf16 [in, out] Matmul-B operand the logits GEMM
+// already consumes, so the forward is untouched and a BF16 head stays byte-exact
+// (identical call, no dequant). Keeping the head quantized end-to-end would save
+// ~2.3 GiB but needs an `lm_head_fp4`-style field on the dense weights plus a
+// forward branch; that is a follow-up, not this fix.
+//
+// ModelOpt vs compressed-tensors global-scale convention: CT stores the value as
+// a DIVISOR and `DequantCtNvfp4WeightToF32` reciprocates it internally, whereas
+// ModelOpt's `weight_scale_2` IS the scale (qwen3_5_weights.cpp:272 assigns it to
+// `scale2` directly). Passing `1/weight_scale_2` as the "disk divisor" makes the
+// shared CT dequant compute the ModelOpt scale exactly.
+}  // namespace
+
+OwnedTensor LoadLmHeadAnyDtype(const TensorResolver& get, const TensorExists& has,
+                               const std::string& name) {
+  const StTensor& w = get(name);
+  VT_CHECK(w.shape.size() == 2, "qwen3_5 dense: expected 2-D weight for " + name);
+
+  if (w.dtype == "BF16") {
+    return LoadBf16Transposed(get, name);  // unchanged byte-for-byte
+  }
+
+  if (w.dtype == "F8_E4M3") {
+    const int64_t out_dim = w.shape[0];
+    const int64_t in_dim = w.shape[1];
+    // Per-output-channel [V,1] (unsloth @ccdaab7e) or a single per-tensor scalar.
+    // Stored BF16 there, F32 elsewhere; normalize both to f32 rows.
+    std::vector<float> row_scale(static_cast<size_t>(out_dim), 1.0F);
+    VT_CHECK(has(name + "_scale"),
+             "qwen3_5 dense: FP8 " + name + " requires " + name + "_scale");
+    const StTensor& sc = get(name + "_scale");
+    const int64_t n_scale =
+        static_cast<int64_t>(sc.nbytes) / (sc.dtype == "BF16" ? 2 : 4);
+    VT_CHECK(n_scale == out_dim || n_scale == 1,
+             "qwen3_5 dense: " + name + "_scale must be per-tensor or [out,1]");
+    for (int64_t r = 0; r < out_dim; ++r) {
+      const int64_t i = (n_scale == 1) ? 0 : r;
+      if (sc.dtype == "BF16") {
+        uint16_t h = 0;
+        std::memcpy(&h, static_cast<const uint8_t*>(sc.data) + i * 2, 2);
+        const uint32_t bits = static_cast<uint32_t>(h) << 16;
+        std::memcpy(&row_scale[static_cast<size_t>(r)], &bits, sizeof(float));
+      } else {
+        std::memcpy(&row_scale[static_cast<size_t>(r)],
+                    static_cast<const uint8_t*>(sc.data) + i * 4, sizeof(float));
+      }
+    }
+    std::vector<uint16_t> dq(static_cast<size_t>(out_dim) * in_dim);
+    for (int64_t r = 0; r < out_dim; ++r) {
+      DequantFp8ToBf16(static_cast<const uint8_t*>(w.data) + r * in_dim,
+                       row_scale[static_cast<size_t>(r)], in_dim,
+                       dq.data() + static_cast<size_t>(r) * in_dim);
+    }
+    MaybeReleaseSourcePages(w.data, w.nbytes);
+    OwnedTensor o = MakeOwned(vt::DType::kBF16, {in_dim, out_dim});
+    dense_loaders::TransposeBf16(dq.data(), out_dim, in_dim,
+                                 reinterpret_cast<uint16_t*>(o.bytes.data()));
+    return o;
+  }
+
+  if (w.dtype == "U8") {
+    const int64_t out_dim = w.shape[0];
+    const int64_t in_dim = w.shape[1] * 2;
+    VT_CHECK(in_dim % 16 == 0,
+             "qwen3_5 dense: NVFP4 in_dim must be a multiple of 16 for " + name);
+    const StTensor& ws = get(name + "_scale");
+    VT_CHECK(ws.dtype == "F8_E4M3",
+             "qwen3_5 dense: expected F8_E4M3 " + name + "_scale");
+    // ModelOpt spells the global scale `weight_scale_2`; compressed-tensors
+    // spells it `weight_global_scale` and stores the reciprocal.
+    float disk_divisor = 0.0F;
+    if (has(name + "_scale_2")) {
+      const float ws2 = ReadF32Scalar(get(name + "_scale_2"));
+      VT_CHECK(ws2 != 0.0F, "qwen3_5 dense: zero " + name + "_scale_2");
+      disk_divisor = 1.0F / ws2;  // ModelOpt scale -> CT divisor convention
+    } else {
+      VT_CHECK(has(name + "_global_scale"),
+               "qwen3_5 dense: NVFP4 " + name + " requires " + name +
+                   "_scale_2 (ModelOpt) or " + name + "_global_scale (CT)");
+      disk_divisor = ReadF32Scalar(get(name + "_global_scale"));
+      VT_CHECK(disk_divisor != 0.0F,
+               "qwen3_5 dense: zero " + name + "_global_scale (divisor)");
+    }
+    std::vector<float> f32(static_cast<size_t>(out_dim) * in_dim);
+    DequantCtNvfp4WeightToF32(static_cast<const uint8_t*>(w.data),
+                              static_cast<const uint8_t*>(ws.data), disk_divisor,
+                              out_dim, in_dim, f32.data());
+    MaybeReleaseSourcePages(w.data, w.nbytes);
+    std::vector<uint16_t> dq(static_cast<size_t>(out_dim) * in_dim);
+    for (size_t i = 0; i < f32.size(); ++i) {
+      uint32_t bits = 0;
+      std::memcpy(&bits, &f32[i], sizeof(bits));
+      // round-to-nearest-even f32 -> bf16, matching DequantFp8ToBf16.
+      const uint32_t lsb = (bits >> 16) & 1U;
+      bits += 0x7FFFU + lsb;
+      dq[i] = static_cast<uint16_t>(bits >> 16);
+    }
+    OwnedTensor o = MakeOwned(vt::DType::kBF16, {in_dim, out_dim});
+    dense_loaders::TransposeBf16(dq.data(), out_dim, in_dim,
+                                 reinterpret_cast<uint16_t*>(o.bytes.data()));
+    return o;
+  }
+
+  VT_CHECK(false, "qwen3_5 dense: unsupported dtype '" + w.dtype + "' for " +
+                      name + "; supported: BF16, F8_E4M3 (+_scale), "
+                      "U8 NVFP4 (+_scale and _scale_2/_global_scale)");
+  return OwnedTensor{};
+}
+
+namespace {
+
 GdnLayerWeights LoadGdnDense(const TensorResolver& get, const TensorExists& has,
                              const std::string& base) {
   const std::string la = base + "linear_attn.";
@@ -395,7 +522,7 @@ Qwen3_5DenseWeights LoadQwen3_5Dense(const std::vector<SafetensorsFile>& shards,
   // The 27B owns an explicit head; smaller Qwen3.5 checkpoints tie logits to
   // the embedding table and omit lm_head.weight.
   if (has("lm_head.weight")) {
-    w.lm_head = LoadBf16Transposed(get, "lm_head.weight");
+    w.lm_head = LoadLmHeadAnyDtype(get, has, "lm_head.weight");
   } else {
     w.tied_lm_head = true;
     w.embed_tokens.nk = true;

tests/CMakeLists.txt

@@ -1337,6 +1337,9 @@ target_include_directories(test_mistral_paged_engine PRIVATE
 # 0 multi-valued cells). Checkpoint-gated, dgx-only.
 vllm_cpp_add_test(test_qwen3_32b_nvfp4a16_load
   vllm/models/test_qwen3_32b_nvfp4a16_load.cpp)
+
+vllm_cpp_add_test(test_qwen3_5_lm_head_dtypes
+  vllm/models/test_qwen3_5_lm_head_dtypes.cpp)
 vllm_cpp_add_test(test_qwen3_32b_nvfp4a16_paged_engine
   vllm/models/test_qwen3_32b_nvfp4a16_paged_engine.cpp)
 target_compile_definitions(test_qwen3_32b_nvfp4a16_paged_engine PRIVATE