diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 53a7287b8..e88490182 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -253,3 +253,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#949](https://github.com/mudler/vllm.cpp/issues/949) | — | Nothing in the tree refuses a borrowed `vt::Tensor` that outlives the object owning its storage, and the ONLY instrument that catches one is `sanitize-cpu`, which is `continue-on-error` — that is how [#904](https://github.com/mudler/vllm.cpp/issues/904) landed. Measured in the #936 review rather than argued: with the #904 fix reverted, a plain Release build with no sanitizer runs the case 18/18 passed, 546 assertions, `rc=0`, because `dtype` lives in the `vt::Tensor` struct and not in the freed buffer, so no ordinary gate can see the dangling read. Three remedies are open and none is foregone: promote the lane once it has a `main` baseline, add a test that fails without a sanitizer, or reject the pattern statically — a prototype detector for a member access chained onto a call returning an owning type by value swept 1777 files with no hit but the defect. Anchors: the owning deleter `src/vllm/model_executor/models/ltx2_device.cpp:1088 @ 800dd082f`, the read `src/vt/cpu/cpu_layernorm.cpp:33 @ 800dd082f`. Listed under `## Owed` in [`ltx2-device-staged-view-uaf.md`](specs/ltx2-device-staged-view-uaf.md) | bug | | [#933](https://github.com/mudler/vllm.cpp/issues/933) | `ENG-EXPERT-STREAM` | Measure gateability of the `llama-cpp-unsloth` oracle by BUILDING it and RUNNING `Qwen3.8-2.4T-A95B UD-Q1_0` on it. The oracle is pinned at `36fe8e1cc` (branch `iq1-narrow`) and records `gateable = no`, because the IQ1_XXXS port is grounded in the fork's SOURCE, read and cited, which is weaker than a running comparison. It is the only place ggml type 66 is defined: the vllm.cpp pin `237ad9b96` ends at `Q1_0 = 41` and `ggml-org` master `ad1de39e0` at `Q2_0 = 42`, while type 66 carries 96.92 % of that checkpoint's parameters. Running it needs the full 370 GiB checkpoint and, per Unsloth's documentation, at least 450 GB of RAM. Until then the ported arm has no running oracle, which is what `gateable = no` makes visible | task | | [#957](https://github.com/mudler/vllm.cpp/issues/957) | `FIX-OFFLOAD-DOCS-957` | `4a183b731` (#887) turned a configured weight offload from ACCEPTED-AND-INERT into a hard startup refusal, and neither public document followed. `docs/WEIGHT-OFFLOAD.md` still said "a budget you set is accepted, reported, and does not free memory" and `docs/USAGE.md:1473` still said "Accepted and inert today", while `RefuseUnsupportedWeightOffload` (`src/vllm/model_executor/weight_offloader.cpp:72-83` @ 2daa3287f) throws from the load path (`src/vllm/entrypoints/model_loader.cpp:1410-1414` @ 2daa3287f) before any weight I/O. `ModelFactory::supports_weight_offload` defaults false and NO model sets it, so every architecture is refused; `tests/vllm/model_executor/test_weight_offloader.cpp:376-379` @ 2daa3287f asserts that count itself. Found auditing the 28 commits `documentation-checkpoint` flags: 27 needed nothing | bug | +| [#940](https://github.com/mudler/vllm.cpp/issues/940) | `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | The FP8 W8A8 linear path is not a shared seam: `ResidentFp8`, `MatmulFp8CutlassD` and `MatmulFp8CutlassPreQuantD` lived in the anonymous namespace of `src/vllm/model_executor/models/qwen3_5.cpp` (`:1458`, `:1495`, `:1517` @ `c7cb59fbb`), so a second model could reach them only by copying them — the hand-rolled parallel path AGENTS.md §"Shared seams" forbids. NVFP4 already had `dense_nvfp4_gemm.h` + `compressed_tensors/schemes/nvfp4.h`; FP8 had neither half. Forced by `MODEL-NEMOTRON-H` ([#517](https://github.com/mudler/vllm.cpp/issues/517)), whose 46 FP8 W8A8 mamba `in_proj`/`out_proj` projections are 36.6% of decode bytes and 27.6% of GEMM FLOPs, and whose `in_proj` produces the fused `zxbcdt` the conv and the SSD scan consume — so that block cannot be split and has no device path at all without the seam. Extracted to `dense_fp8_gemm.h` + `quantization/fp8.h` with Qwen3.5 byte-identity as the gate; spec [`vt-fp8-shared-seam.md`](specs/vt-fp8-shared-seam.md) | bug | diff --git a/.agents/specs/vt-fp8-shared-seam.md b/.agents/specs/vt-fp8-shared-seam.md new file mode 100644 index 000000000..79e1094ef --- /dev/null +++ b/.agents/specs/vt-fp8-shared-seam.md @@ -0,0 +1,206 @@ +# VT-FP8-SHARED-SEAM — the FP8 W8A8 linear path becomes a shared seam + +Issue: [#940](https://github.com/mudler/vllm.cpp/issues/940). +Owning row: `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` (the row that forces it; [#517](https://github.com/mudler/vllm.cpp/issues/517), +spec [`nemotron-h-abi-e2e.md`](nemotron-h-abi-e2e.md)). +Related: [`vt-fp8-w8a8-cpu-arm.md`](vt-fp8-w8a8-cpu-arm.md) — #468/#842's op-tier +registrations, whose §Residual gap names exactly the model-tier half this row +closes; [`perf-fp8-alpha-fold.md`](perf-fp8-alpha-fold.md) — the folded-alpha +lever whose arithmetic this seam carries unchanged. + +AGENTS.md §"Shared seams": *"If a shared seam cannot represent the upstream +behavior, extend it or record one exact tracked exception. Never hand-roll a +parallel path."* + +NVFP4 already honours that: +`include/vllm/model_executor/models/dense_nvfp4_gemm.h` is a real seam with a +policy layer at `compressed_tensors/schemes/nvfp4.h`. FP8 W8A8 did not. The +weight struct `Fp8Weight` was in a header +(`qwen3_5_weights.h:273 @ c7cb59fbb`), but everything that made it *usable* sat +inside one translation unit: + +| Entry point | Was (`qwen3_5.cpp @ c7cb59fbb`) | +|---|---| +| `ResidentFp8` | `:1458` | +| `DenseCublasLtFp8Enabled` | `:1478` | +| `MatmulFp8CutlassD` | `:1495` (CUDA guard `:1497-1498`) | +| `MatmulFp8CutlassPreQuantD` | `:1517` (CUDA guard `:1519-1520`) | + +A second model therefore had three options and the policy forbids two: include a +`.cpp`, copy the entry points, or extend the seam. This row extends it. + +## Scope + +**IN.** + +1. `include/vllm/model_executor/models/dense_fp8_gemm.h` — the four definitions + above, moved VERBATIM, mirroring `dense_nvfp4_gemm.h`'s shape (preamble with + the upstream chain, `namespace vllm::dense_fp8`, header-only, the CUDA guard + travelling with the code it guards). +2. `include/vllm/model_executor/layers/quantization/fp8.h` — `Fp8W8A8LinearMethod` + + `MakeLinearMethod(const OwnedTensor&, const Fp8Weight&)`, mirroring + `compressed_tensors/schemes/nvfp4.h`'s `Nvfp4W4A16LinearMethod` + + `MakeLinearMethod`. +3. `src/vllm/model_executor/models/qwen3_5.cpp` — calls the extracted seam. The + ~14 call sites are UNCHANGED text; what changes is that the three names they + call are now two one-line type adapters plus a `using`. +4. `tests/vllm/model_executor/layers/test_linear_method.cpp` — the two CPU cases + that make the policy layer's selection and its reach into the seam + falsifiable. + +**OUT.** + +- **Wiring NemotronH to this seam.** That is A2-Q under #517, and it is what the + seam exists for, but a model port inside an extraction would make the + byte-identity gate below unreadable. +- **Any numerics, tolerance, guard or dispatch condition.** Specifically the + CUDA-only refusal keyed on `kMatmulFp8CublasLt` is carried across unchanged — + see §Residual gap. +- `ResidentFp8Qkv` / `ResidentFp8Qkvz` and the merged-QKV(z) fp8 path. They are + 35B-specific merged-operand builders, not the general linear seam, and #940 + does not name them. +- The missing device-upload accounting recorded under §Found, not fixed. + +### Why the seam is templated on `Dev`/`DBuf` + +`qwen3_5.cpp` carries its own anonymous-namespace `Dev`/`DBuf` — the KNOWN +DUPLICATION `dense_nvfp4_gemm.h:45-50` records, deliberately, because unifying +the device-glue families is a separate gate-model-touching refactor. Those types +are layout-identical to `dense_attn::Dev`/`DBuf` but are *distinct types*, so a +non-template header could only have been COPIED into qwen3_5.cpp, not called by +it. + +That copy is the failure mode #940 exists to prevent: a seam sitting dead beside +the production path, where "byte-identical" is vacuously true because nothing +routed through it. Templating on the two glue types instead gives ONE definition +with two instantiations — qwen3_5.cpp instantiates it with its own types +(generating the code it had), the `vllm::layers` policy layer instantiates it +with the shared ones. The mutation table below is what turns that from a claim +into a measurement. + +## Upstream chain + +Our loader (`LoadFp8Raw`, `qwen3_5_weights.cpp:423`) accepts both the +compressed-tensors and the ModelOpt spelling of the same per-tensor FP8 W8A8 +checkpoint and reduces them to one `Fp8Weight`. Upstream, all three entry points +delegate to the same shared `fp8_linear.apply_weights`, which is why one method +here covers every spelling: + +| Concern | Upstream (pin `555967922`) | +|---|---| +| compressed-tensors scheme | `compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py:60,201-207` | +| ModelOpt scheme | `modelopt.py:444,531-537` | +| generic fp8 linear | `fp8.py:267,446` | +| static per-tensor act quant | `utils/quant_utils.py:124` `kFp8StaticTensorSym`, handed to `init_fp8_linear_kernel` at `modelopt.py:511-512` → our `vt::QuantFp8Static` | +| per-tensor scaled epilogue | the folded `alpha = input_scale * weight_scale` | +| scheme selection | `base_config.py:180` `get_quant_method` → `MakeLinearMethod` | + +**Header placement.** `quantization/fp8.h`, not `quantization/schemes/fp8.h`. +#940's text names `schemes/nvfp4.h` for a file that actually lives at +`compressed_tensors/schemes/nvfp4.h`, so it is using a shorthand, and inventing a +`quantization/schemes/` directory would mirror nothing upstream. `fp8.py` sits +directly under `quantization/` in vLLM, and `layers/quantization/ +modelopt_mixed_precision.h` is the local precedent for a header-only policy +header in that same place. + +## Design + +`dense_fp8_gemm.h` is header-only and carries, in order: + +- `DenseCublasLtFp8Enabled()` — `VT_DENSE_CUBLASLT_FP8`, default ON. Unchanged + lever, unchanged spelling, unchanged default. +- `ResidentFp8(DevT, const Fp8Weight&)` — the lazy one-shot upload of the raw + fp8 `[N,K]` bytes, owned by the (const) weight's `mutable shared_ptr`. +- `MatmulFp8CutlassD(DevT, x, w, out_dtype)` — `QuantFp8Static` then the + folded-alpha fp8 GEMM (cuBLASLt by default, cutlass under the lever). +- `MatmulFp8CutlassPreQuantD(DevT, a_fp8, w, out_dtype)` — the same GEMM + over an activation a preceding fused epilogue already quantized. Upstream this + is the `x: torch.Tensor | QuantizedActivation` overload of the same apply. + +`quantization/fp8.h` adds nothing computational: `Apply` is +`MatmulFp8CutlassD`, `ApplyPreQuantized` is +`MatmulFp8CutlassPreQuantD`, and `MakeLinearMethod` selects on +`Fp8Weight::Empty()` exactly as the NVFP4 factory selects on +`Nvfp4Weight::Empty()`. + +### Residual gap — carried, not closed + +`MatmulFp8CutlassD` refuses on a host queue because its guard asks for +`kMatmulFp8CublasLt`, registered for kCUDA only, while the ops it would actually +run (`kQuantFp8Static`, `kMatmulFp8Cutlass`) DO have CPU reference arms since +#468/#842. That mismatch is recorded in +[`vt-fp8-w8a8-cpu-arm.md`](vt-fp8-w8a8-cpu-arm.md) §Residual gap and pinned at +`tests/vt/test_ops_fp8_cpu.cpp:445-453`. Widening it is a dispatch change; an +extraction that quietly widened it would be invisible to a byte-identity gate, +which is the whole reason it is out of scope here. The new CPU case re-pins it +at the model tier so it stays visible rather than assumed closed. + +## Risks + +- **The seam lands dead beside the production path.** Mitigated by construction + (one definition, instantiated) and MEASURED by mutations M3–M4 below, which + perturb the header and require a Qwen3.5 CUDA arm to go red. +- **A behaviour change hides inside a move.** Mitigated by moving the bodies + verbatim and by the assertion-count identity below; the only textual change is + `MakeTensor` → `dense_attn::MakeTensor` (identical body, + `dense_device_glue.h:47` vs `qwen3_5.cpp:583`). +- **Template instantiation differs from the inline code.** Both instantiations + are `inline` in the same TU as before; the CPU full gate and the CUDA arm are + what test this rather than the argument. + +## Tests and gates + +Qwen3.5 byte-identity is the gate. Assertion counts before and after must be +identical for every pre-existing suite; the two NEW cases are additive and +declared as such. + +| Suite | Before (`c7cb59fbb`) | After | +|---|---|---| +| `test_qwen3_5_gdn_spec_routing` | 6 cases / 52 assertions | 6 / 52 | +| `test_ops_fp8_cpu` | 4 / 56 | 4 / 56 | +| `test_qwen27_paged_forward` | 29 / 765 | 29 / 765 | +| `test_qwen27_dense_forward` | 9 / 583 | 9 / 583 | +| `test_linear_method` | 6 / 76 | 8 / 88 (+2 new cases) | + +Full CPU gate: clean configure + `cmake --build build -j 12` (0 warnings under +`-Werror`) + `ctest -j 4`. + +Mutation table (the "the seam is LIVE" proof) is recorded in §Evidence. + +## Found, not fixed + +`ResidentFp8` — and its merged siblings `ResidentFp8Qkv` / `ResidentFp8Qkvz` — +`Alloc` + `Copy` the fp8 weight bytes to the device without calling +`vllm::load_stats::AddDeviceUpload` and without the post-upload +`AdoptDeviceBytesAsHost` step. Both are performed by every other resident-weight +helper in the same file: `ResidentWeight` (`qwen3_5.cpp:1009,1016 @ c7cb59fbb`) +and `ResidentNvfp4` (`:1106,1111,1116,1121`), and `dense_nvfp4_gemm.h:294-328` +carries the comment explaining why the pair is mandatory ("this is the one +host->device move of those bytes and it must be accounted and followed by the +same post-upload residency step every other qualifying weight gets", ENG-LOAD +-DIRECT-UPLOAD / #150). + +Consequences, both plausible and neither measured here: the 35B fp8 tower's +upload is missing from load accounting, and its device pages are never re-tagged, +which is the shape of the GB10 weight-residency ATS penalty. Carried across +UNCHANGED — repairing it inside an extraction would be exactly the behaviour +change hidden in a move that this row's gate cannot see. Filed separately. + +## Stop conditions + +- The extraction cannot be made behaviour-preserving — return `NEEDS_DECISION` + with the demonstration rather than adapting the numerics. +- A mutation of the extracted code leaves every Qwen3.5 arm green: the seam is + not on the production path and the byte-identity claim is vacuous. Do not + land; re-wire until a mutation bites. + +## Outcome + +Landed as a pure extraction. See §Evidence in the PR for the before/after +assertion counts, the four-row mutation table, and the full-gate result. + +## Now + +`DONE` — the seam exists and Qwen3.5 routes through it. Wiring +`MODEL-NEMOTRON-H`'s 46 FP8 W8A8 projections to it is A2-Q under #517 and is +NOT part of this row. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index c71a9abde..2f253511a 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -76,6 +76,7 @@ are our reading of their documented behavior, not measurements. | GPTQ | ◐ CPU dequant | ✅ | ✅ | ☐ | | MXFP4 compressed-tensors | ◐ W4A16 Marlin, mem 2.63x less. gate_up FUSION + decode-graph default-ON; #44 3/3, 32B 6/6. **`VT_MARLIN_DENSE` DEFAULT-ON** (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA, byte-faithful, beats MoE (c8 0.969) | ✅ | ✅ | ☐ | | fp8 weights | ✅ | ✅ | ✅ | ☐ | +| Per-tensor FP8 W8A8 linear is a shared seam any model can bind | ✅ `models/dense_fp8_gemm.h` + `layers::Fp8W8A8LinearMethod` (#940), bound via `layers::MakeLinearMethod`. One definition, CUDA only ([spec](../.agents/specs/vt-fp8-shared-seam.md)) | ✅ `Fp8LinearMethod` | ✅ | ☐ | | fp8-tower GDN `in_proj` emits bf16, unlocking packed GDN decode | ◐ `VT_GDN_FP8_IN_BF16` + `VT_GDN_PACKED_DECODE_FP8_TOWER` (inert alone), both default **OFF**, ungated (#339) ([spec](../.agents/specs/perf-fp8-alpha-fold.md)) | ✅ bf16 `out_dtype` | ☐ | ☐ | | Merged fp8 projection folds per-column alpha in the GEMM epilogue | ◐ `VT_FP8_ALPHA_VEC_EPILOGUE`, CUDA only, default off, ungated; refuses split-K under a bf16-D equivalence claim (`claims_splitk1_premise`, default off) | n/a | n/a | n/a | | `vt::MulColVecF32` carries a bf16 store width | ✅ f32 arm byte-identical; bf16 arm rounds once; CPU + CUDA | n/a | ☐ | ☐ | diff --git a/docs/USAGE.md b/docs/USAGE.md index 54e44c857..ff60ca23e 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -257,6 +257,15 @@ refused naming both sides rather than bound half from each. Different components MAY disagree with each other: a `modelopt_mixed` checkpoint really does ship an FP8 tower beside an NVFP4 MLP, and the dense arm reads exactly that. +**Which code runs an FP8 projection is no longer a Qwen3.5 detail.** The +per-tensor FP8 W8A8 residency and GEMM entry points live in +`include/vllm/model_executor/models/dense_fp8_gemm.h`, with the scheme policy in +`include/vllm/model_executor/layers/quantization/fp8.h`, so any model binds them +through `layers::MakeLinearMethod(bf16_weight, fp8_weight)` — the same shape the +NVFP4 W4A16 seam already had. Nothing about running Qwen3.5 changes: the levers +(`VT_DENSE_NATIVE`, `VT_DENSE_CUBLASLT_FP8`) keep their names and defaults, and +the path stays CUDA-only. + Still OWED for the MoE arm, and refused BY NAME rather than discovered as a dtype complaint: an NVFP4 attention or GDN tower, an FP8 shared expert, an FP8 `lm_head`, a per-expert-but-unquantized routed layout, and a non-BF16 stacked diff --git a/include/vllm/model_executor/layers/quantization/fp8.h b/include/vllm/model_executor/layers/quantization/fp8.h new file mode 100644 index 000000000..e22a67a3d --- /dev/null +++ b/include/vllm/model_executor/layers/quantization/fp8.h @@ -0,0 +1,99 @@ +// Per-tensor FP8 (W8A8) LinearMethod + the scheme-selection factory. +// +// UPSTREAM (ported FROM, ground-every-impl rule): +// vllm/model_executor/layers/quantization/fp8.py:267,446 +// Fp8LinearMethod / Fp8LinearMethod.apply — the generic per-tensor fp8 W8A8 +// linear this file mirrors, and the reason this header sits directly under +// quantization/ rather than under compressed_tensors/schemes/: upstream +// `fp8.py` is a quantization-level module, not a compressed-tensors scheme. +// vllm/model_executor/layers/quantization/compressed_tensors/schemes/ +// compressed_tensors_w8a8_fp8.py:60,201-207 (CompressedTensorsW8A8Fp8 — +// the compressed-tensors spelling of the same checkpoint) +// vllm/model_executor/layers/quantization/modelopt.py:444,531-537 +// (ModelOptFp8LinearMethod.apply — the ModelOpt spelling; note all three +// delegate to the SAME `self.fp8_linear.apply_weights`, which is why one +// method here covers every spelling our loader accepts) +// vllm/model_executor/layers/quantization/base_config.py:180 +// QuantizationConfig.get_quant_method(layer, prefix) — the selection +// MakeLinearMethod below mirrors: pick the method ONCE from the +// checkpoint, not per call +// +// Our loader accepts both spellings (`LoadFp8Raw`, +// src/vllm/model_executor/models/qwen3_5_weights.cpp:423) and reduces them to +// one `Fp8Weight`: raw fp8-e4m3fn [N,K] bytes + per-tensor `weight_scale` + +// per-tensor `input_scale` + the folded `alpha = input_scale * weight_scale`. +// That is exactly the state upstream's shared `fp8_linear` consumes. +// +// These methods are a THIN policy wrapper over the byte-exact compute in +// dense_fp8_gemm.h (MatmulFp8CutlassD / MatmulFp8CutlassPreQuantD): identical +// vt:: op sequence, identical operands, identical order — the SAME template +// bodies src/vllm/model_executor/models/qwen3_5.cpp instantiates for the 35B +// production forward, instantiated here with the shared dense_attn::Dev/DBuf. +// Issue #940. +// +// DEVICE REACH — this method refuses on a host queue, and that is inherited, +// not chosen here. `MatmulFp8CutlassD`'s guard asks whether +// `vt::OpId::kMatmulFp8CublasLt` is registered for the running device, and it is +// registered for kCUDA only. #468/#842 registered CPU reference arms for +// `kQuantFp8Static` and `kMatmulFp8Cutlass`, so the OPS exist on the host, but +// the model-layer predicate still names the cuBLASLt op — the residual gap +// pinned at tests/vt/test_ops_fp8_cpu.cpp:445-453. Widening it is a dispatch +// change and belongs to its own row, not to the extraction that created this +// header. +#pragma once + +#include + +#include "vllm/model_executor/layers/linear.h" +#include "vllm/model_executor/models/dense_fp8_gemm.h" // MatmulFp8Cutlass{,PreQuant}D +#include "vllm/model_executor/models/qwen3_5_weights.h" // OwnedTensor, Fp8Weight +#include "vt/ops.h" + +namespace vllm { +namespace layers { + +// Per-tensor FP8 W8A8 plain linear (q/k/v/o_proj, GDN in_proj_qkv/z, out_proj, +// and the mamba in_proj/out_proj MODEL-NEMOTRON-H needs). `Apply` == static +// per-tensor activation quant + the folded-alpha fp8 GEMM, byte-for-byte +// dense_fp8::MatmulFp8CutlassD. +class Fp8W8A8LinearMethod : public LinearMethodBase { + public: + explicit Fp8W8A8LinearMethod(const Fp8Weight* weight) : w_(weight) {} + + DBuf Apply(Dev d, const vt::Tensor& x, vt::DType out_dtype) const override { + return dense_fp8::MatmulFp8CutlassD(d, x, *w_, out_dtype); + } + + // The `QuantizedActivation` overload of the same upstream apply + // (compressed_tensors_w8a8_fp8.py:201-207): `a_fp8` is ALREADY the static + // per-tensor fp8 [M,K] a preceding fused epilogue produced, so the internal + // QuantFp8Static is skipped and only the GEMM runs. Not on LinearMethodBase, + // because a bf16 method has no meaning for a pre-quantized fp8 activation — + // a caller reaches it only after selecting this scheme. + DBuf ApplyPreQuantized(Dev d, const vt::Tensor& a_fp8, + vt::DType out_dtype) const { + return dense_fp8::MatmulFp8CutlassPreQuantD(d, a_fp8, *w_, out_dtype); + } + + const char* Name() const override { return "fp8-w8a8-per-tensor"; } + + private: + const Fp8Weight* w_; +}; + +// --- Selection factory (mirrors get_quant_method) --------------------------- +// The scheme is chosen ONCE, here, from the checkpoint's populated weights: +// exactly one of {bf16, fp8} is present per projection (`LoadFp8Raw` fills the +// fp8 field and leaves the bf16 one EMPTY, and the dequant-at-load fallback does +// the reverse — qwen3_5_weights.cpp:391-405), so a non-empty fp8 weight selects +// the quantized method. Same shape as MakeLinearMethod for NVFP4 in +// compressed_tensors/schemes/nvfp4.h; overloaded on the weight type, so a model +// including both headers gets the right one by argument type. +inline std::unique_ptr MakeLinearMethod( + const OwnedTensor& bf16_w, const Fp8Weight& fp8_w) { + if (!fp8_w.Empty()) return std::make_unique(&fp8_w); + return std::make_unique(&bf16_w); +} + +} // namespace layers +} // namespace vllm diff --git a/include/vllm/model_executor/models/dense_fp8_gemm.h b/include/vllm/model_executor/models/dense_fp8_gemm.h new file mode 100644 index 000000000..74fa33a42 --- /dev/null +++ b/include/vllm/model_executor/models/dense_fp8_gemm.h @@ -0,0 +1,170 @@ +// Shared per-tensor **FP8 W8A8** (fp8-e4m3fn weights, fp8 activations) dense +// GEMM glue — the FP8 half of the QUANT-SCHEME additivity seam, and the sibling +// of dense_nvfp4_gemm.h. +// +// Extracted (behavior-preserving) from the anonymous namespace of +// src/vllm/model_executor/models/qwen3_5.cpp (`ResidentFp8`, +// `DenseCublasLtFp8Enabled`, `MatmulFp8CutlassD`, `MatmulFp8CutlassPreQuantD`) +// so a SECOND model needing an fp8 W8A8 projection can reach the residency + +// GEMM entry points without copying them. Before this header the only ways in +// were `#include`ing a `.cpp` (impossible) or re-typing the entry points into +// another translation unit — the hand-rolled parallel path AGENTS.md §"Shared +// seams" forbids. Issue #940. +// +// The forcing row is MODEL-NEMOTRON-H (#517): Nemotron-3.5-Lightning ships 46 +// FP8 W8A8 projections (the mamba `in_proj`/`out_proj` of 23 layers), 36.6% of +// its decode bytes and 27.6% of its GEMM FLOPs. `in_proj` produces the fused +// `zxbcdt` the conv and the SSD scan consume, so that block cannot be split: +// without this seam its device path does not exist at all. +// +// UPSTREAM CHAIN (ported FROM, cited per the ground-every-impl rule; the +// checkpoint spelling our loader accepts is BOTH compressed-tensors and +// ModelOpt, and upstream both land on the same apply): +// * scheme (compressed-tensors) vllm/model_executor/layers/quantization/ +// compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py:60,201-207 +// (`CompressedTensorsW8A8Fp8.apply_weights` -> `self.fp8_linear`) +// * scheme (ModelOpt) vllm/model_executor/layers/quantization/ +// modelopt.py:444,531-537 (`ModelOptFp8LinearMethod.apply` -> the SAME +// `self.fp8_linear.apply_weights`) +// * generic fp8 linear vllm/model_executor/layers/quantization/ +// fp8.py:267,446 (`Fp8LinearMethod`) +// * static per-tensor act quant vllm/model_executor/layers/quantization/utils/ +// quant_utils.py:124 (`kFp8StaticTensorSym`), which modelopt.py:511-512 +// hands to `init_fp8_linear_kernel` — our vt::QuantFp8Static +// * per-tensor scaled epilogue the folded `alpha = input_scale * +// weight_scale` scalar, mirroring vLLM's per-tensor ScaledEpilogue +// The local primitives this drives are vt::QuantFp8Static + vt::MatmulFp8CublasLt +// (default) / vt::MatmulFp8Cutlass (VT_DENSE_CUBLASLT_FP8=0), both realized in +// src/vt/cuda/cuda_matmul_fp8_cutlass.cu and (for the CPU reference arms of +// kQuantFp8Static / kMatmulFp8Cutlass) src/vt/cpu — see #468/#842. +// +// TEMPLATED ON Dev/DBuf ON PURPOSE — this is what makes the extraction REAL. +// `qwen3_5.cpp` carries its own anonymous-namespace `Dev`/`DBuf` (see the +// KNOWN-DUPLICATION note in dense_nvfp4_gemm.h: unifying the device-glue +// families is a separate, gate-model-touching refactor). A non-template header +// would therefore have had to be *copied* into qwen3_5.cpp rather than called by +// it — a seam that is dead alongside the production path proves nothing, which +// is precisely the failure #940 exists to prevent. Templating on the two glue +// types instead lets qwen3_5.cpp instantiate the ONE body with ITS types +// (generating byte-for-byte the code it had) while the shared layer +// (vllm::layers, dense_attn::Dev/DBuf) instantiates the SAME body with the +// shared ones. One definition, two instantiations, no copy. +// +// Both glue families satisfy the same tiny contract: `DevT` is `{Backend& b; +// Queue& q;}` and `DBufT` is constructible as `DBufT(d, dtype, shape)` with a +// `.t()` tensor view — dense_device_glue.h:42-56,108-130 and +// qwen3_5.cpp's copies of the same. +#pragma once + +#include +#include +#include + +#include "vllm/model_executor/models/dense_device_glue.h" // Dev/DBuf/MakeTensor +#include "vllm/model_executor/models/qwen3_5_weights.h" // Fp8Weight +#include "vt/backend.h" +#include "vt/dtype.h" // VT_CHECK +#include "vt/ops.h" + +namespace vllm { +namespace dense_fp8 { + +using vt::Backend; +using vt::DType; +using vt::Tensor; + +// cuBLASLt FP8 dense GEMM toggle (VT_DENSE_CUBLASLT_FP8, DEFAULT ON when the fp8 +// weights are resident). Routes the fp8 dense projections through vt:: +// MatmulFp8CublasLt (cuBLASLt e4m3 — the native equivalent of vLLM's measured- +// FASTER nvjet_sm121_qqtst fp8 kernels) instead of vt::MatmulFp8Cutlass (our +// cutlass sm120 fp8 GEMM, measured NEUTRAL vs bf16 at M=64/sm_121a). The +// activation quant + fp8-resident weight are IDENTICAL for both — only the GEMM +// backend differs, so both are the same fp8 W8A8 math (vLLM's scheme). +// VT_DENSE_CUBLASLT_FP8=0 restores the cutlass fp8 GEMM (the previous, validated +// path) for the parent's authoritative A/B. +inline bool DenseCublasLtFp8Enabled() { + static const bool on = [] { + const char* e = std::getenv("VT_DENSE_CUBLASLT_FP8"); + return !(e != nullptr && e[0] == '0'); + }(); + return on; +} + +// Device-resident view over an Fp8Weight's raw fp8 [N,K] bytes, uploaded ONCE +// (lazily) and reused across every forward step (mirror ResidentNvfp4). The +// shared_ptr in the (const) weight owns the device buffer for the model lifetime. +template +inline Tensor ResidentFp8(DevT d, const Fp8Weight& w) { + if (!w.d_packed) { + const size_t pb = w.packed.bytes.size(); + void* p = d.b.Alloc(pb); + d.b.Copy(d.q, p, w.packed.bytes.data(), pb); + Backend* bk = &d.b; + w.d_packed = std::shared_ptr(p, [bk](void* q) { bk->Free(q); }); + } + return dense_attn::MakeTensor(w.d_packed.get(), DType::kI8, d.q.device, + {w.n, w.k}); +} + +// y[M,N] = x[M,K] (bf16/f32 device) @ dequant(w).T via a per-tensor W8A8 fp8 +// GEMM: static per-tensor activation quant (vt::QuantFp8Static with the +// checkpoint input_scale) then an fp8 GEMM with the folded alpha +// (= input_scale·weight_scale). By DEFAULT the GEMM is cuBLASLt fp8 (vt:: +// MatmulFp8CublasLt — mirrors vLLM's nvjet_qqtst fp8 dense); VT_DENSE_CUBLASLT_ +// FP8=0 selects the cutlass sm120 fp8 GEMM (vt::MatmulFp8Cutlass). out dtype f32 +// (q/k/v, in_proj_qkv/z sinks) or bf16 (o/out_proj residual sinks). CUDA-only +// (the 35B W8A8 path is CUDA-resident — fp8 fields are populated by DEFAULT on +// the CUDA+cutlass load, VT_DENSE_NATIVE). +// +// The CUDA-only refusal is UNCHANGED by the extraction and is deliberately +// keyed on `kMatmulFp8CublasLt`, which is registered for kCUDA only — a +// "cuBLASLt" kernel on the host would be a lie in the name. That is what the +// pinned expectation at tests/vt/test_ops_fp8_cpu.cpp:445-453 records, and it is +// why the CPU registrations of kQuantFp8Static / kMatmulFp8Cutlass (#468/#842) +// do NOT by themselves make this entry point run on a host queue. Nothing here +// widens it; see the header's issue for the residual gap. +template +inline DBufT MatmulFp8CutlassD(DevT d, const Tensor& x, const Fp8Weight& w, + DType out_dtype) { + const int64_t M = x.shape[0], K = x.shape[1], N = w.n; + VT_CHECK(vt::OpRegistered(vt::OpId::kMatmulFp8CublasLt, d.q.device.type), + "MatmulFp8CutlassD: the fp8 W8A8 path is CUDA-only"); + DBufT a_fp8(d, DType::kI8, {M, K}); + vt::QuantFp8Static(d.q, a_fp8.t(), x, w.input_scale); + Tensor wdev = ResidentFp8(d, w); + DBufT dout(d, out_dtype, {M, N}); + if (DenseCublasLtFp8Enabled()) + vt::MatmulFp8CublasLt(d.q, dout.t(), a_fp8.t(), wdev, w.alpha); + else + vt::MatmulFp8Cutlass(d.q, dout.t(), a_fp8.t(), wdev, w.alpha); + return dout; +} + +// Pre-quantized fp8 analog of MatmulFp8CutlassD: the activation is ALREADY the +// static-quant fp8 [M,K] (produced ONCE — either by RmsNormQuantFp8 or a shared +// quant — and fed to every projection reading it), so this SKIPS the internal +// QuantFp8Static and runs only the fp8 GEMM. The fp8 counterpart of +// MatmulNvfp4Fp4DirectD; each GEMM still applies its own folded alpha (= shared +// input_scale · this projection's weight_scale), so the result is identical to +// MatmulFp8CutlassD(x) when a_fp8 == QuantFp8Static(x, w.input_scale). +// +// Upstream this is the `x: torch.Tensor | QuantizedActivation` overload of the +// same apply (compressed_tensors_w8a8_fp8.py:201-207) — an activation that a +// preceding fused epilogue already quantized is handed to the GEMM as-is. +template +inline DBufT MatmulFp8CutlassPreQuantD(DevT d, const Tensor& a_fp8, + const Fp8Weight& w, DType out_dtype) { + const int64_t M = a_fp8.shape[0], N = w.n; + VT_CHECK(vt::OpRegistered(vt::OpId::kMatmulFp8CublasLt, d.q.device.type), + "MatmulFp8CutlassPreQuantD: the fp8 W8A8 path is CUDA-only"); + Tensor wdev = ResidentFp8(d, w); + DBufT dout(d, out_dtype, {M, N}); + if (DenseCublasLtFp8Enabled()) + vt::MatmulFp8CublasLt(d.q, dout.t(), a_fp8, wdev, w.alpha); + else + vt::MatmulFp8Cutlass(d.q, dout.t(), a_fp8, wdev, w.alpha); + return dout; +} + +} // namespace dense_fp8 +} // namespace vllm diff --git a/src/vllm/model_executor/models/qwen3_5.cpp b/src/vllm/model_executor/models/qwen3_5.cpp index a81f5d125..94b673a1e 100644 --- a/src/vllm/model_executor/models/qwen3_5.cpp +++ b/src/vllm/model_executor/models/qwen3_5.cpp @@ -35,6 +35,7 @@ #include #include +#include "vllm/model_executor/models/dense_fp8_gemm.h" // dense_fp8:: FP8 W8A8 seam (#940) #include "vllm/model_executor/models/dense_nvfp4_gemm.h" // dense_nvfp4::MarlinDenseEnabled #include "vllm/model_executor/model_loader/nvfp4_dequant.h" #include "vt/backend.h" @@ -1452,79 +1453,27 @@ const OwnedTensor& DenseLmHead(const Qwen3_5DenseWeights& weights) { return weights.tied_lm_head ? weights.embed_tokens : weights.lm_head; } -// Device-resident view over an Fp8Weight's raw fp8 [N,K] bytes, uploaded ONCE -// (lazily) and reused across every forward step (mirror ResidentNvfp4). The -// shared_ptr in the (const) weight owns the device buffer for the model lifetime. -Tensor ResidentFp8(Dev d, const Fp8Weight& w) { - if (!w.d_packed) { - const size_t pb = w.packed.bytes.size(); - void* p = d.b.Alloc(pb); - d.b.Copy(d.q, p, w.packed.bytes.data(), pb); - Backend* bk = &d.b; - w.d_packed = std::shared_ptr(p, [bk](void* q) { bk->Free(q); }); - } - return MakeTensor(w.d_packed.get(), DType::kI8, d.q.device, {w.n, w.k}); -} +// --- FP8 W8A8 dense seam (issue #940) --------------------------------------- +// `ResidentFp8`, `DenseCublasLtFp8Enabled`, `MatmulFp8CutlassD` and +// `MatmulFp8CutlassPreQuantD` USED to be defined right here, in this anonymous +// namespace, which meant a second model could not reach them at all — the exact +// "hand-roll a parallel path" trap AGENTS.md §"Shared seams" forbids. Their ONE +// definition now lives in dense_fp8_gemm.h (the FP8 sibling of +// dense_nvfp4_gemm.h) and the three names below are pure type adapters: this +// file keeps its own anonymous-namespace `Dev`/`DBuf` (the KNOWN DUPLICATION +// dense_nvfp4_gemm.h records), so it instantiates the shared template with THOSE +// types and the generated code is what it was. No logic lives here; a change to +// the seam changes this model's arithmetic, which is what makes the extraction +// provable rather than decorative. +using dense_fp8::DenseCublasLtFp8Enabled; -// cuBLASLt FP8 dense GEMM toggle (VT_DENSE_CUBLASLT_FP8, DEFAULT ON when the fp8 -// weights are resident). Routes the fp8 dense projections through vt:: -// MatmulFp8CublasLt (cuBLASLt e4m3 — the native equivalent of vLLM's measured- -// FASTER nvjet_sm121_qqtst fp8 kernels) instead of vt::MatmulFp8Cutlass (our -// cutlass sm120 fp8 GEMM, measured NEUTRAL vs bf16 at M=64/sm_121a). The -// activation quant + fp8-resident weight are IDENTICAL for both — only the GEMM -// backend differs, so both are the same fp8 W8A8 math (vLLM's scheme). -// VT_DENSE_CUBLASLT_FP8=0 restores the cutlass fp8 GEMM (the previous, validated -// path) for the parent's authoritative A/B. -bool DenseCublasLtFp8Enabled() { - static const bool on = [] { - const char* e = std::getenv("VT_DENSE_CUBLASLT_FP8"); - return !(e != nullptr && e[0] == '0'); - }(); - return on; -} - -// y[M,N] = x[M,K] (bf16/f32 device) @ dequant(w).T via a per-tensor W8A8 fp8 -// GEMM: static per-tensor activation quant (vt::QuantFp8Static with the -// checkpoint input_scale) then an fp8 GEMM with the folded alpha -// (= input_scale·weight_scale). By DEFAULT the GEMM is cuBLASLt fp8 (vt:: -// MatmulFp8CublasLt — mirrors vLLM's nvjet_qqtst fp8 dense); VT_DENSE_CUBLASLT_ -// FP8=0 selects the cutlass sm120 fp8 GEMM (vt::MatmulFp8Cutlass). out dtype f32 -// (q/k/v, in_proj_qkv/z sinks) or bf16 (o/out_proj residual sinks). CUDA-only -// (the 35B W8A8 path is CUDA-resident — fp8 fields are populated by DEFAULT on -// the CUDA+cutlass load, VT_DENSE_NATIVE). DBuf MatmulFp8CutlassD(Dev d, const Tensor& x, const Fp8Weight& w, DType out_dtype) { - const int64_t M = x.shape[0], K = x.shape[1], N = w.n; - VT_CHECK(vt::OpRegistered(vt::OpId::kMatmulFp8CublasLt, d.q.device.type), - "MatmulFp8CutlassD: the fp8 W8A8 path is CUDA-only"); - DBuf a_fp8(d, DType::kI8, {M, K}); - vt::QuantFp8Static(d.q, a_fp8.t(), x, w.input_scale); - Tensor wdev = ResidentFp8(d, w); - DBuf dout(d, out_dtype, {M, N}); - if (DenseCublasLtFp8Enabled()) - vt::MatmulFp8CublasLt(d.q, dout.t(), a_fp8.t(), wdev, w.alpha); - else - vt::MatmulFp8Cutlass(d.q, dout.t(), a_fp8.t(), wdev, w.alpha); - return dout; + return dense_fp8::MatmulFp8CutlassD(d, x, w, out_dtype); } -// Pre-quantized fp8 analog of MatmulFp8CutlassD: the activation is ALREADY the -// static-quant fp8 [M,K] (produced ONCE — either by RmsNormQuantFp8 or a shared -// quant — and fed to every projection reading it), so this SKIPS the internal -// QuantFp8Static and runs only the fp8 GEMM. The fp8 counterpart of -// MatmulNvfp4Fp4DirectD; each GEMM still applies its own folded alpha (= shared -// input_scale · this projection's weight_scale), so the result is identical to -// MatmulFp8CutlassD(x) when a_fp8 == QuantFp8Static(x, w.input_scale). -DBuf MatmulFp8CutlassPreQuantD(Dev d, const Tensor& a_fp8, const Fp8Weight& w, DType out_dtype) { - const int64_t M = a_fp8.shape[0], N = w.n; - VT_CHECK(vt::OpRegistered(vt::OpId::kMatmulFp8CublasLt, d.q.device.type), - "MatmulFp8CutlassPreQuantD: the fp8 W8A8 path is CUDA-only"); - Tensor wdev = ResidentFp8(d, w); - DBuf dout(d, out_dtype, {M, N}); - if (DenseCublasLtFp8Enabled()) - vt::MatmulFp8CublasLt(d.q, dout.t(), a_fp8, wdev, w.alpha); - else - vt::MatmulFp8Cutlass(d.q, dout.t(), a_fp8, wdev, w.alpha); - return dout; +DBuf MatmulFp8CutlassPreQuantD(Dev d, const Tensor& a_fp8, const Fp8Weight& w, + DType out_dtype) { + return dense_fp8::MatmulFp8CutlassPreQuantD(d, a_fp8, w, out_dtype); } // --- Merged FP8 QKVParallelLinear (VT_FP8_MERGED_QKV, opt-in). The FP8 (W8A8) diff --git a/tests/vllm/model_executor/layers/test_linear_method.cpp b/tests/vllm/model_executor/layers/test_linear_method.cpp index df3313948..315b49c2e 100644 --- a/tests/vllm/model_executor/layers/test_linear_method.cpp +++ b/tests/vllm/model_executor/layers/test_linear_method.cpp @@ -16,6 +16,7 @@ #include #include "vllm/model_executor/layers/quantization/compressed_tensors/schemes/nvfp4.h" +#include "vllm/model_executor/layers/quantization/fp8.h" #include "vllm/model_executor/model_loader/mxfp4_dequant.h" #include "vt/backend.h" #include "vt/dtype.h" @@ -454,3 +455,107 @@ TEST_CASE("linear_method: bf16 UnquantizedLinearMethod apply == reference Matmul } } } + +// =========================================================================== +// FP8 W8A8 (per-tensor) — the seam extracted from qwen3_5.cpp by #940. +// +// Same two questions this file already asks of NVFP4, asked of FP8: does the +// factory pick the scheme ONCE from the checkpoint (get_quant_method), and is +// the extracted compute REACHED rather than sitting dead beside the model. The +// numeric arm runs on dgx (the fp8 GEMM is CUDA-only, see below); what a host +// build can prove is selection and the inherited device refusal, and both of +// those go red if the guard or the wiring in dense_fp8_gemm.h moves. +namespace { + +// A minimal non-empty per-tensor FP8 W8A8 weight in the shape `LoadFp8Raw` +// produces (qwen3_5_weights.cpp:423): raw e4m3fn [N,K] bytes, per-tensor +// weight_scale + input_scale, folded alpha. Powers of two, so nothing here +// rounds. Bytes stay below 0x7E — 0x7F/0xFF are NaN in e4m3fn. +vllm::Fp8Weight MakeFp8W8A8(int64_t N, int64_t K, uint32_t seed) { + vllm::Fp8Weight w; + w.n = N; + w.k = K; + w.weight_scale = 0.00390625F; + w.input_scale = 0.0078125F; + w.alpha = w.input_scale * w.weight_scale; + w.packed.dtype = DType::kI8; + w.packed.rank = 2; + w.packed.shape[0] = N; + w.packed.shape[1] = K; + w.packed.bytes.resize(static_cast(N * K)); + uint32_t s = seed; + auto* bytes = reinterpret_cast(w.packed.bytes.data()); + for (int64_t i = 0; i < N * K; ++i) { + s = s * 1664525u + 1013904223u; + bytes[static_cast(i)] = static_cast((s >> 16) % 0x7EU); + } + return w; +} + +} // namespace + +TEST_CASE("linear_method: factory selects bf16 vs fp8-w8a8 by weight presence") { + OwnedTensor bf16 = MakeBf16({4, 16}, 3); + vllm::Fp8Weight empty_fp8; // Empty() == true + vllm::Fp8Weight fp8 = MakeFp8W8A8(4, 16, 31); + REQUIRE(empty_fp8.Empty()); + REQUIRE_FALSE(fp8.Empty()); + + // get_quant_method analogue: a bf16 checkpoint => UnquantizedLinearMethod. + auto m_bf16 = layers::MakeLinearMethod(bf16, empty_fp8); + CHECK(std::string(m_bf16->Name()) == "bf16-unquantized"); + + // An fp8-resident checkpoint => the per-tensor W8A8 method, chosen ONCE here. + auto m_fp8 = layers::MakeLinearMethod(bf16, fp8); + CHECK(std::string(m_fp8->Name()) == "fp8-w8a8-per-tensor"); + + // The two overloads coexist: same call spelling, different weight type, and + // the NVFP4 one is unaffected by the FP8 one being in scope. + auto m_fp4 = layers::MakeLinearMethod(bf16, MakeNvfp4W4A16(4, 16)); + CHECK(std::string(m_fp4->Name()) == "compressed-tensors-nvfp4-w4a16"); +} + +// REACHABILITY of the extracted seam from the POLICY layer. Both arms of +// dense_fp8_gemm.h run their `kMatmulFp8CublasLt`-registered guard first, and +// that op is CUDA-only (tests/vt/test_ops_fp8_cpu.cpp:445-453 pins that), so on +// a host queue each must THROW that exact refusal. This is not a test of the +// refusal for its own sake: it is the assertion that `Fp8W8A8LinearMethod:: +// Apply` / `ApplyPreQuantized` actually execute the shared template bodies. +// Deleting either VT_CHECK from dense_fp8_gemm.h, or pointing the method at +// something else, turns this case red. +TEST_CASE("linear_method: the fp8 w8a8 method reaches the shared seam in both arms") { + const int64_t M = 2, K = 16, N = 4; + OwnedTensor bf16 = MakeBf16({N, K}, 11); + OwnedTensor xw = MakeBf16({M, K}, 13); + vllm::Fp8Weight fp8 = MakeFp8W8A8(N, K, 17); + + vt::Queue q{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + vt::Backend& b = vt::GetBackend(vt::DeviceType::kCPU); + vllm::dense_attn::Dev d{b, q}; + vllm::dense_attn::DBuf x(d, DType::kBF16, {M, K}, xw.bytes.data()); + + auto method = layers::MakeLinearMethod(bf16, fp8); + REQUIRE(std::string(method->Name()) == "fp8-w8a8-per-tensor"); + + CHECK_THROWS_WITH_AS(method->Apply(d, x.t(), DType::kF32), + doctest::Contains("MatmulFp8CutlassD: the fp8 W8A8 path is CUDA-only"), + std::runtime_error); + + // The pre-quantized arm (the `QuantizedActivation` overload) reaches its own + // entry point, not the plain one — the message names which. + const auto* fp8_method = + dynamic_cast(method.get()); + REQUIRE(fp8_method != nullptr); + vllm::dense_attn::DBuf a_fp8(d, DType::kI8, {M, K}); + CHECK_THROWS_WITH_AS( + fp8_method->ApplyPreQuantized(d, a_fp8.t(), DType::kBF16), + doctest::Contains("MatmulFp8CutlassPreQuantD: the fp8 W8A8 path is CUDA-only"), + std::runtime_error); + + // The refusal is the OP TABLE's answer, not a hardcoded device test: the two + // CPU reference arms #468/#842 registered DO resolve here, and the model-layer + // predicate still names the cuBLASLt op. That is the residual gap, pinned. + CHECK(vt::OpRegistered(vt::OpId::kQuantFp8Static, vt::DeviceType::kCPU)); + CHECK(vt::OpRegistered(vt::OpId::kMatmulFp8Cutlass, vt::DeviceType::kCPU)); + CHECK_FALSE(vt::OpRegistered(vt::OpId::kMatmulFp8CublasLt, vt::DeviceType::kCPU)); +}