feat(#468): register the static FP8 W8A8 path on the CPU backend — the fp8 seam becomes gateable without a GPU by localai-bot · Pull Request #826 · mudler/vllm.cpp

localai-bot · 2026-08-14T22:01:54Z

Issue: #468
Spec: .agents/specs/vt-fp8-w8a8-cpu-arm.md

What and why

#468's "what done would look like" option 1, verbatim: "a CPU registration for the fp8 matmul sufficient to exercise the wiring (which has value well beyond this lever — it would make the whole fp8 model path CPU-testable)."

kQuantFp8Static, kMatmulFp8Cutlass and kMatmulFp8CublasLt had no CPU registration, so the fp8 W8A8 path was structurally untestable without a GPU. qwen3_5.cpp:1497 refuses by name: "the fp8 W8A8 path is CUDA-only". This registers the static per-tensor path on DeviceType::kCPU and gates it.

Upstream grounding (pin `5559679229bc961848b121ccdeaa8fa5d79bec98`)

modelopt.py:2527 — if quant_algo == "FP8": return ModelOptFp8LinearMethod(...), unconditional; exclusions checked first at :2517-2521
modelopt.py:510-517 — static, hard-coded: init_fp8_linear_kernel(activation_quant_key=kFp8StaticTensorSym, weight_quant_key=kFp8StaticTensorSym, ...)
modelopt.py:528 — input_scale collapses to a scalar via .max()
csrc/quantization/w8a8/fp8/common.cuh:58-77 — x = val * scale under is_scale_inverted=true, fmaxf(-448, fminf(x, 448)), hardware RNE
csrc/libtorch_stable/quantization/w8a8/fp8/common.cu:31 — the reciprocal is formed once, outside the elementwise math
:204-210 — scale.numel()==1 ⇒ one group over the whole tensor

There is no dynamic fallback in this class. Per-token dynamic is a different class, ModelOptFp8PcPtLinearMethod (modelopt.py:539, docstring :546), reached by a different quant_algo.

The scale is applied as `x * (1/s)`, not `x / s`

include/vt/ops.h:1457 and cuda_matmul_fp8_cutlass.cu:318 documented the math as x / input_scale while the code has always computed inv = 1.0f/input_scale; x*inv — which is what upstream does. The code was right and the comment was wrong. Repaired here, in-flow under #468, because the inverse mistake — someone "correcting" the code to match the comment — is a silent 1-ulp divergence on a default-ON 35B path, and near an e4m3 tie one ulp changes the emitted byte.

The gate

tests/vt/test_ops_fp8_cpu.cpp.

G1 — bitwise, zero tolerance. CPU QuantFp8Static against an independently written reference. The reference enumerates all 128 finite e4m3fn magnitudes, decodes each to an exact double from the field layout, and picks nearest with an even-significand tie-break by scanning — a different algorithm from the tree's F32ToFp8 (frexp + nearbyint) and from vllm::F32ToF8E4M3, so agreement is evidence rather than a tautology dressed as a gate. Compares bytes, not Approx: doctest's Approx carries a scale term defaulting to 1.0 and therefore a ~1.19e-5 absolute floor, meaningless for a byte compare.
G2 — CPU vs CUDA, bitwise on identical input, CUDA-gated.
G3 — the GEMM against a double reference that itself reproduces upstream's lossy pipeline (clamp → e4m3 RNE → dequant) before accumulating. A reference computing exact arithmetic would make a wrong implementation look better than upstream and pass.

Preserved deviation

Upstream applies scale_a and scale_b as two epilogue scalars (scaled_mm/cutlass.py:265-267); we fold alpha = input_scale * weight_scale into one f32 (include/vt/ops.h:1501-1511). Unchanged here, and recorded as a ported deviation.

Notes for the reviewer

CUDA behaviour is unchanged; only comments were touched there.
include/vt/fused_recipe.h and tests/vt/test_fused_chain_additivity.cpp are touched — the implementer should state in review whether registering the CPU op legitimately changes what the fused-chain recipe admits, or whether that belongs in a separate change.
The CPU matmul is a correctness reference, not a performance path, and no speed claim is made.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

…on CPU Issue #468 records that the model-layer wiring of VT_GDN_FP8_ALPHA_IN_CONV has no test at any tier, and names the structural reason: the fp8 matmul registers only on CUDA, so a CPU-tier test cannot reach the path at all and the CUDA tier needs a GPU. Its "What done would look like" option 1 is a CPU registration for the fp8 matmul -- valuable well beyond that lever, because it makes the whole fp8 model path CPU-testable. This is the spec for option 1, committed before any implementation. It scopes two CPU registrations (kQuantFp8Static, kMatmulFp8Cutlass), the comment repairs they create the need for, and the consequential change to the fusion additivity test, whose three fp8-terminal recipes currently assert the full composite THROWS on CPU. That assertion is a checker's claim, so changing it takes a spec plus red-before and green-after evidence, which is what this file declares and what the implementation commit carries. Two things are stated up front rather than discovered later. First the residual gap: the model-layer entry points key on kMatmulFp8CublasLt, which deliberately stays CUDA-only, so this row makes the OP seam CPU-reachable and does not claim to make MatmulFp8CutlassD execute on a CPU queue. Second the stale comment: the op contract and the CUDA kernel both describe the quant math as x / input_scale while the code multiplies by the reciprocal, which is what upstream ships. The code is right and the comment is wrong, and left alone it invites someone to "correct" a default-ON 35B path into a divide. Also repoints the #468 roadmap row at this spec and its work branch. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5[1m] [claude-code]

The fp8 matmul registered only on CUDA, so the fp8 model path was structurally untestable on the machine most of this work happens on: a CPU-tier test could not reach it and the CUDA tier needs a GPU. That is the reason #468 gives for the wiring shipping uncovered, and it is what this closes at the op seam. Two CPU registrations, per .agents/specs/vt-fp8-w8a8-cpu-arm.md: kQuantFp8Static -- the scale application, over the fp8-e4m3fn codec already in cpu_ops.cpp. It multiplies by the RECIPROCAL, formed once outside the loop, which is what upstream ships (common.cuh:62 `x = val * scale` with the inverse formed at common.cu:31) and what our CUDA kernel does. Not a divide: the two differ by up to one f32 ulp before the fp8 round, and near an e4m3 tie that ulp changes the emitted byte. kMatmulFp8Cutlass -- a CORRECTNESS REFERENCE. f32 accumulate, one folded alpha, naive triple loop. It makes no speed claim, nothing routes a production model through it, and it says so in the code. No CUDA behavior changes. kMatmulFp8CublasLt stays CUDA-only, so the MODEL-layer predicate still refuses on CPU; the spec records that residual gap and the new test pins it rather than letting it be assumed closed. Comment repairs, in-flow under #468. Three comments said "CUDA only" and become false here. A fourth was ALREADY false and matters most: include/vt/ops.h and cuda_matmul_fp8_cutlass.cu both describe the quant as `x / input_scale` while the code three lines below multiplies by the reciprocal. The code is right and the comment is wrong; left alone it invites someone to "correct" the code on a default-ON 35B path. Both now carry the upstream anchor and say not to. Gate: tests/vt/test_ops_fp8_cpu.cpp. G1 is BITWISE with zero tolerance against an independently written reference -- it enumerates all 128 finite e4m3fn magnitudes, decodes each from the field layout, and picks nearest-even by scanning, which is a different algorithm from both F32ToFp8 (frexp/nearbyint) and vllm::F32ToF8E4M3, so agreement is evidence rather than tautology. Not expressed as doctest Approx, whose scale term defaults to 1.0 and would give a byte compare a ~1.19e-5 floor. G3 checks the GEMM against a double reference that reproduces upstream's LOSSY pipeline -- clamp, e4m3 RNE, dequant -- because an exact-arithmetic reference would let a wrong implementation look better than upstream and pass. G2 (CPU vs CUDA, byte for byte) is committed and CUDA-gated; it is PENDING, not skipped, on this GPU-less host and says so in its output. test_fused_chain_additivity asserted, for all three fp8-terminal recipes, that the full Tier-0 composite THROWS on CPU. Registering the terminal makes that false. The assertions are not deleted to go green -- each is replaced by the strictly stronger byte-exact check the registration makes available, the prefix check is kept beside it so a future tail regression is localised, and cpu_full flips to true for those three rows. Assertions go 19 -> 25. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5[1m] [claude-code]

The mutation series killed every mutant, but M2 (divide instead of reciprocal-multiply) died by 2 assertions, and measuring WHY showed the margin was thinner than the green suggested. `x/s` and `x*(1/s)` agree on almost every input. Over 20000 random values in [-2,2] they never disagree at ANY scale tried. The difference is only visible where an input lands on an e4m3 tie after scaling, which is what G1's constructed tie population exists for -- and even then it is scale-dependent: over that population 10 of 18 candidate scales expose it at all, and of the five G1 shipped with, only 0.0092 did, at 24 of 209 words. So the assertion that keeps the reciprocal form -- the one the repaired comment points at, guarding a default-ON 35B path -- was one scale-list edit away from being silently disarmed. 0.13 (78/209) and 0.77 (82/209) are the strongest detectors measured and are added with that measurement written down beside them, so a later reader prunes the list knowing what it costs. Assertions 44 -> 56, still 4 cases. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5[1m] [claude-code]

G1 and G3 PASS; G2 (CPU vs CUDA, byte for byte) is PENDING on this host because it has no GPU, and is recorded as owed rather than counted. Beyond the mutation table, three things the series measured that the pass/fail column does not carry: M1 moves ~99.7% of bytes at every scale but 1.0 (where ignoring the scale is correctly a no-op); M2 is nearly invisible and dies only because the population contains constructed exact ties AND the scale list contains a detector; and F32ToFp8 saturates in two places, so a reviewer mutating only the obvious guard would wrongly read the gate as blind. Also records that M1/M2/M5a first failed to BUILD on -Werror rather than to assert, and were re-expressed before being counted as verdicts. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5[1m] [claude-code]

localai-bot · 2026-08-14T23:19:27Z

Superseded by #842, opened from row/VT-FP8-W8A8-CPU-ARM-V2. The tree is byte-identical; only commit messages changed.

Reason: scripts/check-commit-trailers.py rejects every commit on this branch —

[attribution] malformed Assisted-by value 'AGENT:claude-opus-5[1m] [claude-code]'

The ASSISTED_BY regex requires a space before a bracket group, so claude-opus-5[1m] is not a valid model token. It is repaired to AGENT:claude-opus-5 [1m] [claude-code] on the new branch.

Repairing commit messages rewrites history, which makes the push non-fast-forward. AGENTS.md prohibits force-pushing and names --force-with-lease explicitly, so the repaired history went to a new branch rather than over this one. This branch is left untouched as a record, not a work surface.

Verification that nothing but the messages moved: HEAD^{tree} is unchanged, and git diff ca7358f59fca20953dfdcfa8b26f43b31f20f9f8 810182ea159d2671ef0dead225ce481318c06574 --stat is empty.

Closing in favour of #842.

…PU backend (#468) (#842) ## Row `VT-FP8-W8A8-CPU-ARM`, owned by roadmap row `PERF-27B-LMHEAD-FP4`. One row per PR. Issue: #468. The issue stays OPEN. See "Honest gaps". ## Before starting - **Issue/PR search and existing claim:** #468 is open and tracks this work. I searched open PRs for overlap on `src/vt/cpu/cpu_ops.cpp`, `include/vt/ops.h` and `tests/vt/test_fused_chain_additivity.cpp`. Only PR #437 (LTX-2.5) touches `include/vt/ops.h`, and it adds enum entries. It does not touch the fp8 contract text. No other claim existed on #468. - **Pull request shape:** one PR carries the spec and its code, per `POLICY-SINGLE-PR-AND-STYLE`. The spec commit precedes the implementation commit on this branch. - **Roadmap row:** the #468 row in `.agents/roadmap_v1.md` now points at `.agents/specs/vt-fp8-w8a8-cpu-arm.md` and at branch `row/VT-FP8-W8A8-CPU-ARM`. - **Anchors inspected at HEAD, not copied from a record:** `src/vt/cpu/cpu_ops.cpp` `F32ToFp8` and `kFp8Max`; `include/vt/ops.h` `QuantFp8Static` and `MatmulFp8Cutlass` contracts; `src/vt/cuda/cuda_matmul_fp8_cutlass.cu` quant kernel and registrar; `src/vllm/model_executor/models/qwen3_5.cpp` `MatmulFp8CutlassD`; `tests/vt/test_fused_chain_additivity.cpp` catalog and its three `CHECK_THROWS`. Upstream at the pinned oracle `5559679229bc961848b121ccdeaa8fa5d79bec98`: `csrc/quantization/w8a8/fp8/common.cuh:58-77`, `csrc/libtorch_stable/quantization/w8a8/fp8/common.cu:31` and `:204-210`, `vllm/model_executor/layers/quantization/modelopt.py:510-513`, `:528` and `:2527`. Every line number above was re-derived at HEAD. Base SHA `b1cd4d8f6bb7ec5f0bd923a75dcc140becc7fdd8`. This branch merges `origin/main` three times, each time at an immutable SHA: `5da1d7f2fa89472c47860da9b31f8959e809acc0`, `3ce5a1dc1b0f1baaeab7598fbf3835abe6d53c2f` and `2f2bce926d673111c9816275906d4df8aa20effe`. **The third merge resolves review finding F2, and it was not a text collision.** `git merge-tree --write-tree origin/main 810182e` exited 1 on `.agents/roadmap_v1.md`. The reason: main **relocated** the intake table out of `roadmap_v1.md` into `.agents/issue-index.md`, and `2f2bce926` carries no `#468` row in the roadmap at all. Reapplying my edit where I authored it would have resurrected a row main had deleted, which is the duplicate a move-plus-edit produces even when both sides merge cleanly. So the edit moved with the table: `roadmap_v1.md` takes the target wholesale and byte-identical, and the `#468` row is reapplied in `issue-index.md`. Verified four ways, because a clean auto-merge can silently interleave rows and still satisfy the obvious checks: 1. The diff against the target is **exactly one `-` and one `+` at the same line index**, file length unchanged. The reviewer states this check as "the target is a strict PREFIX of the result", which assumes the branch only appends; my edit modifies one row in place, so a prefix test cannot hold and would pass vacuously. This is the strictly stronger form, and it forbids an interleave, a reorder, an insertion and a deletion at once. 2. My `#468` row is byte-identical to the row I authored. 3. All **216** keys are byte-identical to the target except mine, and the key set is unchanged: none dropped, none added. 4. No key appears twice. `merge-tree` against `origin/main` now exits 0. ## What changed The static FP8 W8A8 path now registers on the CPU backend. `src/vt/cpu/cpu_ops.cpp` gains two kernels: `QuantFp8StaticKernel` registers `OpId::kQuantFp8Static`, and `MatmulFp8CutlassKernel` registers `OpId::kMatmulFp8Cutlass`. The quant kernel multiplies by the reciprocal of `input_scale` and forms that reciprocal once outside the loop, which mirrors upstream and mirrors our CUDA kernel. The matmul kernel is a correctness reference with f32 accumulation and one folded `alpha`. It makes no speed claim, and no production model routes through it. This is the **first half of option 1** of the two closures #468 names. It makes the fp8 OP seam reachable, and therefore gateable, without a GPU. It is **not** sufficient to exercise the `mixed_scale` wiring #468 set out to cover: the GEMM registered here is `kMatmulFp8Cutlass`, while the model-layer predicate keys on `kMatmulFp8CublasLt`, so that wiring stays unreachable. See "Honest gaps". ## Evidence - [x] `scripts/agent-preflight.sh` passes. Result: `All gates green`, 72 checks, exit 0. - [x] Tests that cover this change: `tests/vt/test_ops_fp8_cpu.cpp` (new, 4 cases, 56 assertions) and `tests/vt/test_fused_chain_additivity.cpp` (1 case, 25 assertions). - [x] Same-change doc obligations: none are owed. No shipped capability moved. Models still refuse the fp8 path on CPU, because the model layer keys on a different op id. See "Honest gaps". Only op-tier test reachability changed, and `AGENTS.md` states that editing `src/`, `include/` or `tests/` alone owes no public document. ### Full gate on the merge result ``` cmake -S . -B build -G Ninja -DVLLM_CPP_BUILD_TESTS=ON (CI's configuration) cmake --build build -j 6 BUILD_EXIT=0 1422 targets 0 warnings ctest --test-dir build -j 4 CTEST_EXIT=0 100% tests passed, 0 tests failed out of 481 scripts/agent-preflight.sh PREFLIGHT_EXIT=1 67 ok, 7 failed -- all 7 are BASELINE, see below check-commit-style / check-commit-trailers / check-issue-index-append-only OK (from merge base) ``` Build flags: `-Wall -Wextra -Werror`, no `CMAKE_BUILD_TYPE`, so `NDEBUG` is off and `assert` stays live. Disk free at finish: 59 GiB. **Read the earlier numbers in this PR's history as void.** Every full gate before this one ran `-DVLLM_CPP_BUILD_EXAMPLES=OFF -DVLLM_CPP_SERVER=OFF`, flags I chose while the box was at 100% disk and then failed to revisit. That configuration builds **925 targets and runs 463 tests**; CI's builds **1422 and runs 481**. The gap is not cosmetic: `test_minimax_music3_e2e_real` links `ApiServer` and cannot BUILD without the server, so the reduced gate reported `100% tests passed` over a target it had never compiled. Same class as F6, an instrument reporting on a state it was not given. Only the CI-configuration run above is binding. **The 7 remaining preflight failures are BASELINE, verified rather than assumed.** `check-release-binary-contract`, `check-release-workflow`, `check-test-registration`, `test_check_release_binary_contract`, `test_release_manifest`, `test_release_pipeline`, `test_check_test_registration`. Each was run in a clean detached worktree at `2f2bce926` itself and each fails there. This branch adds none and touches no release workflow, CI file, or test-registration surface. `issue-index append-only` also failed until I fixed it; it is now green and is the only one that was ever mine. ### G1, G2 and G3 | Gate | Result | |---|---| | G1. Quant is byte-identical to an independent reference. Zero tolerance. | PASS | | G2. CPU output equals CUDA output, byte for byte. | **PENDING. NOT RUN.** This host has no GPU. This gate is neither passed nor skipped. It is owed. | | G3. GEMM matches a `double` reference that reproduces the upstream lossy pipeline. | PASS | G1 builds its reference from the format, not from our code. It enumerates all 128 finite e4m3fn magnitudes, decodes each one from the field layout, and picks the nearest value with an even-significand tie break by scanning. `F32ToFp8` instead uses `frexp` and `std::nearbyint`. The two algorithms differ, so agreement is evidence and not a tautology. G1 compares bytes. It does not use `doctest::Approx`, whose `scale` term defaults to 1.0 and would put a floor near 1.19e-5 on a byte comparison. G3 quantizes through the same clamp and the same round-to-nearest-even that the hardware path uses, then dequantizes. An exact-arithmetic reference would let a wrong implementation sit closer to it than the correct implementation does. The tolerance bounds only the accumulation width, as `4 * K * FLT_EPSILON * alpha * sum|terms|`. ### Mutations Each mutation was applied alone to a restored tree. Each was rebuilt, run, and restored. The compiler exit status appears beside every result, because a mutation that fails to build reads as a passing test. The whole series was re-run on the post-merge tree with identical results. | # | Mutation | compile | `[doctest] test cases:` | `[doctest] assertions:` | `Status:` | |---|---|---|---|---|---| | M0 | both registrations removed. This is the RED-BEFORE. | `0` | `4 \| 0 passed \| 4 failed \| 0 skipped` | `6 \| 1 passed \| 5 failed` | `FAILURE!` | | M1 | `input_scale` ignored | `0` | `4 \| 3 passed \| 1 failed \| 0 skipped` | `56 \| 32 passed \| 24 failed` | `FAILURE!` | | M2 | divide instead of reciprocal multiply | `0` | `4 \| 3 passed \| 1 failed \| 0 skipped` | `56 \| 50 passed \| 6 failed` | `FAILURE!` | | M3 | saturation removed | `0` | `4 \| 3 passed \| 1 failed \| 0 skipped` | `56 \| 28 passed \| 28 failed` | `FAILURE!` | | M4 | round-to-nearest-even replaced by truncation | `0` | `4 \| 3 passed \| 1 failed \| 0 skipped` | `56 \| 28 passed \| 28 failed` | `FAILURE!` | | M5a | kernel ignores `alpha` | `0` | `4 \| 3 passed \| 1 failed \| 0 skipped` | `56 \| 53 passed \| 3 failed` | `FAILURE!` | | M5b | caller folds `weight_scale` only | `0` | `4 \| 3 passed \| 1 failed \| 0 skipped` | `56 \| 53 passed \| 3 failed` | `FAILURE!` | | restored | none | `0` | `4 \| 4 passed \| 0 failed \| 0 skipped` | `56 \| 56 passed \| 0 failed` | `SUCCESS!` | M0 reports 6 assertions and not 56. A changed case count is signal. This is why the series reads `Status:` next to `assertions:`. M1, M2 and M5a first failed to build rather than to assert. The compiler rejected the dead `input_scale`, `inv_scale` and `alpha` under `-Werror=unused-parameter` and `-Wunused-variable`. I re-expressed each mutation with an explicit `(void)` and re-ran it. Only rows with `compile_exit=0` are verdicts. ### M2 goes red, and the margin was itself a finding `x / s` and `x * (1/s)` agree on almost every input. Over 20000 random values in the range -2.0 to 2.0 the two forms never disagreed, at any of 14 scales tried. The difference appears only where a scaled input lands on an e4m3 tie, which is what G1's constructed tie population creates. Even there the effect depends on the scale. Over that population, 10 of 18 candidate scales expose the defect at all. Of the five scales G1 first shipped with, only 0.0092 exposed it, at 24 of 209 words. The mutant died by 2 assertions. The assertion that protects the reciprocal form was one scale-list edit away from silent disarming. I measured 0.13 at 78 of 209 words and 0.77 at 82 of 209 words, added both, and recorded those counts in the test beside the list. M2 now dies by 6 assertions across 3 scales. Two further measurements: M1 moves about 99.7% of output bytes at every scale except 1.0, where ignoring the scale is correctly a no-op, for 17229 of 21595 words overall. `F32ToFp8` also saturates in two places, an early `a >= kFp8Max` return and a late `exp_field > 15` guard, so a reviewer who mutates only the obvious guard would wrongly read the gate as blind. M3 removes both. ## Two files outside the original brief `tests/vt/test_fused_chain_additivity.cpp` changed by 117 lines, and `include/vt/fused_recipe.h` by 4. Both are forced consequences of the registration. `vt::FusedChainComposite` walks a recipe's opcodes and dispatches each step to the standalone `vt::` op on `q.device`. Three catalog recipes end in opcode `FOp::kQuantFp8`, which dispatches to `vt::QuantFp8Static`: `kRmsNormQuantFp8`, `kRmsNormGatedQuantFp8` and `kSiluMulQuantFp8`. Without a CPU registration that dispatch threw. The test asserted the throw with three `CHECK_THROWS` and marked those catalog rows `cpu_full = false`. The registration makes those three assertions false. The proof is a command and not an argument. Check out the base version of that test, then run it against the new registrations: ``` [doctest] test cases: 1 | 0 passed | 1 failed | 0 skipped [doctest] assertions: 21 | 18 passed | 3 failed | [doctest] Status: FAILURE! ``` Three assertions fail. They are exactly the three `CHECK_THROWS`. The other 18 pass unchanged. The change is therefore not scope creep and not a judgement call. I did not delete those assertions to reach green. I replaced each one with the stronger byte-exact check that the registration makes available, which compares the full composite against the standalone-op-sequence golden including the fp8 output. Each driver keeps its prefix check as well, so a future regression in the tail stays localised. The `cpu_full` flag flips to `true` for the three rows. Assertions go from 21 to 25. The `include/vt/fused_recipe.h` change is comment only. It sits inside the opcode table, on the line that described `kQuantFp8` as `CUDA-only`. That is a documented contract statement which this change falsifies. ## A comment that was already wrong `include/vt/ops.h` and `src/vt/cuda/cuda_matmul_fp8_cutlass.cu` both described the quant as `x / input_scale`. The code three lines below each comment multiplies by the reciprocal. The code is correct and the comment was wrong. Left in place, the comment invites a reader to "correct" a default-ON 35B path into a divide. Both comments now carry the upstream anchor and state the hazard. This repair is in flow under #468. ## Speed claims - [x] This PR makes NO speed claim. The new CPU matmul is a correctness reference and its comment says so. ## Honest gaps **G2 is PENDING. It has not run. Do not read this PR as having proved CPU/CUDA byte agreement.** This host has no CUDA device, so one of the three declared gates is unmeasured. The case is committed and CUDA gated. It prints its reason and still asserts the CPU registration, so it can never pass vacuously, but a vacuity guard is not the measurement. Running it needs `ctest -R test_ops_fp8_cpu` in the container build on `dgx.casa` (GB10/sm_121, `vllmcpp-build:gb10`) or on `192.168.68.23` (Thor/sm_110, `vllmcpp-build:aarch64`). The operator has offered to run it; I did not improvise a remote invocation. What is at stake if G2 later fails: G1 already proves the CPU quant matches an independent reference derived from the format, and the CUDA kernel is unchanged by this PR, so a G2 failure would indicate a pre-existing CPU/CUDA divergence rather than a regression introduced here. That is worth knowing either way, which is why the gate stays open rather than being dropped. **This PR does not make the model layer run fp8 on CPU.** `kMatmulFp8CublasLt` stays CUDA only, because a kernel named for cuBLASLt does not belong on the host. `MatmulFp8CutlassD` and `MatmulFp8CutlassPreQuantD` in `src/vllm/model_executor/models/qwen3_5.cpp` gate on that op id, so they still refuse on a CPU queue. The new test pins the refusal with `CHECK_FALSE(vt::OpRegistered(vt::OpId::kMatmulFp8CublasLt, DeviceType::kCPU))`, so the gap stays visible. #468 remains open for it. Option 2 of #468, the `mixed_scale` forwarding test, is untouched, and so is the parked lever. **`test_op_parity` did not reproduce the baseline red.** The task brief named it as base-inherited red under #755 and #672. It passed twice here: at base SHA `b1cd4d8f6` in 1.87 s, and again after the first merge in 4.39 s. My build is CPU only, with `VLLM_CPP_BUILD_EXAMPLES=OFF` and `VLLM_CPP_SERVER=OFF`. Either the red depends on configuration or load, or main had already repaired it by this base. I did not investigate, because it is outside this row. I record the discrepancy rather than resolve it silently. **`test_cpu_x86_llamacpp_floor` did red once.** The first preflight ran at load average 172 and the harness exited 4, `NO_QUIET_WINDOW`, instead of 2. That is the known load-dependent failure in #618. The final preflight passed once the box quietened. **A shared-scratchpad hazard bit this session and could bite a reader.** Another session was writing `ctest.log` into the same scratchpad directory. My first monitor keyed on that filename and would have reported a foreign tree's result as mine. I moved my logs to a uniquely named directory. I also stopped a chained command of mine that would have overwritten that other session's log. **Disk reached 100% mid-session** because of concurrent build trees. I cleared `~/.cache/go-build`, which regenerates, to recover. I did not delete any other session's worktree or build. **F6, and it is the one that mattered.** The G2 case was named `"G2: CPU QuantFp8Static == CUDA QuantFp8Static, byte for byte"`. doctest splits `-tc=` on commas, so that name was **unselectable**. Selecting the one gate this row still owes, by its exact shipped name, measured: ``` [doctest] test cases: 0 | 0 passed | 0 failed | 4 skipped [doctest] assertions: 0 | 0 passed | 0 failed | [doctest] Status: SUCCESS! exit 0 ``` A gate that examines nothing and prints `SUCCESS!` was sitting on the arm nobody has run. Renamed comma-free; the same selector now returns `test cases: 1 | 1 passed`. The whole file stays 4 cases and 56 assertions. The reason is written next to the case, not only in the spec, because it applies to every case name in the tree. **F3. Two shipped comments asserted CPU/CUDA equivalence as fact while G2 has never run.** `cuda_matmul_fp8_cutlass.cu` called the CPU arm "the byte-for-byte mirror of this kernel", and `ops.h` said it "agrees with the CUDA kernel to fp8/bf16 tolerance". Both now state that the equivalence is **declared and owed under G2**, and both name what is actually measured instead: G1 proves the CPU kernel matches an independent e4m3 reference derived from the format, and two implementations each matching a reference is a weaker claim than the two matching each other. This is this PR's own thesis turned on itself. It repaired a comment asserting an unverified contract, then shipped two more. **F4. The spec's reference-tier risk row was refuted by measurement, and the refutation is recorded.** It reasoned from the announcement and never tested the consequence. `MaybeInstallReferenceTier` declines only while the CPU provider count is zero (`op_provider.cpp:213-214`); these registrations make it one, so on a unified-memory device those ops flip from refusing by name to installing a host kernel over device pointers. Measured: **SIGSEGV on GB10, exit 139** (#844). The `correct but slow` banner is not mitigation, it is a misleading label. This is debt this PR **creates**, and #844 owns the fix, which is broader than this row. **The relocated intake table changed its merge CONTRACT, and I got it wrong first.** `.agents/issue-index.md` is append-only, carries `merge=union` in `.gitattributes`, and says "Never edit a row" in its own preamble. I moved my `#468` edit there along with the table and carried the keyed-record discipline with it, so `scripts/check-issue-index-append-only.py` refused the branch. The review had prescribed "the target's file is a strict PREFIX of the result", which is exactly the append-only test; I judged it vacuous for an in-place edit and substituted a weaker in-place-diff check that my own edit satisfied by construction. The prescribed check was right. Resolved by making **no edit at all**: the file is byte-identical to main, and main's existing `#468` row already carries the three-way linkage, since the issue is linked from the index, from the spec, and from this PR. **This branch supersedes `row/VT-FP8-W8A8-CPU-ARM`, and PR #826 opened from it.** `scripts/check-commit-trailers.py` rejected every commit on that branch: `[attribution] malformed Assisted-by value 'AGENT:claude-opus-5[1m] [claude-code]'`. The regex requires a space before a bracket group, so `claude-opus-5[1m]` is not a valid model token. Repairing it rewrote the commit messages, which makes the push non-fast-forward. Force-pushing is prohibited by `AGENTS.md`, including `--force-with-lease`, so the repaired history was pushed to this new branch instead and the old branch is left untouched as a record. The **tree is byte-identical**: `HEAD^{tree}` is unchanged and `git diff ca7358f59..810182e --stat` is empty. Only commit messages differ. **Commit prefixes use the issue number, not the row ID.** My commits read `spec(#468)` and `feat(#468)`. `.agents/style/commits.md` landed on main after those commits and asks for `type(ROW-ID)`. `scripts/check-commit-style.py --range 3ce5a1d..HEAD` reports `OK: commit writing style`, so the checker accepts them. I did not rewrite pushed history to change them. **One commit message carries wrong arithmetic.** The implementation commit says the additivity assertions went from 19 to 25. The measured base count is 21. I corrected this in the spec rather than by rewriting pushed history, and the correcting commit explains why. 🤖 Generated with [Claude Code](https://claude.com/claude-code) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

mudler added 4 commits August 14, 2026 19:03

localai-bot mentioned this pull request Aug 14, 2026

feat(VT-FP8-W8A8-CPU-ARM): register the static fp8 W8A8 path on the CPU backend (#468) #842

Merged

4 tasks

localai-bot closed this Aug 14, 2026

localai-bot mentioned this pull request Aug 15, 2026

feat(LTX25-KEYFRAMES-ABS-POS): the trained keyframe bias both shipped DiTs were refused over (#658) #678

Closed

Navigation Menu

feat(#468): register the static FP8 W8A8 path on the CPU backend — the fp8 seam becomes gateable without a GPU - #826

feat(#468): register the static FP8 W8A8 path on the CPU backend — the fp8 seam becomes gateable without a GPU#826
localai-bot wants to merge 4 commits into
mainfrom
row/VT-FP8-W8A8-CPU-ARM

localai-bot commented Aug 14, 2026

Uh oh!

localai-bot commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

localai-bot commented Aug 14, 2026

What and why

Upstream grounding (pin 5559679229bc961848b121ccdeaa8fa5d79bec98)

The scale is applied as x * (1/s), not x / s

The gate

Preserved deviation

Notes for the reviewer

Uh oh!

localai-bot commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Upstream grounding (pin `5559679229bc961848b121ccdeaa8fa5d79bec98`)

The scale is applied as `x * (1/s)`, not `x / s`