feat(rocm): wvSplitK skinny GEMM for decode M<=4 — the #487 decode GEMM lever

.agents/issue-index.md

@@ -388,6 +388,7 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1198](https://github.com/mudler/vllm.cpp/issues/1198) | — | Three specs assert `model_loader.cpp` behaviour the loader no longer has, found while verifying line citations for conversion under [#1143](https://github.com/mudler/vllm.cpp/issues/1143): `safetensors-windowed-load.md:63,108` quotes a `shards.clear()` that does not exist (the shard vector is a `shared_ptr` released by the deferred-expert closure, `model_loader.cpp:1636-1643`), `gguf-dflash-draft.md:17-18` calls `LoadDflashDraft` "still typed on `std::vector<SafetensorsFile>`" when it takes a `SharedHeadSource`, its `A5`/`B2` rows plan around a GGUF refusal the loader says at `:906-910` is GONE, and `model-factory-registry.md:91` cites `IsDenseArch` which survives only in a comment saying the registry superseded it. Filed rather than repaired because a citation sweep can see the claim is false and not what the true statement is — that is the owning row's judgement. Owned under `## Owed` in [`citation-anchor-freshness.md`](specs/citation-anchor-freshness.md) | bug |
 | [#1199](https://github.com/mudler/vllm.cpp/issues/1199) | — | `scripts/check-symbol-anchors.py --upstream-root` resolved 354 `vllm/...::Symbol` citations against the parity pin `555967922`: **343 fresh, 0 stale**, which is the measurement that decides the symbol convention over line anchors, since the same pin advance broke every line anchor [#1139](https://github.com/mudler/vllm.cpp/issues/1139) examined. The 11 that did not resolve name a FILE absent at the pin: six `.agents/model-matrix.md` rows (`olmo.py`, `olmo2.py`, `ouro.py`, `persimmon.py`, `plamo2.py`, `fuyu.py` — `registry.py:658` routes `OlmoForCausalLM` to the `transformers` fallback and `registry.py:765` lists `FuyuForCausalLM` as removed at 0.25.0, so our rows claim a mirror source vLLM deleted), two malformed `vllm/tests/kernels/...` paths, `vllm/v1/worker/gpu/worker.py` for `gpu_worker.py`, and one fixture-text false positive. Filed rather than repaired because the model-matrix half is a claim about vLLM, not a path edit. Owned under `## Owed` in [`citation-anchor-freshness.md`](specs/citation-anchor-freshness.md) | bug |
 | [#1213](https://github.com/mudler/vllm.cpp/issues/1213) | `ENV-LEASE-RUNTIME-STAGING` | `AGENTS.md` stated that a leased worker "has no compiler, no downloader and no Python, so it cannot produce a runtime in place", and `.agents/environment.md` carried the matching clause twice for `dgx:gpu0`. All three negatives are false. `rc describe dgx:gpu0` states that a job runs as root in an Ubuntu 24.04 container carrying `git`, `curl`, `wget`, `ssh`, `gcc`, `g++`, `make`, `cmake`, `ninja`, `pkg-config`, `python3`, `pip` and `venv`, and it instructs the reader to install anything missing; the one limit it names is the absent CUDA toolkit. Two jobs then compiled inside a lease on 2026-08-18: `claude/mudler-ubuntu-box/qwen38-gate` apt-installed `cuda-nvcc-13-0` from the `ubuntu2404/sbsa` lane and built this tree 1791/1791 to `BUILD_RC=0` (`/mnt/nas_share/rc/qwen38-gate/out-main/cfg.log` records `nvcc` 13.0.88 and `CUDA feature cutlass-fp8: ENABLED for [121a]`), and `/mnt/nas_share/rc/mtp_test/build.sh` cloned `github.com/mudler/llama.cpp` from inside a job and left a 97 MB `libggml-cuda.so` on the share. **Why it matters:** "the lease cannot produce a runtime in place" is the stated basis for treating the pinned vLLM oracle as unreachable from a lease, and that oracle is the denominator for every speed-parity number the project owes, so the premise needs re-testing rather than inheriting. This claims nothing about a model run; #1185 owns that and stays open. FIXED IN FLOW: the `AGENTS.md` paragraph and both `.agents/environment.md` clauses now say what the measurement supports, and the four real limits (no preinstalled CUDA toolkit, global installs leak until the pod restarts, CIFS `/workspace` holds no symlink so build in `/tmp` and `cp -rL`, and `-j 4` because unconstrained parallelism OOM-reboots the box) plus the host-versus-container egress distinction ride with the correction. | record |
+| [#1183](https://github.com/mudler/vllm.cpp/issues/1183) | `BACKEND-ROCM` | ROCm skinny GEMM architecture eligibility caches the first device | bug |
 | [#1222](https://github.com/mudler/vllm.cpp/issues/1222) | `BACKEND-ROCM` | Qwen3.5 paged-engine gate reports success when a required greedy, anchor, or near-tie-gap artifact is absent | bug |
 | [#1225](https://github.com/mudler/vllm.cpp/issues/1225) | `SPEC-DSPARK-BLOCK-SIZE-GUARD` | A DSpark speculative length below the draft's block is accepted silently. `ResolveDspark` carries upstream's `k >= dspark_block_size` hard error (`include/vllm/config/speculative.h:179-185`, from `vllm/config/speculative.py:1003-1027` @ `555967922`) and both production call sites pass `std::nullopt` for `n_predict` and for `dspark_block_size` (`src/vllm/entrypoints/model_loader.cpp:881-883` and `:1675-1677`), so the floor reaches no user path and only `tests/vllm/config/test_speculative_dspark.cpp:99-107` drives it. Nothing in our draft path reads the checkpoint's block key — the block layout is sized by `k` alone (`include/vllm/v1/worker/gpu/spec_decode/dspark/speculator.h:56`) and no weight is block-shaped — so a short `k` raises no shape error and drafts a structurally wrong block while the tokens keep flowing. A literal port does NOT close it: `dspark_block_size` appears in no pinned file but `speculative.py`, and neither published Qwen3 draft sets it (`deepseek-ai/dspark_qwen3_4b_block7` and `RadixArk/Qwen3.8-27B-DSpark` @ `85ef153b` both carry `block_size: 7`, no `n_predict`), while upstream's `block_size` normalization at `:945-961` is Gemma4-only — so upstream accepts `k=6` on both sides of vllm#52197. Closing it for the lane we ship needs `block_size` as the floor fallback, one tracked divergence argued in the spec and the commit | bug |
 | [#1226](https://github.com/mudler/vllm.cpp/issues/1226) | `ENG-CUDAGRAPH-DEDUP` | The owed device A/B for #1162 RAN on `dgx:gpu0` 2026-08-18 (GB10, driver 580.173.02, nvcc 13.0.88, `rc` job f88d484b, gated commit `72de552c8`, whose four dedup sources are byte-identical to the merged `2a976eb9f`) and it SPLIT. CORRECTNESS PASSED and closes [#1184](https://github.com/mudler/vllm.cpp/issues/1184): 12/12 cells exit 0, zero `invalid device function` and zero `engine-fatal` where the pre-fix head `e4ce5571a` died after exactly one replay, ON replays as often as OFF (60=60, 33=33, 43=43), and `--output-token-ids` is IDENTICAL over 10/10 comparisons with the three OFF/OFF controls passing FIRST and three workloads hashing to three DIFFERENT values. THE BENEFIT IS REFUTED for the case the row was filed for: `N == M` in every `VT_CUDA_GRAPH_DEDUP=1` cell — 3 graphs to 3 execs on sizes [24 16 8], 2 to 2 on [16 8], 2 to 2 on [32 24] — with the registry count CLIMBING 1→1, 2→2, 3→3, so more than one capture reached it and the 1:1 is a measurement rather than the single-capture artefact of the first attempt. The cause is structural and was pre-registered before the run: `AppendKernelPayload` hashes (`func`, `gridDim.{x,y,z}`, `blockDim.{x,y,z}`, `sharedMemBytes`) at `src/vt/graph_dedup_runtime.h:121-128` and the memcpy payload hashes the copy extent, so the padded batch dimension is IN THE KEY, no candidate group forms and `cudaGraphExecUpdate` is never attempted. That contradicts the row's own premise, and SGLang keys the same fields (`cuda_graph_dedup_mixin.py:105-114`), so whatever folds upstream is not decode buckets either. NO throughput or memory number is recorded — clocks unpinned AND the ON arm allocated exactly as many executables as OFF. `VT_CUDA_GRAPH_DEDUP` therefore stays OFF and the default flip is NOT JUSTIFIED on this evidence. THE OPEN HYPOTHESIS this issue owns, deliberately not decided by the record that filed it: a COARSER key keeping the function addresses and topology but dropping the launch dimensions and memcpy extents would let two padded buckets form a candidate group at all, and the probe-before-fold design means a wrong grouping costs one wasted `cudaGraphExecUpdate` probe and a private executable rather than a wrong replay, so it is a cost question needing its own spec, a red-first discrimination test and a device probe-refusal rate. Also owed here: whether ANY other capture site or two models sharing the process-singleton registry can fold (only the Qwen3 dense decode driver was exercised), and a supporting `orin:gpu0` leg, BLOCKED because the Jetson 540.4.0 driver cannot run a CUDA 13 runtime (`cudaGetDeviceCount err=35`; a CUDA 12.x toolkit is the untried route). Honest gaps carried into the record: per-shape replay counts are unavailable, so B's ~30-per-shape is arithmetic over a printed TOTAL; the driver's "N captured size(s)" counts SLOTS not captures; the container's own cuBLASLt was never re-tested at CUDA 13.0. Full evidence [`benchmark-record.md`](benchmark-record.md) entry `ENG-CUDAGRAPH-DEDUP W4`; spec [`eng-cudagraph-dedup.md`](specs/eng-cudagraph-dedup.md) `## Outcome` | perf |

.agents/specs/rocm-skinny-gemm.md

@@ -0,0 +1,111 @@
+# ROCm wvSplitK skinny GEMM — review rework (PR #506)
+
+The original #506 ported upstream's `wvSplitK_hf_sml_` kernel body faithfully
+(the reviewer verified the anchors and the arithmetic) but dropped the dispatch
+preconditions around it. This spec records the rework; the kernel math is
+unchanged.
+
+## The three guards, restored (all verified against the pin `555967922`)
+
+1. **Feature-dim guards** (upstream `utils.py:181` `m > 8 and 0 < n <= 5` with
+   `m = weight.shape[0]`, plus `skinny_gemms.cu:1217` `M_in % _YTILE == 0`):
+   with the donor's naming mapped onto ours (`out[M_tokens, N_features]`), the
+   kernel's YTILE=2 stores write `C[m + y + n*M_features]` unguarded for `y<2` —
+   on odd N the last wave writes `C[N]` (two bytes past the buffer), and at
+   N==1 the first wave already writes out of bounds. Our dispatch now requires
+   `N > 8 && (N % 2) == 0`, everything else falls through to the BLAS path.
+2. **Arch guard**: the port carries only the wave32 reduction arm
+   (`__shfl_xor(x,16)`); upstream branches to `ROW_BCAST15/31` on gfx9
+   (wave64). The gfx9 arm is NOT ported, so dispatch now refuses non-wave32
+   architectures via `CapabilityFromGcnArch(DeviceArchName())` (gfx11xx/gfx12xx
+   only), rather than compiling and silently producing wrong sums on gfx9.
+3. The `N > 8` lower bound (upstream `m > 8`) — folded into (1).
+
+## The test, ported for real this time
+
+`tests/kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_kernel`
+@ pin — preserved: the applicable NKM factor list (tokens 1–4 = our template
+arms), the xavier on/off scaling, and the **elementwise** tolerance
+(`atol = eps_bf16 * sqrt(K)`, `rtol = 1e-2`; torch assert_close semantics) in
+place of the aggregate NMSE. Added guard-boundary cases the upstream suite
+implies: features ≤ 8 and odd features must route to BLAS and stay correct,
+odd K declines, and a K%512 ≠ 0 shape exercises the K-tail. Every case runs
+into a **sentinel-padded output buffer** (0xDEAD guard band) so any residual
+out-of-bounds store fails the test outright. Deferred with reason recorded:
+fp16 (port is bf16-only), bias (the `vt::MatmulBT` seam has no bias operand),
+padded strides (our dispatch precondition is contiguous rows).
+
+Mutation proof: with the `N % 2` guard removed, the odd-features case corrupts
+the sentinel band and the case fails; with it restored, green.
+
+## Boundaries
+
+- Kernel body unchanged from the reviewed port.
+- The gfx9 (wave64) arm remains owed — a future port of the ROW_BCAST
+  reduction, gated the same way.
+- `VT_ROCM_SKINNY=0` remains the A/B rollback; the allowlist carries it once,
+  in main's re-sorted layout.
+
+## Issue #1183 repair
+
+[Issue #1183](https://github.com/mudler/vllm.cpp/issues/1183) found that the
+architecture guard cached the first device's architecture for the process.
+
+### Diagnosis
+
+`SkinnyGemmArchOk(int device_index)` stored `DeviceArchName(device_index)` in a
+function-static string. A gfx11 call initialized that string as eligible. A
+later gfx9 or unknown device then reused the gfx11 result and could reach the
+wave32-only kernel. The four devices on the repair host are gfx1100, so the
+test uses a controlled resolver instead of claiming heterogeneous hardware.
+
+### Decision
+
+The production call and the test now use the same HIP-free predicate. A
+per-thread vector keys each result by resolver and device index. The first call
+for a key resolves and parses the architecture. Later calls read one boolean
+without a HIP query or a process-wide mutex. The resolver key prevents the test
+resolver from contaminating a production result in the same process.
+
+The guard continues to accept only gfx11 and gfx12. It refuses gfx9 and every
+unknown architecture. The shape, dtype, rollback, fallback, and GetBlas rules
+remain unchanged.
+
+### Rejected alternatives
+
+- A single first-device value repeats the defect and is unsafe after a device
+  hop.
+- An uncached `DeviceArchName` call adds a HIP property query to every skinny
+  GEMM dispatch.
+- A process-wide keyed map needs synchronization on the decode path.
+
+### Evidence
+
+The test-only refactor first preserved the faulty cache. This command compiled
+the production predicate and its deterministic device-hop test:
+
+```sh
+env LD_LIBRARY_PATH=/opt/rocm/lib:/opt/rocm/lib/llvm/lib flock /home/vikash/gpu.lock cmake --build build-hip --target test_rocm_arch -j2
+```
+
+The build exited 0. The next command exited 1 before the fix:
+
+```sh
+env LD_LIBRARY_PATH=/opt/rocm/lib:/opt/rocm/lib/llvm/lib flock /home/vikash/gpu.lock build-hip/tests/test_rocm_arch '--test-case=skinny GEMM architecture eligibility follows device hops'
+```
+
+The gfx9 and unknown checks received `true`. Resolver counts were `{1,0,0,0}`
+instead of `{1,1,1,1}`. After the keyed cache change, the same test passed 1 of
+1 cases and 6 of 6 assertions with exit 0.
+
+The mutation replaced the keyed cache call with one function-static resolved
+architecture. The build exited 0, and the same focused test exited 1 with the
+same three assertion failures. After restoration, SHA-256 values for the
+header, production caller, and test matched their pre-mutation values. A fresh
+rebuild and focused run then passed 1 of 1 cases and 6 of 6 assertions.
+
+### Outcome
+
+Architecture eligibility follows the requested device on every device-hop
+sequence. A repeated key does not query the resolver again. The gfx9 wave64 arm
+remains refused and owed as recorded in `## Boundaries`.

CMakeLists.txt

@@ -1532,6 +1532,7 @@ if(VLLM_CPP_HIP)
     src/vt/rocm/rocm_gdn_postconv.hip
     src/vt/rocm/rocm_gdn_scan.hip
     src/vt/rocm/rocm_gdn_fused.hip
+    src/vt/rocm/rocm_skinny_gemm.hip
     src/vt/rocm/rocm_ops.hip)
   if(VLLM_CPP_HIP_ARCHITECTURES)
     set_source_files_properties(
@@ -1552,6 +1553,7 @@ if(VLLM_CPP_HIP)
       src/vt/rocm/rocm_gdn_postconv.hip
       src/vt/rocm/rocm_gdn_scan.hip
       src/vt/rocm/rocm_gdn_fused.hip
+      src/vt/rocm/rocm_skinny_gemm.hip
       src/vt/rocm/rocm_ops.hip
       PROPERTIES HIP_ARCHITECTURES "${VLLM_CPP_HIP_ARCHITECTURES}")
   endif()

docs/USAGE.md

@@ -95,7 +95,16 @@ MoE-path coverage is partial: `MoeRouterTopK` (f32/bf16 logits, ungrouped
 softmax, no bias) and `MoeSiluMul` are native; the remaining chain
 (`kSharedExpertGate`, `kMoeCombine`/`kMoeCombineGate`, and the grouped quant
 expert GEMM) is not registered yet, so MoE-bearing models still throw on
-those ops. On a
+those ops.
+
+### ROCm decode GEMM routing (wvSplitK skinny path)
+
+Decode-shaped GEMMs (M<=4, bf16) route to a split-K skinny-GEMM kernel (a port
+of vLLM's `wvSplitK`) instead of the 128x128-tile rocBLAS GEMM that dominates
+decode GPU time ([#487](https://github.com/mudler/vllm.cpp/issues/487)). On by
+default where it fits; `VT_ROCM_SKINNY=0` restores the BLAS path for A/B.
+
+On a
 discrete card there is no CPU fallback tier, so a model whose layers call an op
 that is not registered yet fails loudly with `vt: no kernel for op N on device
 type 5` — that is the memory-safety design working, not a crash. Run with

include/vt/rocm/rocm_skinny_gemm_arch.h

@@ -0,0 +1,53 @@
+// ROCm wvSplitK architecture eligibility. This header stays free of HIP
+// headers so a controlled resolver can gate device-hop behavior on any host.
+#pragma once
+
+#include <string>
+#include <vector>
+
+#include "vt/rocm/rocm_arch.h"
+
+namespace vt::rocm {
+
+using SkinnyGemmArchResolver = std::string (*)(int) noexcept;
+
+// The wvSplitK port carries only the wave32 reduction arm. Upstream uses
+// ROW_BCAST15/31 on gfx9 wave64 devices, and that arm is not ported. The
+// predicate accepts gfx11 and gfx12 only. An unknown architecture refuses.
+namespace detail {
+
+class SkinnyGemmArchCache {
+ public:
+  bool Eligible(int device_index, SkinnyGemmArchResolver resolve) {
+    for (const Entry& entry : entries_) {
+      if (entry.device_index == device_index && entry.resolve == resolve) {
+        return entry.eligible;
+      }
+    }
+
+    const auto cap = CapabilityFromGcnArch(resolve(device_index));
+    const bool eligible = cap.has_value() && (cap->first == 11 || cap->first == 12);
+    entries_.push_back(Entry{device_index, resolve, eligible});
+    return eligible;
+  }
+
+ private:
+  struct Entry {
+    int device_index;
+    SkinnyGemmArchResolver resolve;
+    bool eligible;
+  };
+
+  std::vector<Entry> entries_;
+};
+
+}  // namespace detail
+
+inline bool SkinnyGemmArchOk(int device_index, SkinnyGemmArchResolver resolve) {
+  // Each worker reads a device property once per device. Per-thread storage
+  // keeps the decode path free of a process-wide lock.
+  static thread_local detail::SkinnyGemmArchCache cache;
+  return cache.Eligible(device_index, resolve);
+}
+
+}  // namespace vt::rocm

scripts/env-doc-allowlist.txt

@@ -171,6 +171,7 @@ VT_RMSNORM_GATED_FAST
 VT_ROCM_GEMM_COMPUTE
 VT_ROCM_GEMV
 VT_ROCM_HIPBLASLT
+VT_ROCM_SKINNY
 VT_SILU_FP4_FAST
 VT_SPEC_TRACE
 VT_SWIZZLE_IN_QUANT