perf(cuda): head_dim-128 arm for the warp-split-KV decode kernel, gated OFF (#382) - #425
Merged
Conversation
…ed OFF
PagedAttentionDecodeOptKernel is reachable only at `d == 32 * kDecEpl` (256),
so head_dim 128 -- what Qwen3-dense, Llama and Mistral actually use -- falls to
the generic block kernel. This adds the EPL=4 instantiation behind
VT_ATTN_DECODE_D128, DEFAULT OFF.
The kernel is templated on elements-per-lane. EPL == 8 forwards through
LoadRowN<8> to the existing LoadRow8, so the d256 path keeps the same
transactions, the same order and the same accumulation -- the d256 launcher
still instantiates the default EPL and is byte-for-byte what it was. EPL == 4
adds one 64-bit load per lane for bf16 and one 128-bit for f32; 32 lanes * 4
elems == 128, so a warp still covers exactly one head-dim row.
No GQA carve-out: PagedAttentionDecodeGqaKernel sits inside the d == 256 branch,
so at head_dim 128 it can never run, and excluding qpk == kDecGqaQG would strand
exactly those models (e.g. Qwen3-32B) on the block kernel this arm replaces.
Gated OFF because it is correctness-complete but NOT byte-exact against the
block kernel it replaces: the two reduce the KV sequence in a different ORDER
(warp-strided online softmax vs the block kernel's per-tile loop), so a greedy
anchor can move at an exact bf16 tie. OFF keeps every existing golden
byte-identical. This mirrors how the FA2 decode GQA group-swap landed --
correctness-complete and gated OFF -- before it was flipped ON against the full
gate. The flip is a separate change and owes the near-tie razor, the
distributional gate, and regen under the ratified-tie rule.
The env var is also the same-binary A/B your perf protocol requires: one binary,
VT_ATTN_DECODE_D128=0/1, no rebuild between arms.
EVIDENCE (sm_110 / Jetson AGX Thor, Qwen3-1.7B-NVFP4A16, W4A16 Marlin build):
Kernel numerics, EPL=4 vs the generic block kernel, 8 context lengths
straddling the 32-token page boundary, 2048 elements each:
ctx max_abs max_rel bitwise%
1 0 0 100.0000
31 0 0 100.0000
32 0 0 100.0000
33 0 0 100.0000
64 0 0 100.0000
100 3.725e-09 0 99.9512
685 0 0 100.0000
1000 6.104e-05 5.076e-03 99.9512
Six of eight are bit-for-bit identical. The worst case is 0.5% relative on
small-magnitude outputs at ctx 1000 -- named plainly because that is the
magnitude at which a near-tie flips.
Comparator self-test: seeding 1-ULP corruption into 3 of the 2048 elements of
the ctx=31 case (offsets 5, 1024, 2047) moves that row from 100.0000% to
99.8535% bitwise with max_abs 4.883e-04, and no other row changes. So the
comparison is demonstrably able to fail and localizes correctly. This is a
COMPARATOR self-test, not a kernel negative control.
Serving A/B, same harness / model / config, c=1,2,4,8, 128 tokens, temp 0:
OFF 81.630 / 162.738 / 316.255 / 477.070 tok/s
ON 131.023 / 258.953 / 508.139 / 936.048 tok/s
ratio 1.61x / 1.59x / 1.61x / 1.96x
Build: the single TU compiles clean on the device (nvcc, sm_110, Release,
`ninja CMakeFiles/vllm.dir/src/vt/cuda/cuda_paged_attn.cu.o`, exit 0).
LIMITATIONS, stated because they bound what the numbers above prove:
- ONE model, ONE head_dim, ONE arch. Not measured anywhere but sm_110.
- The serving A/B is a SINGLE repetition per point (~1 s per point) and was
taken across two builds, not through the new flag.
- No model-level token-exact gate was run against this arm.
- LoadRowN<4, float> and the windowed (HasWindow=true) EPL=4 instantiation
are compiled but unexercised by the evidence above.
- The GQA-fused kernel remains d256-only; this change does not widen it.
Refs #382.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [ClaudeCode]
The cherry-picked change (Filip Sajdak, authorship preserved) refs #382, but that issue was never in the roadmap intake table. AGENTS.md requires the number in three places that must agree -- the issue table, the row's spec, and the PR body -- and only two of the three were true. Placed under `KERNEL-ATTN-PAGED`, which is the row that anchors `src/vt/cuda/cuda_paged_attn.cu` and therefore owns the decode-opt kernel this arm extends. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
Keeps the branch current while CI is queue-blocked (#274). No conflicts. Amended: git merge had already created this commit, so the message that was meant to accompany it never applied and it carried git's default -- which has no trailers at all and reds the trailer gate. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
Keeps Filip Sajdak's d128 decode arm current while CI is queue-blocked (#274). No conflicts; the kernel commit is untouched and still authored by him. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
force-pushed
the
row/KERNEL-ATTN-DECODE-D128
branch
from
August 11, 2026 21:13
2a65376 to
cb5f151
Compare
This was referenced Aug 13, 2026
localai-bot
pushed a commit
to joral/vllm.cpp
that referenced
this pull request
Aug 17, 2026
…efault OFF FOLLOWING_AGENTS_PROTOCOL Implements .agents/specs/rocm-decode-attn-d128.md, committed ahead of this change. The ROCm mirror of PR mudler#425's CUDA arm for issue mudler#382. Every fast decode kernel in rocm_paged_attn.hip was gated to d == 256 || d == 512, so head_dim 128 -- Qwen3-dense, Llama and Mistral -- fell straight through to PagedAttnOnline. LoadRowEplBf16/StoreRowEplBf16 gain an EPL=4 (uint2) case, and both decode dispatch switches gain a d == 128 branch, so that geometry reaches PagedAttnDecodeGqaBf16 (qg=2, fused) or PagedAttnDecodeOptBf16T (per-head) instead. Gated VT_ATTN_DECODE_D128, DEFAULT OFF -- the same env var, default and reason as the merged CUDA arm, because the warp-strided online softmax reduces the KV sequence in a different ORDER than PagedAttnOnline, so a greedy anchor can move at an exact bf16 tie and OFF keeps every golden byte-identical. No allowlist entry is owed: the CUDA arm already added VT_ATTN_DECODE_D128. Testing: a new bf16 Qwen3-geometry (GQA 2, head_dim 128) cross-device case, where the suite previously had no bf16 coverage of any EPL-templated kernel. Because the arm ships OFF and its flag is read into a static const bool -- once per process -- the default registration can only ever gate the fallback, so tests/CMakeLists.txt registers a second invocation of the same binary with the flag set. Verified non-vacuous against the issue mudler#463 trap: 1 case, 6 assertions, not zero. Scope note: this commit is the SCALAR arm only. The rocWMMA d=128 decode spike that was originally developed alongside it is split into its own spec and its own PR -- it is a spike with its own (unfiled) issue, and bundling the two put 474 lines of two unrelated kernels in one review. The `|| decode_wmma` disjunct in the bf16_decode_opt gate belongs to that change, not this one: it only means anything once a second opt-in kernel for this head size exists. Row: BACKEND-ROCM Issue: mudler#382 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-sonnet-5 [Claude Code] Assisted-by: Claude:claude-opus-5 [Claude Code]
localai-bot
pushed a commit
that referenced
this pull request
Aug 17, 2026
… OFF Implements [`.agents/specs/rocm-decode-attn-d128.md`](https://github.com/mudler/vllm.cpp/blob/main/.agents/specs/rocm-decode-attn-d128.md), which landed ahead of this change as #564. The ROCm half of #382. The CUDA half merged as #425 (`66399617`); this mirrors it, adopting that arm's flag, default and stated reason rather than inventing new ones. ## What changes `d == 128` — the Qwen3-dense / Llama / Mistral head size — reaches the fast decode kernels instead of falling through to the generic `PagedAttnOnline`. `LoadRowEplBf16`/`StoreRowEplBf16` gain an `EPL=4` (`uint2`) case beside the existing `EPL=8`/`16`; the dispatch gates and the two launch switches gain a `d == 128` arm. No new kernel and no new algorithm — the kernel bodies were already generic over `EPL`. **Default OFF, opt in with `VT_ATTN_DECODE_D128=1`** — the same env var, default and reason as the merged CUDA arm. The arm is correctness-complete but not byte-exact against the kernel it replaces: warp-strided online softmax reduces the KV sequence in a different **order** than `PagedAttnOnline`'s per-tile loop, so a greedy anchor can move at an exact bf16 tie. Shipping OFF keeps every existing golden byte-identical. The flip owes the near-tie razor, a distributional gate and regen under the ratified-tie rule, and per the spec must be argued **per backend** — see the reversal below. That is what keeps #382 open. ## Reviewer note Spec §4 item 3 writes the gate as `(d == 128 && (decode_d128 || decode_wmma))`. This commit implements it **without** the `decode_wmma` disjunct, which is what the same item's "Forward reference" paragraph instructs: `VT_ATTN_DECODE_WMMA` does not exist in the tree, and the flag lands with the rocWMMA arm on its own branch. The difference is intentional; it is visible in the diff before the note explaining it is. ## Evidence gfx1200 (RX 9060 XT, RDNA4, 32 CU), ROCm 7.2.3, `$GPU_LOCK` held. All figures are a same-binary flag A/B — no rebuild between arms — at 1024-token synthetic prompt, 128 generated, greedy, seed 0, 2 reps per cell agreeing within ~1%. | Model | head_dim | decode path | TPOT OFF | TPOT ON | speedup | |---|---|---|---|---|---| | Qwen3-0.6B | 128 | `qg=2` fused | 42.53 ms | 11.78 ms | **3.61x** | | Qwen3-1.7B | 128 | `qg=2` fused | 52.85 ms | 21.93 ms | **2.41x** | | Qwen3-4B | 128 | `qg=4` per-head | 81.89 ms | 39.22 ms | **2.09x** | | Qwen3.5-0.8B | 256 | — (control) | 23.76 ms | 23.55 ms | 1.01x | Qwen3-4B has no GQA fusion at any head_dim, so its 2.09x isolates the `EPL` widening from the fusion. **Qwen3.5-0.8B is the negative control and it earned its keep.** Its `head_dim` is 256, so the `d == 128` gate provably cannot reach it. Its first OFF rep came in a 33% outlier at 31.14 ms, which a blind 2-rep average would have reported as a ~1.2x "win" for a model the flag cannot affect. Re-run three times: 23.86 / 23.75 / 23.68 against ON's 23.52 / 23.57. End-to-end output throughput rises less than TPOT on the same runs (0.6B 2.48x, 1.7B 2.05x, 4B 2.02x) because they carry a 1024-token prefill the flag does not touch. TPOT isolates decode; throughput dilutes it. ### Concurrency — the advantage grows, it does not compress Qwen3-1.7B, `--num-prompts` = 2x concurrency: | Conc | tok/s OFF | tok/s ON | ratio | TPOT ratio | |---|---|---|---|---| | 1 | 12.89 | 24.66 | 1.91x | 2.40x | | 2 | 23.27 | 47.45 | 2.04x | 2.45x | | 4 | 39.10 | 86.86 | 2.22x | 2.46x | | 8 | 58.97 | 147.35 | **2.50x** | **2.77x** | | 16 | 78.43 | 227.08 | **2.90x** | **3.18x** | This refuted the prediction made before the run, which reasoned that a tiny grid at concurrency 1 flatters the fast kernel. The dominant effect is the reverse: from c8 to c16 the fallback scales only **1.33x** against the arm's **1.54x**, and scaling efficiency at c16 relative to perfect-linear-from-c1 is **38% OFF against 58% ON**. `PagedAttnOnline` is the batch-scaling bottleneck, not merely slow per call, so the win is largest in the regime a server actually runs in. The c1 row reproduces an independently-run four-model sweep to within ~1% (52.85/21.93 there vs 53.40/22.26 here). ### Correctness - `ctest -R 'rocm|cross_device'` **5/5**, chained directly to the exact-SHA push. - New case: "paged attention at Qwen3 geometry (bf16, GQA 2, head_dim 128) matches the CPU oracle", looped over `RegisteredDevices()`, NMSE <= 5e-4 vs the CPU oracle plus `OpProviderStats::declines == 0`. Genuinely new coverage — the existing generic cross-device test runs at `d=8, f32` and never reached any bf16 `EPL`-templated kernel, so none of them had bf16 coverage in this suite. (The merged CUDA arm shipped with no test at all.) - Because the arm ships OFF **and** its flag is read into a `static const bool` — once per process — the default registration can only ever gate the fallback. `tests/CMakeLists.txt` adds a second invocation with the flag set, same shape as the existing `test_dense_gateup_fused_marlin_off_*` pair. Verified non-vacuous against the #463 trap: 1 case, 6 assertions, not zero. - Full `ctest` 448/455. The 7 failures are **proven** pre-existing, not asserted: a clean `main` `2784dd7b` worktree built from source with none of this code fails the identical set (only `test_op_parity`'s index shifts 403 -> 404, from the added registration). They are a missing `shellcheck`, an mmap-RSS assertion, a JSON type error, and the `SharedExpertGate` ROCm registration gap owed to unmerged #509. - `agent-preflight` fails 9, a strict **subset** of that same baseline's 10 (differing only by `role-undeclared`). `check-commit-trailers` and `check-doc-checkpoint` both pass against this base. ## Carried finding #382 measured this same `EPL=4` arm **1.6x slower** on sm_110 / Jetson AGX Thor, where gfx1200 measures it 2-3.6x faster. Recorded, not reconciled — different kernels, different fallbacks, different memory systems. It is why the default-ON flip must be argued per backend rather than once, and it is preserved in the spec rather than averaged away. ## Against the pinned oracle: 6.35x to 1.75x slower on per-token decode Measured after the tables above, with **both sides in the same container**, oracle = vLLM `555967922` in its production configuration via `vllm bench serve`. Qwen3-0.6B, 1024 in / 128 out, concurrency 1, **8 prompts**, warmup discarded, **3 reps**: | | TPOT reps | mean | vs oracle | |---|---|---|---| | ours, flag unset | 42.54 / 42.46 / 42.19 | 42.40 ms | 6.35x slower | | ours, `VT_ATTN_DECODE_D128=1` | 11.97 / 11.38 / 11.66 | **11.67 ms** | **1.75x slower** | | vLLM `555967922` | 6.57 / 6.90 / 6.58 | 6.68 ms | — | Running our binary against the container's ROCm rather than the host's is a substitution, so it was proved inert first: in-container matches native at 42.79 vs 42.53 ms unset, and 12.03 vs 11.78 ms with the flag. The prompt count is load-bearing. At `--num-prompts 2` the oracle returned TPOT **6.96 ms and 13.45 ms on consecutive reps**, a ~2x spread averaging to a plausible-looking and entirely fictional number. At 8 prompts with a discarded warmup both sides hold to ~±0.3 ms. This number lived only in a PR comment, which a squash merge does not carry into the tree. It is now in the spec's §5 and appended to `.agents/benchmark-record.md`, with its caveats attached rather than trailing. ## The container/glibc blocker was RETRACTED An earlier revision of this body, and the spec's §6, said the oracle re-measure was blocked on a Nix-glibc vs container-glibc ABI mismatch. **That diagnosis was wrong and is retracted.** Our binary runs inside the pinned oracle container; the earlier failures were self-inflicted (`LD_LIBRARY_PATH` exported container-wide, which breaks the container's own tools, plus a bind mount that silently yielded nothing and looked exactly like a missing ELF interpreter). §6 now reads "not run — **not blocked**", and the WMMA-spec cross-reference is gone. A false blocker in the record is worse than no record, because it stops the next person from trying. ## Not claimed **This does not close #488.** That issue asks for a **per-call** kernel comparison and explicitly asserts no cause. The number above is **per-token latency** with asymmetric harnesses — the oracle over HTTP via `vllm bench serve`, ours in-process — so TPOT is the only comparable axis, and TTFT, E2EL and end-to-end throughput carry the oracle's HTTP and tokenizer overhead and are directional only. It is not the same-tool per-call trace `AGENTS.md` wants before a throughput claim. `rocprofv3` is present in the container and our binary traces under it; what is still owed is decode-phase windowing on the oracle side, or the trace compares our decode against vLLM's model load and graph capture. One board, one shape. **`docs/BENCHMARKS.md`'s ROCm axis stays PENDING**, and the row this PR adds is marked DIRECTIONAL and sits beside the existing row rather than overwriting it. **The flag-ON arm still has no proof it REACHES the new kernel, now filed as #1134.** `RegisteredDevices()` (`tests/vt/test_backend_cross_device.cpp:84-96`) enumerates `{kCUDA, kMETAL, kVULKAN, kXPU, kROCM}` and excludes `kCPU`, so on a CPU-only runner — which is what CI has — the new case reports 1 test case, 0 assertions, exit 0, for **both** registrations. On ROCm the case's only backend assertion is `OpProviderStats::declines == 0`, counted at **provider** granularity, so it is identical with the flag set and unset. §9 stop condition 2 is left OPEN. The spec disclosed this honestly; what was missing is the issue `AGENTS.md` requires for a known gap not fixed in flow. Searched before filing: not a duplicate of #463 (the unset-weights-env-var shape, which does not describe the `declines` half), #785 (a kernel that never LAUNCHES behind a dead `#if`) or #900 (same family, LTX-2.5 subject). Also out of scope and named in the spec's new `## Owed` section: the dtype gap (ROCm's decode-opt is bf16-only at every head_dim, so 4 of 5 dtype combinations still fall to `PagedAttnOnline` at `d=128` — pre-existing, inherited, not introduced), `qg=4`/`qg=8` fusion, `d=128` prefill, and the rocWMMA arm. ## Record repairs carried in the final commit `docs(BACKEND-ROCM): retract the blocker, keep the oracle number, and file the gap`, on top of joral's commits, which are untouched. It carries the retraction above; the oracle number into §5 and `.agents/benchmark-record.md`; `docs/BENCHMARKS.md` and `docs/STATUS.md` reconciled; §7's stale "**two** flag-on ctest registrations" corrected to one, matching what `8aedd780` already fixed in §4 item 3 and the Test-coverage section; a literal `## Owed` heading over the owed list; and #1134 filed and appended to `.agents/issue-index.md`. Two comment-only edits at `rocm_paged_attn.hip:330` and `:455`, which still enumerated "EPL=8 → d=256, EPL=16 → d=512" without the new `EPL=4` case although the top-of-file comment at `:264` had been updated. The branch is **rebased** onto `origin/main` `d1e5e9bc` — it was 39 commits behind, and the rebase drops the earlier `merge: upstream/main` commit. Gates rerun from the worktree with explicit SHAs: `check-commit-trailers`, `check-commit-style`, `check-doc-checkpoint`, `check-public-doc-tables`, `check-agent-record` and `check-pr-size`, all OK. Not rebuilt and not re-run on hardware: every gfx1200 figure here is joral's, unchanged. Row: BACKEND-ROCM Issue: #382 Issue: #1134 Spec: #564 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands @filipsajdak's #383 on current
main, with authorship preserved (git logshowsauthor=Filip Sajdak). Issue #382. Supersedes #383, which is red for reasons that are not its own.The work is @filipsajdak's, unchanged
PagedAttentionDecodeOptKernelis reachable only atd == 32 * kDecEpl(256), so head_dim 128 — what Qwen3-dense, Llama and Mistral actually use — never reaches it and falls to the generic block kernel. This adds theEPL=4instantiation behindVT_ATTN_DECODE_D128, default OFF.EPL == 8forwards throughLoadRowN<8>to the existingLoadRow8, so the d256 path keeps the same transactions, order and accumulation — byte-for-byte what it was.Gated OFF for an honest reason: it is correctness-complete but not byte-exact against the block kernel, because the two reduce the KV sequence in a different order, so a greedy anchor can move at an exact bf16 tie. That mirrors how the FA2 decode GQA group-swap landed (#48) before being flipped ON (#49).
Why #383 was red, and why none of it was the PR
Both failures were stale-base — it sat 46 commits behind:
cuda-fat-buildfailed the gencode audit on seven files, every one of them Marlin:Not one is
cuda_paged_attn.cu, the only source the PR touches. That is precisely the defect #407 fixed — "Marlin gencode drifted from the feature table" — which landed after this branch was cut.agent-recordfailed onaudit-live-rows, which reads live branch state at run time. Reproduced at the PR tip:--checkreturns rc=0, both before and after merging main. It was transient record state onmain(an ACTIVE row whose branch had gone), since repaired.One thing added
#382was never in the roadmap intake table.AGENTS.mdwants the number in three places that must agree — the issue table, the row's spec, the PR body — and only two were true. Linked underKERNEL-ATTN-PAGED, the row that anchorssrc/vt/cuda/cuda_paged_attn.cu.That record commit is mine; the kernel commit is Filip's and is untouched.
Evidence
preflight rc=0 on the merged tree;
check-agent-recordOK. The original commit already carried compliant trailers and passescheck-commit-trailerson its own range.The lever is default-OFF, so no golden moves. Flipping it ON is a separate change that owes the near-tie razor and the distributional gate, as Filip's own commit message states.
🤖 Generated with Claude Code