perf(fa2): port decode GQA group-swap into d128 varlen launcher (gated OFF) - #48
Closed
localai-bot wants to merge 1 commit into
Closed
perf(fa2): port decode GQA group-swap into d128 varlen launcher (gated OFF)#48localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
…d OFF) Ports vLLM's FA2 seqlenq_ngroups_swapped decode optimization into LaunchDecodeVarlenFA2Bf16 (gate VT_FA2_DECODE_GQA_SWAP, default OFF), so the Qwen3-dense head_dim-128 varlen decode launches (batch, kv_heads) instead of (batch, hq): the ngroups query heads of a KV group pack into seqlen_q, KV is read once per group, and the num_splits heuristic sees batch*kv_heads. This halves the CTA count at batch>=2, the exact over-wave #47 localized as the MXFP4 c2-c8 gap (ours decode flash grid (batch,hq) vs vLLM's (batch,kv_heads); decode flash 63.7us c2 / 218us c8 vs vLLM 41.7/151). Presented WITHOUT a materialized transpose via kv-major-group-minor strides, a 1:1 mirror of the already-shipped d256 LaunchDecodeFA2Bf16 swap (the vendored flash_fwd_kernel.h get_lse_tile/combine already honor the flag in both the num_splits==1 direct-write and >1 combine paths). OFF path is byte-identical to the shipped plain-varlen reduction; ON is non-byte-exact only when num_splits>1 (split reduction order changes -> near-tie, toward vLLM's own numerics). Blast radius: the shared d128 varlen decode launcher (Qwen3-dense 0.6B/4B/8B + MXFP4-8B); the 27B/35B gate models decode on their own d256 path, untouched. Correctness (dgx GB10, sm_121a, CUDA 13.0): - RED-first op test: test_ops_paged_attn 5/5 varlen-d128 cases, 280/280 assn (both GQA ratios x batch{1,2,4,8} x short+long ctx; swap_launches==1 proves the swapped grid engaged; swap-vs-plain near-tie; MHA qpk==1 inert). RED proven: wrong swapped o_head_stride -> 26528 violations. Full binary 28/28 cases / 454679 assertions, no regression. - compute-sanitizer memcheck: 0 errors, 0 bytes leaked. - #44 MXFP4 e2e smoke (default async/graphed) swap-ON: 3/3 deterministic TOKEN-EXACT + coherent, byte-identical to swap-OFF (no token flip e2e). Default stays OFF: the c1-c8 x3 binding re-bench + default flip is the recorded next step (parity-enablers: flip only on a measured speed win with no regression). #47 projects flash alone closes ~28%@c2 / ~55%@c8 of the gap. Records: STATUS/BENCHMARKS/FEATURES + kernel/quantization matrices + state (KERNEL-FA2-GQA-SWAP) + NOW + benchmark-record + parity-ledger. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
localai-bot
marked this pull request as ready for review
August 6, 2026 03:06
Collaborator
Author
|
Landed on main as the squash (mudler authorship). 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…te (#48) row/KERNEL-FA2-GQA-SWAP squash. LaunchDecodeVarlenFA2Bf16 gains the upstream mha_fwd_kvcache ngroups swap (logical q/o reshape via strides, no transpose; h=kv_heads, seqlen_q=ngroups; num_splits on swapped dims; 1:1 mirror of the shipped d256 LaunchDecodeFA2Bf16). Gate VT_FA2_DECODE_GQA_SWAP default OFF, OFF path untouched. Blast radius: Qwen3-dense d128 family only (0.6B/4B/8B); gate models already swap on d256; Voxtral MHA inert. Gates: op RED-first 5/5 (280 assn; wrong-stride injection fails loudly 26,528 violations), full binary 28/28 (454,679), memcheck 0/0, #44 smoke swap-ON 3/3 token-exact AND byte-identical to OFF (graphed). Flip conditioned on the c1-c8 re-bench + 0.6B/4B e2e (next campaign); #47 projects ~28% c2 / ~55% c8 gap closure. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
localai-bot
pushed a commit
that referenced
this pull request
Aug 11, 2026
…ated OFF (#382) Lands Filip Sajdak's work from PR #383 via #425. Merged --no-ff rather than squashed ON PURPOSE: a squash would re-author the commit to whoever pressed the button, and this is his change. `git log` keeps `author=Filip Sajdak` on the kernel commit. PagedAttentionDecodeOptKernel was reachable only at d == 32 * kDecEpl (256), so head_dim 128 -- what Qwen3-dense, Llama and Mistral actually use -- never reached it and fell to the generic block kernel. This adds the EPL=4 instantiation behind VT_ATTN_DECODE_D128, DEFAULT OFF. EPL == 8 still forwards through LoadRowN<8> to the existing LoadRow8, so the d256 path is byte-for-byte what it was. Gated OFF for an honest reason rather than caution: it is correctness-complete but NOT byte-exact against the block kernel, because the two reduce the KV sequence in a different ORDER, so a greedy anchor can move at an exact bf16 tie. Flipping it ON is a separate change owing the near-tie razor and the distributional gate, exactly as the FA2 decode GQA group-swap did in #48/#49. #383's CI red was never its own: the gencode audit failed on seven files, every one of them Marlin, while this change touches only cuda_paged_attn.cu. That was the drift #407 fixed, and the branch predated it by 46 commits. Its agent-record red was audit-live-rows reading transient record state on main; reproduced at the tip as rc=0 both before and after merging main. Carried in: issue #382 was never in the roadmap intake table, so only two of the three required links agreed. Filed under KERNEL-ATTN-PAGED, the row that anchors src/vt/cuda/cuda_paged_attn.cu. That commit is mine; the kernel commit is his and is untouched. Gate: scripts/agent-preflight.sh rc=0 on the merged tree. CI is queue-blocked (#274) -- every recent run is queued or cancelled -- so the operator's own gate run is the authority here, as AGENTS.md prescribes. The lever is default-OFF, so no golden moves. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports vLLM's FA2
seqlenq_ngroups_swappeddecode optimization intoLaunchDecodeVarlenFA2Bf16(gateVT_FA2_DECODE_GQA_SWAP, default OFF), the #47-localized MXFP4 c2-c8 lever. The Qwen3-dense head_dim-128 varlen decode now launches(batch, kv_heads)instead of(batch, hq), halving the CTA count at batch>=2 (the exact over-wave #47 measured). A 1:1 mirror of the already-shipped d256LaunchDecodeFA2Bf16swap (the vendored kernel's get_lse_tile/combine already honor the flag). OFF path byte-identical; ON is a split-reduction near-tie toward vLLM's numerics.Blast radius: the shared d128 varlen decode launcher (Qwen3-dense 0.6B/4B/8B + MXFP4-8B). 27B/35B gate models decode on their own d256 path, untouched.
Correctness (dgx GB10, sm_121a, CUDA 13.0):
test_ops_paged_attn5/5 varlen-d128 cases, 280/280 assn (both GQA ratios x batch{1,2,4,8} x short+long ctx;swap_launches==1proves the grid engaged; swap-vs-plain near-tie; MHA-inert). RED proven: wrong swapped stride -> 26528 violations. Full binary 28/28 / 454679 assn, no regression.Default stays OFF. The c1-c8 x3 binding re-bench + default flip is the recorded next step (parity-enablers). #47 projects flash alone closes ~28%@c2 / ~55%@c8 of the gap.
🤖 Generated with Claude Code