fix(gemv): instantiate mmvq_q4k_rows for K=6656 (Muse Glimmer) - #879
Closed
Tet-9 wants to merge 1 commit into
Closed
fix(gemv): instantiate mmvq_q4k_rows for K=6656 (Muse Glimmer)#879Tet-9 wants to merge 1 commit into
Tet-9 wants to merge 1 commit into
Conversation
The batched-row Q4_K launcher (launch_mmvq_q4k_rows / launch_mmvq_rows_f32) already supported K=2048/4096/5120/6144 but not K=6656 (Muse Glimmer's hidden size), even though the single-row launch_mmvq_q4k already handles K=6656 via si_mmvq_q4k_kfixed_kernel<__nv_bfloat16, 26>. Any batched verify-path call (e.g. speculative decode) against Muse Glimmer at M>1 silently fell back or declined. Adds NSUPER=26 instantiations of si_mmvq_q4k_rows_exact_kernel and extends both dispatch functions' K guards and branches to cover 6656. Verified on RTX 5090: batched output byte-identical to N serial single-row calls, 1.79x faster than the serial fallback at K=6656 (runtime/examples/mmvq_q4k_rows_6656_check.cpp).
Closed — RTX 5090 checkbox not tickedThis PR touches To submit for review:
Non-speed runtime fixes that should not be evaluated need a maintainer Automated by rtx5090-required CI. Maintainers and |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The batched-row Q4_K launchers (
launch_mmvq_q4k_rows/launch_mmvq_rows_f32inkernels/csrc/cuda/gemm/gemv.cu) already supported K=2048/4096/5120/6144 but notK=6656 — Muse Glimmer's hidden size — even though the single-row
launch_mmvq_q4kalready handles K=6656 via
si_mmvq_q4k_kfixed_kernel<__nv_bfloat16, 26>.Any batched verify-path call (e.g. speculative decode, M>1) against Muse Glimmer
hit the unsupported-K guard and either silently declined or fell back to N
serial single-row calls instead of the fused batched path.
Change
si_mmvq_q4k_rows_exact_kernelfor
{__nv_bfloat16, float} × {MMAX=6, MMAX=8}, matching the existingpattern for K=2048/4096.
launch_mmvq_q4k_rowsto coverK=6656.
launch_mmvq_rows_f32already had a K=6656 guard entry but its fallbackdispatch macro silently mapped any unlisted K (including 6656) to KB=24 —
fixed to dispatch KB=26 explicitly for K=6656.
runtime/examples/mmvq_q4k_rows_6656_check.cpp, isolating this kernel atK=6656 from the model loader and KV cache (same pattern as
nvfp4_gemm_check.cpp).Verification
Tested on RTX 5090 (sm_120):
launch_mmvq_q4kcalls at K=6656.
for N=4 rows, MN=8192).
K=6656 rows: batch 0.0312 ms, 4 serial calls 0.0558 ms, 1.79x
[PASS] K=6656 batched-row Q4_K matches serial reference
Also re-ran the existing
dflash_gdn_checkpoint_gpu_test(K=2048/4096 paths)to confirm no regression — still passes.