opencl: quant lm_head / decode GEMV and medium-batch GEMM optimizations (speculative decoding/MTP) - #26477
Draft
wanghqc wants to merge 14 commits into
Draft
opencl: quant lm_head / decode GEMV and medium-batch GEMM optimizations (speculative decoding/MTP)#26477wanghqc wants to merge 14 commits into
wanghqc wants to merge 14 commits into
Conversation
Adreno decode- and verify-path matmul optimizations for K-quant weights, plus two fusions and a multi-row f16 GEMV. lm_head / long-vocab GEMV: four-output-per-work-item q4_K and q6_K variants that share one activation read across four output rows, and a tiled-wide q4_K/q6_K decode GEMV for the long-vocab lm_head/embed, including a GPU path for the q6_K lm_head that previously fell back to the CPU. Spec/MTP verify: multi-column (N=2..4) verify GEMVs for q4_K/q6_K, so the draft-token verify lm_head runs one dispatch instead of N separate GEMVs. Decode GEMV bandwidth: split-K-across-workgroups q4_K GEMV for the small-M attention projections, with the K-split as a launch parameter capped by the kernel's real max workgroup size. At the original 4-wave split the reduction order is unchanged, so results are byte-identical. Medium-batch GEMM: a cooperative-K q4_K/q6_K GEMM for the verify batch regime that stages the activation in local memory. Fusions (default on, opt-out env): FFN gate+up q4_K GEMV with the GLU in one decode dispatch, and rms_norm + mul(weight) + add(residual) in one kernel. Both are byte-identical to the unfused path on the shapes they fire. Multi-row f16 decode GEMV: register-blocks several output rows per subgroup for the f16 attention projections and lm_head, cutting activation re-reads. Each optimization has a GGML_OPENCL_* opt-out or opt-in. test-backend-ops MUL_MAT and MUL_MAT_ID report 0 FAIL on the affected quant types.
…Adreno build
The kernel_convert_block_q{4,6}_k_tiled_ns members are declared under
GGML_OPENCL_USE_ADRENO_KERNELS but were registered unconditionally, so the
non-Adreno build failed to compile. Guard the registration to match the
declaration and the use sites.
ggml_cl_mul_mat_q4_k_glu_fused is Adreno-only and aborts on the non-Adreno
path, but the fusion dispatch was ungated, so on other drivers the FFN
{MUL_MAT, MUL_MAT, GLU} subgraph fired and aborted at the first decode token.
Gate the dispatch with GGML_OPENCL_USE_ADRENO_KERNELS to match the executor.
kernel_gemv_noshuffle_q4_k_f32_glu reads the noshuffle image layout, which is only produced at set_tensor time when use_adreno_kernels() accepts the weight (ne0 >= 512 && ne1 >= 512). Below that threshold the weight stays in the plain q4_K layout and the fused kernel misreads it. Require use_adreno_kernels() on both projections, mirroring the existing use_q4k_tiled() carve-out. Latent rather than shipping corruption: real FFN gate/up weights are far above 512x512. test-backend-ops MUL_MAT_VEC_FUSION 140/140 on Adreno X2-90 and X1-85.
… row stride The multi-row f16 decode GEMV casts the src0 row pointer to half4, and to half8 in the register-blocked variants. The dispatch gate only requires ne00 % 4 == 0, which constrains the elements in a row, not the byte stride between rows: nb01/nb02/nb03 and the view offset of a strided src0 are unconstrained, so a row can land 2-byte aligned and the vector load is then undefined behaviour. kernel_mul_mat_f16_f32_mrow now checks its own row pointer and falls back to the scalar loop. The register-blocked variants have no scalar path, so the host selects them only when the offset and every row stride are 8- or 16-byte aligned. No functional or performance change on any shape dispatched today: every mrow shape in test-backend-ops is already aligned, so the guard is a no-op there. MUL_MAT 925/0 on the Adreno X2-90, unchanged.
…r call sites
Upstream added a stride parameter to get_scale_min_k4 so it can read a
row-interleaved scales layout. The kernels this branch adds use a row-major
contiguous layout, so they pass stride = 1, which is identical arithmetic to
the previous 7-argument form.
Without this the OpenCL program fails to build at runtime ("too few arguments
to function call, expected 8, have 7") and the whole backend drops out. The
host build cannot catch it: embedded .cl sources are compiled by the driver on
device, so only test-backend-ops on real hardware surfaces it.
Adreno 840: 8189/8189, 0 FAIL, where the backend previously failed to
initialise.
… to win Split-K buys GPU time by spending an extra kernel launch: every eligible GEMV also enqueues a reduce over the per-slice partials. Launch cost is a per-device property, and the +3.36% the gate was built on does not travel. Measured tg32: Adreno X2-90 +3.36% gemma-4 E4B Adreno 840 -1.3% Qwen3.5-4B-Q4_K_M Adreno 850 -20.0% Qwen3-1.7B-Q4_K_M The kernel is not at fault. On the 850 split-K makes the GPU strictly faster - this GEMV drops from 43.7 to 34.0 us/call - and still costs a fifth of decode, because each added reduce dispatch runs ~550 us of host round-trip against 2.7 us of GPU work. Default on for X2E only, where it is measured; X1E is unmeasured and therefore excluded. GGML_OPENCL_Q4K_GEMV_SPLITK forces either way. test-backend-ops: Adreno 840 8189/8189 and Adreno 850 MUL_MAT 926/926, both 0 FAIL.
…med) The previous commit excluded X1E from the q4_K split-K default as unmeasured and invited widening once a datapoint existed. It now exists and it says do not widen: Adreno X1-85, Qwen3.5-4B-Q4_K_M, tg32, -0.7% - neutral to slightly negative, the same direction as the Adreno 840. Comment only, no behaviour change. Recorded in the source because the previous wording actively invited someone to widen the gate.
The 64-row tiled q4_K/q6_K lm_head layout regresses the A7X hard, so the
default now keys on the generations with measured evidence instead of being
unconditional. GGML_OPENCL_{Q4K,Q6K}_GEMV_TILED forces either way; the convert
and every dispatch site read the same predicate.
The set_tensor convert stores q4_K scales row-interleaved and the base GEMV/GEMM kernels read that layout, but the splitk, mc3, glu-fused, cok, r1, kimg and o4 variants still addressed scales as row-major contiguous with stride 1. Each computed wrong values on any multi-superblock K, on every device, whenever it was dispatched; the op-test suite never reaches these variants, so only end-to-end decode showed it. Address scales as sb*12*M + row with stride M in all seven kernels.
The large-m escape routes vocab-scale q4_K/q6_K lm_head weights to the flat
GEMV instead of the noshuffle GEMV. Its rationale, that gemv_noshuffle drops
for large M, predates the o4 kernel added earlier in this series, which now
serves the same long-vocab shapes and is faster. Measured on Qwen3-4B-Q4_K_M,
q6_K lm_head 151936x2560, tg128, flat vs o4:
Adreno 840 -10.3%
Adreno X2-90 -3.7%
Park the escape behind GGML_OPENCL_FLAT_LARGE_M=1 for both types and let
large-m weights take the noshuffle GEMV, which is upstream's routing for these
shapes.
The o4 lm_head/embed GEMVs dispatch ceil(ne01/4 / 64)*64 quads, so any vocab with ne01 % 256 == 128 spawned 32 tail quads whose unguarded vstore4 wrote 128 rows past dst - test-backend-ops trips its dst sentinel wherever the o4 path is the default (ne01 >= 32768). Guard the store; the tail lanes still run the loop so the subgroup broadcasts stay uniform.
get_tensor has a restore path for every other Adreno weight layout (noshuffle,
trans4_ns) but none for the tiled one, so reading a tiled weight back returned
tiled bytes reinterpreted as block_q4_K/q6_K. test-backend-ops builds its CPU
reference by copying the weights out of the backend, so it compared a correct
GPU result against a reference computed from garbage, and the tiled GEMVs
looked like they miscomputed at multi-superblock K.
Measured on the Adreno 840 with the tiled kernels on, before this change: the
convert writes exactly the layout it documents, the GEMV reproduces the ggml
reference on all 32768 rows, and dst is correct at end of graph, while a
read-back of the weight differs from the source in 9254238 of 9437184 bytes.
Add the inverse pack for both types. Also require use_adreno_kernels in
use_q4k_tiled/use_q6k_tiled: the convert gate only tested ne01, so a weight the
GEMV would decline could still be converted to the tiled layout and then read
by the generic kernel as plain SOA.
With the read-back correct, the X2E/A8X default set earlier in this series
stands on its own measurements. Qwen3-4B-Q4_K_M (q6_K lm_head 151936x2560),
tg128, tiled vs o4:
Adreno 840 +11.9% (16.93 vs 15.14 tok/s)
Adreno X2-90 +6.9% (35.24 vs 32.87 tok/s)
Measure this one on a cold device: these kernels are far more clock-sensitive
than the o4 route they replace, and on a heat-soaked Adreno 840 the ranking
inverts to a 4-5% loss. The A7X regresses on this layout and is unaffected.
test-backend-ops -o MUL_MAT, this commit against the same binary with the
tiled kernels forced off: Adreno 840 919/919 both ways, Adreno X2-90 918 OK /
0 FAIL both ways. Greedy output byte-identical to the tiled-off arm on both.
github-actions
Bot
added
ggml
kernel_gemv_noshuffle_q8_0_f32 puts one output row per lane and splits K only across the subgroups of a single workgroup, so M is the sole source of workgroup parallelism. gpt-oss-20b's K and V projections are M=512 = 8 workgroups on a 16-CU X2: half the device idle, and the kernel measures 48 GB/s where the M=2880/4096 projections in the same decode graph reach 122-123. Add a split-K variant mirroring the q4_0/q4_K ones already here -- each (kslice, subgroup) pair reduces a disjoint set of K-blocks into a per-slice partial, reusing kernel_gemv_splitk_reduce_f32 to sum them. Gated to M<=1024 (larger M already fills the compute units and the extra reduce dispatch only costs). Enabled where it is measured to win, matching the q4_K split-K gate directly above: X2-90 +2.8% tg32 @D4096 on gpt-oss; Adreno 840 (12 CU) NEUTRAL on Llama-3.2-3B-Q8_0 (+0.5% @d0, 0.0% @D4096 -- its K/V projection is M=1024 = 16 workgroups, which already fills 12 CUs, so there is no under-fill to recover). Unmeasured on X1E/A7X/A6X, and the q4_K split-K measured -0.7% on X1E, so the default is not widened on absence of evidence. GGML_OPENCL_Q8_GEMV_SPLITK forces either way. The new kernel is placed BEFORE the base kernel in the .cl: on A6X no kernel may be defined after one that uses a subgroup builtin. Verified: test-backend-ops MUL_MAT 918 OK / 0 FAIL, greedy output byte-identical with base-twice and fix-twice controls.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR has comprehensive decode-side optimizations on K-quant dense models for Adreno GPUs, including optimization for skinny GEMM that are essential for speculative decoding and MTP (multi-token prediction):
Fleet defaults vs upstream, Adreno 840 (A8X), Qwen3-4B-Q4_K_M:
Per switch, against the route each replaces:
Note the wins live in the decode GEMV and the medium-batch GEMM, not long prefill.
Additional information
Gating:
Which models see the gain.
Gated on a q4_K/q6_K lm_head/embed (tiled additionally needs vocab ≥ 32768, ne00 ≥ 512, and for q6_K vocab % 128 == 0; the cooperative-K GEMM needs a q4_K GEMM at n_q ∈ [2..8]).
Scanned over a 108-GGUF local model set: 65 eligible, 43 not.
-Q8_0build (24 — the lm_head stays Q8_0);Intel NEO GPU
RMS_NORM_MUL_ROPE8/72 → 72/72.Requirements