gfx1151 (Strix Halo): fuse attn_k+v into single MMVQ dispatch

[jeffli-xilinx]

Summary

Fuse attn_k + attn_v weight matrices into a single MMVQ dispatch on Strix Halo (gfx1151) to improve wave occupancy during decode.

• Concatenates wk and wv weights into a single wkv_concat tensor at model load, so the graph emits one MUL_MAT instead of two separate attn_k/attn_v dispatches
• On gfx1151 (40 CUs = 80 SIMDs, max 16 waves/SIMD), this doubles the MMVQ row count from N=1024 to N=2048, increasing wave occupancy from 12.8 to 16.0 waves/SIMD (full occupancy within the 24-VGPR budget)
• Gated to gfx1151 only via runtime CC detection (ggml_backend_reg_get_proc_address)
• No kernel changes — the MMVQ kernel sees a normal matrix with more rows

Effective bandwidth (rocprofv3 PMC FETCH_SIZE, Radeon 8060S LPDDR5X 230 GB/s peak)

Kernel	N (separate)	BW (separate)	N (fused)	BW (fused)	BW gain
Q4_K attn_k/v	1024	149-178 GB/s (65-77%)	2048	160-184 GB/s (70-80%)	+3-7%
Q6_K attn_k/v	1024	175-201 GB/s (76-87%)	2048	210-223 GB/s (91-97%)	+6-24%

End-to-end throughput (llama-bench tg128, gfx1151)

Model	Baseline (t/s)	KV Fusion (t/s)	Change
Qwen2.5-0.5B Q4_K_M	303.5	309.7	+2.0%
Qwen3-1.7B Q4_K_M	140.3	141.8	+1.0%
Qwen3-4B Q4_K_M	71.2	71.9	+1.0%
Qwen2.5-7B Q4_K_M	45.8	45.9	+0.2%
Qwen3-8B Q4_K_M	41.5	41.5	0.0%
Qwen3.5-9B Q4_0	39.8	39.6	-0.5%

Small models (0.5B-4B) where attn K/V dispatches are a larger fraction of total decode time see +1-2% improvement. Large models (7B+) are flat — expected since FFN matmuls dominate.

Implementation

1. src/llama-model.h: Add wkv_concat field to llama_layer
2. src/llama-model.cpp: Post-load weight concatenation in load_tensors():
   • Runtime CC detection via ggml_backend_reg_get_proc_address("ggml_backend_cuda_get_device_cc") — checks first layer only (single-GPU assumption for Strix Halo iGPU)
   • Copies wk then wv bytes into a single tensor on the same backend buffer type
   • Guards: same type, same ne[0], same buffer, no existing wqkv fusion
   • Logs LLAMA_LOG_INFO when fusion activates
3. src/llama-graph.cpp: New fused K+V branch in build_qkv():
   • No bias/clamp models (Qwen3): Uses ggml_view_3d directly (zero-copy split, same pattern as existing wqkv)
   • Bias models (Qwen2.5): Uses ggml_view_2d → ggml_cont → ggml_add(bias) → ggml_reshape_3d
4. Guarded by loras->empty() && !layer.wk_s && !layer.wv_s — falls back to separate path for LoRA and NVFP4

Memory overhead

~2× K+V weight size per layer (e.g. ~118 MB for Qwen3-4B Q4_K). On Strix Halo's unified memory (31 GiB) this is acceptable. Follow-up could skip loading original wk/wv when fusion is active (harder due to LoRA/NVFP4 fallback).

Test plan

• All 6 models run successfully under llama-bench (Q4_K_M and Q4_0)
• Models with biases (Qwen2.5-0.5B, Qwen2.5-7B) work correctly
• Models without biases (Qwen3 family) work correctly
• Text generation produces coherent output (llama-cli smoke test)
• No regression on large models
• Full 42-test regression suite passed (28 LLM + 14 VLM)
• Verify LoRA fallback path (LoRA active → uses separate K/V)

🤖 Generated with Claude Code

[liangliangchang]

LGTM

[liangliangchang]

Summary

Solid gfx1151-targeted optimization: concatenating wk+wv at load time turns two decode MMVQ dispatches (N=1024) into one (N=2048), improving wave occupancy on Strix Halo. Graph changes mirror the existing wqkv pattern and benchmarks (+1-2% on small models) support the approach.

Verdict: Approve with minor revisions.

---

What looks good

1. Occupancy rationale is credible for gfx1151 (40 CUs / 80 SIMDs). Two N=1024 MMVQ dispatches sit at ~12.8 waves/SIMD; one N=2048 dispatch reaches ~16.0.

2. Zero-copy ggml_view_3d split matches the fused QKV layout math. Bias/clamp branch correctly uses cont before add/clamp.

3. Safe fallbacks: LoRA active, NVFP4 scales (wk_s/wv_s), and wqkv models skipped.

4. Device gating correctly narrowed to gfx1151 only (Strix Halo), not all RDNA 3.5.

5. get_proc_address for device CC avoids coupling llama-model.cpp to ggml-cuda.h.

---

Issues and suggestions

1. PR title/description are stale (minor)

Latest code gates on gfx1151 only, not all RDNA 3.5:

constexpr int cc_gfx1151 = 0x1000000 + 0x1151;
fuse_kv = (cc == cc_gfx1151);

Update title/body to say "Strix Halo (gfx1151)".

2. PR claims "No kernel changes" — inaccurate (minor)

Includes substantial mmvq.cu changes (has_gate template param + MMVQ_PARAMETERS_RDNA3_5 table split). Reasonable companion fix — bias-only fusion no longer pays gate VGPR/shared-mem cost (VGPR 40→24 on gfx1151) — but should be documented in the PR body.

3. Missing buffer consistency check (medium)

Concat allocation uses wk's buffer type, but there is no check that wv lives on the same buffer/device:

if (layer.wk->type != layer.wv->type)       continue;
if (layer.wk->ne[0] != layer.wv->ne[0])     continue;
// missing: wv->buffer == wk->buffer

With layer splitting or tensor overrides, this could produce silently wrong results. Suggest adding:

if (!layer.wv->buffer || layer.wv->buffer != layer.wk->buffer) continue;

4. VRAM duplication is permanent (medium)

wkv_concat is allocated in addition to existing wk/wv tensors, which remain loaded. For Qwen3-4B Q4_K that is ~118 MB extra per the PR. On Strix Halo unified memory this may be fine, but:

• Document this prominently in the PR/merge notes
• Consider a follow-up to skip loading wk/wv when fusion is active (harder because of LoRA/NVFP4 fallback)
• Log at LLAMA_LOG_INFO when fusion activates so users know why VRAM jumped

5. Device detection uses only the first wk layer (low)

The CC check loops layers but breaks after the first wk buffer. Fine for single-GPU Strix Halo; add a comment explaining the assumption for multi-GPU edge cases.

6. Bias-model path adds cont copies (low)

For Qwen2.5 (wk_b/wv_b), fused path does one MMVQ + two ggml_cont + two bias adds. Still a net win (+2% on 0.5B), but zero-copy path only applies to no-bias models (Qwen3).

7. LoRA fallback untested (low)

Test plan item "Verify LoRA fallback path" is still unchecked. Quick smoke test with active LoRA recommended.

8. has_gate mmvq refactor affects all RDNA3.5 (low)

Splitting RDNA3.5 out of RDNA2 parameter table affects gfx1150/gfx1153 too, not just gfx1151. Confirm no regression there even though KV fusion is gfx1151-only.

---

Correctness

Primary case (Qwen3, no bias, no clamp):

Q  = wq @ cur
kv = wkv_concat @ cur          (single MMVQ, N = n_embd_k + n_embd_v)
K  = view_3d(kv, offset=0)
V  = view_3d(kv, offset=n_embd_kv)

Layout matches wqkv path. GQA OK when wk.ne[1] == wv.ne[1] == n_embd_kv.

Qwen2.5 (bias): single fused MMVQ, then split + cont + per-head bias. Semantically equivalent to separate path.

---

Pre-merge checklist

• Update PR title/body for gfx1151-only scope
• Document mmvq.cu changes
• Add wv->buffer == wk->buffer guard
• Add LLAMA_LOG_INFO when fusion activates
• LoRA smoke test
• Confirm mmvq table split on gfx1150/1153

---

Final verdict

Approve with minor revisions. Well-targeted for Strix Halo decode. Main concerns: VRAM overhead (expected tradeoff), missing same-buffer guard, PR docs don't match gfx1151-only scope or mmvq kernel changes.

[jeffli-xilinx]

Summary

Solid gfx1151-targeted optimization: concatenating wk+wv at load time turns two decode MMVQ dispatches (N=1024) into one (N=2048), improving wave occupancy on Strix Halo. Graph changes mirror the existing wqkv pattern and benchmarks (+1-2% on small models) support the approach.

Verdict: Approve with minor revisions.

What looks good

1. Occupancy rationale is credible for gfx1151 (40 CUs / 80 SIMDs). Two N=1024 MMVQ dispatches sit at ~12.8 waves/SIMD; one N=2048 dispatch reaches ~16.0.
2. Zero-copy ggml_view_3d split matches the fused QKV layout math. Bias/clamp branch correctly uses cont before add/clamp.
3. Safe fallbacks: LoRA active, NVFP4 scales (wk_s/wv_s), and wqkv models skipped.
4. Device gating correctly narrowed to gfx1151 only (Strix Halo), not all RDNA 3.5.
5. get_proc_address for device CC avoids coupling llama-model.cpp to ggml-cuda.h.

Issues and suggestions

1. PR title/description are stale (minor)

Latest code gates on gfx1151 only, not all RDNA 3.5:

constexpr int cc_gfx1151 = 0x1000000 + 0x1151;
fuse_kv = (cc == cc_gfx1151);

Update title/body to say "Strix Halo (gfx1151)".

2. PR claims "No kernel changes" — inaccurate (minor)

Includes substantial mmvq.cu changes (has_gate template param + MMVQ_PARAMETERS_RDNA3_5 table split). Reasonable companion fix — bias-only fusion no longer pays gate VGPR/shared-mem cost (VGPR 40→24 on gfx1151) — but should be documented in the PR body.

3. Missing buffer consistency check (medium)

Concat allocation uses wk's buffer type, but there is no check that wv lives on the same buffer/device:

if (layer.wk->type != layer.wv->type)       continue;
if (layer.wk->ne[0] != layer.wv->ne[0])     continue;
// missing: wv->buffer == wk->buffer

With layer splitting or tensor overrides, this could produce silently wrong results. Suggest adding:

if (!layer.wv->buffer || layer.wv->buffer != layer.wk->buffer) continue;

4. VRAM duplication is permanent (medium)

wkv_concat is allocated in addition to existing wk/wv tensors, which remain loaded. For Qwen3-4B Q4_K that is ~118 MB extra per the PR. On Strix Halo unified memory this may be fine, but:

• Document this prominently in the PR/merge notes
• Consider a follow-up to skip loading wk/wv when fusion is active (harder because of LoRA/NVFP4 fallback)
• Log at LLAMA_LOG_INFO when fusion activates so users know why VRAM jumped

5. Device detection uses only the first wk layer (low)

The CC check loops layers but breaks after the first wk buffer. Fine for single-GPU Strix Halo; add a comment explaining the assumption for multi-GPU edge cases.

6. Bias-model path adds cont copies (low)

For Qwen2.5 (wk_b/wv_b), fused path does one MMVQ + two ggml_cont + two bias adds. Still a net win (+2% on 0.5B), but zero-copy path only applies to no-bias models (Qwen3).

7. LoRA fallback untested (low)

Test plan item "Verify LoRA fallback path" is still unchecked. Quick smoke test with active LoRA recommended.

8. has_gate mmvq refactor affects all RDNA3.5 (low)

Splitting RDNA3.5 out of RDNA2 parameter table affects gfx1150/gfx1153 too, not just gfx1151. Confirm no regression there even though KV fusion is gfx1151-only.

Correctness

Primary case (Qwen3, no bias, no clamp):

Q  = wq @ cur
kv = wkv_concat @ cur          (single MMVQ, N = n_embd_k + n_embd_v)
K  = view_3d(kv, offset=0)
V  = view_3d(kv, offset=n_embd_kv)

Layout matches wqkv path. GQA OK when wk.ne[1] == wv.ne[1] == n_embd_kv.

Qwen2.5 (bias): single fused MMVQ, then split + cont + per-head bias. Semantically equivalent to separate path.

Pre-merge checklist

• Update PR title/body for gfx1151-only scope
• Document mmvq.cu changes
• Add wv->buffer == wk->buffer guard
• Add LLAMA_LOG_INFO when fusion activates
• LoRA smoke test
• Confirm mmvq table split on gfx1150/1153

Final verdict

Approve with minor revisions. Well-targeted for Strix Halo decode. Main concerns: VRAM overhead (expected tradeoff), missing same-buffer guard, PR docs don't match gfx1151-only scope or mmvq kernel changes.

Addressed all comments in the latest commit.
has_gate change has been removed as it was not needed.

[jeffli-xilinx]

Regression Results — Strix Halo (gfx1151, xconucstrhalo40)

42 passed, 0 failed, 0 skipped on commit be84ef43 (rebased on latest gfx11, has_gate removed, review fixes applied).

Board: Strix Halo gfx1151, 40 CUs, 31 GiB unified memory, ROCm nightly 7.13.

LLM Results (28 tests)

Model	Context	Decode (t/s)	Prefill (t/s)	Sanity
Qwen2.5-0.5B-Instruct Q4_K_M	128	307.89	8946.62	2/2
Qwen2.5-0.5B-Instruct Q4_K_M	3968	281.65	9543.55	2/2
Qwen2.5-3B-Instruct Q4_K_M	128	87.11	2707.65	2/2
Qwen2.5-3B-Instruct Q4_K_M	3968	82.19	2583.80	2/2
Qwen2.5-7B-Instruct Q4_K_M	128	45.61	1485.21	2/2
Qwen2.5-7B-Instruct Q4_K_M	3968	42.76	1282.87	2/2
Qwen3-1.7B Q4_K_M	128	141.03	4504.91	2/2
Qwen3-1.7B Q4_K_M	3968	107.57	3385.41	2/2
Qwen3-4B Q4_K_M	128	71.35	2092.82	2/2
Qwen3-4B Q4_K_M	3968	59.63	1650.56	2/2
Qwen3-8B Q4_K_M	128	41.48	1268.21	2/2
Qwen3-8B Q4_K_M	3968	37.27	1105.81	2/2
Qwen3.5-4B Q4_K_M	128	60.40	1913.52	2/2
Qwen3.5-9B Q4_K_M	128	36.47	1137.62	2/2
Llama-2-7B Q4_K_M	128	46.42	1327.48	2/2
Llama-2-7B Q4_K_M	1920	34.57	1161.17	2/2
Gemma-2B Q4_K_M	128	111.85	3800.51	2/2
Gemma-2B Q4_K_M	8000	104.00	2286.61	2/2
SmolLM2-1.7B-Instruct Q4_K_M	128	145.78	4187.91	2/2
SmolLM2-1.7B-Instruct Q4_K_M	8000	43.37	1191.34	2/2
Llama-3.1-8B-Instruct Q4_K_M	128	42.50	1251.09	2/2
Qwen3-30B-A3B-Instruct-2507 Q4_K_M	128	75.18	1168.49	2/2
Gemma-4-E2B-IT Q4_K_M	128	95.80	2995.78	0/2
Gemma-4-E2B-IT Q4_K_M	3968	90.49	2659.72	0/2
Qwen3.5-35B-A3B Q4_K_M	128	54.85	1164.06	2/2
Qwen3.6-35B-A3B Q4_K_M	128	55.09	1103.41	2/2
Qwen3.6-35B-A3B Q4_K_M	4096	54.03	1354.76	2/2
Qwen3.6-27B Q4_K_M	4096	12.15	351.16	2/2
GLM-4.7-Flash Q4_K_M	128	55.75	986.27	2/2
Gemma-4-12B-it Q4_K_M	128	26.73	855.56	2/2
Gemma-3-12B-IT Q4_K_M	4096	23.72	776.69	2/2

VLM Results (11 tests)

Model	Decode (t/s)	Prefill (t/s)	Sanity
Gemma-3-4B-IT	66.48	533.24	1/1
Qwen2.5-VL-3B-Instruct	82.47	244.28	1/1
Qwen2.5-VL-7B-Instruct	43.81	205.06	1/1
Qwen3-VL-4B-Instruct	65.34	605.71	1/1
Gemma-4-26B-A4B-IT	46.09	755.82	1/1
Gemma-4-31B-IT	10.40	293.46	1/1
MiniCPM-V-2_6	44.39	357.82	1/1
MiniCPM-o-2_6	44.40	356.88	1/1
Janus-Pro-1B	147.05	3238.28	1/1
Janus-Pro-7B	43.32	1066.31	1/1
Gemma-3-12B-IT (VLM)	25.47	78.37	1/1

Notes

• Gemma-4-E2B-IT sanity check failure is a pre-existing chat template issue (empty completions), not related to this PR — benchmarks run fine.
• GGML_CUDA_DISABLE_GRAPHS=1 required on this board; ROCBLAS_USE_HIPBLASLT=1 set per regression defaults.
• Env: ROCm nightly 7.13, llama_models_rocm.yaml defaults (10 reps, -fa 1 --poll 50 -ctk f16 -ctv f16).

[roberteg16]

AFAIK, llama.cpp folks want the graph level code backend agnostic

[jeffli-xilinx]

Good point. The graph-level code (llama-graph.cpp) is backend-agnostic — the wkv_concat branch emits standard ggml_mul_mat + ggml_view ops, no CUDA-specific calls. Any backend that handles MUL_MAT benefits from the reduced dispatch count.

The backend-specific part is limited to llama-model.cpp: the CC detection via ggml_backend_reg_get_proc_address("ggml_backend_cuda_get_device_cc") that gates whether to create wkv_concat at load time. This keeps the CUDA/HIP coupling to a runtime proc-address lookup rather than a compile-time dependency — same pattern used elsewhere in the codebase for backend capability probing.

If this moves upstream, the CC gate could be generalized (e.g., a backend hint for "supports fused KV") to avoid the CUDA-specific string.

[mgehre-amd]

I wouldn't make the optimization depend on available memory. This can make the performance (which depends on the optimization) unpredictable.