cuda: extract Q2_0 elements via __byte_perm - #25603
Conversation
github-actions
Bot
added
testing
|
@khosravipasha how is the cuda pr coming along? |
|
Mostly ready. Waiting for metal to merge as policy is to have one PR open at a time. Can send ours as draft PR here instead of waiting. This is the bracnh planning to submit https://github.com/PrismML-Eng/llama.cpp/tree/pr/q2_0-cuda Interesting speed up, need to take a closer look tomorrow. |
|
The CUDA changes should be scheduled after #24127 . |
|
I've been interested in running the Q2_0 version of Bonsai-27b, so I rebased onto current master. I'm planning to have this PR wait for @khosravipasha's PR as well as #25628. test-backend-ops perfreference unpacking (c3a6223): this PR (16a019d): llama-batched-benchmerged output of reference unpacking (c3a6223):
this PR (16a019d):
|
|
@dfriehs Was this rebased after our latest branch? If yes let me do some testing on my side today, I am probably okay with just merging this one too, was there any big change when doing the rebase? |
No, it's still the commit from the prism branch as before. As far as I could tell, there was no difference between that commit and the PR branch except for fixing
All the changes in
I don't mind either way, it's your call. One PR might be less work for reviewers, so if you don't mind I'd err on that side. Can't have two PRs out of draft, however, so it might need to wait. |
|
Sorry, if you meant this as in
then yes, this is post mmq refactor (rebased on |
Q2_0 (ternary) GPU kernels for CUDA and Vulkan, cherry-picked from the upstream-targeted PrismML PR branches (ggml-org#25603 cuda, ggml-org#25430 vulkan). Upstream already carries the Q2_0 type + CPU backend (ggml-org#24448) and the qwen35 arch, so these kernels are the only gap for running Ternary-Bonsai-27B (Q2_g64).
|
I see thanks, I meant both, I did the rebasing and did a bit more test. Wanted to add some speed numbers, and also pointin people to the the right gguf models. |
dfriehs
changed the title
|
@khosravipasha, if you don't mind I'd appreciate some tests on your side again. The changes should have only touched the integer unpacking so I expect the logits to match exactly, and KLD between 9b2a088 and a487841 is pretty much just noise for me: For the benchmarks I still see the same improvements as above, around +15-40% tg, +8% pp on a RTX 3090. I'm planning to have the PR leave draft once #26264 is closed. |
|
@dfriehs Thanks for adding this and rebeasing, got some solid speed ups on L40S, should be similar on other GPUs and accurary is good, more details below ========
Commands# build before/after
git worktree add --detach bp-before 9b2a08881
git worktree add --detach bp-after a4878413e
for d in bp-before bp-after; do
cmake -S $d -B $d/build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89 -DCMAKE_BUILD_TYPE=Release -G Ninja
cmake --build $d/build -j --target llama-bench llama-perplexity
done
# speed
./bp-before/build/bin/llama-bench -m <model>.gguf -ngl 99 -fa 1 -p 512 -n 128 -r 3
./bp-after/build/bin/llama-bench -m <model>.gguf -ngl 99 -fa 1 -p 512 -n 128 -r 3
# KL (before vs after)
./bp-before/build/bin/llama-perplexity -m <model>.gguf -f wiki.test.raw -c 512 --chunks 20 -ngl 99 \
--save-all-logits before.bin
./bp-after/build/bin/llama-perplexity -m <model>.gguf -f wiki.test.raw -c 512 --chunks 20 -ngl 99 \
--kl-divergence --kl-divergence-base before.binSpeed (t/s)
KL divergence (before vs after)
|
|
Should be a simple merge now. 🚀
before
after
|
am17an
added
the
merge ready
Port of ggml-org#25603 (upstream commit 15e755f) adapted to this fork's group-size-128 Q2_0. Upstream's tile-load lives in mmq-load-tiles.cuh after their MMQ refactor; ours is still in mmq.cuh, so this is a manual port rather than a cherry-pick. The technique is group-size agnostic: it consumes 4 int16 (32 codes) per Q8_1 chunk, and that chunk size comes from Q8_1's 32-element block rather than from QK2_0, so the iqs*4 indexing is unchanged for g128. The decode path previously summed unsigned codes and applied a deferred -sum(act) correction; __byte_perm yields signed symbols directly, so the tail becomes d2*d8*sumi. Both reduce to d2*d8*sum((c-1)*q8).
Squashed overlay of the custom-hardware fork (45 files, +3255/-11) on top of upstream master: - MoE expert cache: GPU-accelerated LRU cache for CPU-resident MoE experts (ggml-cuda/moe-cache.cu, backend hooks, fit placement, --moe-cache flag), toggleable via -DGGML_MOE_CACHE - maple arch + TQ ternary quants + Q2_0_128 type (ggml + CUDA kernels, GGUF type-42 remap), toggleable via -DGGML_MAPLE - q2_0/q4_0 vecdot byte_perm port (upstream ggml-org#25603) - server reasoning_effort -> thinking_budget mapping - ROCm bring-up fixes: HIP stream-priority/popcnt portability, clang prefetch casts, loader mmap-bypass reverted (caused GPU load hang)
Squashed overlay of the custom-hardware fork (45 files, +3255/-11) on top of upstream master: - MoE expert cache: GPU-accelerated LRU cache for CPU-resident MoE experts (ggml-cuda/moe-cache.cu, backend hooks, fit placement, --moe-cache flag), toggleable via -DGGML_MOE_CACHE - maple arch + TQ ternary quants + Q2_0_128 type (ggml + CUDA kernels, GGUF type-42 remap), toggleable via -DGGML_MAPLE - q2_0/q4_0 vecdot byte_perm port (upstream ggml-org#25603) - server reasoning_effort -> thinking_budget mapping - ROCm bring-up fixes: HIP stream-priority/popcnt portability, clang prefetch casts, loader mmap-bypass reverted (caused GPU load hang)
6 participants
Overview
Unpack Q2_0 elements via __byte_perm, leading to quite a substantial boost in t/s especially for single decode (+15-40% tg, +8% pp).
Additional information
test-backend-ops testpasses before and with d19075f, and KL divergence is 0 between the two. I have not tested KL divergence against CPU as the CPU pass would take 6+ hours on my machine.I'm not able to test HIP/ROCm or MUSA. If either don't support
__byte_perm, I will add a fallback path.test-backend-ops perf
before d19075f:
with d19075f:
llama-batched-bench
merged output of
llama-batched-bench -no-kvu -ngl all -fit off -ub 1024 -npp 0,256,4096 -ntg 256 -npl 1,2,4,8 -m Ternary-Bonsai-8B-Q2_0_g64.ggufand
llama-batched-bench -no-kvu -ngl all -fit off -ub 1024 -npp 16384,32768 -ntg 256 -npl 1 -m Ternary-Bonsai-8B-Q2_0_g64.ggufbefore d19075f:
with d19075f:
Requirements