CUDA: add Q2_0 support - #25707
Merged
Merged
Conversation
github-actions
Bot
added
testing
|
@khosravipasha need help on this ? |
Contributor
Author
|
@cfournel Thanks, this is mostly good for initial PR I think. Mostly needs a review. |
|
@khosravipasha |
khosravipasha
force-pushed
the
pr/q2_0-cuda
branch
from
July 22, 2026 23:46
5eec798 to
d33aa4d
Compare
Contributor
Author
|
@mata-p thanks for the heads up and testing. |
|
Thanks! |
am17an
approved these changes
Jul 23, 2026
am17an
left a comment
am17an
left a comment
Contributor
There was a problem hiding this comment.
Looks ok to me from static analysis.
Collaborator
|
This pr works great with sd.cpp and their ternary flux.2 klein 4b quant. 
edit: @khosravipasha did you know that unlike the binary, the ternary version preserves the editing capabilities for the model? |
Contributor
Author
|
@Green-Sky cute cat :) |
dfriehs
reviewed
Jul 26, 2026
|
|
||
| #define DECL_MMQ_CASE(type) \ | ||
| template void mul_mat_q_case<type>(ggml_backend_cuda_context & ctx, const mmq_args & args, cudaStream_t stream) \ | ||
|
|
Contributor
There was a problem hiding this comment.
I think Q2_0 needs a declaration here as well, like Q1_0 got in #25778
Contributor
Author
There was a problem hiding this comment.
added it I believe, and rebased with new master
khosravipasha
force-pushed
the
pr/q2_0-cuda
branch
from
July 29, 2026 17:27
d33aa4d to
01fa7d0
Compare
Contributor
Author
am17an
added
the
merge ready
|
is the provided dpsark drafter in the repo supposed to work in this PR? and the following router-mode ini config (abbreviated) Load logs |
Contributor
Author
|
@MaxKruse dspark drafter still experimental so will need our fork in near future. We will be doing a major refactor our of our fork after this PR is merged so will see how best to support that in mainline, for now need to wait until llama.cpp's dspark version gets merged and stable, our implemention is very experimental and not ready for being upstreamed here. |
epoyraz
added a commit
to epoyraz/Bonsai-demo
that referenced
this pull request
Jul 31, 2026
The Q2_0 CUDA merge covers normal decoding only. The paired *dspark-Q4_1*.gguf drafter does not load on mainline llama.cpp, which rejects it with "tensor 'dspark.fc.weight' has offset ..., expected ..."; a user hit exactly this on ggml-org/llama.cpp#25707 while testing the merged CUDA backend. Per the upstream reply on that thread, the dspark implementation is experimental, not ready to be upstreamed, and waiting on llama.cpp's own dspark support to merge and stabilize. Without this the preceding "no fork needed" line reads as if the fork is now optional everywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kashif
pushed a commit
to kashif/llama.cpp
that referenced
this pull request
Aug 2, 2026
huaxel
pushed a commit
to huaxel/CachyLLama
that referenced
this pull request
Aug 2, 2026
smalinin
pushed a commit
to smalinin/llama.cpp
that referenced
this pull request
Aug 4, 2026
khosravipasha
added a commit
to PrismML-Eng/Bonsai-demo
that referenced
this pull request
Aug 5, 2026
* docs: mark Q2_0 CUDA backend as merged in mainline llama.cpp ggml-org/llama.cpp#25707 ("CUDA: add Q2_0 support") was merged upstream on 2026-07-30 (9b2a088), so the ternary migration table no longer needs the "in review" row. Also refreshes the surrounding prose that enumerated which backends run Q2_0 on mainline, which the merge (and the earlier Vulkan merge, #25430) had left stale: CPU, Metal, Vulkan, and CUDA now all run the group-64 files on stock llama.cpp. x86 AVX-512-VNNI remains the only outstanding item, and the demo still ships the fork binaries for the group-128 *-Q2_0.gguf files until the format migration completes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: note that the dspark drafter is still fork-only The Q2_0 CUDA merge covers normal decoding only. The paired *dspark-Q4_1*.gguf drafter does not load on mainline llama.cpp, which rejects it with "tensor 'dspark.fc.weight' has offset ..., expected ..."; a user hit exactly this on ggml-org/llama.cpp#25707 while testing the merged CUDA backend. Per the upstream reply on that thread, the dspark implementation is experimental, not ready to be upstreamed, and waiting on llama.cpp's own dspark support to merge and stabilize. Without this the preceding "no fork needed" line reads as if the fork is now optional everywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: tighten ternary upstream status; clarify PQ2_0 is reserved/experimental * docs: correct dspark upstream status — mainline has DSpark (#25173), our drafter fork-only due to packing (#26337) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Pasha Khosravi <khosravipasha@gmail.com>
satindergrewal
pushed a commit
to satindergrewal/llama.cpp
that referenced
this pull request
Aug 11, 2026
satindergrewal
pushed a commit
to satindergrewal/llama.cpp
that referenced
this pull request
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Follow up PR after the
Q2_0CPU (#24448) and Metal (#25419) backends. This PR adds the CUDA backend forQ2_0.Supported Models
Supported Models: Ternary-Bonsai-{27B, 8B, 4B, 1.7B}.
Note: In our huggingface repos ggufs ending with
_Q2_0_g64.ggufare packed for this PR;Q2_0.ggufones are the older group-128 packing from our fork. After all backends merge we will rename and replace with the official Q2_0 (group size 64).Correctness: F16 vs Q2_0 logits KL
To test for correctness we compare the logits from Q2_0 vs F16. Since the Ternary-Bonsai models are natively ternary and it packs perfectly into 2-bits (Q2_0), if we compre logits from Q2_0 vs F16 we should get a KL close to 0.
Speeds:
llama-benchCUDA (NVIDIA L40S)Q2_0 g64 vs g128; speed/size tradeoff (for the curious)
The g128 is built from our fork, but wanted to compare speeds if anyone is curious.
Prompt processing is similar as expected; token generation is slightly slower for the 8B as expected from the larger size for g64 vs g128. For 27B token generation looks very similar.
Requirements
I have read and agree with the contributing guidelines: Yes
AI usage disclosure: Initial kernels were for group size 128, for the migration to group size 64 and rebasing to master used AI to make the changes (some cuda files were moved and refactored). Mostly followed either Q4_0 or Q1_0.
Manually reviewed all the changes and ran both speed and KL tests for correctness with a fresh build. The Q2_0 group 128 has been used by many people before and they behaved well, the main extra tests here were done to make sure newly converted g64 ggufs match the PR.