feat: CUDA Q2_0 support (Bonsai-27B ternary/binary on Qwen3.6 arch) - #2
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Carries upstream PR ggml-org#25707 (CUDA Q2_0, authored by PrismML's khosravipasha) onto
perf, enabling PrismML's Bonsai-27B family (Qwen3.6-derived,qwen35arch — already registered upstream) to run on CUDA. No PrismML fork code is carried: Q1_0/Q2_0 CPU+Metal are already in mainline (whichperftracks); this adds the last missing piece, CUDA Q2_0, from the maintainer's own upstream PR. When ggml-org#25707 merges upstream, a routineperfsync deduplicates it.Format note: upstream Q2_0 uses group-size 64 (18-byte block). PrismML publishes matching ggufs as
*_Q2_g64.ggufalongside their older fork-format group-128 files. Use the g64 variant — the group-128 file will not load (block-size mismatch, verified: 420 MB overrun).Verified on RTX 3060 (full offload,
-ngl 99 -fa 1):Both generate coherent text (e.g. ternary Q2_0: "The capital of France is → Paris."). Q2_0 CUDA output matches the CPU codec.
6GB-card reality (these are dense models, no MoE offload): the 3.53 GiB binary fits a 6GB card fully offloaded with room for KV; the 7.05 GiB ternary needs ~8GB fully offloaded, or partial CPU offload on 6GB.