ggml-cpu: x86 AVX512-VNNI repack GEMV/GEMM for Q1_0 and Q2_0 - #86
Merged
Conversation
Q2_0 previously had no repack path at all and Q1_0 only a NEON one, so batched CPU mul_mat for both formats fell back to per-row vec_dot on x86. Add: - block_q2_0x4 (4-row, 8-byte-chunk interleave) with generic repack, gemv and gemm implementations - AVX512-VNNI gemv/gemm kernels for the existing q1_0 4x8 layout and the new q2_0 4x8 layout; sum(qy) is computed once per activation sub-block and the horizontal reduction happens once per output tile - repack type selection on AVX512-VNNI CPUs for both formats
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Adds x86 AVX512-VNNI repacked GEMV/GEMM support for Q1_0 and Q2_0.
Changes:
- Adds Q2_0 4-row repacking and generic fallbacks.
- Implements optimized Q1_0/Q2_0 x86 kernels.
- Adds architecture fallbacks and runtime dispatch.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
ggml/src/ggml-cpu/repack.h |
Declares Q2_0 layouts and kernels. |
ggml/src/ggml-cpu/repack.cpp |
Implements repacking, fallbacks, templates, and dispatch. |
ggml/src/ggml-cpu/arch/x86/repack.cpp |
Adds AVX512-VNNI kernels. |
ggml/src/ggml-cpu/arch-fallback.h |
Maps generic fallbacks across architectures. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Cast to size_t/int64_t before the nrow*nblocks and i*nblocks multiplications so they cannot overflow int before widening. Resolves CodeQL alerts 630/631.
khosravipasha
approved these changes
Jul 18, 2026
khosravipasha
left a comment
khosravipasha
left a comment
Collaborator
There was a problem hiding this comment.
LGTM,
We can polish later and upstream the groups size 64 version in llama.cpp.
Doing another prebuilt release so merging this now.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
x86 (AVX512-VNNI) repack GEMV/GEMM for Q1_0 and Q2_0
Adds CPU repack (interleaved multi-row GEMM) support for the low-bit formats on x86. Until now the repack path existed only for Q1_0 on NEON; Q2_0 had none anywhere, so both formats ran batch-flat on x86 — the same GFLOPS at n=1 and n=512 — while q4-family formats gained 2-2.5x from their repack GEMMs.
What's included
repack.h/repack.cpp: newblock_q2_0x4layout (4-row interleave, 8-byte chunks so each column's 32 codes stay contiguous), repack function, generic gemv/gemm fallbacks, template instantiations, and selection-table entries routing Q1_0 and Q2_0 to*_4x8_q8_0onavx512 && avx512_vnni(NEON entries untouched).arch/x86/repack.cpp: AVX512-VNNI gemv+gemm forq1_0_4x8(existing NEON layout; pshufb + maskz_set1 bit-expand) andq2_0_4x8(carry-free 2-bit unpack). Both share the structural wins the profiling pointed at: onesum(qy)dpbusd per activation sub-block reused across all 4 columns, per-lane fp32 partials with a single horizontal reduction per output tile.arch-fallback.h: q2_0 generic defines for all non-x86 arches (ARM/riscv builds verified locally).Measured (EPYC Zen 4, two independent boxes agree within 2%)
Kernel (single-thread, m=4096 k=14336):
End-to-end (llama-bench, t=8, r=3):
Numerics (stated openly)
Known limits / follow-ups
test-backend-opsdoes not allocate repack buffers in this tree, so repack kernels have no unit coverage — validated here end-to-end instead (engagement confirmed via verbose load: ~500 tensors repacked, plus the e2e/KLD gates above). Making test-backend-ops exercise extra buffer types is a follow-up worth its own change.