metal: default-on nr1=2 multi-column Q2_0 matvec (spec-decode verify) - #64
Merged
Conversation
Mirror the Q1_0 default to Q2_0: for ne11>=2 use the nr1=2 multi-column variant that reads each streamed q2_0 weight group once per 2 src1 columns, instead of the mul_mv_ext route. Measured (M5 Pro, in-code microbench): nr1_2 = 93.2us vs 122us ext at ne11=2 (+31%); ne11=4 via 2 passes 171 vs 183. ne11==3 is carved out -- that is the nr1_3 occupancy cliff (195 vs 152 ext, tpb=16) that kept this path opt-in; three columns stay on ext. Output is identical (pure matvec routing); base decode (ne11=1) is unaffected. GGML_METAL_Q2_0_NR1=1 restores the old routing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Make the
nr1=2multi-column Q2_0 matvec the default forne11 >= 2(the small-batch / spec-decode verify path), mirroring the Q1_0 path — instead of requiring the opt-inGGML_METAL_Q2_0_NR1env var.Why
The
nr1=2variant reads each streamed q2_0 weight group once per 2src1columns, versus themul_mv_extroute that re-streams per column. In-code microbench (M5 Pro): nr1_2 = 93.2 µs vs 122 µs ext at ne11=2 (+31%); ne11=4 via 2 passes 171 vs 183. Q1_0 already defaults to this; this brings Q2_0 (ternary targets) to parity — it helps the DSpark verify path (which runs the target at ne11 = draft depth).It was previously opt-in only because of the
nr1_3occupancy cliff atne11==3(195 vs 152 µs ext, tpb=16). This change carves outne11==3(stays on ext), so only the measured-win widths engage.How
ne11 >= 2 && ne11 != 3→nr1=2;ne11==3→ unchanged (ext);ne11==1(base decode) → unchanged.GGML_METAL_Q2_0_NR1=1restores the old routing;=2/3/4still force as before.Testing