Skip to content

metal: default-on nr1=2 multi-column Q2_0 matvec (spec-decode verify) - #64

Merged
bri-prism merged 1 commit into
prismfrom
perf/metal-q2_0-nr1-default
Jul 14, 2026
Merged

metal: default-on nr1=2 multi-column Q2_0 matvec (spec-decode verify)#64
bri-prism merged 1 commit into
prismfrom
perf/metal-q2_0-nr1-default

Conversation

@bri-prism

Copy link
Copy Markdown

What

Make the nr1=2 multi-column Q2_0 matvec the default for ne11 >= 2 (the small-batch / spec-decode verify path), mirroring the Q1_0 path — instead of requiring the opt-in GGML_METAL_Q2_0_NR1 env var.

Why

The nr1=2 variant reads each streamed q2_0 weight group once per 2 src1 columns, versus the mul_mv_ext route that re-streams per column. In-code microbench (M5 Pro): nr1_2 = 93.2 µs vs 122 µs ext at ne11=2 (+31%); ne11=4 via 2 passes 171 vs 183. Q1_0 already defaults to this; this brings Q2_0 (ternary targets) to parity — it helps the DSpark verify path (which runs the target at ne11 = draft depth).

It was previously opt-in only because of the nr1_3 occupancy cliff at ne11==3 (195 vs 152 µs ext, tpb=16). This change carves out ne11==3 (stays on ext), so only the measured-win widths engage.

How

  • ne11 >= 2 && ne11 != 3nr1=2; ne11==3 → unchanged (ext); ne11==1 (base decode) → unchanged.
  • GGML_METAL_Q2_0_NR1=1 restores the old routing; =2/3/4 still force as before.
  • Output-identical — pure matvec routing, no numerics change.

Testing

  • Ternary base decode (llama-bench tg128, ne11=1): 27.06 → 27.13 tok/s = unchanged (change correctly scoped to multi-column).
  • DSpark verify (tlg6l1 × ternary-cont6k, k=4), default settings now auto-engages nr1=2: runs clean, accept 0.757 / τ 4.03, output correct.

Mirror the Q1_0 default to Q2_0: for ne11>=2 use the nr1=2 multi-column
variant that reads each streamed q2_0 weight group once per 2 src1 columns,
instead of the mul_mv_ext route. Measured (M5 Pro, in-code microbench):
nr1_2 = 93.2us vs 122us ext at ne11=2 (+31%); ne11=4 via 2 passes 171 vs 183.

ne11==3 is carved out -- that is the nr1_3 occupancy cliff (195 vs 152 ext,
tpb=16) that kept this path opt-in; three columns stay on ext. Output is
identical (pure matvec routing); base decode (ne11=1) is unaffected.
GGML_METAL_Q2_0_NR1=1 restores the old routing.
@bri-prism
bri-prism merged commit 887f007 into prism Jul 14, 2026
18 of 33 checks passed
@khosravipasha
khosravipasha deleted the perf/metal-q2_0-nr1-default branch July 14, 2026 05:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant