oQ-MTP performance on M1 not as expected

[s-n-t]

I've tested Qwen3.6-27B-oQ5-fp16-mtp and Qwen3.6-27B-oQ8-fp16-mtp generated from the original weights, acceptance seems to be high:

2026-05-06 20:12:45,019 - omlx.patches.mlx_lm_mtp.batch_generator - INFO - MTP[1] finish=length tokens=128 cycles=68 accept=58/68 (85.3%) emits[init=2,draft=58,bonus=58,verify=10] timing[backbone=1305.3ms mtp=324.7ms sample=6296.6ms cache=17.4ms]
2026-05-06 20:12:45,056 - omlx.scheduler - INFO - Cache phase timings: cleanup_finished_sync=0.1ms/2, store_cache_main_eval=8.0ms/2, store_cache_main_prep=8.1ms/2
2026-05-06 20:13:12,767 - omlx.patches.mlx_lm_mtp.batch_generator - INFO - MTP path activated for uid=2 (model has mtp_forward, batch=1)
2026-05-06 20:13:21,280 - omlx.patches.mlx_lm_mtp.batch_generator - INFO - MTP[2] finish=length tokens=128 cycles=68 accept=58/68 (85.3%) emits[init=2,draft=58,bonus=58,verify=10] timing[backbone=1323.1ms mtp=325.9ms sample=6410.3ms cache=18.2ms]

But oQ5 runs slower with MTP enabled and while oQ8 is improved it's not as much as I think is expected?

MTP ON

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.6-27B-oQ5-fp16-mtp
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          7308.2       65.63   140.1 tok/s    15.4 tok/s      15.643    73.6 tok/s    19.80 GB
pp4096/tg128         28089.0       65.23   145.8 tok/s    15.5 tok/s      36.373   116.1 tok/s    21.22 GB
pp8192/tg128         56390.1       64.54   145.3 tok/s    15.6 tok/s      64.586   128.8 tok/s    22.25 GB
pp16384/tg128       115483.6       70.19   141.9 tok/s    14.4 tok/s     124.397   132.7 tok/s    23.75 GB
pp32768/tg128       244621.0       75.65   134.0 tok/s    13.3 tok/s     254.228   129.4 tok/s    26.74 GB

MTP OFF

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.6-27B-oQ5-fp16-mtp
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          7318.8       59.80   139.9 tok/s    16.9 tok/s      14.913    77.2 tok/s    19.46 GB
pp4096/tg128         28026.7       61.19   146.1 tok/s    16.5 tok/s      35.798   118.0 tok/s    20.88 GB
pp8192/tg128         56369.0       62.09   145.3 tok/s    16.2 tok/s      64.254   129.5 tok/s    21.91 GB
pp16384/tg128       115442.2       66.20   141.9 tok/s    15.2 tok/s     123.849   133.3 tok/s    23.41 GB
pp32768/tg128       244644.6       73.36   133.9 tok/s    13.7 tok/s     253.961   129.5 tok/s    26.41 GB

MTP ON

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.6-27B-oQ8-fp16-mtp
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          7207.5       75.86   142.1 tok/s    13.3 tok/s      16.841    68.4 tok/s    28.81 GB
pp4096/tg128         27623.5       70.30   148.3 tok/s    14.3 tok/s      36.552   115.6 tok/s    30.26 GB
pp8192/tg128         55456.9       71.22   147.7 tok/s    14.2 tok/s      64.502   129.0 tok/s    31.29 GB
pp16384/tg128       113586.4       78.26   144.2 tok/s    12.9 tok/s     123.526   133.7 tok/s    32.79 GB
pp32768/tg128       240711.8       80.76   136.1 tok/s    12.5 tok/s     250.969   131.1 tok/s    35.79 GB

MTP OFF

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.6-27B-oQ8-fp16-mtp
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          7201.1       89.31   142.2 tok/s    11.3 tok/s      18.543    62.1 tok/s    28.34 GB
pp4096/tg128         27628.8       91.76   148.3 tok/s    11.0 tok/s      39.283   107.5 tok/s    29.80 GB
pp8192/tg128         55460.2       92.42   147.7 tok/s    10.9 tok/s      67.198   123.8 tok/s    30.82 GB
pp16384/tg128       113573.6       95.73   144.3 tok/s    10.5 tok/s     125.732   131.3 tok/s    32.32 GB
pp32768/tg128       240794.7      102.56   136.1 tok/s     9.8 tok/s     253.819   129.6 tok/s    35.32 GB

Happy to test anything else if it helps?

[beamivalice]

I experienced the same thing with MTPLX. For lower quant, the cost of token prediction was eating up the real generation time. Even with n=1. I am using MBP M1 Max 32GB. It could only go lower if we address the bottleneck of the prediction token.

MTPLX n=3 I got 13.9 tok/s · 4313 tokens · 2696 thinking · ttft 26.87s · 173.9 ms/verify · 1566 verifies.

oMLX 4bits/fp16 model. 2262t prompt · 3712t generated · prefill 135 tok/s · gen 19 tok/s · ttft 16.76s. 215.97s total.

MTPLX n=1 I got 16.6 tok/s · 3684 tokens · 1945 thinking · ttft 26.79s · 96.1 ms/verify · 2067 verifies.

[johnlovesgoats]

I think it isnt implemented that well for mlx

we could try without oQ quant?

[edeltoaster]

I got a 30% improvement with the Jundot/Qwen3.6-27B-oQ4-mtp on an M4 Pro 64GB.

[himefei]

I got a 30% improvement with the Jundot/Qwen3.6-27B-oQ4-mtp on an M4 Pro 64GB.

I can't even get it to work, my own oQ quants with preserve mtp on can't even toggle the model setting for MTP. Downloaded same model, toggled MTP switch, can't get it to geneate anything.

[MartinKuhl]

I noticed a significant drop in the HUMANEVAL & LIVECODEBENCH benchmark results when using Qwen3.6-27B-oQ4-mtp compared to Qwen3.6-27B-4bit.

[himefei]

Tbh, I never liked MLX Quantz, let alone any of its variants.

[svilendotorg]

Adding a data point + analysis for omlx HEAD on M1 32GB. Same pattern as @s-n-t and the M1 Max comment: MTP fires correctly with 91-95% acceptance, but gives no speedup vs AR — and the per-request log breakdown points squarely at the sampler.

Setup

• omlx 0.3.9.dev1 (HEAD-7395ca5) via brew install --HEAD jundot/omlx/omlx
• MacBook M1, 32 GB unified, macOS 26.4.1
• Model: Jundot/Qwen3.6-27B-oQ4-mtp
• temp=0.6 / top_p=0.95 / top_k=20, thinking disabled, cold reload between modes, 500-token coding prompt, mean of 3 timed runs (warmup excluded)

Results

mtp_enabled=false:  9.11 tok/s         (1.00x baseline)
mtp_enabled=true:   8.72-9.10 tok/s    (0.96x — slight slowdown)

Per-request timing on a representative MTP run:

MTP[0] tokens=302 cycles=157 accept=144/157 (91.7%)
       emits[init=2,draft=144,bonus=143,verify=13]
       timing[backbone=718ms mtp=1294ms sample=29755ms cache=15ms]

That's roughly 98 ms per generated token in the sampler alone, dwarfing everything else combined. GPU forward (backbone + mtp = 2 s out of 32 s) is fast; the CPU-side sample dispatch is doing 93% of wall time. Speculation can't help here: even reducing GPU forward cycles by 90% leaves the sampler unchanged.

(@s-n-t's logs above show the same pattern — sample=6296ms for 128 tokens = ~49 ms/token on oQ5/fp16. Mine is ~2x worse on oQ4, possibly because the smaller draft-LM-head increases sample dispatch frequency. Either way the sampler dominates.)

Sanity check — not a memory/bandwidth issue

I initially suspected M1's 68 GB/s unified bandwidth saturating with the 16 GB trunk. To rule it out, I rebooted, applied sudo sysctl iogpu.wired_limit_mb=27000 (raises GPU memory cap from default ~24 GB to 27 GB), and re-ran cold:

mtp_enabled=false:  8.96 / 8.52 / 9.11 tok/s  (mean 8.86)
mtp_enabled=true:   9.06 / 9.20 / 9.03 tok/s  (mean 9.10)

Identical band, sampler timing unchanged (sample=28057ms post vs 29755ms pre on similar token counts). VRAM headroom changes nothing.

Possible cause

The 0.3.8 release notes flagged a related fix:

Fix mlx-lm sample_utils mx.compile RNG state bug ... omlx now ships its own sampler at omlx/utils/sampling.py without the @partial(mx.compile, inputs=mx.random.state, ...) decorator.

If that workaround doesn't extend to the MTP path's per-position sampling (or HEAD has reintroduced a similar mx.compile fence on MTP-only sampling code), it would match this profile — small CPU dispatches that should be cheap but are doing repeated python↔Metal boundary crossings, multiplied by the depth-and-cycle structure of MTP.

A Metal-fused sampler (or extending the mx.compile-bypass workaround to MTP paths) would unlock the 2x+ claim on M1/M2/M3 base/Pro hardware. Until then MTP only pays off on Max/Ultra where the sampler floor is small relative to GPU forward — matching @edeltoaster's 30% improvement on M4 Pro 64GB.

Happy to share the AR-vs-MTP test harness (curl + admin login script) if useful for repro under different settings.

[zwqjoy]

I think MTP performance on M1 not as expected as far

[deepsweet]

I've come up with the detailed benchmark and conclusions regarding general FP16 advantages for M1/M2. MTP is a bit of a different thing, but still applicable. People on Reddit say that MTP really shines only for coding, but on prose the acceptance rate falls off.

[deepsweet]

Main branch finally allowed me to quantize oQ-MTP on my own and the results are the same as for others here.

M2 Max 30 GPU cores:

Benchmark Model: Qwen3.6-35B-A3B-MLX-oQ5-FP16
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          1227.8       13.59   834.0 tok/s    74.2 tok/s       2.954   390.0 tok/s    23.87 GB
pp4096/tg128          4105.4       13.90   997.7 tok/s    72.5 tok/s       5.870   719.5 tok/s    24.64 GB
pp8192/tg128          9762.2       14.46   839.2 tok/s    69.7 tok/s      11.598   717.3 tok/s    24.99 GB
pp16384/tg128        20645.3       16.25   793.6 tok/s    62.0 tok/s      22.709   727.1 tok/s    25.83 GB
pp32768/tg128        47931.2       20.02   683.6 tok/s    50.4 tok/s      50.473   651.7 tok/s    27.33 GB

Benchmark Model: Qwen3.6-35B-A3B-MLX-oQ5-FP16-MTP
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          1247.2       14.42   821.0 tok/s    69.9 tok/s       3.079   374.1 tok/s    24.39 GB
pp4096/tg128          4173.1       15.23   981.5 tok/s    66.2 tok/s       6.108   691.6 tok/s    25.16 GB
pp8192/tg128          9719.0       15.91   842.9 tok/s    63.3 tok/s      11.740   708.7 tok/s    25.51 GB
pp16384/tg128        21003.0       16.88   780.1 tok/s    59.7 tok/s      23.147   713.4 tok/s    26.35 GB
pp32768/tg128        48663.9       20.89   673.4 tok/s    48.2 tok/s      51.318   641.0 tok/s    27.85 GB

@jundot sorry for pinging you a lot today regarding the MTP, but is there anything bfloat16 specific? Does the float16 switch in UI fully propagates all the way down to MTP?

[jundot]

Quick check on my side: fp16 does propagate all the way to MTP. The cast in oq.py applies to mtp.* weights as well when preserve_mtp is on, and there's no hardcoded float dtype in the MTP runtime path. So if speed is off, the cause is somewhere else. I'll dig into it tomorrow.

[beamivalice]

This is not the case with current 0.3.9dev2 - verified. M1/M2's MTP got speed up correctly. Mine from 20t/s -> 21-23t/s

Make sure to use fp16 mtp model

Benchmark Model: Qwen3.6-27B-oQ4-fp16-mtp

Single Request Results

Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem
pp1024/tg128 6986.0 43.69 146.6 tok/s 23.1 tok/s 12.535 91.9 tok/s 16.74 GB
pp4096/tg128 26972.3 44.54 151.9 tok/s 22.6 tok/s 32.628 129.5 tok/s 18.16 GB
pp8192/tg128 55839.6 46.61 146.7 tok/s 21.6 tok/s 61.760 134.7 tok/s 19.19 GB

[deepsweet]

Current HEAD:

Benchmark Model: Qwen3.6-35B-A3B-MLX-oQ5-FP16
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          1206.5       13.26   848.7 tok/s    76.0 tok/s       2.890   398.6 tok/s    23.87 GB
pp4096/tg128          4150.3       14.13   986.9 tok/s    71.3 tok/s       5.945   710.6 tok/s    24.64 GB
pp8192/tg128          9848.4       15.03   831.8 tok/s    67.0 tok/s      11.758   707.6 tok/s    24.99 GB
pp16384/tg128        20863.9       16.70   785.3 tok/s    60.4 tok/s      22.984   718.4 tok/s    25.83 GB
pp32768/tg128        48876.4       20.13   670.4 tok/s    50.1 tok/s      51.433   639.6 tok/s    27.33 GB

Benchmark Model: Qwen3.6-35B-A3B-MLX-oQ5-FP16-MTP
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          1210.0       13.77   846.3 tok/s    73.2 tok/s       2.959   389.4 tok/s    24.39 GB
pp4096/tg128          4068.7       14.88  1006.7 tok/s    67.7 tok/s       5.958   708.9 tok/s    25.16 GB
pp8192/tg128          9575.2       15.72   855.5 tok/s    64.1 tok/s      11.572   719.0 tok/s    25.51 GB
pp16384/tg128        20825.2       17.03   786.7 tok/s    59.2 tok/s      22.988   718.3 tok/s    26.35 GB
pp32768/tg128        48787.9       20.40   671.6 tok/s    49.4 tok/s      51.378   640.3 tok/s    27.85 GB

MTP is definitely not activated for the first one and activated for the second, double checked…

@jundot have you had a chance to investigate this? Maybe @svilendotorg idea above about the custom sampler?

Seen GGUF-people claiming +30%, which matches this thread report with M4.

[beamivalice]

You are using MOE model. It's hardly an improvement in GGUF as well.