In M1MAX 64g， the speed is same as mlx normal model,27b, 19t/s， use MTP 19t/s,

[zwqjoy]

Problem

In M1MAX 64g， the speed is same as mlx normal model,27b, 19t/s， use MTP 19t/s,

Proposal

In M1MAX 64g， the speed is same as mlx normal model,27b, 19t/s， use MTP 19t/s,

[s-n-t]

Am experiencing similar - maybe some benchmarks will help understand the cause?

python3.11 -m mlx_lm generate --prompt "write a quicksort function in python" --model ~/.mtplx/models/Youssofal--Qwen3.6-27B-MTPLX-Optimized-Speed-fp16 --max-tokens 32768
==========
Here's a thinking process:
...
==========
Prompt: 17 tokens, 42.003 tokens-per-sec
Generation: 1687 tokens, 20.371 tokens-per-sec

mtplx tune --model Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed-fp16
...
MTPLX Tune
Results written to ...

AR     14.8 tok/s   1.00x
D1     21.3 tok/s   1.44x
D2     21.4 tok/s   1.45x  BEST
D3     20.9 tok/s   1.41x

Best for this Mac: D2, 1.45x AR
Saved: Web UI starts will use depth 2 for this model.

The AR number is way down on normal mlx_lm generation?

[uxbug]

M2 MAX 13.4 tok/s Is this normal?

$ mtplx start --host 0.0.0.0 --port 1234 --api-key sk-1234 --model /local/models/Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed-FP16 --reasoning off --temperature 0.6 --top-p 0.95 --top-k 20
MTPLX start
[1/4] Checking model: /local/models/Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed-FP16
model ready: /local/models/Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed-FP16
[2/3] Starting local MTPLX server for the browser chat...
      Loading model: /local/models/Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed-FP16
      Keep this terminal open. The next step is the model load.

  ███╗   ███╗ ████████╗ ██████╗  ██╗      ██╗  ██╗
  ████╗ ████║ ╚══██╔══╝ ██╔══██╗ ██║      ╚██╗██╔╝
  ██╔████╔██║    ██║    ██████╔╝ ██║       ╚███╔╝
  ██║╚██╔╝██║    ██║    ██╔═══╝  ██║       ██╔██╗
  ██║ ╚═╝ ██║    ██║    ██║      ███████╗ ██╔╝ ██╗
  ╚═╝     ╚═╝    ╚═╝    ╚═╝      ╚══════╝ ╚═╝  ╚═╝

  Native MTP speculative decoding · Apple Silicon

╭─ MTPLX 0.3.6 ─────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                   │
│      Model  mtplx-qwen36-27b-optimized-speed-fp16                                                 │
│    Profile  sustained (Sustained: long-context native-MTP path with bounded memory)               │
│        API  http://127.0.0.1:1234/v1                                                              │
│       Mode  Sustained MTP                                                                         │
│    Browser  http://127.0.0.1:1234/                                                                │
│  Listening  0.0.0.0:1234 (all interfaces)                                                         │
│    Loading  /local/models/Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed-FP16  │
│     Warmup  16 tokens                                                                             │
│       Auth  API key required                                                                      │
│       Stop  Ctrl-C                                                                                │
│                                                                                                   │
╰───────────────────────────────────────────────────────────────────────────────────────────────────╯

[1/6] Server config ready: listening on 0.0.0.0:1234 (all interfaces)
      Local API Base URL: http://127.0.0.1:1234/v1
[2/6] Model resolved: /local/models/Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed-FP16
[3/6] Runtime contract verified
      Loading the model can take about a minute on first start.

[4/6] Preparing Sustained MTP runtime
[4/6] Checking local acceleration runtime
      This may take a few seconds.
[4/6] Runtime checks complete
[5/6] Loading model weights: /local/models/Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed-FP16
      This is the long step; MTPLX is mapping the model into MLX.
      Model load in progress (this may take a minute).
[5/6] Model loaded in 9.9s
[5/6] Installing native-MTP draft head
[5/6] Context window: 262144 tokens
[6/6] Warming model with 16 tokens
[6/6] Warmup complete in 3.1s (7.87 tok/s)

MTPLX is ready.
Listening: 0.0.0.0:1234 (all interfaces)
Local Chat UI: http://127.0.0.1:1234/
Local OpenAI API Base URL: http://127.0.0.1:1234/v1
Model: mtplx-qwen36-27b-optimized-speed-fp16
Reasoning history: preserve (policy auto)
API key: required
Health check: http://127.0.0.1:1234/health
Keep this terminal open. Press Ctrl-C to stop MTPLX.
Opening chat UI in your browser...
{"event": "mtplx_openai_generation", "prompt_tokens": 89, "completion_tokens": 7, "elapsed_s": 1.366942, "tok_s": 5.120921, "seed": 264618655, "attempts": 1, "blank_retries": 0, "text_preview": "xxx"}
{"event": "mtplx_openai_generation", "prompt_tokens": 17, "completion_tokens": 1328, "elapsed_s": 98.390003, "tok_s": 13.497306, "seed": 1080791169, "attempts": 1, "blank_retries": 0, "text_preview": "xxx"}

[youssofal]

@uxbug Could you please run: MTPLX bench tune and sent me the output?

[youssofal]

In M1MAX 64g， the speed is same as mlx normal model,27b, 19t/s， use MTP 19t/s,

@zwqjoy On M1 max the speedup won't be 2x but it should be 1.2x - 1.4x. I believe the issue is you are using 3 MTP heads. On the M1 max using MTP=2 or MTP=1 would result in better speeds.

Please run: MTPLX bench tune and send the output here.

[s-n-t]

I'm gonna go ahead and give yo mine :) (M1 Max 32 core GPU)

[tune] model: /Users/service/.mtplx/models/Youssofal--Qwen3.6-27B-MTPLX-Optimized-Speed-fp16
[tune] diagnostic telemetry active; use --no-telemetry for clean speed comparison
[tune] close heavy apps now for cleaner results before measurements start
[tune] fans may get loud during tuning and will be restored afterward
[max] fan tool: thermalforge (/Users/service/.mtplx/bin/thermalforge)
[max] baseline fans: min 1316 RPM, max 1328 RPM
[max] running thermalforge max...
[max] waiting 1.0s for the daemon to update target RPM...
[max] target accepted: 3500 RPM; waiting for actual fans to ramp...
[max] fans ramped: actual 3112-3157 RPM; target 3500 RPM
[tune] artifacts: /Users/service/outputs/cli/tune/tune-20260516-164455
[tune] running isolated candidates: AR, D1, D2, D3
[tune] AR (1/4) starting; log: /Users/service/outputs/cli/tune/tune-20260516-164455/ar.log
[tune] AR finished in 17.1s: 14.42 tok/s
[tune] AR telemetry: scope=generation | power pkg=27.6W cpu=3.4W ane=0.0W gpu=24.3W | freq M=1.40GHz GPU=1.05GHz | temp core_avg=38.5C core_max=51.1C gpu_avg=44.6C | util M=42.8% GPU=68.9% | fans=3478rpm | samples=6 | window=13.3s
[tune] settling 5.0s before D1
[tune] D1 (2/4) starting; log: /Users/service/outputs/cli/tune/tune-20260516-164455/d1.log
[tune] D1 finished in 13.2s: 19.74 tok/s
[tune] D1 telemetry: scope=generation | power pkg=47.6W cpu=2.7W ane=0.0W gpu=45.8W | freq M=1.41GHz GPU=1.29GHz | temp core_avg=39.4C core_max=52.4C gpu_avg=48.3C | util M=42.7% GPU=92.1% | fans=3505rpm | samples=4 | window=9.7s
[tune] settling 5.0s before D2
[tune] D2 (3/4) starting; log: /Users/service/outputs/cli/tune/tune-20260516-164455/d2.log
[tune] D2 finished in 13.0s: 20.25 tok/s
[tune] D2 telemetry: scope=generation | power pkg=39.9W cpu=3.0W ane=0.0W gpu=36.9W | freq M=1.39GHz GPU=1.13GHz | temp core_avg=39.9C core_max=52.7C gpu_avg=48.9C | util M=43.1% GPU=70.7% | fans=3511rpm | samples=4 | window=9.5s
[tune] settling 5.0s before D3
[tune] D3 (4/4) starting; log: /Users/service/outputs/cli/tune/tune-20260516-164455/d3.log
[tune] D3 finished in 13.1s: 19.95 tok/s
[tune] D3 telemetry: scope=generation | power pkg=44.8W cpu=2.5W ane=0.0W gpu=44.4W | freq M=1.33GHz GPU=1.29GHz | temp core_avg=40.1C core_max=53.3C gpu_avg=49.1C | util M=41.6% GPU=83.9% | fans=3507rpm | samples=4 | window=9.6s
MTPLX Tune
Results written to /Users/service/outputs/cli/tune/tune-20260516-164455

AR 14.4 tok/s 1.00x
D1 19.7 tok/s 1.37x
D2 20.3 tok/s 1.40x
D3 20.0 tok/s 1.38x BEST

Best for this Mac: D3, 1.38x AR
Tie-break: D3 was within 2.0% of raw fastest D2; preferred the deeper depth.
Not saved: save disabled.

AR: telemetry=scope=generation | power pkg=27.6W cpu=3.4W ane=0.0W gpu=24.3W | freq M=1.40GHz GPU=1.05GHz | temp core_avg=38.5C core_max=51.1C gpu_avg=44.6C | util M=42.8% GPU=68.9% | fans=3478rpm | samples=6 | window=13.3s
D1: verify_ms=72.49 acceptance=MTP1 93/98 (94.90%)
D1: telemetry=scope=generation | power pkg=47.6W cpu=2.7W ane=0.0W gpu=45.8W | freq M=1.41GHz GPU=1.29GHz | temp core_avg=39.4C core_max=52.4C gpu_avg=48.3C | util M=42.7% GPU=92.1% | fans=3505rpm | samples=4 | window=9.7s
D2: verify_ms=101.05 acceptance=MTP1 65/68 (95.59%) | MTP2 59/68 (86.76%)
D2: telemetry=scope=generation | power pkg=39.9W cpu=3.0W ane=0.0W gpu=36.9W | freq M=1.39GHz GPU=1.13GHz | temp core_avg=39.9C core_max=52.7C gpu_avg=48.9C | util M=43.1% GPU=70.7% | fans=3511rpm | samples=4 | window=9.5s
D3: verify_ms=128.33 acceptance=MTP1 51/54 (94.44%) | MTP2 49/54 (90.74%) | MTP3 38/53 (71.70%)
D3: telemetry=scope=generation | power pkg=44.8W cpu=2.5W ane=0.0W gpu=44.4W | freq M=1.33GHz GPU=1.29GHz | temp core_avg=40.1C core_max=53.3C gpu_avg=49.1C | util M=41.6% GPU=83.9% | fans=3507rpm | samples=4 | window=9.6s
artifacts: /Users/service/outputs/cli/tune/tune-20260516-164455

The thing is as mentioned earlier, mlx_lm runs this at 20 t/s WITHOUT MTP, so the 14.4 AR seems wrong???

[uxbug]

mtplx max --install was executed before executing MTPLX bench tune. This was not installed before. The following is the output.

$ mtplx bench tune --model Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed-FP16 --no-telemetry
[tune] model: Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed-FP16
[tune] diagnostic telemetry disabled; speed comparison is cleaner
[tune] close heavy apps now for cleaner results before measurements start
[tune] fans may get loud during tuning and will be restored afterward
[max] fan tool: thermalforge (~/.mtplx/bin/thermalforge)
[max] baseline fans: min 0 RPM, max 0 RPM
[max] running `thermalforge max`...
[max] waiting 1.0s for the daemon to update target RPM...
[max] target accepted: 6800 RPM; waiting for actual fans to ramp...
[max] fans ramped: actual 6422-6450 RPM; target 6800 RPM
[tune] artifacts: ~/MTPLX/outputs/cli/tune/tune-20260517-005118
[tune] running isolated candidates: AR, D1, D2, D3
[tune] AR (1/4) starting; log: ~/MTPLX/outputs/cli/tune/tune-20260517-005118/ar.log
[tune] AR finished in 19.9s: 15.15 tok/s
[tune] settling 5.0s before D1
[tune] D1 (2/4) starting; log: ~/MTPLX/outputs/cli/tune/tune-20260517-005118/d1.log
[tune] D1 finished in 17.0s: 19.25 tok/s
[tune] settling 5.0s before D2
[tune] D2 (3/4) starting; log: ~/MTPLX/outputs/cli/tune/tune-20260517-005118/d2.log
[tune] D2 finished in 17.9s: 19.12 tok/s
[tune] settling 5.0s before D3
[tune] D3 (4/4) starting; log: ~/MTPLX/outputs/cli/tune/tune-20260517-005118/d3.log
[tune] D3 finished in 17.5s: 19.30 tok/s
MTPLX Tune
Results written to ~/MTPLX/outputs/cli/tune/tune-20260517-005118

AR     15.2 tok/s   1.00x
D1     19.2 tok/s   1.27x
D2     19.1 tok/s   1.26x
D3     19.3 tok/s   1.27x  BEST

Best for this Mac: D3, 1.27x AR
Not saved: save disabled.

D1: verify_ms=76.91 acceptance=MTP1 93/98 (94.90%)
D2: verify_ms=110.70 acceptance=MTP1 65/68 (95.59%) | MTP2 59/68 (86.76%)
D3: verify_ms=133.25 acceptance=MTP1 51/54 (94.44%) | MTP2 49/54 (90.74%) | MTP3 38/53 (71.70%)
artifacts: ~/MTPLX/outputs/cli/tune/tune-20260517-005118

[youssofal]

Thanks for posting the numbers. This was useful.

v0.3.7 fixes the confusing part of the benchmark output: Tune now treats the headline tok/s as decode speed, and keeps prefill / end-to-end speed separate. It also keeps bench tune --no-telemetry as the clean path for comparing AR, D1, D2, and D3 on the same machine.

I also checked this against mlx_lm on the same local model family and prompt size. The honest answer is: mlx_lm AR can be faster than MTPLX AR in some cases, but the tuned MTP depth is the number to compare. On my M5 Max run, MTPLX D3 beat both MTPLX AR and the comparable mlx_lm generation result. On M1/M2 Max machines, the gain can be much smaller, and Tune may choose D1/D2/D3 depending on the hardware.

Release: https://github.com/youssofal/MTPLX/releases/tag/v0.3.7

Upgrade with:

python3 -m pip install -U mtplx

I’m closing this release-tracking issue because the reporting and tuning path are fixed. If v0.3.7 still shows tuned MTP no faster than mlx_lm on the exact same model, prompt, and token count, please post the fresh output and I’ll treat that as a separate hardware/runtime performance bug.

[youssofal]

Handled in v0.3.7. If fresh same-model/same-prompt numbers still show a runtime performance problem, please open a new focused report with the new output.

[youssofal]

Quick correction from my side: I closed this too broadly.

v0.3.7 fixes the confusing reporting around Tune: headline tok/s is decode speed, and prefill / end-to-end speed are now kept separate. That part is fixed.

But the bigger question here is still valid: the reported mlx_lm generate result is from a much longer generation, while mtplx tune is using a short benchmark prompt. Those are not the same test, and the gap between short and long generations needs a proper apples-to-apples benchmark before I call the performance question closed.

So I’m reopening this. The next step is a long-generation benchmark mode that compares MTPLX and mlx_lm on the same model family, prompt shape, sampling settings, and output length. That will make it clear whether this is just benchmark mismatch or a real long-generation runtime gap we need to fix.

[taozhiyuai]

@youssofal

taozhiyu@TAOZHIYUs-MacBook-Pro M-OLLAMA % mtplx bench tune --model ~/.mtplx/models/Youssofal--Qwen3.6-27B-MTPLX-Optimized-Speed --retune --max
[tune] model: /Users/taozhiyu/.mtplx/models/Youssofal--Qwen3.6-27B-MTPLX-Optimized-Speed
[tune] diagnostic telemetry active; use --no-telemetry for clean speed comparison
[tune] close heavy apps now for cleaner results before measurements start
[tune] fans may get loud during tuning and will be restored afterward
[max] fan tool: thermalforge (/Users/taozhiyu/.mtplx/bin/thermalforge)
[max] baseline fans: min 0 RPM, max 0 RPM
[max] running thermalforge max...
[max] waiting 1.0s for the daemon to update target RPM...
[max] target accepted: 5349-5777 RPM; waiting for actual fans to ramp...
[max] fans ramped: actual 6007-6368 RPM; target 5349-5777 RPM
[tune] artifacts: /Users/taozhiyu/Downloads/M-OLLAMA/outputs/cli/tune/tune-20260521-055230
[tune] running isolated candidates: AR, D1, D2, D3
[tune] AR (1/4) starting; log: /Users/taozhiyu/Downloads/M-OLLAMA/outputs/cli/tune/tune-20260521-055230/ar.log
[tune] AR finished in 17.5s: 21.28 tok/s
[tune] AR telemetry: scope=generation | temp core_avg=55.8C core_max=66.9C gpu_avg=42.4C | fans=5564rpm | samples=14 | window=11.4s | notes=powermetrics: sudo: a password is required
[tune] settling 5.0s before D1
[tune] D1 (2/4) starting; log: /Users/taozhiyu/Downloads/M-OLLAMA/outputs/cli/tune/tune-20260521-055230/d1.log
[tune] D1 finished in 9.3s: 36.59 tok/s
[tune] D1 telemetry: scope=generation | temp core_avg=58.7C core_max=71.8C gpu_avg=46.4C | fans=5551rpm | samples=8 | window=6.2s | notes=powermetrics: sudo: a password is required
[tune] settling 5.0s before D2
[tune] D2 (3/4) starting; log: /Users/taozhiyu/Downloads/M-OLLAMA/outputs/cli/tune/tune-20260521-055230/d2.log
[tune] D2 finished in 8.7s: 41.52 tok/s
[tune] D2 telemetry: scope=generation | temp core_avg=60.6C core_max=76.1C gpu_avg=47.2C | fans=5547rpm | samples=7 | window=5.6s | notes=powermetrics: sudo: a password is required
[tune] settling 5.0s before D3
[tune] D3 (4/4) starting; log: /Users/taozhiyu/Downloads/M-OLLAMA/outputs/cli/tune/tune-20260521-055230/d3.log
[tune] D3 finished in 9.2s: 37.21 tok/s
[tune] D3 telemetry: scope=generation | temp core_avg=63.2C core_max=77.9C gpu_avg=47.9C | fans=5562rpm | samples=8 | window=6.2s | notes=powermetrics: sudo: a password is required
MTPLX Tune
Results written to /Users/taozhiyu/Downloads/M-OLLAMA/outputs/cli/tune/tune-20260521-055230

AR 21.3 tok/s 1.00x
D1 36.6 tok/s 1.72x
D2 41.5 tok/s 1.95x BEST
D3 37.2 tok/s 1.75x
Speed shown is decode tok/s; prefill is tracked separately.

Best for this Mac: D2, 1.95x AR
Not saved: save disabled.

AR: telemetry=scope=generation | temp core_avg=55.8C core_max=66.9C gpu_avg=42.4C | fans=5564rpm | samples=14 | window=11.4s | notes=powermetrics: sudo: a password is required
D1: verify_ms=46.25 acceptance=MTP1 91/100 (91.00%)
D1: telemetry=scope=generation | temp core_avg=58.7C core_max=71.8C gpu_avg=46.4C | fans=5551rpm | samples=8 | window=6.2s | notes=powermetrics: sudo: a password is required
D2: verify_ms=56.41 acceptance=MTP1 64/69 (92.75%) | MTP2 59/68 (86.76%)
D2: telemetry=scope=generation | temp core_avg=60.6C core_max=76.1C gpu_avg=47.2C | fans=5547rpm | samples=7 | window=5.6s | notes=powermetrics: sudo: a password is required
D3: verify_ms=71.43 acceptance=MTP1 52/60 (86.67%) | MTP2 43/60 (71.67%) | MTP3 37/59 (62.71%)
D3: telemetry=scope=generation | temp core_avg=63.2C core_max=77.9C gpu_avg=47.9C | fans=5562rpm | samples=8 | window=6.2s | notes=powermetrics: sudo: a password is required
artifacts: /Users/taozhiyu/Downloads/M-OLLAMA/outputs/cli/tune/tune-20260521-055230