Native MTP speculative decoding (Qwen3.5/3.6 reference implementation)#990

Open

AirRunner wants to merge 29 commits into

ml-explore:mainfrom

AirRunner:feat/mtp-native

Commits on Apr 25, 2026

feat: native MTP speculative decoding for Qwen3.5
AirRunner
committed
fix(mtp): eliminate SSM state contamination on draft rejection
AirRunner
committed
fix(mtp): server integration (yield types, cache fallback, batching)
AirRunner
committed
fix(mtp): address @janhilgard code review feedback (double-norm, quant_predicate)
AirRunner
committed
feat(mtp): add --mtp CLI flag for generate and server
AirRunner
committed
test(mtp): add unit tests for MTP speculative decoding
AirRunner
committed
fix(mtp): warn when --mtp flag is used with a model without MTP head
AirRunner
committed
style: apply black and isort formatting
AirRunner
committed
fix(mtp): stack per-expert MTP weights for MoE models in sanitize()

AirRunner
and
Thump604
committed
fix(mtp): raise clear error when config has MTP but weights do not
AirRunner
committed
feat(mtp): add probabilistic draft acceptance for stochastic samplers
AirRunner
committed
refactor(mtp): clean up mtp_generate_step and add dynamic MTP/batch switching
AirRunner
committed
fix(mtp): always leave 1 token for _step_backbone in _prefill
AirRunner
committed
fix(mtp): correct prev_tokens management for logits processors
AirRunner
committed
refactor(mtp): thread prev_tokens explicitly through mtp_generate_step
AirRunner
committed

Commits on Apr 26, 2026

refactor(cache): declare rollback_state as class attribute on ArraysCache
AirRunner
committed

Commits on Apr 29, 2026

Commits on May 4, 2026

fix(mtp): remove spurious mtp_cache trim on draft rejection
AirRunner
committed

Commits on May 5, 2026

fix(mtp): use residual sampling on rejection at temp>0
AirRunner
committed

Commits on May 6, 2026

fix(mtp): reduce residual sampling to 1 sync, correct z=0 fallback
AirRunner
committed

Commits on May 7, 2026

Commits on May 8, 2026

fix(mtp): clear Metal allocator cache every 256 tokens during decode
AirRunner
committed

Commits on May 9, 2026

style(mtp): move u after _step_backbone
AirRunner
committed

Commits on May 11, 2026

qwen3_5: remove mtp.fc exclusion from quant_predicate
AirRunner
committed

Commits on May 15, 2026