MTP fp16 for M1/M2

[beamivalice]

Referring to this PR from mlx-lm, someone found out that using --dtype float16 will help M1/M2 class processor to take advantage from MTP properly.

Our oQ should be able to convert MTP with FP16 for M1/M2 as well.

[nmqanh]

Just curious, when creating oQ quant you can click advantaged settings -> select dtype float16 which will create oQ quant with fp16 suffix, does that work for you already?

[crazyi]

@deepsweet
I found that hw.optional.arm.FEAT_BF16: 1 is supported on the M2 chip. If the M2 chip supports BF16, why does converting to FP16 still result in performance improvements?
#604
Why the M2 is more advanced that it seemed

[beamivalice]

Just curious, when creating oQ quant you can click advantaged settings -> select dtype float16 which will create oQ quant with fp16 suffix, does that work for you already?

The current oQ strips out MTP layer entirely.

[deepsweet]

@deepsweet I found that hw.optional.arm.FEAT_BF16: 1 is supported on the M2 chip. If the M2 chip supports BF16, why does converting to FP16 still result in performance improvements? #604 Why the M2 is more advanced that it seemed

M2 has a general hardware support of BF16, but not the matrix multiplication:

Apple M2 Metal does not support matrix multiplication on BFloat16:

MPSMatrixMultiplication.mm:3260: failed assertion `Input data type must be one
of MPSDataTypeFloat32, MPSDataTypeFloat16, MPSDataTypeInt8, or
MPSDataTypeInt16.'

From https://jakub-m.github.io/llama2-rs

This is a layered and quite shady story, so I've tested it for myself empirically.

[deepsweet]

https://github.com/deepsweet/metal-fp32-bf16-fp16 ❤️‍🔥