[BUG] GLM-4-32B-0414 only produces gibberish

[Kliffcom]

OS

Linux

GPU Library

CUDA 12.x

Python version

3.10

Pytorch version

2.6.0+cu124

Model

https://huggingface.co/LatentWanderer/THUDM_GLM-4-32B-0414-6.5bpw-h8-exl2

Describe the bug

When using the model examples/chat.py it only produces nonsense output.

Reproduction steps

Run python ./examples/chat.py -m THUDM_GLM-4-32B-0414-6.5bpw-h8-exl2 --mode glm and give it any prompt.

Expected behavior

It should reply to the prompt as intended.

Logs

` -- Model: THUDM_GLM-4-32B-0414-6.5bpw-h8-exl2
-- Options: ['gpu_split: 20,20']
-- Loading tokenizer...
-- Loading model...
-- Loading model...
-- Prompt format: glm
-- System prompt:

You are a helpful AI assistant.

User: Who are you

siegeseice

siegele e ebe ce e treats
cosea ceta seatste te pe ce se coast cops te taipse t se
t t te cop cap te taits taup seat taic ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta t ta ta ta ta
ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta_print ta ta ta ta taic ta ta ta ta_tr ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta tata ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta_t ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta_tr ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta ta tac ta ta ta ta ta ta ta ta ta tacer ta tata ta ta ta ta ta ta ta ta ta`

Additional context

I pulled the latest dev branch and run pip install . --upgrade before trying to use the model. Changing the temperature or sampler does not seem to change anything.
The exl3 version works as expected, both quants were created using the same files.

Acknowledgements

• I have looked for similar issues before submitting this one.
• I understand that the developers have lives and my issue will be answered when possible.
• I understand the developers of this program are human, and I will ask my questions politely.

[turboderp]

I'm aware that GLM4-32B is broken at the moment. I'll probably push out a new release soon without a fix for that particular model, because PyTorch 2.7 is out now with Blackwell support, and the fix for this model is very non-trivial. Essentially the model produces intermediate states in one particular layer that exceed the FP16 range, and supporting that would require a whole new set of GEMM kernels. It was much easier to add this to V3. The whole point of V3 (much of it, anyway) is the ability to accommodate quirks like this more easily and without accruing more and more tech debt for every new, quirky model that needs special attention.

In this case the fix would be a separate quantized GEMM kernel with FP32 (or BF16) output, some toggles to enable that in the relevant places, a toggle in the cuBLAS matmul function to optionally do a mixed-mode GEMM using cublasGemmEx instead of cublasHgemm for large batch sizes, a separate code path in the MLP module to route those FP32 values into the output RMS norm layer, and a whole lot of testing to verify that a) it works and b) nothing else broke along the way.

Alternatively, the MLP down projection for the first layer could simply be rescaled at load time (by a factor of 50 or so), but then the output norm for that layer needs a toggle to allow it to revert to the right scale (after upcasting in the kernel), since it is of course a nonlinearity.

Either way, I'm just not sure I have the bandwidth for GLM4-32B at the moment, or that I could have anything done before the model becomes irrelevant anyway. 🤷