GLM 4.5 Air much slower since an update. Still not fixed so I roll back to a previous version. Big drop in speed from 6 tk/s to 0.5

[Gobeda]

I am on Linux Mint 22, I have an RTX 3070 Mobile with 64 GB of RAM and a Ryzen 5800H, and I use in llama-server (for Silly Tavern)
I run the following command:
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 ./llama-server -m ../text-generation-webui-main/models/GLM-4.5-Air-UD-Q3_K_XL-00001-of-00002.gguf --n-gpu-layers 99 --threads 7 --prio 2 -fa --batch-size 2048 --ubatch-size 2048 --ctx-size 12000 --jinja -ot "blk.([0-1]).ffn.=CUDA0,.ffn_._exps=CPU"

Since an update between 5 of august and 14th (compiled from llama) for GLM 4.5 Air the tk/s in generation are around 0.5 token per second much slower than my previous usual of 6 tokens per second while if I revert back to versions like llama.cpp-9515c6131aecaccc955fdedcfe16c3e030aaefcb (last update 5th of August) is back to my normal speeds. Am I the only one experiencing this? Any help in knowing what is the cause of the problem?

[JohannesGaessler]

Do a git bisect to identify the exact commit causing the issue.

[Gobeda]

At first I thought was CUDA graphs, because my gpu was idling a 0% even with CUDA enabled so works at 0.5 tk/s then I did another build of the lastest git version and is working fine again when the first build wasn't really using the GPU? Forced -DCMAKE_CUDA_ARCHITECTURES=86 and now is working fine running again at normal speed.

Before with SillyTavern for the same chat in Sillytavern.

slot launch_slot_: id 0 | task 0 | processing task
slot update_slots: id 0 | task 0 | new prompt, n_ctx_slot = 12032, n_keep = 0, n_prompt_tokens = 9004
slot update_slots: id 0 | task 0 | kv cache rm [0, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.227454
slot update_slots: id 0 | task 0 | kv cache rm [2048, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 4096, n_tokens = 2048, progress = 0.454909
slot update_slots: id 0 | task 0 | kv cache rm [4096, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 6144, n_tokens = 2048, progress = 0.682363
slot update_slots: id 0 | task 0 | kv cache rm [6144, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 8192, n_tokens = 2048, progress = 0.909818
slot update_slots: id 0 | task 0 | kv cache rm [8192, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 9004, n_tokens = 812, progress = 1.000000
slot update_slots: id 0 | task 0 | prompt done, n_past = 9004, n_tokens = 812
slot release: id 0 | task 0 | stop processing: n_past = 10511, truncated = 0
slot print_timing: id 0 | task 0 | 
prompt eval time = 131329.17 ms / 9004 tokens ( 14.59 ms per token, 68.56 tokens per second)
eval time = 2781076.18 ms / 1508 tokens ( 1844.21 ms per token, 0.54 tokens per second)
total time = 2912405.35 ms / 10512 tokens
srv update_slots: all slots are idle
srv log_server_r: request: POST /chat/completions 127.0.0.1 200

Now with the new build

srv update_slots: all slots are idle
srv params_from_: Chat format: Hermes 2 Pro
slot launch_slot_: id 0 | task 0 | processing task
slot update_slots: id 0 | task 0 | new prompt, n_ctx_slot = 12032, n_keep = 0, n_prompt_tokens = 9003
slot update_slots: id 0 | task 0 | kv cache rm [0, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.227480
slot update_slots: id 0 | task 0 | kv cache rm [2048, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 4096, n_tokens = 2048, progress = 0.454959
slot update_slots: id 0 | task 0 | kv cache rm [4096, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 6144, n_tokens = 2048, progress = 0.682439
slot update_slots: id 0 | task 0 | kv cache rm [6144, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 8192, n_tokens = 2048, progress = 0.909919
slot update_slots: id 0 | task 0 | kv cache rm [8192, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 9003, n_tokens = 811, progress = 1.000000
slot update_slots: id 0 | task 0 | prompt done, n_past = 9003, n_tokens = 811
slot release: id 0 | task 0 | stop processing: n_past = 10018, truncated = 0
slot print_timing: id 0 | task 0 | 
prompt eval time = 53977.89 ms / 9003 tokens ( 6.00 ms per token, 166.79 tokens per second)
eval time = 156356.89 ms / 1016 tokens ( 153.89 ms per token, 6.50 tokens per second)
total time = 210334.78 ms / 10019 tokens
srv update_slots: all slots are idle
srv log_server_r: request: POST /chat/completions 127.0.0.1 200

Seems that is back to normal.