Eval bug: Qwen 3.6 crashes llama-server with prompts more than 132 tokens

[samteezy]

Name and Version

(base) root@llama-new:~/llm-stack/bin# ./llama-cli --version
version: 9860 (fdb1db8)
built with GNU 12.4.0 for Linux x86_64

Build flags:

-DGGML_CUDA=ON \
-DGGML_ZENDNN=ON \
-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=FLAME \
-DGGML_CUDA_FA_ALL_QUANTS=ON \

Operating systems

Linux

GGML backends

CUDA

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Oct_29_23:50:19_PDT_2024
Cuda compilation tools, release 12.6, V12.6.85
Build cuda_12.6.r12.6/compiler.35059454_0

Hardware

16GB V100 with an AMD EPYC 7052. Running in an Ubuntu 26 LXC under Proxmox.

Models

unsloth/Qwen3.6-35B-A3B-UD-Q4_K_XL

Problem description & steps to reproduce

I normally do really basic testing of models by copying and pasting a new article from NPR, and at first I thought it was an issue with the prompt. But I'm realizing that it's got to do with the length of the prompt itself. I'm running successive lorem ipsum generated content to see where the cutoff is.

68 words (132 tokens): works
69 words (133 tokens): crashes

It works fine with Gemma 4 and both the Ornith 9B and 35B models (based on Qwen 3.5), and unsloth's Qwen 3.5-122B also works.

env ZENDNNL_MATMUL_ALGO=1
      /root/llm-stack/bin/llama-server
      --port ${PORT}
      --no-mmap
      -lv 5
      --model /mnt/models/unsloth/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
      --temp 0.6
      --top-p 0.95
      --ctx-size 32000
      --reasoning-preserve
      --spec-type draft-mtp
      --spec-draft-n-max 2
      -ctk q8_0
      -ctv q8_0

First Bad Commit

No response

Relevant log output ( -lv 6 with a 69-word lorem ipsum prompt)

1.39.062.467 I slot operator(): id 3 | task 296 | new prompt, n_ctx_slot = 32000, n_keep = 0, task.n_tokens = 133
1.39.062.469 I slot operator(): id 3 | task 296 | checking checkpoint with [127, 127] against 124...
1.39.062.470 I slot operator(): id 3 | task 296 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see #13194 (comment))
1.39.062.472 I slot operator(): id 3 | task 296 | erased invalidated context checkpoint (pos_min = 127, pos_max = 127, n_tokens = 128, n_swa = 0, pos_next = 0, size = 63.066 MiB)
1.39.068.922 D res send: sending result for task id = 296
1.39.068.924 D res send: task id = 296 pushed to result queue
1.39.068.927 I slot operator(): id 3 | task 296 | cached n_tokens = 0, memory_seq_rm [0, end)
1.39.068.957 I srv stream_sessi: conv_id= (empty=1)
1.39.068.964 D srv stop: all tasks already finished, no need to cancel
1.39.069.218 D slot operator(): id 3 | task 296 | main/do_checkpoint = no, pos_min = -1, pos_max = -1
1.39.069.224 D srv decode: n_batch (effective) = 2048, off = 0
1.39.069.226 D set_adapters_lora: adapters = (nil)
1.39.069.226 D adapters_lora_are_same: adapters = (nil)
1.39.069.227 D set_embeddings: value = 0
1.39.070.938 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.941 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.942 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.942 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.943 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.944 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.944 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.945 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.946 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.946 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.947 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.947 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.948 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.948 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.949 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.961 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.964 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.965 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.965 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.966 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.967 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.968 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.968 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.969 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.970 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.971 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.972 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.973 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.973 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.975 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.975 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.976 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.977 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.979 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.979 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.980 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.981 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.982 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.983 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.983 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.984 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.985 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.986 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.986 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.988 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.988 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.989 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.990 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.990 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.991 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.993 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.993 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.994 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.995 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.996 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.996 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.998 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.998 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.070.999 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.000 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.000 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.001 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.002 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.003 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.003 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.005 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.005 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.007 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.008 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.008 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.009 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.010 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.010 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.011 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.012 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.013 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.013 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.015 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.015 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.015 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.017 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.017 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.018 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.019 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.021 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.021 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.022 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.023 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.023 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.025 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.025 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.026 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.027 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.027 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.028 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.029 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.030 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
1.39.071.030 D ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture

[samteezy]

I think it's something to do with the automatic "fit" capability.

I noticed a similar issue trying to run a lower Deepseek v4 quant (one of antirez's).

Wondering if it was an outdated CUDA problem, I rebuilt llama.cpp to include Vulkan support and the issue persisted there, so it's not a CUDA thing.

0.26.287.883 D slot launch_slot_: id  3 | task -1 | launching slot : {"id":3,"n_ctx":32768,"speculative":true,"is_processing":false}
0.26.287.929 D common_sampler_init: prefill token: 248045 = <|im_start|>
0.26.287.933 D common_sampler_init: prefill token: 74455 = assistant
0.26.287.934 D common_sampler_init: prefill token: 198 = 

0.26.287.934 D common_sampler_init: prefill token: 248068 = <think>
0.26.287.935 D common_sampler_init: prefill token: 198 = 

0.26.287.949 I slot launch_slot_: id  3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist 
0.26.287.954 I slot launch_slot_: id  3 | task -1 | sampler params: 
	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
	dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 32768
	top_k = 20, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.700
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
0.26.287.955 I slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
0.26.287.955 I slot process_sing: id  0 | task -1 | saving idle slot to prompt cache
0.26.287.956 I slot prompt_clear: id  0 | task -1 | clearing prompt with 0 tokens
0.26.288.185 I slot process_sing: id  1 | task -1 | saving idle slot to prompt cache
0.26.288.188 I slot prompt_clear: id  1 | task -1 | clearing prompt with 0 tokens
0.26.288.365 I slot process_sing: id  2 | task -1 | saving idle slot to prompt cache
0.26.288.367 I slot prompt_clear: id  2 | task -1 | clearing prompt with 0 tokens
0.26.288.545 D que    start_loop: update slots
0.26.288.547 D srv  update_slots: posting NEXT_RESPONSE
0.26.288.550 D que          post: new task, id = 1, front = 0
0.26.288.553 I slot update_slots: id  3 | task 0 | new prompt, n_ctx_slot = 32768, n_keep = 0, task.n_tokens = 133
0.26.288.556 D res          send: sending result for task id = 0
0.26.288.556 D res          send: task id = 0 pushed to result queue
0.26.288.560 I slot update_slots: id  3 | task 0 | cached n_tokens = 0, memory_seq_rm [0, end)
0.26.288.578 D srv          stop: all tasks already finished, no need to cancel
0.26.288.755 D slot update_slots: id  3 | task 0 | main/do_checkpoint = no, pos_min = -1, pos_max = -1
0.26.288.757 D srv  update_slots: decoding batch, n_tokens = 129
0.26.288.759 D set_adapters_lora: adapters = (nil)
0.26.288.759 D adapters_lora_are_same: adapters = (nil)
0.26.288.760 D set_embeddings: value = 0

So then I tried setting --cpu-moe and it was able to complete a response. Makes me think it's something to do with how -fit works with Qwen 3.6 and Deepseek v4.

[0cc4m]

Try without ZENDNN and without your BLAS vendor setting.

[jhughes2112]

I have been troubleshooting a problem for the past few days where llama-server is giving a 400 error, and seems to increase likelihood the longer the message is. Not going to hijack this, but it's with two different PCs and two different models, both running the latest built Docker image. I'm rebuilding native for Windows to see if the problem is still there. Just a data point. (Edit: I am testing against Qwen 3.6 27B and 35B on separate machines.)

[TheOneWhoWill]

Might be a GPU specific bug. I couldn't reproduce it on a 5070 ti, same vram, same command (minus the mtp flag because this isn't an mtp model, I don't know why you have that flag). I tested on 120 tokens of Lorem, and then 218, well beyond your reported "limit"

[jhughes2112]

@TheOneWhoWill Sadly, when Unsloth released these Qwen 3.6 MTP models, they didn't put mtp in the name, so it matches the non-mtp versions: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF?show_file_info=Qwen3.6-35B-A3B-UD-Q4_K_S.gguf

[TheOneWhoWill]

@jhughes2112 Alright, let me retry with this model

[TheOneWhoWill]

@jhughes2112 Still couldn't reproduce, likely gpu specific

[Atomic-Germ]

Could you try that with a vanilla cpu-only build, then cpu with zen, and so on down the hardware list? That, and run without any of your flags, then add one at a time. If it doesn't crash right away on the CPU one, we can get a fix on where it's actually choking. I doubt many of us have an epic to test on, so you're going to have to do all that.

Hopefully it's revealing enough to fix

[samteezy]

Sorry for the delay in response, as I've not been home a ton this holiday weekend.

Per @0cc4m 's advice, rebuilding without ZenDNN works as expected. Adding back in BLAS/BLIS works as expected, too.

Only when ZenDNN is included in the build do I get this issue where I need to specify offloading behavior using --cpu-moe or similar params.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.