Eval bug: Thinking model with thinking disabled cannot use /apply-template with final assistant turn

[gabe-l-hart]

Name and Version

$./bin/llama-cli --version
version: 6161 (291f531cd)
built with Apple clang version 17.0.0 (clang-1700.0.13.5) for arm64-apple-darwin24.6.0

Operating systems

Mac

GGML backends

Metal

Hardware

M3 Max 64GB

Models

• Thinking model: IBM Granite 3.2 8b
• Non-thinking model: IBM Granite Code 2b

Problem description & steps to reproduce

Description

This issue is a bug in a recently introduced change to support the "enable_thinking" toggle for Qwen3. It relates to this comment thread: https://github.com/ggml-org/llama.cpp/pull/13196/files#r2282737348.

The problem is that in response to https://github.com/ggml-org/llama.cpp/pull/13196/files#r2134714258, the logical condition of inputs.enable_thinking was inverted in a056e53. The result is that for a model that does support thinking, but needs to explicitly disable it (--reasoning-budget 0), attempting to apply the chat template (either for /apply-template or a full prefill operation), will result in triggering the error "Assistant response prefill is incompatible with enable_thinking.".

Repro

# Boot the server with Granite 3.2 8B and thinking disabled
./bin/llama-server -hf ibm-research/granite-3.2-8b-instruct-GGUF --jinja --reasoning-budget 0

# Send apply-template with a final assistant turn
curl http://localhost:8081/apply-template -d '{"messages": [{"role": "user", "content": "hello world"}, {"role": "assistant", "content": "hi hi"}]}'

Response

{"error":{"code":500,"message":"Assistant response prefill is incompatible with enable_thinking.","type":"server_error"}}

Expected

{"prompt":"<|start_of_role|>system<|end_of_role|>Knowledge Cutoff Date: April 2024.\nToday's Date: August 18, 2025.\nYou are Granite, developed by IBM. You are a helpful AI assistant.<|end_of_text|>\n<|start_of_role|>user<|end_of_role|>hello world<|end_of_text|>\n<|start_of_role|>assistant<|end_of_role|>hi hi"}

First Bad Commit

a056e53

Relevant log output

main: server is listening on http://127.0.0.1:8081 - starting the main loop
srv  update_slots: all slots are idle
got exception: {"code":500,"message":"Assistant response prefill is incompatible with enable_thinking.","type":"server_error"}
srv  log_server_r: request: POST /apply-template 127.0.0.1 500

[arichiardi]

Detected this as well when the request contains

"chat_template_kwargs": {"enable_thinking": "false"}"

which is my default for GLM-4.5 and Qwen3-Coder

@gabe-l-hart did you find a workaround by any chance?

[gabe-l-hart]

@arichiardi it's not a workaround, but the logic in this PR should fix it. I'd love the extra testing help!

#15404

[createthis]

@gabe-l-hart @ggerganov What are the chances this line contains a logic inversion typo?

llama.cpp/tools/server/utils.hpp

Line 727 in b730706

if ( (!inputs.enable_thinking) || inputs.chat_template_kwargs.find("enable_thinking") != inputs.chat_template_kwargs.end()) {

Why is it triggering when thinking is disabled? This seems directly opposed to the error message:

Assistant response prefill is incompatible with enable_thinking.

Is the error message wrong or is the if statement wrong? I think the if statement should be changed to simply:

if (inputs.enable_thinking) {

Why do we care about chat_template_kwargs? All we care about is whether or not thinking is enabled.

[gabe-l-hart]

@createthis I think it is a logic inversion bug, but it's a bit more complex than simply swapping the condition because the default value for enable_thinking is true for all models, even those that don't support the keyword arg. I have a fix for this ready to go in #15404 (as well as details of my exploring to figure out what caused this).

[createthis]

@gabe-l-hart Isn't enable_thinking disabled by --reasoning-budget 0 these days? I just wonder if your patch is doing more work than it has to.

[createthis]

I thought I read that this changed in #13771

[gabe-l-hart]

Yep, that's correct, so the issue that caused the authors to flip that bool is that for non-reasoning models, they would be forced to set --reasoning-budget 0 or hit this condition because the default value is true even for non-reasoning models (I'm on my phone so can't find the commit/comment thread, but it's linked in my PR). If we're ok with this error returning for non-thinking models that don't explicitly disable thinking, then simply flipping the condition is a fine fix, but that feels like it introduces another awkward unexpected behavior.

[createthis]

I see. Thanks for the context!