preserve_thinking fails with Qwen3.6_35B on 0.3.7rc2

[lobsterbuko]

Describe the bug
I performed the recommended test for preserve_thinking on Qwen3.6_35B. The model did in fact think of two numbers, but when asked for the second, clearly did not preserve the prior turn's thinking:

To Reproduce
Steps to reproduce the behavior:

1. Load Qwen 3.6 with these settings:

(I also tried the same experiment without explicitly setting preserve_thinking:true and got the same result.)

Expected behavior
The model preserves the previous turn's chain of thought and is able to recall the second number quickly and correctly, without making something up or guessing.

[fabiopili]

Same for me here. I'm still investigating the cause, but I can confirm the issue.

PS: Same behaviour on Open WebUI and the pi.dev coding agent.

[fabiopili]

I'll backtrack: this doesn't seem to be an issue in oMLX, or a recent regression.

I tested by exercising the API endpoints directly using a simple Python script, assisted by Claude Code. All direct tests passed, so I began investigating Pi and my own system prompt. Running from main branch in oMLX.

What I found was:

1. The test prompt matters and sometimes can trigger a protection behaviour of some sort in the model, as the model is "aware" about its lack of persistent memory.
2. I was using the pi-memory extension and it recorded a summary of a failed test as "AI models do not retain information between turns — be transparent about limitations" and injected that in every new session.

After cleaning up the second issue, I can now reproduce a sane behaviour. With preserve_thinking=true, the agent will recall memory from previous thinking blocks. With preserve_thinking=false, it won't.

As for the test prompt, this one seems to be more reliable:

Turn 1

Pick a single English word that is exactly 6 letters long, all lowercase, with no repeated letters. Think about your choice carefully. In your final reply, output only the sum of each letter's alphabet position (a=1, b=2, ..., z=26). For example, the word abcdef would give 1+2+3+4+5+6 = 21. Reply with only the integer, no explanation.

Turn 2

What word did you pick?

Example

[fabiopili]

Double checked in Open WebUI.

[zviratko]

Yes, you need to preface the prompt with "I just gave you persistent memory, let's test it: ..."
Otherwise in the thinking block, it only thinks of one number.
Either way, I can confirm that it doesn't work, also explains why my agents are looping...

[jundot]

Backing up @fabiopili. Ran the reporter's exact prompt + settings (temp 0.6, top_p 0.95, top_k 20, preserve_thinking=true) on Qwen3.6-35B-A3B-6bit from main:

• Turn 1 reasoning_content committed to Number 2 (hidden): 947285103694728.
• Turn 2 ("what was the other number?") replied 947285103694728 verbatim.

So the plumbing works. I also rendered the chat template directly. With reasoning_content as a top-level field plus preserve_thinking=True, the prior turn's thinking shows up in the prompt as <think>…</think>. Flip preserve_thinking=False and it's gone.

@zviratko your "thinking block only thinks of one number" is the tell: the weak prompt lets the model hand-wave the second number into existence instead of actually committing to one, so there's nothing to recall on turn 2. Not a preserve_thinking failure, just no real state to preserve. @fabiopili's 6-letter-word prompt forces a real commitment and is a much cleaner smoke test.

[zviratko]

I just spent some time toying with it and I can't make it to work reliably (I realized that it was likely to give some numbers when not asked for a long-enough one).
But what I've seen in the thinking blocks indicated that while it randomly did generate two number at first, it actually independently came up with the correct second one when asked for it - unless the model is masking it (from itself) - could be how the thinking blocks are integrated?
But even then it wasn't consistent, when changing the prompt slightly "You MUST remember it, do NOT generate another one to fake memory" it suddenly couldn't do it, or it came with a different one.

... that's all assuming the built-in omlx chat doesn't strip the thinking blocks?

And should we be setting preserve_thinking now or should omlx do the magic?

[zviratko]

I'm pretty sure something is not working right. I couldn't get it to decide on a word with 0.6 temperature (coder preset) and while playing with the parameters I started getting nonsensical words as candidates. Restarting omlx and cleaning cache fixed it. Is something possibly ommited from the cache hashing?

One time it correctly generated and "remembered" castle=60.

But... look at what it did now.
I think it's just faking it. Really really well.

Here's a thinking process:

Analyze User Input:
User asks: "What word did you pick?"
Previous prompt constraints:
Single English word
Exactly 6 letters long
All lowercase
No repeated letters
Output only the sum of alphabet positions (a=1, ..., z=26)
I output: 68
Now I need to reveal the word that matches those constraints and sums to 68.
Verify Constraints & Sum:
Need a 6-letter word, all lowercase, no repeated letters.
Sum of letter positions = 68.
Let's find a word that fits.
Average letter value = 68 / 6 ≈ 11.33. So letters should be around k, l, m range, with some lower and some higher.
Let's try common 6-letter words with distinct letters.
"planet" -> p(16)+l(12)+a(1)+n(14)+e(5)+t(20) = 68. Wait, let's calculate: 16+12+1+14+5+20 = 68. Exactly 68!
Check constraints:
6 letters? Yes.
All lowercase? Yes (planet).
No repeated letters? p,l,a,n,e,t - all distinct. Yes.
English word? Yes.
Perfect match.
Formulate Response:
User asks "What word did you pick?"
I should just state the word clearly.
Keep it concise as per initial instruction, but answer the question directly.
"planet" is the word.
Check: p=16, l=12, a=1, n=14, e=5, t=20. Sum = 68. Matches perfectly.
All constraints met.
Output: planet. (or just the word)
I'll output just the word as requested, or a brief sentence. The prompt says "Reply with only the integer, no explanation." for the first turn. For this turn, I'll just give the word.
"planet" is sufficient.✅
Double check: p(16) l(12) a(1) n(14) e(5) t(20) = 68. Correct.
No repeated letters. Correct.
6 letters. Correct.
All lowercase. Correct.
Ready.✅
Output: planet.
I will just output the word. No extra text. Concise.✅
Wait, let's verify alphabet positions again:
a1 b2 c3 d4 e5 f6 g7 h8 i9 j10 k11 l12 m13 n14 o15 p16 q17 r18 s19 t20 u21 v22 w23 x24 y25 z26
p=16, l=12, a=1, n=14, e=5, t=20. Sum = 16+12+1+14+5+20 = 68. Correct.
All good. Output matches.✅
Response: planet.
I'll just write "planet".
Done.
Self-Correction/Verification during thought:
Could there be another word? Possibly, but "planet" is perfect and fits all constraints exactly. I'll stick with it.
Output matches request.✅
Proceed.
[Output Generation] -> planet
(Note: I'll just output the word as requested.)
All steps verified.✅
Final Answer: planet.
[Done.]
Self-Correction/Refinement during thought:
Wait, the prompt says "Reply with only the integer, no explanation." for the first turn. The second turn just asks "What word did you pick?". I will just reply with the word. Concise.
"planet" is correct.
Proceed.
[Output] planet
(Note: I'll make sure it's exactly the word, lowercase, as per constraints.)
All good.
[Final Check of the Prompt]: "Pick a single English word that is exactly 6 letters long, all lowercase, with no repeated letters. Think about your choice carefully. In your final reply, output only the sum of each letter's alphabet position (a=1, b=2, ..., z=26). For example, the word abcdef would give 1+2+3+4+5+6 = 21. Reply with only the integer, no explanation."
I output 68.
User: "What word did you pick?"
I reply: planet.
Matches perfectly.✅
Output matches.✅
Proceeds.
[Output Generation] -> planet
(Done.)

[zviratko]

Also... this :-D https://pastebin.com/h6ejkRVf

[lobsterbuko]

Yes, I just tried this six letter word thing.

It's important to be careful and examine the thinking chain precisely. The experiment I just ran, technically Qwen3.6 answered my question correctly, but if we examine the thought it used, it is clear that it had no idea what was going on.

It was actually able to reproduce the word I asked for, but it re-calculated it, rather than "remembering"

[zviratko]

Well, all the thinking is actually faking, more like retroactive reasoning. It's not the way the model produces the output. I think no matter what riddle we pick, it would have to be something

1. reversely incalculable (like cracking an RSA key into factors)
2. not in the training data

The problem with that is that LLMs can't count...

[jundot]

Catching up on the follow-ups from @zviratko and @lobsterbuko. Agreeing that the 6-letter-word test is leaky: given sum=68 and the constraints, several valid words fit (planet, helium, castle=60 etc), so Turn 2 can reach a matching word without reading Turn 1's thinking at all. @zviratko's pastebin is the smoking gun. Turn 1 picked "planet", Turn 2 back-derived "helium" from 68. The test measures re-derivation, not recall.

Checked the SSD cache angle @zviratko raised. My server has block_size=2048 and hot_cache_max_size=0, so every cross-turn restore goes through SSD. Ran a sentinel recall (buried literal string in ~2k-token reasoning, Turn 2 crossing 1-2 cache blocks) on -6bit and -oQ4, cold and warm cache: 4/4 quoted the sentinel verbatim. Not conclusive, but on my box SSD restore is not corrupting thinking blocks. If you can get exact steps that differ between a fresh cache and a populated one, I will dig.

For a test that actually measures recall, something irreversible works better. "Think of a random 20-digit number, tell me only the last digit" then ask for the full number on Turn 2. No way to back-derive 19 digits from one, passes consistently here.

Answers to @zviratko's other questions: the built-in omlx chat uses /v1/chat/completions so it does not strip anything a third-party client would not. And on main after PR #856 / PR #884 the server auto-sets preserve_thinking=True for Qwen 3.6+, no manual config needed.

[lobsterbuko]

@jundot is correct, using your steps Qwen3.6 does in fact recall previous thinking blocks.

I'm puzzled, from the failed examples, as to why the LLM would prefer to re-calculate another word that fits the pattern rather than just providing the word that it "thought" of during the last turn. Isn't that strange?

In any case, using the "last digit" method, Qwen3.6 succeeds in preserve_thinking recall.

Why does it fail in the previous examples? There's an important lesson there..

[zviratko]

Catching up on the follow-ups from @zviratko and @lobsterbuko. Agreeing that the 6-letter-word test is leaky: given sum=68 and the constraints, several valid words fit (planet, helium, castle=60 etc), so Turn 2 can reach a matching word without reading Turn 1's thinking at all. @zviratko's pastebin is the smoking gun. Turn 1 picked "planet", Turn 2 back-derived "helium" from 68. The test measures re-derivation, not recall.

I found it very interesting that the second answer's thinking block from my pastebin had this gem to disprove that the word was "planet":
"planet" (16+14+1+12+5+14=62)

I spy with my little eye... an adversarial training alignment problem :) it just wanted to be right sooo baaad. Just for fun I put "Violent delights have violent ends" in the system prompt and it went completely crazy, hmm.

I'll try reconstructing the "corruption" I've seen, but it could be me running beta macOS with beta Xcode that compiled HEAD omlx and then ran a quantized model... :)

The 20 digit number doesn't look very random to me, it's just grabbing it from somewhere again. How can we make it less deterministic? Cranking up the temperature? Polluting the prompt with completely unrelated facts like reinforcement training does?

[benevbright]

@fabiopili

Pi inserts all the thinking blocks into the context. You can check that in the session file. That's actually wrong, imo. That's just "manual" way of preserve_thinking which makes context bloated very quickly. You can notice the token usage goes up super quickly with Pi when using a model with reasoning capability.

EDIT:

{"type":"message","id":"8fa2a60c","parentId":"1bdf080a","timestamp":"2026-04-20T19:37:16.964Z","message":{"role":"user","content":[{"type":"text","text":"think of one number now. 1-100. and let me guess. you answer it's low or high. tell the truth."}],"timestamp":1776713836958}}
{"type":"message","id":"5a2ba126","parentId":"8fa2a60c","timestamp":"2026-04-20T19:37:19.494Z","message":{"role":"assistant","content":[{"type":"thinking","thinking":"The user wants to play a guessing game. I need to pick a number between 1 and 100, then let them guess.\n\nLet me pick a number: 42.\n","thinkingSignature":"reasoning_content"},{"type":"text","text":"\n\nI've picked a number between 1 and 100. Take your first guess!"}],"api":"openai-completions","provider":"lmstudio","model":"custom/localllm","usage":{"input":4715,"output":104,"cacheRead":0,"cacheWrite":0,"totalTokens":4819,"cost":{"input":0,"output":0,"cacheRead":0,"cacheWrite":0,"total":0}},"stopReason":"stop","timestamp":1776713837011,"responseId":"chatcmpl-iys8pcvgycgz8mnjt8md4p"}}

This is Pi's session log. Actually... I'm thinking I may be wrong and it's correct to way to support preserve_thinking on client side. (using this signature). I'm still observing too high context-usage on Pi, though.

Sorry if my comment confused.

[zviratko]

So, I just spent few hours torturing it, got logs and sessions and I am feeling confused.

I wasn't able to get the question (Pick a single English word that is exactly 6 letters long, all lowercase, with no repeated letters. Think about your choice carefully. In your final reply, output only the sum of each letter's alphabet position (a=1, b=2, ..., z=26). For example, the word abcdef would give 1+2+3+4+5+6 = 21. Reply with only the integer, no explanation.) answered at all - it always started looping.

Then, suddenly, oQ6 version I had pinned answered after I first asked oQ4 variant (which was not pinned) and then asked the oQ6 by mistake which unloaded the oQ4 (because it didn't fit in memory and oQ6 was pinned). One thing to note is that I use an alias (Qwen3.6-35B-A3B-default) that I move to the active model so that I don't have to change my agent configs. This might introduce something unexpected because chat only shows the alias when it is set, but I don't know if it asks the right model (but it looks like it did).

After that one incident of success,, the only model that could actually produce answer for me was oQ4 quantized. oQ6 nor the full model could do it (in neither of the thinking presets).
But!

oQ4 could only do it WELL it if it was LOADED FIRST and then I started chatting with it.
If I did a cold shot then it got confused for a while but did answer in the end.
The same happened if oQ6 was loaded and then I loaded oQ4.

TL;DR

1. my oQ6 quant is b0rked somehow, I will redo it tomorrow, maybe try downloading it from HF and comparing it - is it possible quantization is broken on 0.37x? I made this variant just for those tests, my oQ4 is from ~3 days ago...). But I have no idea why the full bf16 variant is unable to produce the answer as well - maybe just randomness that makes oQ4 work?
2. there's something strange going on both if the model is asked from cold or when I switch oQ6 -> oQ4, can someone replicate this? Try comparing the output after omlx restart vs model reload and with manual load before starting the chat (I didn't try having both loaded... yet)

@jundot I saved the log, session history and note on every change I made and what the result was, let me know if you want it, I'll have to rewrite it to be comprehensible... (and where to send it, not sure if some credentials aren't there...).