[BUG]gemma 3 27b exl2 loops nonsense afterwards 2-3 correct paragraphs

[ciprianveg]

OS

Windows

GPU Library

CUDA 12.x

Python version

3.12

Pytorch version

Tabbyapi latest

Model

No response

Describe the bug

[BUG]i just updated tabbyapi to be able to use the latest exllama 0.29 version with gemma 3 27b exl2, but asking it to tell a kid story ends up looping nonsense afterwards 2-3 correct paragraphs. I tried with the turboderp 4, 5 and 6bpw models, i limited to 20000 and to 10000 context length in tabby api and tried with cache mode q4, q6 and FP16. Same behaviour..

Reproduction steps

Tabbyapi latest version with exllama 0.29. Any of the gemma-3 27b 4, 5, 6bpw models. Cache q4 q6 or fp16, tell a kids story with a wizard. Nonsense loop after a couple of good paragraphs

Expected behavior

Non nonsense loops

Logs

No response

Additional context

No response

Acknowledgements

• I have looked for similar issues before submitting this one.
• I understand that the developers have lives and my issue will be answered when possible.
• I understand the developers of this program are human, and I will ask my questions politely.

[turboderp]

First of all can you verify that you have 0.2.9 installed, and that TabbyAPI isn't somehow using 0.2.8-dev in its own venv? This is precisely how Gemma3 failed up until the most recent fixes right before the release.

[ciprianveg]

Hello,
in my tabby pull and update output i see:
Requirement already satisfied: pytz>=2020.1 in c:\ai\tabby\tabbyapi\venv\lib\site-packages (from pandas->exllamav2@ https://github.com/kingbri1/exllamav2/releases/download/v0.2.9/exllamav2-0.2.9+cu128.torch2.7.0-cp310-cp310-win_amd64.whl->tabbyAPI==0.0.1) (2024.2)
is some other way to assure it gets updated?

[ciprianveg]

to be sure it is not a windows issue caused by drivers, i tried it also in docker and i have the same behaviour:

it looks like it starts looping nonsense some paragraphs later, but it still does. My query is to tell the kids a long story about a beach near our home

[ciprianveg]

it looks like the context cache quantization creates the loop issue, even at q8. On docker if I use fp16 i managed it to work but only with a small context, 6000, even if there was free space on my 3090, if i increased it it started the nonsense loop after some paragraphs with fp16, too. It looks like the issue is the context memory handling. With q4-q8, there are issues, with f16 if you do not leave 3-4Gb free on my 3090, there are issues, maybe it doesn't compute correctly the memory needs... with vllm I have the same issues with --kv-cache-dtype fp8 quantization of the cache, but without it, i can use 11000 context using 23.4Gb of my 3090.. The issue should be caused by the new gemma-3 context cache implementation..

[meigami0]

Same here. When using max_seq_len >8192 (e.g., 16384), it easily triggers the loop issue.
I suspect this might be related to Gemma-3’s 1024-token sliding_window being insufficient for generation windows in long contexts, potentially causing attention pattern conflicts.

[Downtown-Case]

+1, I had a similar issue and just switched away from Gemma 3 TBH.

[Downtown-Case]

Addendum: I am not sure what commit I last tried it on.

[hidoba]

The original unquantized Gemma 3 has activations way over 65500 (which is a max value for float16 which exllamav2 uses): https://www.unsloth.ai/blog/gemma3

I don't think original Gemma 3 models are going to work well.

Google also released QAT versions (their own quantizations optimized at the time of training), but I'm not sure if their format is one of the supported strategies in exllamav2.

[Downtown-Case]

I extracted the q4_0 QAT to a huggingface format myself. Didn't work well at long context for me, but I'll upload it if anyone wants to try.

[r0mar0ma]

Not sure is this the same issue but I was getting a garbage from any gemma-3 (4b, 12b, 27b) with any exl2 quantization for input approx 7000+ tokens (maybe less, haven't checked).

The output was like:
cause Causeধারységразອນ Causeбомble Vitamin المعادगित causeধার體的 ant Cause vitaminળ सख्त cause causes causeধারбут পরিচালনার spiking Causeразбом causeອນ钾‌ನ cause vitamin Cause cause..........
and so on up to requested max_new_tokens.

But this doesn't happen with abliterated (mlabonne) versions from HF. That versions has a lot of parameters in config that is not specified in original versions.
So just as experiment tried to copy "text_config" (and "vision_config" just in case) sections in config.json from abliterated exl2 to "original" exl2 and got totally success (all of 4b, 12b and 27b with different exl2 quantizations) even for 90000+ input tokens.

Probably exllamav2 just using different default values for some of parameters, so it looks like it's more of a configuration than quantization issue.

[Downtown-Case]

@r0mar0ma

That explains a lot! Yeah, I ran into a similar issue.

FYI Llama.cpp just got sliding window attention sorted out for gemma 3, so (combined with the QAT quants, and with nothing against the wonderfull exllamav2/v3) it's likely a more optimal way to run this specific family, especially at longer contexts.