[BUG] ExllamaV2 version >0.2.8 broken for mistral 7b(v0.2) models on Nvidia 2060

[IceFog72]

OS

Windows

GPU Library

CUDA 12.x

Python version

3.11

Pytorch version

2.7.0+cu128

Model

No response

Describe the bug

After 0.2.8 all next ExllamaV2 libs make mistral 7b v0.2 models give gibberish (v0.1 is ok). It's like rope is way off. I see same things if I use tabby on Google colab with 7b-12b nemo, it return text with lost coherence, just random words.

running torch-2.6.0+cu124 next to 0.3.1 not help (but it starts return words instead of gibberish, still incoherent)

torch-2.6.0

Reproduction steps

Run tabby with ExllamaV2 0.2.9+ on nvidia 20xx/google colab using mistral v0.2 7b models / 12b nemo

Expected behavior

To work like ExllamaV2 0.2.8 with PyTorch version 2.6.0+cu124

Logs

No response

Additional context

No response

Acknowledgements

• I have looked for similar issues before submitting this one.
• I understand that the developers have lives and my issue will be answered when possible.
• I understand the developers of this program are human, and I will ask my questions politely.

[IceFog72]

!git clone https://github.com/theroyallab/tabbyAPI
%cd tabbyAPI
!git reset --hard f2c7da2

one of last working commits for me with 0.2.8
after when tabby updated to 0.2.9 I could still use it if I downgrade exl2 to 0.2.8
now tabby requires 0.3.1

[Vhallo]

Gave IceMedovukhaRP-7b a quick test and it works completely fine with exllamav2-0.3.1+cu128.torch2.7.0-cp313-cp313-win_amd64.whl & latest TabbyAPI.

12B Nemo models don't have any problems either. So at least it's not a general issue.

[IceFog72]

Gave IceMedovukhaRP-7b a quick test and it works completely fine with exllamav2-0.3.1+cu128.torch2.7.0-cp313-cp313-win_amd64.whl & latest TabbyAPI.

12B Nemo models don't have any problems either. So at least it's not a general issue.

Windows, Linux? 20xx card? 30xx+ with flash attention on?

[Vhallo]

Windows & 40xx + FA, so possible that no FA is causing issues.

[Originalimoc]

It also breaks Qwen2.5-7B-8bpw. 30xx+FA good. 20xx compatibility mode nope.

[Ph0rk0z]

Y'all tried to install/upgrade xformers? Without FA you get either torch or xformers attention. If you want to use old version with tabby, just patch out the version check.