Eval bug: Ternary Bonsai 27B don't run

[Filippide8864]

Name and Version

CUDA 12 llama.cpp v2.25.0 for windows (LM Studio)

Operating systems

Windows

GGML backends

CUDA

Hardware

i5 13gen + RTX 4050 Laptop

Models

Ternary Bonsai 27B

Problem description & steps to reproduce

When I try to run the model, it crash.

First Bad Commit

No response

Relevant log output

Logs

[kth8]

#25707

[khosravipasha]

Yeah still pending on CUDA, also make sure to download the Ternary-Bonsai-27B-Q2_g64.gguf. The Q2_0g64 one is the one that will be the offical one that works.

Also even after above merged need to wait for LM Studio to pick up the changes.

In the meantime can try our demo repo which setups up the right dependencies and downloads the rights models for you
https://github.com/PrismML-Eng/Bonsai-demo

[gh055]

I can run the Q2_0g64 version on my RTX 5080 w/ llama.cpp-cuda (b10075), but output speed is only abysmal 1-2 t/s. Are any of my options wrong?

--host 0.0.0.0 --port 8080 
--mlock --no-mmap 
--n-gpu-layers 99 
--flash-attn on 
--numa distribute 
--batch-size 128 
--threads 14 
--tools read_file,get_datetime 
--cpu-moe 
--cache-type-k q8_0 
--cache-type-v q8_0 
--ctx-size 32768 
--temp 0.7 
--top-p 0.95 
--top-k 20 
--reasoning off

[khosravipasha]

@gh055

The cuda PR just merged today, can try again it should work from now in future llama.cpp releases
#25707

[gh055]

I tested with b10216 and it works now. Thanks!