Eval bug: error loading model: unknown model architecture: 'bailingmoe3' in latest b10269-1.5.1

[dineshr93]

followed instructions from https://huggingface.co/AtomicChat/Ling-3.0-flash-GGUF

wget https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant/releases/download/b10269-1.5.1/llama-turboquant-linux-x64-rocm.tar.gz
tar xzf llama-turboquant-linux-x64-rocm.tar.gz
cd build/bin

./llama-server -m ~/models/Ling-3.0-flash-GGUF/Ling-3.0-flash-AD-Q5_K_M-00001-of-00002.gguf --host 0.0.0.0 --port 8888 --jinja -ngl 99 -c 262144
0.00.020.573 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.022.107 I srv    load_model: loading model '/home/username/models/Ling-3.0-flash-GGUF/Ling-3.0-flash-AD-Q5_K_M-00001-of-00002.gguf'
0.00.066.194 E llama_model_load: error loading model: unknown model architecture: 'bailingmoe3'
0.00.066.206 E llama_model_load_from_file_impl: failed to load model
0.00.066.249 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
0.00.108.526 E llama_model_load: error loading model: unknown model architecture: 'bailingmoe3'
0.00.108.531 E llama_model_load_from_file_impl: failed to load model
0.00.108.537 E cmn  common_init_: failed to load model '/home/username/models/Ling-3.0-flash-GGUF/Ling-3.0-flash-AD-Q5_K_M-00001-of-00002.gguf'
0.00.108.543 E srv    load_model: failed to load model, '/home/username/models/Ling-3.0-flash-GGUF/Ling-3.0-flash-AD-Q5_K_M-00001-of-00002.gguf'
0.00.108.546 I srv    operator(): operator(): cleaning up before exit...
0.00.109.748 E srv  llama_server: exiting due to model loading error

Operating systems

Linux

GGML backends

HIP

Hardware

GGML backends is rocm (so selected hip) llama.cpp

AMD Ryzen AI Max+ 395(128GB)" chip with OS "Ubuntu 26.04 LTS" AMD Radeon 8060S (gfx1151)

Models

https://huggingface.co/AtomicChat/Ling-3.0-flash-GGUF/tree/main/AD-Q5_K_M

Problem description & steps to reproduce

wget https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant/releases/download/b10269-1.5.1/llama-turboquant-linux-x64-rocm.tar.gz
tar xzf llama-turboquant-linux-x64-rocm.tar.gz
cd build/bin



./llama-server -m ~/models/Ling-3.0-flash-GGUF/Ling-3.0-flash-AD-Q5_K_M-00001-of-00002.gguf --host 0.0.0.0 --port 8888 --jinja -ngl 99 -c 262144

Relevant log output

Logs

./llama-server -m ~/models/Ling-3.0-flash-GGUF/Ling-3.0-flash-AD-Q5_K_M-00001-of-00002.gguf --host 0.0.0.0 --port 8888 --jinja -ngl 99 -c 262144
0.00.020.573 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.022.107 I srv    load_model: loading model '/home/username/models/Ling-3.0-flash-GGUF/Ling-3.0-flash-AD-Q5_K_M-00001-of-00002.gguf'
0.00.066.194 E llama_model_load: error loading model: unknown model architecture: 'bailingmoe3'
0.00.066.206 E llama_model_load_from_file_impl: failed to load model
0.00.066.249 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
0.00.108.526 E llama_model_load: error loading model: unknown model architecture: 'bailingmoe3'
0.00.108.531 E llama_model_load_from_file_impl: failed to load model
0.00.108.537 E cmn  common_init_: failed to load model '/home/username/models/Ling-3.0-flash-GGUF/Ling-3.0-flash-AD-Q5_K_M-00001-of-00002.gguf'
0.00.108.543 E srv    load_model: failed to load model, '/home/username/models/Ling-3.0-flash-GGUF/Ling-3.0-flash-AD-Q5_K_M-00001-of-00002.gguf'
0.00.108.546 I srv    operator(): operator(): cleaning up before exit...
0.00.109.748 E srv  llama_server: exiting due to model loading error

[dineshr93]

update: instead of ready made build I manually built the binary using the Rocm and vulkan build and both worked. Serving the llm via Vulkan served better performance of 33 avg t/s. while rocm llama-server binary gave avg 28 t/s.

Building

Rocm build:
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

vulkan build: (sudo apt install libvulkan-dev vulkan-tools glslang-tools)
cmake -S . -B buildvulk \
  -DGGML_VULKAN=ON \
  -DCMAKE_BUILD_TYPE=Release \
  && cmake --build buildvulk --config Release -j$(nproc)

Serving

serving with Rocm build:
LD_LIBRARY_PATH=/home/dinesh/llamacpps/llama.cpp/atomic-llama-cpp-turboquant/build/bin build/bin/llama-server -m ~/models/Ling-3.0-flash-GGUF/Ling-3.0-flash-AD-Q5_K_M-00001-of-00002.gguf --host 0.0.0.0 --port 8888 --jinja -ngl 99 -c 131072 --load-mode none -fa off

serving with Vulkan build:
LD_LIBRARY_PATH=/home/dinesh/llamacpps/llama.cpp/atomic-llama-cpp-turboquant/buildvulk/bin buildvulk/bin/llama-server -m ~/models/Ling-3.0-flash-GGUF/Ling-3.0-flash-AD-Q5_K_M-00001-of-00002.gguf --host 0.0.0.0 --port 8888 --jinja -ngl 99 -c 131072 --no-mmap -fa on

The readymade binaries in the release is not working.

[worthant]

Hi @dineshr93! Thanks for finding that issue, I'll investigate it ASAP.