Compile bug: zero-size array ‘gemm_gemv_kernels’ / invalid feature modifier ‘sme’

[jpm-canonical]

Git commit

a7417f5

Operating systems

Linux

GGML backends

CPU

Problem description & steps to reproduce

I'm trying to create a portable arm64 build, which includes KleidiAI to have optimizations which are selected during runtime.

To make the binary portable to all arm64 CPUs, I use the -DGGML_NATIVE=OFF flag. To enable arm64 support I add -DGGML_CPU_AARCH64=ON, followed by -DGGML_CPU_KLEIDIAI=ON to enable KleidiAI. The build fails with this combination with the log added below.

When I remove -DGGML_NATIVE=OFF, the build succeeds. But my understanding is that it builds for the host CPU's features, so the resulting binary is not portable. Please correct me if my understanding is wrong.

Is there a recommended way to build llama.cpp for arm64, being portable to different versions of the CPU, while still getting the best optimization available using KleidiAI?

First Bad Commit

No response

Compile command

cmake -B build -DGGML_NATIVE=OFF -DGGML_CPU_AARCH64=ON -DGGML_CPU_KLEIDIAI=ON -DLLAMA_CURL=OFF

Relevant log output

$ rm -rf build

$ cmake -B build -DGGML_NATIVE=OFF -DGGML_CPU_AARCH64=ON -DGGML_CPU_KLEIDIAI=ON -DLLAMA_CURL=OFF
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "2.43.0") 
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: aarch64
-- GGML_SYSTEM_ARCH: ARM
-- Including CPU backend
-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- ARM detected
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E - Failed
-- ARM feature FMA enabled
-- Using KleidiAI optimized kernels if applicable
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
-- Adding CPU backend variant ggml-cpu:  
-- Configuring done (3.5s)
-- Generating done (0.2s)
-- Build files have been written to: /home/ubuntu/llama.cpp/build

$ cmake --build build --config Release
[  0%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml.c.o
[  1%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml.cpp.o
[  1%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml-alloc.c.o
[  1%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-backend.cpp.o
[  1%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-opt.cpp.o
[  3%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-threading.cpp.o
[  3%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml-quants.c.o
[  3%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/gguf.cpp.o
[  3%] Linking CXX shared library ../../bin/libggml-base.so
[  3%] Built target ggml-base
[  4%] Building C object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.c.o
[  4%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.cpp.o
[  4%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/repack.cpp.o
[  4%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/hbm.cpp.o
[  6%] Building C object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/quants.c.o
[  6%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/traits.cpp.o
[  6%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/amx/amx.cpp.o
[  6%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/amx/mmq.cpp.o
[  7%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/binary-ops.cpp.o
[  7%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/unary-ops.cpp.o
[  7%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/vec.cpp.o
[  9%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ops.cpp.o
[  9%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/llamafile/sgemm.cpp.o
[  9%] Building C object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/arch/arm/quants.c.o
[  9%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/arch/arm/repack.cpp.o
[ 10%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/kleidiai/kleidiai.cpp.o
[ 10%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/kleidiai/kernels.cpp.o
/home/ubuntu/llama.cpp/ggml/src/ggml-cpu/kleidiai/kernels.cpp:28:30: error: zero-size array ‘gemm_gemv_kernels’
   28 | static ggml_kleidiai_kernels gemm_gemv_kernels[] = {
      |                              ^~~~~~~~~~~~~~~~~
gmake[2]: *** [ggml/src/CMakeFiles/ggml-cpu.dir/build.make:300: ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/kleidiai/kernels.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1752: ggml/src/CMakeFiles/ggml-cpu.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2

[slaren]

Try:
-DGGML_BACKEND_DL=ON -DGGML_NATIVE=OFF -DGGML_CPU_ALL_VARIANTS=ON -DGGML_CPU_KLEIDIAI=ON

[jpm-canonical]

Good idea, going the good direction, but that also seems to have an issue. Getting Failed to get ARM features on sme. Looks like it is being checked against the host during setup and compilation.

jpmeijers@pi-2404-server:~/llama.cpp$ rm -rf build

jpmeijers@pi-2404-server:~/llama.cpp$ cmake -B build -DGGML_BACKEND_DL=ON -DGGML_NATIVE=OFF -DGGML_CPU_ALL_VARIANTS=ON -DGGML_CPU_KLEIDIAI=ON -DLLAMA_CURL=OFF
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "2.43.0") 
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: aarch64
-- GGML_SYSTEM_ARCH: ARM
-- Including CPU backend
-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- ARM detected
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E - Failed
-- ARM feature FMA enabled
-- Using KleidiAI optimized kernels if applicable
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
-- Adding CPU backend variant ggml-cpu-armv8.0_1: -march=armv8-a 
-- ARM detected
-- ARM feature DOTPROD enabled
-- ARM feature FMA enabled
-- Using KleidiAI optimized kernels if applicable
-- Adding CPU backend variant ggml-cpu-armv8.2_1: -march=armv8.2-a+dotprod GGML_USE_DOTPROD
-- ARM detected
-- ARM feature DOTPROD enabled
-- ARM feature FMA enabled
-- ARM feature FP16_VECTOR_ARITHMETIC enabled
-- Using KleidiAI optimized kernels if applicable
-- Adding CPU backend variant ggml-cpu-armv8.2_2: -march=armv8.2-a+dotprod+fp16 GGML_USE_DOTPROD;GGML_USE_FP16_VECTOR_ARITHMETIC
-- ARM detected
-- ARM feature DOTPROD enabled
-- ARM feature SVE enabled
-- ARM feature FMA enabled
-- ARM feature FP16_VECTOR_ARITHMETIC enabled
-- Using KleidiAI optimized kernels if applicable
-- Adding CPU backend variant ggml-cpu-armv8.2_3: -march=armv8.2-a+dotprod+fp16+sve GGML_USE_DOTPROD;GGML_USE_FP16_VECTOR_ARITHMETIC;GGML_USE_SVE
-- ARM detected
-- ARM feature DOTPROD enabled
-- ARM feature SVE enabled
-- ARM feature MATMUL_INT8 enabled
-- ARM feature FMA enabled
-- ARM feature FP16_VECTOR_ARITHMETIC enabled
-- Using KleidiAI optimized kernels if applicable
-- Adding CPU backend variant ggml-cpu-armv8.6_1: -march=armv8.6-a+dotprod+fp16+sve+i8mm GGML_USE_DOTPROD;GGML_USE_FP16_VECTOR_ARITHMETIC;GGML_USE_SVE;GGML_USE_MATMUL_INT8
-- ARM detected
-- ARM feature DOTPROD enabled
-- ARM feature SVE enabled
-- ARM feature MATMUL_INT8 enabled
-- ARM feature FMA enabled
-- ARM feature FP16_VECTOR_ARITHMETIC enabled
-- Using KleidiAI optimized kernels if applicable
-- Adding CPU backend variant ggml-cpu-armv8.6_2: -march=armv8.6-a+dotprod+fp16+sve+i8mm+sve2 GGML_USE_DOTPROD;GGML_USE_FP16_VECTOR_ARITHMETIC;GGML_USE_SVE;GGML_USE_MATMUL_INT8;GGML_USE_SVE2
-- ARM detected
cc1: error: invalid feature modifier ‘sme’ in ‘-march=armv9.2-a+dotprod+fp16+sve+i8mm+sme’
cc1: note: valid arguments are: fp simd crc lse fp16 rcpc rdma dotprod aes sha2 crypto sha3 sm4 fp16fml sve profile rng memtag sb ssbs predres sve2 sve2-sm4 sve2-aes sve2-sha3 sve2-bitperm tme i8mm f32mm f64mm bf16 flagm pauth ls64 mops cssc; did you mean ‘sm4’?
CMake Warning at ggml/src/ggml-cpu/CMakeLists.txt:220 (message):
  Failed to get ARM features
Call Stack (most recent call first):
  ggml/src/CMakeLists.txt:295 (ggml_add_cpu_backend_variant_impl)
  ggml/src/CMakeLists.txt:329 (ggml_add_cpu_backend_variant)


-- Using KleidiAI optimized kernels if applicable
-- Adding CPU backend variant ggml-cpu-armv9.2_1: -march=armv9.2-a+dotprod+fp16+sve+i8mm+sme GGML_USE_DOTPROD;GGML_USE_FP16_VECTOR_ARITHMETIC;GGML_USE_SVE;GGML_USE_MATMUL_INT8;GGML_USE_SME
-- ARM detected
cc1: error: invalid feature modifier ‘sme’ in ‘-march=armv9.2-a+dotprod+fp16+sve+i8mm+sve2+sme’
cc1: note: valid arguments are: fp simd crc lse fp16 rcpc rdma dotprod aes sha2 crypto sha3 sm4 fp16fml sve profile rng memtag sb ssbs predres sve2 sve2-sm4 sve2-aes sve2-sha3 sve2-bitperm tme i8mm f32mm f64mm bf16 flagm pauth ls64 mops cssc; did you mean ‘sm4’?
CMake Warning at ggml/src/ggml-cpu/CMakeLists.txt:220 (message):
  Failed to get ARM features
Call Stack (most recent call first):
  ggml/src/CMakeLists.txt:295 (ggml_add_cpu_backend_variant_impl)
  ggml/src/CMakeLists.txt:330 (ggml_add_cpu_backend_variant)


-- Using KleidiAI optimized kernels if applicable
-- Adding CPU backend variant ggml-cpu-armv9.2_2: -march=armv9.2-a+dotprod+fp16+sve+i8mm+sve2+sme GGML_USE_DOTPROD;GGML_USE_FP16_VECTOR_ARITHMETIC;GGML_USE_SVE;GGML_USE_MATMUL_INT8;GGML_USE_SVE2;GGML_USE_SME
-- Configuring done (6.2s)
-- Generating done (0.3s)
-- Build files have been written to: /home/jpmeijers/llama.cpp/build

jpmeijers@pi-2404-server:~/llama.cpp$ cmake --build build --config Release
[  0%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml.c.o
[  0%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml.cpp.o
[  0%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml-alloc.c.o
[  0%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-backend.cpp.o
[  0%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-opt.cpp.o
[  1%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-threading.cpp.o
[  1%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml-quants.c.o
[  1%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/gguf.cpp.o
[  1%] Linking CXX shared library ../../bin/libggml-base.so
[  1%] Built target ggml-base
[  1%] Building CXX object ggml/src/CMakeFiles/ggml-cpu-armv9.2_2-feats.dir/ggml-cpu/arch/arm/cpu-feats.cpp.o
[  1%] Built target ggml-cpu-armv9.2_2-feats
[  1%] Building C object ggml/src/CMakeFiles/ggml-cpu-armv9.2_2.dir/ggml-cpu/ggml-cpu.c.o
cc1: error: invalid feature modifier ‘sme’ in ‘-march=armv9.2-a+dotprod+fp16+sve+i8mm+sve2+sme’
cc1: note: valid arguments are: fp simd crc lse fp16 rcpc rdma dotprod aes sha2 crypto sha3 sm4 fp16fml sve profile rng memtag sb ssbs predres sve2 sve2-sm4 sve2-aes sve2-sha3 sve2-bitperm tme i8mm f32mm f64mm bf16 flagm pauth ls64 mops cssc; did you mean ‘sm4’?
gmake[2]: *** [ggml/src/CMakeFiles/ggml-cpu-armv9.2_2.dir/build.make:76: ggml/src/CMakeFiles/ggml-cpu-armv9.2_2.dir/ggml-cpu/ggml-cpu.c.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:2080: ggml/src/CMakeFiles/ggml-cpu-armv9.2_2.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2