[Help] How to use llama.cpp:server-vulkan on Asahi Linux (Apple M1) ?

[Synseria]

Description

Hello,

I'm trying to use the ghcr.io/ggml-org/llama.cpp:server-vulkan Docker image on my Mac with Asahi Linux (Apple M1), but I'm having issues getting the GPU to work. I'm not very familiar with GPU or Vulkan configuration, so any help would be appreciated.

What I'm trying to achieve:

I want to run llama.cpp:server-vulkan using my Apple M1 GPU on Asahi Linux.

Here's my simplified Deployment example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-vulkan
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llama-vulkan
  template:
    metadata:
      labels:
        app: llama-vulkan
    spec:
      containers:
      - name: llama-vulkan
        image: ghcr.io/ggml-org/llama.cpp:server-vulkan
        securityContext:
          privileged: true
        env:
        - name: LLAMA_ARG_N_GPU_LAYERS
          value: "100"
        - name: LLAMA_ARG_N_PARALLEL
          value: "1"
        - name: LLAMA_ARG_CTX_SIZE 
          value: "8192"
        - name: LLAMA_ARG_MODEL
          value: /models/Llama-3.2-3B-Instruct-Q4_0.gguf
        volumeMounts:
        - name: models
          mountPath: /models
        - name: dev-dri
          mountPath: /dev/dri
      volumes:
      - name: models
        persistentVolumeClaim:
          claimName: llama-models-pvc
      - name: dev-dri
        hostPath: 
          path: /dev/dri 
          type: Directory

What I've tried:

• Ensured /dev/dri is correctly mounted in the container.
• Installed vulkan-tools.
• Confirmed that vulkaninfo works on the host and detects the GPU:

Vulkan Instance Version: 1.4.304
Devices: count = 2
    GPU id = 0 (Apple M1 (G13G B1))
    GPU id = 1 (llvmpipe (LLVM 18.1.8, 128 bits))

• Running the container with --privileged and mounting /dev/dri.

Problem:

When I run the container, I get the following logs:

warning: no usable GPU found, --gpu-layers option will be ignored
warning: one possible reason is that llama.cpp was compiled without GPU support
warning: consult docs/build.md for compilation instructions

I suspect it might be related to:

• Incompatibility with the Apple M1 GPU.
• Misconfiguration of Vulkan or the Docker image.

Additional Information:

• I am using Asahi Linux on an Apple M1 Mac.
• The GPU is detected as Apple M1 (G13G B1) by vulkaninfo.
• I'm not very familiar with Vulkan, GPU configuration, or Docker GPU passthrough.

What I need help with:

• Is the llama.cpp:server-vulkan image compatible with the Apple M1 GPU?
• Do I need to compile the image myself with special settings?
• Is there any additional configuration needed for Vulkan on Asahi Linux?
• Any tips on debugging GPU passthrough in Docker with Vulkan?

Any guidance or tips would be greatly appreciated!