RamaLama won't recognize RX5700XT

[Split7fire]

The whole story starts in @RealVishy comment in #2503. That comment said, that RX5700XT is working good with RamaLama on Linux.

I'm using Bluefin-dx and tried to run RamaLama with no avail. I created an issue here ublue-os/bluefin#2197. And got an suggestion to post an issue here.

I'm using RamaLama bundled with distro:

❯ /usr/bin/ramalama -v
ramalama version 0.5.2

Testing:

/usr/bin/ramalama --debug run llama3.2
run_cmd:  podman inspect quay.io/ramalama/rocm:0.5
Working directory: None
Ignore stderr: False
Ignore all: True
exec_cmd:  podman run --rm -i --label RAMALAMA --security-opt=label=disable --name ramalama_Eef0KsY5uh --pull=newer -t --device /dev/dri --device /dev/kfd -e HIP_VISIBLE_DEVICES=0 --mount=type=bind,src=/var/home/vlad/.local/share/ramalama/models/ollama/llama3.2:latest,destination=/mnt/models/model.file,ro quay.io/ramalama/rocm:latest llama-run -c 2048 --temp 0.8 -v /mnt/models/model.file
Loading modelggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no                                                                                                                     
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 5700 XT, gfx1010:xnack- (0x1010), VMM: no, Wave Size: 32

~ took 5s 

Any help appreciated.

[ericcurtin]

Maybe you need this fix?

#802

[Split7fire]

Maybe you need this fix?

#802

Thanks for your suggestion. I tried:

❯ /usr/bin/ramalama --debug --ngl=999 run llama3.2
usage: ramalama [-h] [--container] [--debug] [--dryrun] [--engine ENGINE] [--gpu] [--image IMAGE] [--nocontainer] [--runtime {llama.cpp,vllm}] [--store STORE] [-v]
                {help,containers,ps,convert,info,list,ls,login,logout,pull,push,rm,run,serve,stop,version} ...
ramalama: error: unrecognized arguments: --ngl=999

~ 

[ericcurtin]

It's --ngl 999 rather than --ngl=999

[Split7fire]

It's --ngl 999 rather than --ngl=999

Yeah, I tried that too

❯ /usr/bin/ramalama --debug --ngl 999 run llama3.2
usage: ramalama [-h] [--container] [--debug] [--dryrun] [--engine ENGINE] [--gpu] [--image IMAGE] [--nocontainer] [--runtime {llama.cpp,vllm}] [--store STORE] [-v]
                {help,containers,ps,convert,info,list,ls,login,logout,pull,push,rm,run,serve,stop,version} ...
ramalama: error: argument subcommand: invalid choice: '999' (choose from help, containers, ps, convert, info, list, ls, login, logout, pull, push, rm, run, serve, stop, version)

~ 

[ericcurtin]

You need to put it after the run command I think

[Split7fire]

You need to put it after the run command I think

Nope.

❯ /usr/bin/ramalama --debug run --ngl 999 llama3.2
usage: ramalama [-h] [--container] [--debug] [--dryrun] [--engine ENGINE] [--gpu] [--image IMAGE] [--nocontainer] [--runtime {llama.cpp,vllm}] [--store STORE] [-v]
                {help,containers,ps,convert,info,list,ls,login,logout,pull,push,rm,run,serve,stop,version} ...
ramalama: error: unrecognized arguments: --ngl

~ 

[ericcurtin]

Can you try updating the version of ramalama, this ngl thing was added recently enough