unable to run ramalama using --runtime vllm on macOS

[benoitf]

trying ramalama on macOS 15 using --runtime vllm

I got

Trying to pull quay.io/modh/vllm:rhoai-2.18-cuda...
Error: choosing an image from manifest list docker://quay.io/modh/vllm:rhoai-2.18-cuda: no image found in image index for architecture "arm64", variant "v8", OS "linux"

it seems to fetch a cuda image while I am on Apple silicon.

[benoitf]

https://docs.vllm.ai/en/latest/getting_started/installation/cpu/index.html

it's not optimal but it can run on macOS

$ podman run --rm -it -p 8090:8000 localhost/vllm-cpu-env --model TinyLlama/TinyLlama-1.1B-Chat-v1.0                                                             INFO 02-13 10:52:31 __init__.py:190] Automatically detected platform cpu.
INFO 02-13 10:52:31 api_server.py:840] vLLM API server version 0.7.3.dev116+g578087e5
...

[ericcurtin]

I think you found your answer in the above doc:

"vLLM has experimental support for macOS with Apple silicon. For now, users shall build from the source vLLM to natively run on macOS."

This is one for the vLLM folks

[ericcurtin]

A general issue around vLLM should be opened if not already, could do with some Containerfile work.

llama.cpp is more suitable for macOS runtime today

[benoitf]

I don't see why it's being closed as fixed

The user experience is bad. It tries to fetch an image that does not exists

It should report a good error message

[benoitf]

"vLLM has experimental support for macOS with Apple silicon. For now, users shall build from the source vLLM to natively run on macOS."

some ramalama images are building llama.cpp from sources so it could also build vLLM for a later arm/macOS usage
I'm not talking about running vLLM locally on my laptop but being containerized (where it's already compiled)