vllm: Cannot select GPU device

[scetu]

Platform

Linux/Ubuntu

Lemonade Version

10.4.0

GPU / APU Model

AMD Ryzen™ AI 9 HX PRO 370 w/ Radeon™ 890M × 24

Component

Other / Not sure

Bug Description

Cannot run vLLM backend, due to hardware configuration of laptop with AMD Ryzen AI 9 + Nvidia dGPU.

2026-05-10 21:13:12.716 [Info] (Server) Ensuring model loaded: Qwen3.5-4B-vLLM 
2026-05-10 21:13:12.716 [Info] (vLLM) Loading model: Qwen3.5-4B-vLLM
2026-05-10 21:13:12.955 [Info] (vLLM) Starting vllm-server on port 8001...
2026-05-10 21:13:12.957 [Info] (ProcessManager) Process started successfully, PID: 54798
2026-05-10 21:13:16.356 [Info] (Process) Traceback (most recent call last):
2026-05-10 21:13:16.356 [Info] (Process)   File "<frozen runpy>", line 198, in _run_module_as_main
2026-05-10 21:13:16.356 [Info] (Process)   File "<frozen runpy>", line 88, in _run_code
2026-05-10 21:13:16.356 [Info] (Process)   File "/var/lib/lemonade/.cache/lemonade/bin/vllm/rocm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 25, in <module>
2026-05-10 21:13:16.356 [Info] (Process)     from vllm.config import ModelConfig, VllmConfig
2026-05-10 21:13:16.356 [Info] (Process)   File "/var/lib/lemonade/.cache/lemonade/bin/vllm/rocm/lib/python3.12/site-packages/vllm/config/__init__.py", line 6, in <module>
2026-05-10 21:13:16.356 [Info] (Process)     from vllm.config.compilation import (
2026-05-10 21:13:16.356 [Info] (Process)   File "/var/lib/lemonade/.cache/lemonade/bin/vllm/rocm/lib/python3.12/site-packages/vllm/config/compilation.py", line 22, in <module>
2026-05-10 21:13:16.356 [Info] (Process)     from vllm.platforms import current_platform
2026-05-10 21:13:16.356 [Info] (Process)   File "<frozen importlib._bootstrap>", line 1412, in _handle_fromlist
2026-05-10 21:13:16.356 [Info] (Process)   File "/var/lib/lemonade/.cache/lemonade/bin/vllm/rocm/lib/python3.12/site-packages/vllm/platforms/__init__.py", line 277, in __getattr__
2026-05-10 21:13:16.356 [Info] (Process)     platform_cls_qualname = resolve_current_platform_cls_qualname()
2026-05-10 21:13:16.356 [Info] (Process)                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-05-10 21:13:16.356 [Info] (Process)   File "/var/lib/lemonade/.cache/lemonade/bin/vllm/rocm/lib/python3.12/site-packages/vllm/platforms/__init__.py", line 240, in resolve_current_platform_cls_qualname
2026-05-10 21:13:16.356 [Error] (Process)     raise RuntimeError(
2026-05-10 21:13:16.356 [Error] (Process) RuntimeError: Only one platform plugin can be activated, but got: ['cuda', 'rocm']
2026-05-10 21:13:17.173 [Error] (WrappedServer) vllm-server process has terminated with exit code: 1
2026-05-10 21:13:17.173 [Error] (WrappedServer) This usually means:
2026-05-10 21:13:17.173 [Error] (WrappedServer)   - Missing required drivers or dependencies
2026-05-10 21:13:17.173 [Error] (WrappedServer)   - Incompatible model file
2026-05-10 21:13:17.173 [Error] (WrappedServer)   - Try running the server manually to see the actual error
2026-05-10 21:13:22.178 [Warn] (ProcessManager) Process did not respond to SIGTERM, using SIGKILL
2026-05-10 21:13:22.178 [Info] (ProcessManager) Process terminated, waiting for GPU driver cleanup...
2026-05-10 21:13:24.178 [Error] (Router) Backend load failed: vllm-server failed to start within timeout
2026-05-10 21:13:24.178 [Warn] (Router) Load failed with non-file-not-found error, evicting all models and retrying...
2026-05-10 21:13:24.178 [Info] (Router) Evicting all models (0 total)
2026-05-10 21:13:24.178 [Info] (Router) All models evicted

System Details Report

Hardware Information:

• Processor: AMD Ryzen™ AI 9 HX PRO 370 w/ Radeon™ 890M × 24
• Graphics 1: AMD Radeon™ 890M Graphics
• Graphics 2: NVIDIA RTX PRO 1000 Blackwell Generation Laptop GPU

Steps to Reproduce

1. Download vLLM model + backend
2. Try to run the model on laptop with AMD Ryzen AI + dGPU

Expected vs Actual Behavior

Expected: Have ability to select GPU for interference with vLLM. There is not such setting now.

Log Output

Additional Context

No response

[jeremyfowers]

Is the real issue here that we don’t have a CUDA backend for vLLM? That’s less of a bug and more of a gaping feature/support gap.

[scetu]

I do not think so, on this kind of hardware, based on error message, you need to select only one GPU either ROCm or CUDA. vLLM currently see both and cannot pick just one GPU. Solution would be to have simmilar ability to select Backend for model, like for Lama.cpp, where in Model Settings, user can pick device which should be used (ROCm or CUDA or Vulkan). There is no such option for vLLM now.

[galuszkak]

I have working patch for this issue - I would like to share here idea how this would be solved and if this is acceptable (I did help myself with LLM to patch this - so before I raise any pull request I would like to get input on the approach)...

So in my view:

• We should probably add a vLLM ROCm (especially if people have more than 1 AMD GPU / APU) device option in Lemonade (vllm.device / UI model option), passed to the backend as ROCR_VISIBLE_DEVICES.
• Always launch Lemonade’s vLLM ROCm backend with VLLM_TARGET_DEVICE=rocm.
• Patch the lemonade-sdk/vllm-rocm package during build so its bundled vLLM honors VLLM_TARGET_DEVICE=rocm before probing CUDA. This prevents hybrid AMD/NVIDIA systems from activating both CUDA and ROCm platform plugins.

This keeps the CUDA suppression inside vLLM ROCm package instead of relying on a Lemonade-side Python startup shim, while Lemonade only passes explicit backend/device intent.

[fl0rianr]

Thanks @galuszkak for suggesting a fix, I think the diagnosis is right: the issue is not mainly missing CUDA support, but that vLLM can see both CUDA and ROCm even though Lemonade already selected vllm:rocm.

But I don’t think ROCR_VISIBLE_DEVICES alone fixes this, because it only affects ROCm device visibility. It does not stop vLLM’s CUDA/NVML platform probe from seeing an NVIDIA GPU before ROCm device selection matters.

I’d also prefer not to add a generic vllm.device option for this, because that mixes backend selection with device selection. If we add a CUDA vLLM backend later, Lemonade should choose exactly one backend (vllm:rocm or vllm:cuda), and then only devices for that backend should be visible to vLLM.

So my approach is: keep the selected backend authoritative. For vllm:rocm, prevent CUDA/NVIDIA from being selected during vLLM platform initialization, without changing ROCm device visibility.

[bong-water-water-bong]

This error is from vLLM's platform detection:

```
RuntimeError: Only one platform plugin can be activated, but got: ['cuda', 'rocm']
```

Your system has both an NVIDIA dGPU (RTX PRO 1000) and an AMD iGPU (Radeon 890M). The vLLM ROCm build detects both CUDA and ROCm platforms and can only activate one.

Workaround: Set `CUDA_VISIBLE_DEVICES=""` in the vLLM server environment to hide the NVIDIA GPU from vLLM's platform detection. You can do this by setting the environment variable before starting lemond, or by configuring it in the backend settings.

This is an upstream vLLM issue with hybrid AMD+NVIDIA systems, not a Lemonade code bug.

[galuszkak]

@bong-water-water-bong this isn't a fix - ROCm uses CUDA_VISIBLE_DEVICES - ergo if you put it empty you will not see AMD devices... https://rocm.docs.amd.com/en/latest/conceptual/gpu-isolation.html#cuda-visible-devices

[fl0rianr]

Indeed, I wanted to say exactly the same...

[bong-water-water-bong]

Fixed by #2459 — sets VLLM_TARGET_DEVICE=rocm in the vLLM subprocess environment. Companion fix in vllm-rocm#20 preloads an NVML stub that intercepts nvmlInit/nvmlDeviceGetCount, keeping the CUDA platform inactive on hybrid AMD+NVIDIA systems.