Failed on AMD ROCm GPU with vLLM

[alexhegit]

Request to support it with AMD ROCm GPU.

# DFlash on ROCm/vLLM Deployment Notes


## Goal

Try to run the following speculative decoding pair with vLLM on ROCm:

- Target model: `Qwen3.5-27B`
- Draft model: `z-lab/Qwen3.5-27B-DFlash`


## What was verified

### Container start command

```bash
docker run -d --name dflash-vllm-rocm-nightly \
  --ipc host \
  --network host \
  --privileged \
  --cap-add SYS_ADMIN \
  --cap-add SYS_PTRACE \
  --device /dev/kfd:/dev/kfd \
  --device /dev/dri:/dev/dri \
  --device /dev/mem:/dev/mem \
  --group-add render \
  --security-opt seccomp=unconfined \
  -e GLOO_SOCKET_IFNAME=eth0 \
  -e NCCL_SOCKET_IFNAME=eth0 \
  -e VLLM_HOST_IP=10.50.0.59 \
  -v /mnt/volume_atl1_d4t/HF:/model:ro \
  --entrypoint bash \
  rocm/vllm-dev:nightly \
  -lc "exec vllm serve /model/Qwen/Qwen3.5-27B \
    --speculative-config '{\"method\":\"dflash\",\"model\":\"/model/Qwen/Qwen3.5-27B-DFlash\",\"num_speculative_tokens\":15}' \
    --tensor-parallel-size 1 \
    --dtype auto \
    --enable-prefix-caching \
    --max-model-len 8192 \
    --max-num-batched-tokens 32768 \
    --gpu-memory-utilization 0.9 \
    --host 0.0.0.0 \
    --port 18000 \
    --served-model-name qwen35-27b-dflash \
    --api-key amd \
    --trust-remote-code"

Failure path A: default ROCm backend

When not forcing --attention-backend flash_attn, the service failed during backend selection.

Key error:

ValueError: No valid attention backend found for rocm with AttentionSelectorConfig(... use_non_causal=True).
Reasons: {ROCM_ATTN: [non-causal attention not supported], ROCM_AITER_UNIFIED_ATTN: [non-causal attention not supported], TRITON_ATTN: [non-causal attention not supported]}.

Interpretation

This means:

1. DFlash internally requires use_non_causal=True
2. Current ROCm backends available to vLLM do not support that execution mode
3. Therefore DFlash cannot run on this path even though it is recognized by vLLM

Important clarification:

• Removing --attention-backend flash_attn does not remove the need for non-causal attention
• The need for non-causal attention comes from the DFlash execution path itself

Failure path B: forcing --attention-backend flash_attn

When forcing:

--attention-backend flash_attn

the run progressed much farther:

• target model loaded successfully
• draft model loaded successfully
• weights were loaded
• compilation/profile steps ran

Then it failed with:

AssertionError: FlashAttention version not detected.

Even though pip list | grep flash showed:

flash_attn 2.8.3

Short summary

DFlash is recognized by ROCm vLLM nightly, but the run still fails because the ROCm attention backends do not support the non-causal attention required by DFlash. Forcing flash_attn avoids that specific selection path but then fails later because FlashAttention is not successfully detected by vLLM.

[Martin-Atrin]

Tested another ROCm path on an AMD Strix Halo iGPU box and got slightly farther, but it still does not survive first decode.

Setup:

• Hardware: AMD Ryzen AI Max+ 395 / Radeon 8060S (Strix Halo, unified memory)
• Runtime base: docker.io/vllm/vllm-openai-rocm:nightly
• Container style: isolated distrobox / podman-based test box, not our production vLLM container
• Target model: Qwen/Qwen3.5-4B
• Drafter: z-lab/Qwen3.5-4B-DFlash
• vLLM: 0.19.1rc1.dev133+ge80e63392.rocm721
• flash-attn reported by Python: 2.8.3

What changed versus the earlier ROCm failure:

• I worked around the AssertionError: FlashAttention version not detected path by patching the ROCm get_flash_attn_version() path to return 2 when upstream flash_attn is actually importable.
• With that patch in place, vLLM no longer dies during startup/version detection.
• The server fully initializes, loads both the target and drafter, finishes warmup, and starts listening on the API port.

Relevant startup logs after the patch:

• Using FLASH_ATTN backend (selected via --attention-backend).
• Using FlashAttention version 2
• base model loaded successfully
• drafter loaded successfully
• API server came up and /v1/models responded correctly

However, the first real generation request still kills the engine.

New fatal error on first decode:

TypeError: flash_attn_varlen_func() got an unexpected keyword argument 'out'

Trace lands in:

• vllm/v1/attention/backends/flash_attn.py
• call site inside FlashAttentionImpl.forward(...)
• invoked through torch.ops.vllm.unified_attention_with_output(...)

So the current picture on ROCm seems to be:

1. DFlash is recognized.
2. With forced --attention-backend flash_attn and a local patch, startup can succeed.
3. The remaining blocker is now a runtime API mismatch between vLLM's ROCm FlashAttention call path and the installed upstream flash_attn implementation.

In other words, on this machine it progressed from:

• backend selection failure
• to FA version detection failure
• to a real first-token execution failure in flash_attn_varlen_func

This does not look like a pure hardware incompatibility anymore; it looks more like ROCm flash-attn / vLLM interface drift.

If useful, I can also share the exact launch flags and the tiny local patch used to get past the FA version assertion.

[MichelNivard]

Did anyone get it to work on AMD eventually?