Performance of llama.cpp on AMD GPU (MI300X)

[yeahdongcn]

I have got an opportunity to use the AMD Developer Cloud to test the AMD Instinct MI300X.

Below are some test results I gathered during the evaluation.

https://huggingface.co/unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF:

root@6-4-0-gpu-mi300x1-192gb-devcloud-atl1:~/llama.cpp# ./build/bin/llama-bench -m /root/.cache/llama.cpp/unsloth_DeepSeek-R1-0528-Qwen3-8B-GGUF_DeepSeek-R1-0528-Qwen3-8B-Q4_K_M.gguf -ngl 999
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen3 8B Q4_K - Medium         |   4.68 GiB |     8.19 B | ROCm       | 999 |           pp512 |       5839.67 ± 5.30 |
| qwen3 8B Q4_K - Medium         |   4.68 GiB |     8.19 B | ROCm       | 999 |           tg128 |        161.47 ± 0.30 |

build: f5e96b36 (5873)

https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF:

root@6-4-0-gpu-mi300x1-192gb-devcloud-atl1:~/llama.cpp# ./build/bin/llama-bench -m /root/.cache/llama.cpp/unsloth_DeepSeek-R1-Distill-Llama-70B-GGUF_DeepSeek-R1-Distill-Llama-70B-Q4_K_M.gguf -ngl 999
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| llama 70B Q4_K - Medium        |  39.59 GiB |    70.55 B | ROCm       | 999 |           pp512 |        949.85 ± 0.74 |
| llama 70B Q4_K - Medium        |  39.59 GiB |    70.55 B | ROCm       | 999 |           tg128 |         36.31 ± 0.08 |

build: f5e96b36 (5873)

https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF:

root@6-4-0-gpu-mi300x1-192gb-devcloud-atl1:~/llama.cpp# ./build/bin/llama-bench -m /root/.cache/llama.cpp/unsloth_DeepSeek-R1-0528-GGUF_DeepSeek-R1-0528-UD-TQ1_0.gguf -ngl 999
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| deepseek2 671B IQ1_S - 1.5625 bpw | 150.51 GiB |   671.03 B | ROCm       | 999 |           pp512 |        116.99 ± 0.76 |
| deepseek2 671B IQ1_S - 1.5625 bpw | 150.51 GiB |   671.03 B | ROCm       | 999 |           tg128 |         33.14 ± 0.04 |

build: f5e96b36 (5873)

Kimi K2 Q2_K https://huggingface.co/gabriellarson/Kimi-K2-Instruct-GGUF (#14642 #14654):

root@rocm-jupyter-gpu-mi300x8-1536gb-devcloud-atl1:~/llama.cpp# ./build/bin/llama-bench -m ~/.cache/llama.cpp/gabriellarson_Kimi-K2-Instruct-GGUF_Kimi-K2-Instruct-Q2_K-00001-of-00013.gguf -ngl 999 
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 8 ROCm devices:
  Device 0: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 1: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 2: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 3: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 4: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 5: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 6: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 7: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| deepseek2 671B F16             | 347.25 GiB |  1026.41 B | ROCm       | 999 |           pp512 |        100.27 ± 0.49 |
| deepseek2 671B F16             | 347.25 GiB |  1026.41 B | ROCm       | 999 |           tg128 |         38.14 ± 0.11 |

build: 273ea092 (5892)

Kimi K2 Q4_K_M https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF (#14642 #14654):

root@rocm-jupyter-gpu-mi300x8-1536gb-devcloud-atl1:~/llama.cpp# ./build/bin/llama-batched-bench \
    --model ./unsloth/Kimi-K2-Instruct-GGUF/unsloth_Kimi-K2-Instruct-GGUF_Q4_K_M_Kimi-K2-Instruct-Q4_K_M-00001-of-00013.gguf \
    --numa numactl \
    --threads 32 \
    --ctx-size 131072 \
    --n-gpu-layers 999 \
    -ub 4096 -b 4096 \
    --seed 3407 \
    --temp 0.6 \
    --min-p 0.01 \
    --flash-attn \
    -npp 4096 \
    -ntg 1024 \
    -npl 1

main: n_kv_max = 131072, n_batch = 4096, n_ubatch = 4096, flash_attn = 1, is_pp_shared = 0, n_gpu_layers = 999, n_threads = 32, n_threads_batch = 32

|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
|  4096 |   1024 |    1 |   5120 |  109.385 |    37.45 |  183.934 |     5.57 |  293.319 |    17.46 |

llama_perf_context_print:        load time =  102419.85 ms
llama_perf_context_print: prompt eval time =  110295.55 ms /  4112 tokens (   26.82 ms per token,    37.28 tokens per second)
llama_perf_context_print:        eval time =  183907.84 ms /  1024 runs   (  179.60 ms per token,     5.57 tokens per second)
llama_perf_context_print:       total time =  395738.95 ms /  5136 tokens

root@rocm-jupyter-gpu-mi300x8-1536gb-devcloud-atl1:~/llama.cpp# ./build/bin/llama-bench -m ./unsloth/Kimi-K2-Instruct-GGUF/unsloth_Kimi-K2-Instruct-GGUF_Q4_K_M_Kimi-K2-Instruct-Q4_K_M-00001-of-00013.gguf -ngl 999
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 8 ROCm devices:
  Device 0: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 1: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 2: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 3: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 4: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 5: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 6: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 7: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| deepseek2 671B Q4_K - Medium   | 578.14 GiB |  1026.41 B | ROCm       | 999 |           pp512 |        118.32 ± 0.51 |
| deepseek2 671B Q4_K - Medium   | 578.14 GiB |  1026.41 B | ROCm       | 999 |           tg128 |         38.09 ± 0.09 |

build: 6ef6aa1c (5898)

[olegshulyakov]

@yeahdongcn you can format it like Performance of llama.cpp with Vulkan "Performance of llama.cpp with ROCm"