gpus run-benchmark --backend hip ships a gfx906-only kernel and SIGSEGVs on every other AMD GPU

[willgriffin]

Summary

mesh-llm gpus run-benchmark --backend hip segfaults deterministically on
gfx1151 (Strix Halo). The cause is visible in the shipped binary: the benchmark's
HIP kernel is the only fat binary in mesh-llm compiled for a single
architecture, gfx906, while every other kernel carries the full nine-target
set. On any non-gfx906 device the kernel cannot resolve and HIP dereferences a
null.

This is distinct from #966 — that was the llama.cpp native runtime missing
RDNA3/3.5 kernels, and PR #1044 fixed it correctly. The runtime is fine here
(evidence below). This is the mesh-llm-gpu-bench kernel only.

Evidence

Enumerating offload bundle entry IDs in the v0.74.0 launcher:

$ grep -a -o -E "hip[v0-9]*-amdgcn-amd-amdhsa--gfx[0-9a-z]+" mesh-llm | sort | uniq -c
    133 hipv4-amdgcn-amd-amdhsa--gfx1100
    133 hipv4-amdgcn-amd-amdhsa--gfx1101
    133 hipv4-amdgcn-amd-amdhsa--gfx1102
    133 hipv4-amdgcn-amd-amdhsa--gfx1103
    133 hipv4-amdgcn-amd-amdhsa--gfx1151
    133 hipv4-amdgcn-amd-amdhsa--gfx1200
    133 hipv4-amdgcn-amd-amdhsa--gfx1201
      1 hipv4-amdgcn-amd-amdhsa--gfx906     <-- the outlier
    133 hipv4-amdgcn-amd-amdhsa--gfx90a
    133 hipv4-amdgcn-amd-amdhsa--gfx942

134 __CLANG_OFFLOAD_BUNDLE__ markers, 133 of which are built for all nine
targets. Exactly one is built for gfx906 alone.

The bundled runtime is correct, for contrast — libggml-hip.so.0.17.0 has 133
bundles for each of the nine targets including gfx1151, and the resolver picks
it properly:

$ mesh-llm doctor
Native runtime:
  status: ok
  selected: meshllm-native-runtime-linux-x86_64-rocm
  flavor: rocm
  selected runtime version: 0.74.0

Note that strings silently returns nothing on binaries this size — grep -a is
what surfaces these.

Stack

amd::Symbol::getDeviceKernel returning null is the signature of "no code object
for this device":

#0  amd::Symbol::getDeviceKernel(amd::Device const&)            (libamdhip64.so.7 + 0x48a610)
#1  hip::ihipLaunchKernel_validate                              (libamdhip64.so.7 + 0x341866)
#2  hip::ihipModuleLaunchKernel                                 (libamdhip64.so.7 + 0x341fc1)
#3  hip::ihipLaunchKernel                                       (libamdhip64.so.7 + 0x386914)
#4  hip::hipLaunchKernel_common                                 (libamdhip64.so.7 + 0x34a2b5)
#5  hip::hipLaunchKernel                                        (libamdhip64.so.7 + 0x34aa3d)
#6  mesh_llm_gpu_bench_hip_main_program_main                    (mesh-llm + 0x22c0fab9)

Identical fault address and instruction offset on every run, across many
invocations:

mesh-llm[45729]: segfault at 8a ip 00007c7def28a610 error 4 in libamdhip64.so.7.2.53211[489610,...]

Reproduction

$ mesh-llm gpus run-benchmark --backend hip
# → SIGSEGV, coredump

Also fires indirectly: mesh-llm serve --llama-flavor rocm spawns this as a
side-car, so every serve start on an affected GPU drops a coredump. The parent
survives, so it is easy to miss.

Environment

mesh-llm	0.74.0 (release artifact, not a source build)
GPU	AMD Ryzen AI Max+ 395 / Radeon 8060S, gfx1151
ROCm	7.2.3
OS / kernel	NixOS 26.11, Linux 7.1.3
rocminfo ISAs	amdgcn-amd-amdhsa--gfx1151, amdgcn-amd-amdhsa--gfx11-generic

Reproduces with HSA_OVERRIDE_GFX_VERSION both set to 11.5.1 and explicitly
unset, so it is not an override-matching artifact.

Impact

Every AMD GPU that is not gfx906 (Vega 20) loses gpus run-benchmark --backend hip and gains a coredump per serve start. Anything keying off the benchmark's
capacity numbers gets nothing on those devices.

Suggested fix

Pass the same --offload-arch list to the gpu-bench HIP compile that the
llama.cpp stage already uses. gfx906 is hipcc's default when no
--offload-arch is given, which is consistent with the benchmark kernel simply
never being told its targets.

The list already exists in .github/workflows/release.yml as
LLAMA_STAGE_AMDGPU_TARGETS (lines 614 and 1441):

gfx90a;gfx942;gfx1100;gfx1101;gfx1102;gfx1103;gfx1151;gfx1200;gfx1201

I could not find the gpu-bench HIP compile invocation from the outside —
crates/mesh-llm-gpu-bench has no build.rs and no build-dependencies — so I
can't point at the exact line, but reusing that variable looks like the whole
change.

Happy to test a candidate build on gfx1151.

[i386]

Investigation update (source-level, against v0.74.0 and v0.75.0):

Confirmed the underlying compile-path mismatch.

• In v0.74.0, the HIP benchmark build invokes hipcc -O3 -std=c++17 and never supplies --offload-arch.
• The ROCm build does pass LLAMA_STAGE_AMDGPU_TARGETS to build-llama.sh, and enables gpu-bench-hip for Cargo, but that environment does not reach the HIP benchmark build script (build-linux.sh). This explains why the llama.cpp runtime can contain the expected targets while the benchmark object falls back to a single compiler default.
• v0.75.0 moved the benchmark out of the host binary and into scripts/package-native-runtime.sh, but build_gpu_benchmark_tool() still invokes HIPCC without any --offload-arch list. The release recipe still sets LLAMA_STAGE_AMDGPU_TARGETS only around the package command (Justfile), so the same omission remains in the shipped ROCm benchmark tool. The failing code moved, but gpus run-benchmark --backend hip ships a gfx906-only kernel and SIGSEGVs on every other AMD GPU #1183 is still actionable for v0.75.0.

The smallest fix appears to be to parse/reuse LLAMA_STAGE_AMDGPU_TARGETS (with the existing SKIPPY_AMDGPU_TARGETS fallback if appropriate) and pass each configured target to HIPCC, for example one --offload-arch=<target> argument per target, for both Linux and Windows runtime-tool packaging. Please add a build test that captures the HIPCC argv and verifies all configured targets are present; ideally also validate a packaged benchmark on gfx1151. The release list currently contains nine AMD targets.

I did not run a ROCm build locally, but the source path confirms the reported cause. I would keep this issue open until the runtime benchmark tool is rebuilt with explicit targets and a release artifact is validated.

[i386]

Fix submitted in draft PR #1198.

The patch passes the configured ROCm target list to the packaged HIP benchmark compiler and adds regression coverage for the generated HIPCC arguments and runtime manifest. Full script test suite: 386 passed, 7 skipped. ROCm hardware validation remains pending.

[i386]

Fix should be in next release @willgriffin - give it a go and let me know how you go. feel free to reopen this one.