AMD ROCm support

[hhenkel]

I'm not really deep in the whole topic, but from reading your explanation in the README.md, it seems like a great achievement, which is currently limited to Nvidia and Apple hardware.

What would be necessary, to get this running on AMD Hardware, like GPUs or Strix Halo systems?

[xXRex45Xx]

I'm not really deep in the whole topic, but from reading your explanation in the README.md, it seems like a great achievement, which is currently limited to Nvidia and Apple hardware.

What would be necessary, to get this running on AMD Hardware, like GPUs or Strix Halo systems?

I'm in the same boat as you. We'll have to wait for either a vulkan or rocm backend support. At the moment, it only works on cpus, nvidia and mac.

Edit: checkout https://github.com/unixsysdev/llama-turboquant , they have rocm backend support working

[NatsuGatsu]

I really hope they add ROCm support soon.

[apollo-mg]

I really hope they add ROCm support soon.

Try TheTom's repo. I'm using it with a 9070XT/ROCM 7.2 + turboquant.