Does llama.cpp place one expert entirely on a single device?

[Imagium719]

Hi there! I have a question regarding the behavior of llama.cpp during multi-GPU inference for MoE models. Specifically, does llama.cpp aim to place each expert entirely on a single device as much as possible, or does it slice experts and distribute them across multiple devices (using tensor parallelism)? This is quite important because the former approach has minimal requirements for GPU interconnect bandwidth, while the latter would require significantly higher bandwidth. Thanks in advance for your insights!

[fairydreaming]

The default behavior is to split the model by whole layers, so it doesn't slice the experts. There's no tensor parallelism in this case.
But there is a --split-mode option that controls how tensors are split. If you provide --split-mode row llama.cpp will split model not by layers, but by tensor rows. While it's not exactly the same as MegatronLM tensor parallelism that other backends use, it parallelizes tensor processing and requires fast inter-GPU bandwidth interconnect, so it's somewhat similar.