mesh-llm has been unable to join the mesh network

[peijs]

Hello, I am using the mesh-llm version 0.74.0, and I have packaged the binary file mesh-llm into Docker for running. The model I am using is DeepSeek-V4-Flash-UD-Q4_K_XL-00001-of-00005.gguf. Under normal circumstances, both servers can load and run this model. However, I now want to distribute the model file slices across 16 4090 graphics cards on two servers, using the "--split" parameter. After running, the two servers have not been able to join the mesh node network. Below are the log files of the two nodes.
Server configuration:
CPU: Dual AMD EPYC 7402, 2 CPUs, each with 24 cores, with hyper-threading enabled; total 96 logical CPUs (48 physical cores)
Memory: 512GB
GPU：8 × RTX 4090 24GB
Architecture: x86_64

-----------master node

docker run -it --rm
--net=host
--gpus all
-v /data/mesh-llm-home:/root/.mesh-llm
-v /share/realcugan/models/DeepSeek-V4-Flash-GGUF:/models
mesh-llm:latest
mesh-llm serve
--model /models/UD-Q4_K_XL/DeepSeek-V4-Flash-UD-Q4_K_XL-00001-of-00005.gguf
--port 8080
--console 3131
--listen-all
--llama-flavor cuda
--split

----------slave node

docker run -it --rm
--net=host
--gpus all
-v /data/mesh-llm-home:/root/.mesh-llm
-v /share/realcugan/models/DeepSeek-V4-Flash-GGUF:/models
mesh-llm:latest
mesh-llm serve --join "xxxx"
--model /models/UD-Q4_K_XL/DeepSeek-V4-Flash-UD-Q4_K_XL-00001-of-00005.gguf
--port 8080
--console 3131
--listen-all
--llama-flavor cuda
--split

-----------master node logs

│21:45:52 INFO Peer 39ed31747f RTT: 0ms (direct)
│21:46:33 INFO model=local-gguf/sha256-be7bf53f25296d0c: Split waiting for peers: split runtime needs at least two participating nodes for local-gguf/sha256-be7bf53f25
│ eligible [8d2047ca70:507.2GB cached=155.1GB missing=0.0GB rtt=0ms transfer=true]; excluded [39ed31747f:stage_inventory_empty]; blockers [stage_inventory
│ nodes=[39ed31747f]]; next_step: Wait for stage inventory refresh or prepare the requested package on this peer.
│21:46:51 INFO Peer 39ed31747f RTT: 1ms (direct)
│21:47:33 INFO model=local-gguf/sha256-be7bf53f25296d0c: Split waiting for peers: split runtime needs at least two participating nodes for local-gguf/sha256-be7bf53f25
│ eligible [8d2047ca70:507.2GB cached=155.1GB missing=0.0GB rtt=0ms transfer=true]; excluded [39ed31747f:stage_inventory_empty]; blockers [stage_inventory
│ nodes=[39ed31747f]]; next_step: Wait for stage inventory refresh or prepare the requested package on this peer.
│21:47:52 INFO Peer 39ed31747f RTT: 4ms (direct)
│21:48:34 INFO model=local-gguf/sha256-be7bf53f25296d0c: Split waiting for peers: split runtime needs at least two participating nodes for local-gguf/sha256-be7bf53f25
│ eligible [8d2047ca70:507.2GB cached=155.1GB missing=0.0GB rtt=0ms transfer=true]; excluded [39ed31747f:stage_inventory_empty]; blockers [stage_inventory
│ nodes=[39ed31747f]]; next_step: Wait for stage inventory refresh or prepare the requested package on this peer.
│21:48:52 INFO Peer 39ed31747f RTT: 0ms (direct)

----------slave node logs

│21:48:37 INFO Peer 8d2047ca70 RTT: 0ms (direct) ││
│21:48:52 INFO Peer 8d2047ca70 RTT: 0ms (direct) [path info] ││
│21:49:27 INFO model=local-gguf/sha256-be7bf53f25296d0c: Split waiting for peers: split runtime needs at least two participating nodes for local-gguf/sha256-be7bf53f25296d0c; found 1 ││
│ eligible [39ed31747f:994.2GB cached=155.1GB missing=0.0GB rtt=0ms transfer=true]; excluded [8d2047ca70:stage_inventory_empty]; blockers [stage_inventory_empty=1 ││
│ nodes=[8d2047ca70]]; next_step: Wait for stage inventory refresh or prepare the requested package on this peer. ││
│21:49:37 INFO Peer 8d2047ca70 RTT: 0ms (direct) ││
│21:50:28 INFO model=local-gguf/sha256-be7bf53f25296d0c: Split waiting for peers: split runtime needs at least two participating nodes for local-gguf/sha256-be7bf53f25296d0c; found 1 ││
│ eligible [39ed31747f:994.2GB cached=155.1GB missing=0.0GB rtt=0ms transfer=true]; excluded [8d2047ca70:stage_inventory_empty]; blockers [stage_inventory_empty=1 ││
│ nodes=[8d2047ca70]]; next_step: Wait for stage inventory refresh or prepare the requested package on this peer. ││
│21:50:37 INFO Peer 8d2047ca70 RTT: 0ms (direct) ││
│21:51:29 INFO model=local-gguf/sha256-be7bf53f25296d0c: Split waiting for peers: split runtime needs at least two participating nodes for local-gguf/sha256-be7bf53f25296d0c; found 1 ││
│ eligible [39ed31747f:994.2GB cached=155.1GB missing=0.0GB rtt=0ms transfer=true]; excluded [8d2047ca70:stage_inventory_empty]; blockers [stage_inventory_empty=1

[peijs]

logs.txt

[peijs]

@i386 @wesbillman @claude

[i386]

I investigated the logs and the v0.74.0 code. This is not a mesh-admission or --join failure:

• Both nodes report the other peer with a direct RTT/path.
• The split coordinator resolves the local model and reports the local cached bytes.
• The failure is specifically that each remote StageInventory response is empty, so each peer is excluded as stage_inventory_empty.

The root cause is the layout of this particular sharded GGUF. The exact public first shard, DeepSeek-V4-Flash-UD-Q4_K_XL-00001-of-00005.gguf, is only about 5.26 MB and contains GGUF metadata but no weight tensors; the remaining shards contain the large layer weights.

In v0.74.0, stage inventory resolution opens only the first-shard path and calls infer_layer_count(&candidate).ok()?. infer_layer_count derives the count from tensor layer_index values, so a metadata-only first shard returns an error/no layer count. The .ok()? then silently turns that into None, and the node advertises an empty inventory.

This also explains why the local side gets as far as it does: direct-GGUF package identity reads the layer count from GGUF compact metadata instead of requiring first-shard tensors.

The code fix should make direct-GGUF inventory use the same metadata/package identity path as split startup, enumerate/validate all shards, and avoid swallowing the inventory-resolution error. A regression test should use a metadata-only first shard.

For now, practical workarounds are to run without --split, use a package-backed split artifact whose manifest contains the layer count, or merge the GGUF shards into one file (the latter is expensive at this size). Current main still contains the same inventory logic, so upgrading from 0.74.0 alone is not sufficient until this path is patched.

[i386]

Fix is available in stacked draft PR #1199: #1199

It is based on #1194 and fixes direct split-GGUF inventory discovery when the first shard contains metadata but no tensors. The regression test confirms the layer count and total shard size are advertised correctly.

[peijs]

Got it, thank you very much for the explanation. I will wait for this issue to be fixed in a future new release. Thanks a lot.

[i386]

Fix should be out today @peijs