Problem
Skippy placement considers capacity and inter-node RTT, but not the usable bandwidth of each adjacent stage edge. That can produce a low-latency plan whose cold prefill is still dominated by activation transfer.
The client sends compact text only to stage 0. The large traffic is between stages: every prompt token's boundary activation crosses each adjacent edge once. For a 128K prompt:
- typical Qwen, width 2,048 with F16 wire: about 512 MiB per edge
- Inkling, width 6,144 with F32 wire: exactly 3 GiB per edge
- Inkling with a validated F16/Q8 wire would fall to 1.5 GiB/768 MiB respectively
Prefill chunks pipeline across stages, so wall time tends toward the slowest edge rather than the sum of all edge times. Decode sends one token at a time, so RTT remains important there.
Proposed change
Make split placement aware of both directed edge bandwidth and RTT:
- measure or advertise usable peer-to-peer bandwidth as an optional, additive capability
- estimate boundary bytes from prompt/context size, activation width, selected wire dtype, and any activation sideband multiplier
- score prefill placement by its bottleneck adjacent edge while retaining RTT weight for decode
- prefer co-located or high-bandwidth adjacency for high-volume boundaries; do not confuse client-to-stage-0 ingress with inter-stage bandwidth
- expose estimated bytes and transfer time per edge in plan/readiness diagnostics
- use a conservative fallback when bandwidth is unknown, preserving mixed-version mesh compatibility
Quantisation must use the actual validated activation wire dtype; KV quantisation does not reduce boundary traffic.
Done when
- deterministic planner tests choose higher-bandwidth adjacency for a large-context/wide-activation case with otherwise similar capacity and RTT
- small-context/decode-oriented cases still prefer lower RTT where appropriate
- plan diagnostics show the bandwidth assumption, boundary volume, bottleneck edge, and predicted transfer floor
- a real multi-node run confirms the selected plan and observed prefill transport are directionally consistent
Related: #1025 covers decode/speculative latency hiding; this issue covers placement for activation bandwidth.
Problem
Skippy placement considers capacity and inter-node RTT, but not the usable bandwidth of each adjacent stage edge. That can produce a low-latency plan whose cold prefill is still dominated by activation transfer.
The client sends compact text only to stage 0. The large traffic is between stages: every prompt token's boundary activation crosses each adjacent edge once. For a 128K prompt:
Prefill chunks pipeline across stages, so wall time tends toward the slowest edge rather than the sum of all edge times. Decode sends one token at a time, so RTT remains important there.
Proposed change
Make split placement aware of both directed edge bandwidth and RTT:
Quantisation must use the actual validated activation wire dtype; KV quantisation does not reduce boundary traffic.
Done when
Related: #1025 covers decode/speculative latency hiding; this issue covers placement for activation bandwidth.