From 59707ce7bfa61dc5a6f1a551944ecdb9c08522c4 Mon Sep 17 00:00:00 2001 From: zcxGGmu <72263081+zcxGGmu@users.noreply.github.com> Date: Sun, 2 Aug 2026 22:47:14 +0800 Subject: [PATCH] docs : clarify tensor split context sizing Document that tensor split shards the KV cache and can allow larger context sizes, while MTP/speculative decoding should be validated separately. --- docs/multi-gpu.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/multi-gpu.md b/docs/multi-gpu.md index 0d9eea7c2fb8..d6f2e7d325da 100644 --- a/docs/multi-gpu.md +++ b/docs/multi-gpu.md @@ -85,8 +85,10 @@ llama-cli -m model.gguf -sm tensor -ctk f16 -ctv f16 - `--flash-attn off` or (`--flash-attn auto` resolving to `off` when it isn't supported) is a hard error. - KV cache types must be non-quantized: `f32`, `f16`, or `bf16`. Support for quantized KV cache is not implemented and trying to use it will result in an error. +- Because tensor parallelism splits the KV cache across GPUs, some models can fit a larger `--ctx-size` than they can with `layer` mode on the same hardware. `--fit` is disabled for `tensor`, so set the context size explicitly and validate memory use. +- If combining tensor parallelism with MTP/speculative decoding, validate stability separately. If a model becomes unstable, retry with MTP/speculative decoding disabled before reducing the context size. - Mark this configuration as experimental in your tooling: validate output quality before deploying. -- `--split-mode tensor`is not implemented for all architectures. The following will fail with *"LLAMA_SPLIT_MODE_TENSOR not implemented for architecture '...'"*: +- `--split-mode tensor` is not implemented for all architectures. The following will fail with *"LLAMA_SPLIT_MODE_TENSOR not implemented for architecture '...'"*: - **MoE / hybrid:** Grok, MPT, OLMoE, DeepSeek2, GLM-DSA, Nemotron-H, Nemotron-H-MoE, Granite-Hybrid, LFM2-MoE, Minimax-M2, Mistral4, Kimi-Linear, Jamba, Falcon-H1 - **State-space / RWKV-style:** Mamba, Mamba2 (and the hybrid Mamba-attention models above)