[Klaud Cold][agentic experiment][Variant F] Kimi-K3 B200 agg TP8xPP2 agentic — SimpleCPU offload + prefix-cache retention / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——SimpleCPU 卸载 + 前缀缓存留存 - #2372
Conversation
…ecipe Aggregated TP8 x PP2 across 2 B200 nodes (16 GPUs), plain TP (no expert parallelism) for the agentic-coding trace replay. Dedicated bring-up image vllm/vllm-openai:kimi-k3 with VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1, fastsafetensors load format, kimi_k3 tool-call/reasoning parsers. Model pre-staged at /lustre/fsw/models/Kimi-K3; launch_b200-dgxc.sh gains the kimik3/fp4 model-path mapping, the agentic recipe overlay, and the agentic cache default_mounts used by the GB200/GB300 agentic paths. 中文：新增 Kimi-K3 MXFP4 B200 聚合式 TP8xPP2 Dynamo-vLLM 智能体编码基准测试配方 （2 节点 / 16 GPU，纯 TP，不启用专家并行(EP)）。使用专用 bring-up 镜像 vllm/vllm-openai:kimi-k3（VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1、 fastsafetensors 加载格式、kimi_k3 工具调用/推理解析器）。模型已预置于 /lustre/fsw/models/Kimi-K3；启动器 launch_b200-dgxc.sh 增加 kimik3/fp4 模型路径映射、智能体配方覆盖及智能体缓存挂载。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
中文：在更新日志条目与 MODELS 表格行中补充 PR #2355 链接。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oke test The cquil11/srt-slurm-nv cam/sa-submission-q2-2026 fork rejected the recipe (benchmark.aiperf_server_metrics: Unknown field). Switch the b200-dgxc agentic clone to upstream NVIDIA/srt-slurm v1.0.36 (validated in #2302/#2341), drop the aiperf_server_metrics field, pin dynamo wheel/router to 1.2.1 (the combination validated with v1.0.36), and reduce the bring-up to a single conc-8 smoke test. 中文：cquil11/srt-slurm-nv 分支的 srtctl 校验拒绝了配方字段 benchmark.aiperf_server_metrics（Unknown field）。将 b200-dgxc 智能体路径改用 上游 NVIDIA/srt-slurm v1.0.36（已在 #2302/#2341 验证），移除该字段，dynamo wheel/router 固定为 1.2.1，并将 bring-up 缩减为单并发（conc 8）冒烟测试。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dynamo-vllm worker entrypoint rejected --enable-auto-tool-choice --tool-call-parser kimi_k3 (unrecognized arguments; different arg parser than vllm serve). Chat parsing happens at the dynamo frontend — same convention as the DSv4 GB300 agentic recipes. Keep --reasoning-parser kimi_k3 (accepted by the worker). Also drop the explicit max-model-len and let vLLM derive the native 1M window from the model config, mirroring the agentic recipe convention. 中文：dynamo-vllm worker 入口不接受 --enable-auto-tool-choice 与 --tool-call-parser kimi_k3（unrecognized arguments，与 vllm serve 的参数解析器 不同），聊天解析由 dynamo 前端处理，与 DSv4 GB300 智能体配方约定一致；保留 worker 可接受的 --reasoning-parser kimi_k3。同时移除显式 max-model-len， 由 vLLM 从模型配置推导原生 1M 上下文窗口。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Third sweep attempt: the engine loaded and served (TP8xPP2 healthy in ~14 min), but dynamo 1.2.1's rust frontend tokenizer rejects Kimi-K3's tiktoken model_type 'kimi_k3' (supported: kimi, kimi_k2, kimi_k25, deepseek_v3), so the model never registered and all chat completions returned 404, aborting the AgentX warmup. Switch to the 1.2.0.dev20260426 wheel used by the DSv4 GB300/B200 Dynamo-vLLM recipes. Upstream published v1.4.0-kimi-k3-dev.1 (2026-07-27) as the day-zero K3 build if this wheel also lacks support. 中文：第三次扫描中引擎已成功加载并提供服务（TP8xPP2 约 14 分钟就绪），但 dynamo 1.2.1 的 rust 前端分词器不支持 Kimi-K3 的 tiktoken model_type 'kimi_k3'，模型未能注册，所有请求返回 404，AgentX 预热中止。改用 DSv4 GB300/B200 Dynamo-vLLM 配方所用的 1.2.0.dev20260426 wheel；如仍不支持， 上游已于 2026-07-27 发布 day-zero 构建 v1.4.0-kimi-k3-dev.1。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin dynamo to ba83080ecd31c1ce918559e576d3c5bc9e092ff1 ("feat: Added
support for Kimi-K3", tag v1.4.0-kimi-k3-dev.1) via srt-slurm's
hash-cached source install: it adds the kimi_k3 tiktoken tokenizer to the
rust frontend (dynamo <=1.2.1 404s every request because the model never
registers) and accepts the kimi_k3 tool-call/reasoning parser worker args,
so restore --enable-auto-tool-choice --tool-call-parser kimi_k3
--reasoning-parser kimi_k3.
中文：将 dynamo 固定到 day-zero Kimi-K3 提交 ba83080（"feat: Added support
for Kimi-K3"，标签 v1.4.0-kimi-k3-dev.1），通过 srt-slurm 的哈希缓存源码
安装：该提交为 rust 前端新增 kimi_k3 tiktoken 分词器（dynamo <=1.2.1 因模型
无法注册而全部返回 404），worker 亦支持 kimi_k3 解析器参数，故恢复
--enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3。
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the vLLM OpenAI-frontend spellings (--enable-auto-tool-choice / --tool-call-parser) with dynamo's namespaced worker args: --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3. 中文：将 vLLM OpenAI 前端风格参数（--enable-auto-tool-choice / --tool-call-parser）替换为 dynamo 命名空间的 worker 参数： --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fifth sweep attempt: the day-zero dynamo registered the kimi_k3 tiktoken tokenizer and the engine served, but all warmup requests got 400 — aiperf's conv-aware routing emits nvext.session_control, a removed POC field this dynamo build rejects (schema moved to router/routing_constraints/ agent_hints). Opt out via AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0, matching the GB300 aggregate AgentX recipes; a single aggregate worker has no P/D routing to bind anyway. 中文：第五次扫描中 day-zero dynamo 已成功注册 kimi_k3 tiktoken 分词器并正常 服务，但全部预热请求返回 400——aiperf 的会话感知路由会发送 nvext.session_control（已被移除的 POC 字段，schema 已迁移至 router/routing_constraints/agent_hints）。通过 AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0 关闭，与 GB300 聚合式 AgentX 配方 一致；单聚合 worker 本无需 P/D 路由绑定。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sixth sweep attempt (both A and C variants): warmup requests 500 then the model 503s — the image's first decode step crashes in the KDA hybrid-state postprocess (mamba_hybrid.py postprocess_state, IndexError: index_fill_(): Expected dtype int64 for index; torch requires an int64 index but the runner passes the int32 idx_mapping). Ship an in-container patch through srt-slurm's setup_script hook (same pattern as configs/patches/ vllm_numa_bind_hash_fix.py): coerce the index with .long(), idempotent, refuses to run if the image layout changed. 中文：第六次扫描（A、C 两个变体一致）：预热请求先 500、随后模型 503——镜像 首个解码步在 KDA 混合状态后处理中崩溃（mamba_hybrid.py postprocess_state， IndexError: index_fill_() 需要 int64 索引，但 runner 传入 int32 idx_mapping）。 通过 srt-slurm 的 setup_script 钩子在容器内打补丁：将索引用 .long() 转换， 幂等，且镜像布局变化时拒绝执行。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Serve Kimi-K3 directly with vllm serve (srt-slurm PR #278 frontend.type: vllm, branch kylliang/direct-aggregate-vllm): no dynamo frontend/worker/ router, which removes the dynamo tokenizer/schema gaps entirely, and the OpenAI-frontend flags --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 become legitimate. PR #278 validates single-node only, so ship patches/srt-slurm-pr278-direct-vllm-multinode.patch extending it to vLLM-native multi-node serve (--master-addr/--nnodes/--node-rank, headless non-leader ranks) for the 2-node TP8xPP2 topology. Keeps the mamba_hybrid index-dtype container patch (engine bug is frontend-agnostic). 中文：智能体实验变体 D——通过 srt-slurm PR #278（frontend.type: vllm）直接以 vllm serve 提供服务：去除 dynamo 前端/worker/router，从根本上规避 dynamo 的 分词器与 schema 兼容问题，OpenAI 前端参数 --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 因此可用。PR #278 仅 支持单节点，故新增补丁将其扩展为 vLLM 原生多节点 serve（--master-addr/ --nnodes/--node-rank，非主节点 headless），以运行 2 节点 TP8xPP2 拓扑。保留 mamba_hybrid 索引类型容器补丁（引擎缺陷与前端无关）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2359。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same engine-level OOM as the dynamo-frontend variants: the flashinfer trtllm MXFP4 MoE kernel allocates a ~1.6 GiB runtime workspace outside vLLM's memory pool on the first forward; at 0.95 a 178 GiB B200 has only ~1.35 GiB free. 中文：与 dynamo 前端变体相同的引擎级 OOM：flashinfer trtllm MXFP4 MoE 内核在 首个前向时于 vLLM 显存池外分配约 1.6 GiB 工作区，0.95 下仅剩约 1.35 GiB。 改为 0.90。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Inherited from the closed dynamo-frontend variants (#2355/#2358): at gpu-mem-util 0.90 the first long-context MLA prefill OOM'd on a 2.92 GiB transient while 3.39 GiB sat reserved-but-unallocated (fragmentation). Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and drop NCCL_CUMEM_ENABLE. 中文：继承自已关闭的 dynamo 前端变体（#2355/#2358）：0.90 显存利用率下首个 长上下文 MLA 预填充因 2.92 GiB 瞬时分配 OOM，而 3.39 GiB 处于已保留未分配 状态（碎片化）。设置 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True 并 移除 NCCL_CUMEM_ENABLE。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…refill OOM)" This reverts commit 4370988. The superseded direct-vllm run served the agentic benchmark for 24 minutes on the original env (NCCL_CUMEM_ENABLE=1, no expandable_segments) without any OOM — the allocator change was precautionary carryover from the closed dynamo-frontend variants and was never justified by evidence from this serving path. Restore the env that was demonstrably running. 中文：回滚 4370988。被中断的 direct-vllm 运行在原始环境 （NCCL_CUMEM_ENABLE=1、未设 expandable_segments）下已稳定运行智能体基准测试 24 分钟且无 OOM——该分配器改动只是从已关闭的 dynamo 前端变体沿袭的预防性 措施，并无本服务路径上的证据支持。恢复已被验证可运行的环境。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep prefix-cache blocks alive across agentic turn gaps, matching the GB200/GB300 AgentX recipes. 中文：新增 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768，使前缀缓存块在智能体 回合间隔内保持留存，与 GB200/GB300 AgentX 配方一致。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
中文：将智能体并发列表从单点 8 扩展为 1/8/16/32。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Engine init hard-fails on Kimi-K3 with the GB200/GB300 AgentX value: "VLLM_PREFIX_CACHE_RETENTION_INTERVAL (32768) must be non-negative and a multiple of scheduler_block_size (3145728)" — the KDA hybrid architecture gives K3 a 3.1M-token scheduler block. Default retention served fine in the earlier runs, so drop the override. 中文：移除 VLLM_PREFIX_CACHE_RETENTION_INTERVAL——Kimi-K3 的 KDA 混合架构使 scheduler_block_size 达 3145728，GB200/GB300 AgentX 的 32768 取值导致引擎 初始化直接失败（必须为其整数倍）。此前运行证明默认留存策略可正常服务， 故不再覆盖。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…fload Variant of the direct-vllm Kimi-K3 B200 agg TP8xPP2 agentic bring-up (#2359) with KV offload to CPU DRAM: SimpleCPUOffloadConnector (kv_role kv_both, lazy_offload false, enable_cross_layers_blocks true), cpu_bytes_to_use_per_rank 299,875,000,000 (the framework's 0.80-utilization budget for cluster:b200-dgxc: 2399 GB/node / 8 ranks), PYTHONHASHSEED=42 for cross-rank block-key hashing, explicit enable-prefix-caching, and dram/vllm-simple kv-offload labeling in the master entry. 中文：直接 vllm serve 的 Kimi-K3 B200 聚合式 TP8xPP2 智能体实验变体 E （基于 #2359）：通过 SimpleCPUOffloadConnector 将 KV 卸载至 CPU 内存 （kv_role kv_both、lazy_offload false、enable_cross_layers_blocks true）， cpu_bytes_to_use_per_rank 为 299,875,000,000（框架按 0.80 利用率对 cluster:b200-dgxc 的预算：每节点 2399 GB / 8 个 rank），设置 PYTHONHASHSEED=42 保证各 rank 前缀块键一致，显式启用前缀缓存，并在主配置 中以 dram/vllm-simple 标注 KV 卸载。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2370。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With enable_cross_layers_blocks true, KV-cache initialization crashes on Kimi-K3's KDA hybrid: "shape '[64980, 64, 576]' is invalid for input of size 199618560" — a 12x element mismatch equal to the MLA layers per PP stage (24 gated-MLA layers / PP2), i.e. the cross-layer block folding mishandles the hybrid-model geometry. Run with cross-layer blocks off, matching the GB200 connector configs. 中文：enable_cross_layers_blocks 为 true 时，Kimi-K3 的 KDA 混合架构在 KV 缓存初始化阶段崩溃（元素数量差 12 倍，恰为每个 PP 阶段的 MLA 层数 24/2）——跨层块折叠未正确处理混合架构几何。改为关闭跨层块，与 GB200 连接器配置一致。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Variant E (#2370) plus VLLM_PREFIX_CACHE_RETENTION_INTERVAL=3145728 to extend prefix-cache block retention across agentic turn gaps. 3145728 (= one scheduler block), not the GB recipes' 32768: Kimi-K3's KDA hybrid gives scheduler_block_size 3145728 and the engine hard-rejects non-multiples (verified earlier on this PR family). 中文：智能体实验变体 F——在变体 E（#2370）基础上增加 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=3145728，延长前缀缓存块在智能体回合 间隔内的留存。取 3145728（恰为一个调度块），而非 GB 配方的 32768： Kimi-K3 的 KDA 混合架构使 scheduler_block_size 为 3145728，引擎硬性要求 取值为其整数倍（此前已在本 PR 系列验证）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2372。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
中文：变体 F 并发曲线改为 1/2/4/8/16（新增 2 与 4，移除 32）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30324861886 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30324938760 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "3145728" | ||
| # No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: the GB200/GB300 AgentX value | ||
| # (32768) hard-fails engine init on Kimi-K3 — the KDA hybrid gives it a | ||
| # scheduler_block_size of 3145728 and the interval must be a multiple of | ||
| # it ("VLLM_PREFIX_CACHE_RETENTION_INTERVAL (32768) must be non-negative | ||
| # and a multiple of scheduler_block_size (3145728)"). Default retention | ||
| # served fine in earlier runs. | ||
| NCCL_CUMEM_ENABLE: "1" |
There was a problem hiding this comment.
🟡 In agg-b200-tp8pp2-agentic.yaml, line 82 sets VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "3145728" (the whole point of variant F), but the comment block right below it (lines 83-88) is a leftover from variant E that says "No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: ... Default retention served fine in earlier runs" — directly contradicting the line above. This is a copy-paste artifact that should be deleted; it has no runtime effect (YAML comment) but will mislead future maintainers about whether retention is configured.
Extended reasoning...
This is a documentation-only defect introduced by copying the variant-E recipe forward without cleaning up its comments. Line 82 of the new file adds VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "3145728" under aggregated_environment — this is the single defining change of variant F per the PR description, extending prefix-cache block retention across agentic turn gaps. Immediately below it, lines 83-88 retain a comment block that begins # No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: the GB200/GB300 AgentX value (32768) hard-fails engine init on Kimi-K3 ... and ends ... Default retention served fine in earlier runs.
That comment is a verbatim explanation for why the variable was deliberately omitted — which was true in variant E (#2370), where this recipe did not set the retention interval and relied on the engine default. When this PR (variant F) added the env var one line above, the stale "we chose not to set this" comment was never deleted, so the file now asserts two contradictory things back-to-back: the code sets the interval to 3145728, while the comment directly above the NCCL_CUMEM_ENABLE line claims no retention interval is set and default retention "served fine."
Proof of the contradiction, step by step:
- Read line 82:
VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "3145728"— the interval IS configured. - Read the very next lines (83-88):
# No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: ... Default retention served fine in earlier runs.— this states the interval is NOT configured and default retention is used. - Both statements cannot be true of the same file at the same time; a maintainer reading top-to-bottom hits an immediate contradiction with no indication which one is authoritative for this variant.
- Tracing history confirms the leftover: the omission comment matches variant E's ([Klaud Cold][agentic experiment][Variant E] Kimi-K3 B200 agg TP8xPP2 agentic — direct vllm serve + SimpleCPU KV offload / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——直接 vllm serve + SimpleCPU KV 卸载 #2370) rationale for not setting the var, and this PR's entire stated purpose (per the PR description and perf-changelog entry) is adding the var — so the comment should have been deleted, not kept alongside the new setting.
No existing check catches this because it is a YAML comment — it has zero effect on parsing, vllm serve startup, or engine behavior, so no CI job or schema validation would ever flag it. The only cost is to future readers/maintainers of this recipe (or anyone diffing variant E vs F) who would be confused about whether prefix-cache retention is actually enabled here.
Fix: delete the stale comment block (lines 83-88, the "No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: ..." lines) since it no longer reflects this variant's configuration. The surrounding comment directly above line 82 already documents the current rationale (why 3145728 was chosen over the GB recipes' 32768), so no replacement text is needed — just removal of the contradictory leftover.
Set VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0: any positive value must be a multiple of Kimi-K3's KDA-hybrid scheduler_block_size (3145728), so 0 is the only setting below one 3.1M-token scheduler block. Also apply the expandable_segments allocator fix proven on the identical SimpleCPU connector config in #2370 (2.36 GiB MLA prefill transient OOM'd while 2.63 GiB sat reserved-but-unallocated); NCCL_CUMEM_ENABLE unchanged. 中文：将 VLLM_PREFIX_CACHE_RETENTION_INTERVAL 设为 0（任何正值都必须是 Kimi-K3 KDA 混合架构 scheduler_block_size 3145728 的整数倍，0 是唯一低于一个 3.1M token 调度块的取值），并应用已在 #2370 相同 SimpleCPU 连接器配置上验证 的 expandable_segments 分配器修复（2.36 GiB MLA 预填充瞬时分配 OOM，而 2.63 GiB 处于已保留未分配状态）；NCCL_CUMEM_ENABLE 保持不变。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ntic-direct-vllm-simplecpu-retention # Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30324938760 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30326422231 |
vLLM's config validation rejects the F combination at startup: "KV connector SimpleCPUOffloadConnector is incompatible with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True unless enable_cumem_allocator is also enabled" — torch's VMM allocator can remap KV virtual addresses under pinned/registered connector memory. Enable the cumem allocator as the error prescribes (KV allocations route through CuMemAllocator's pool with expandable_segments auto-disabled for KV), matching the GB300 aggregate AgentX recipe. 中文：vLLM 配置校验在启动时拒绝 F 的组合（SimpleCPUOffloadConnector 与 expandable_segments:True 不兼容，除非同时启用 enable_cumem_allocator）： torch 的 VMM 分配器可能在连接器持有已固定/注册内存时重映射 KV 虚拟地址。 按报错提示启用 cumem 分配器（KV 分配经 CuMemAllocator 池，其中对 KV 自动 关闭 expandable_segments），与 GB300 聚合式 AgentX 配方一致。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ntic-direct-vllm-simplecpu-retention # Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30337986946 |
Ninth sweep attempt died during CUDA-graph capture with "custom_all_reduce.cuh:164 'invalid argument'": vLLM's custom all-reduce registers IPC buffers that the VMM-backed CuMemAllocator invalidates. Pair enable-cumem-allocator with disable-custom-all-reduce exactly as the GB300 aggregate AgentX recipe does; TP all-reduce falls back to NCCL. 中文：第九次扫描在 CUDA 图捕获阶段报 custom_all_reduce.cuh:164 'invalid argument'：vLLM 自定义 all-reduce 注册的 IPC 缓冲会被基于 VMM 的 CuMemAllocator 失效。按 GB300 聚合式 AgentX 配方的 做法，将 enable-cumem-allocator 与 disable-custom-all-reduce 成对启用；TP all-reduce 回退到 NCCL。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
Summary
Agentic experiment (Variant F) — SimpleCPU offload (from the closed Variant E, #2370) plus
VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0.Identical to Variant E (direct
vllm servevia srt-slurm PR #278 + multinode patch, SimpleCPUOffloadConnector CPU DRAM KV offload with 299,875,000,000 bytes/rank,enable_cross_layers_blocks: false,PYTHONHASHSEED=42, explicit prefix caching), with two changes:VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0— any positive value must be a multiple of Kimi-K3's KDA-hybridscheduler_block_size(3,145,728; the GB recipes' 32768 is hard-rejected at engine init), so 0 is the only setting below one 3.1M-token scheduler block.PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True— inherited from [Klaud Cold][agentic experiment][Variant E] Kimi-K3 B200 agg TP8xPP2 agentic — direct vllm serve + SimpleCPU KV offload / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——直接 vllm serve + SimpleCPU KV 卸载 #2370's own failure evidence: on this exact connector config a 2.36 GiB MLA prefill transient OOM'd while 2.63 GiB sat reserved-but-unallocated.NCCL_CUMEM_ENABLE=1unchanged.Related experiments
vllm serve, GPU-resident KV — [Klaud Cold][agentic experiment][Variant D] Kimi-K3 B200 agg TP8xPP2 agentic — direct vllm serve (srt-slurm PR 278) / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——直接 vllm serve（srt-slurm PR 278） #2359 (c1/c16/c32 already green)中文说明
智能体实验变体 F——SimpleCPU 卸载（来自已关闭的变体 E，#2370）加
VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0。与变体 E 完全一致（直接 vllm serve、SimpleCPUOffloadConnector CPU 内存 KV 卸载、每 rank 299,875,000,000 字节、
enable_cross_layers_blocks: false、PYTHONHASHSEED=42、显式前缀缓存），有两处变更：VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0——任何正值都必须是 Kimi-K3 KDA 混合架构scheduler_block_size（3,145,728）的整数倍，0 是唯一低于一个调度块的取值。PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True——继承自 [Klaud Cold][agentic experiment][Variant E] Kimi-K3 B200 agg TP8xPP2 agentic — direct vllm serve + SimpleCPU KV offload / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——直接 vllm serve + SimpleCPU KV 卸载 #2370 在相同连接器配置上的失败证据（2.36 GiB MLA 预填充瞬时分配 OOM，2.63 GiB 处于已保留未分配状态）；NCCL_CUMEM_ENABLE=1保持不变。🤖 Generated with Claude Code