[Klaud Cold][agentic experiment][Variant E] Kimi-K3 B200 agg TP8xPP2 agentic — direct vllm serve + SimpleCPU KV offload / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——直接 vllm serve + SimpleCPU KV 卸载 - #2370
Conversation
…ecipe Aggregated TP8 x PP2 across 2 B200 nodes (16 GPUs), plain TP (no expert parallelism) for the agentic-coding trace replay. Dedicated bring-up image vllm/vllm-openai:kimi-k3 with VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1, fastsafetensors load format, kimi_k3 tool-call/reasoning parsers. Model pre-staged at /lustre/fsw/models/Kimi-K3; launch_b200-dgxc.sh gains the kimik3/fp4 model-path mapping, the agentic recipe overlay, and the agentic cache default_mounts used by the GB200/GB300 agentic paths. 中文：新增 Kimi-K3 MXFP4 B200 聚合式 TP8xPP2 Dynamo-vLLM 智能体编码基准测试配方 （2 节点 / 16 GPU，纯 TP，不启用专家并行(EP)）。使用专用 bring-up 镜像 vllm/vllm-openai:kimi-k3（VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1、 fastsafetensors 加载格式、kimi_k3 工具调用/推理解析器）。模型已预置于 /lustre/fsw/models/Kimi-K3；启动器 launch_b200-dgxc.sh 增加 kimik3/fp4 模型路径映射、智能体配方覆盖及智能体缓存挂载。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
中文：在更新日志条目与 MODELS 表格行中补充 PR #2355 链接。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oke test The cquil11/srt-slurm-nv cam/sa-submission-q2-2026 fork rejected the recipe (benchmark.aiperf_server_metrics: Unknown field). Switch the b200-dgxc agentic clone to upstream NVIDIA/srt-slurm v1.0.36 (validated in #2302/#2341), drop the aiperf_server_metrics field, pin dynamo wheel/router to 1.2.1 (the combination validated with v1.0.36), and reduce the bring-up to a single conc-8 smoke test. 中文：cquil11/srt-slurm-nv 分支的 srtctl 校验拒绝了配方字段 benchmark.aiperf_server_metrics（Unknown field）。将 b200-dgxc 智能体路径改用 上游 NVIDIA/srt-slurm v1.0.36（已在 #2302/#2341 验证），移除该字段，dynamo wheel/router 固定为 1.2.1，并将 bring-up 缩减为单并发（conc 8）冒烟测试。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dynamo-vllm worker entrypoint rejected --enable-auto-tool-choice --tool-call-parser kimi_k3 (unrecognized arguments; different arg parser than vllm serve). Chat parsing happens at the dynamo frontend — same convention as the DSv4 GB300 agentic recipes. Keep --reasoning-parser kimi_k3 (accepted by the worker). Also drop the explicit max-model-len and let vLLM derive the native 1M window from the model config, mirroring the agentic recipe convention. 中文：dynamo-vllm worker 入口不接受 --enable-auto-tool-choice 与 --tool-call-parser kimi_k3（unrecognized arguments，与 vllm serve 的参数解析器 不同），聊天解析由 dynamo 前端处理，与 DSv4 GB300 智能体配方约定一致；保留 worker 可接受的 --reasoning-parser kimi_k3。同时移除显式 max-model-len， 由 vLLM 从模型配置推导原生 1M 上下文窗口。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Third sweep attempt: the engine loaded and served (TP8xPP2 healthy in ~14 min), but dynamo 1.2.1's rust frontend tokenizer rejects Kimi-K3's tiktoken model_type 'kimi_k3' (supported: kimi, kimi_k2, kimi_k25, deepseek_v3), so the model never registered and all chat completions returned 404, aborting the AgentX warmup. Switch to the 1.2.0.dev20260426 wheel used by the DSv4 GB300/B200 Dynamo-vLLM recipes. Upstream published v1.4.0-kimi-k3-dev.1 (2026-07-27) as the day-zero K3 build if this wheel also lacks support. 中文：第三次扫描中引擎已成功加载并提供服务（TP8xPP2 约 14 分钟就绪），但 dynamo 1.2.1 的 rust 前端分词器不支持 Kimi-K3 的 tiktoken model_type 'kimi_k3'，模型未能注册，所有请求返回 404，AgentX 预热中止。改用 DSv4 GB300/B200 Dynamo-vLLM 配方所用的 1.2.0.dev20260426 wheel；如仍不支持， 上游已于 2026-07-27 发布 day-zero 构建 v1.4.0-kimi-k3-dev.1。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin dynamo to ba83080ecd31c1ce918559e576d3c5bc9e092ff1 ("feat: Added
support for Kimi-K3", tag v1.4.0-kimi-k3-dev.1) via srt-slurm's
hash-cached source install: it adds the kimi_k3 tiktoken tokenizer to the
rust frontend (dynamo <=1.2.1 404s every request because the model never
registers) and accepts the kimi_k3 tool-call/reasoning parser worker args,
so restore --enable-auto-tool-choice --tool-call-parser kimi_k3
--reasoning-parser kimi_k3.
中文：将 dynamo 固定到 day-zero Kimi-K3 提交 ba83080（"feat: Added support
for Kimi-K3"，标签 v1.4.0-kimi-k3-dev.1），通过 srt-slurm 的哈希缓存源码
安装：该提交为 rust 前端新增 kimi_k3 tiktoken 分词器（dynamo <=1.2.1 因模型
无法注册而全部返回 404），worker 亦支持 kimi_k3 解析器参数，故恢复
--enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3。
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the vLLM OpenAI-frontend spellings (--enable-auto-tool-choice / --tool-call-parser) with dynamo's namespaced worker args: --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3. 中文：将 vLLM OpenAI 前端风格参数（--enable-auto-tool-choice / --tool-call-parser）替换为 dynamo 命名空间的 worker 参数： --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fifth sweep attempt: the day-zero dynamo registered the kimi_k3 tiktoken tokenizer and the engine served, but all warmup requests got 400 — aiperf's conv-aware routing emits nvext.session_control, a removed POC field this dynamo build rejects (schema moved to router/routing_constraints/ agent_hints). Opt out via AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0, matching the GB300 aggregate AgentX recipes; a single aggregate worker has no P/D routing to bind anyway. 中文：第五次扫描中 day-zero dynamo 已成功注册 kimi_k3 tiktoken 分词器并正常 服务，但全部预热请求返回 400——aiperf 的会话感知路由会发送 nvext.session_control（已被移除的 POC 字段，schema 已迁移至 router/routing_constraints/agent_hints）。通过 AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0 关闭，与 GB300 聚合式 AgentX 配方 一致；单聚合 worker 本无需 P/D 路由绑定。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sixth sweep attempt (both A and C variants): warmup requests 500 then the model 503s — the image's first decode step crashes in the KDA hybrid-state postprocess (mamba_hybrid.py postprocess_state, IndexError: index_fill_(): Expected dtype int64 for index; torch requires an int64 index but the runner passes the int32 idx_mapping). Ship an in-container patch through srt-slurm's setup_script hook (same pattern as configs/patches/ vllm_numa_bind_hash_fix.py): coerce the index with .long(), idempotent, refuses to run if the image layout changed. 中文：第六次扫描（A、C 两个变体一致）：预热请求先 500、随后模型 503——镜像 首个解码步在 KDA 混合状态后处理中崩溃（mamba_hybrid.py postprocess_state， IndexError: index_fill_() 需要 int64 索引，但 runner 传入 int32 idx_mapping）。 通过 srt-slurm 的 setup_script 钩子在容器内打补丁：将索引用 .long() 转换， 幂等，且镜像布局变化时拒绝执行。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Serve Kimi-K3 directly with vllm serve (srt-slurm PR #278 frontend.type: vllm, branch kylliang/direct-aggregate-vllm): no dynamo frontend/worker/ router, which removes the dynamo tokenizer/schema gaps entirely, and the OpenAI-frontend flags --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 become legitimate. PR #278 validates single-node only, so ship patches/srt-slurm-pr278-direct-vllm-multinode.patch extending it to vLLM-native multi-node serve (--master-addr/--nnodes/--node-rank, headless non-leader ranks) for the 2-node TP8xPP2 topology. Keeps the mamba_hybrid index-dtype container patch (engine bug is frontend-agnostic). 中文：智能体实验变体 D——通过 srt-slurm PR #278（frontend.type: vllm）直接以 vllm serve 提供服务：去除 dynamo 前端/worker/router，从根本上规避 dynamo 的 分词器与 schema 兼容问题，OpenAI 前端参数 --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 因此可用。PR #278 仅 支持单节点，故新增补丁将其扩展为 vLLM 原生多节点 serve（--master-addr/ --nnodes/--node-rank，非主节点 headless），以运行 2 节点 TP8xPP2 拓扑。保留 mamba_hybrid 索引类型容器补丁（引擎缺陷与前端无关）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2359。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same engine-level OOM as the dynamo-frontend variants: the flashinfer trtllm MXFP4 MoE kernel allocates a ~1.6 GiB runtime workspace outside vLLM's memory pool on the first forward; at 0.95 a 178 GiB B200 has only ~1.35 GiB free. 中文：与 dynamo 前端变体相同的引擎级 OOM：flashinfer trtllm MXFP4 MoE 内核在 首个前向时于 vLLM 显存池外分配约 1.6 GiB 工作区，0.95 下仅剩约 1.35 GiB。 改为 0.90。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Inherited from the closed dynamo-frontend variants (#2355/#2358): at gpu-mem-util 0.90 the first long-context MLA prefill OOM'd on a 2.92 GiB transient while 3.39 GiB sat reserved-but-unallocated (fragmentation). Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and drop NCCL_CUMEM_ENABLE. 中文：继承自已关闭的 dynamo 前端变体（#2355/#2358）：0.90 显存利用率下首个 长上下文 MLA 预填充因 2.92 GiB 瞬时分配 OOM，而 3.39 GiB 处于已保留未分配 状态（碎片化）。设置 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True 并 移除 NCCL_CUMEM_ENABLE。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…refill OOM)" This reverts commit 4370988. The superseded direct-vllm run served the agentic benchmark for 24 minutes on the original env (NCCL_CUMEM_ENABLE=1, no expandable_segments) without any OOM — the allocator change was precautionary carryover from the closed dynamo-frontend variants and was never justified by evidence from this serving path. Restore the env that was demonstrably running. 中文：回滚 4370988。被中断的 direct-vllm 运行在原始环境 （NCCL_CUMEM_ENABLE=1、未设 expandable_segments）下已稳定运行智能体基准测试 24 分钟且无 OOM——该分配器改动只是从已关闭的 dynamo 前端变体沿袭的预防性 措施，并无本服务路径上的证据支持。恢复已被验证可运行的环境。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep prefix-cache blocks alive across agentic turn gaps, matching the GB200/GB300 AgentX recipes. 中文：新增 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768，使前缀缓存块在智能体 回合间隔内保持留存，与 GB200/GB300 AgentX 配方一致。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
中文：将智能体并发列表从单点 8 扩展为 1/8/16/32。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Engine init hard-fails on Kimi-K3 with the GB200/GB300 AgentX value: "VLLM_PREFIX_CACHE_RETENTION_INTERVAL (32768) must be non-negative and a multiple of scheduler_block_size (3145728)" — the KDA hybrid architecture gives K3 a 3.1M-token scheduler block. Default retention served fine in the earlier runs, so drop the override. 中文：移除 VLLM_PREFIX_CACHE_RETENTION_INTERVAL——Kimi-K3 的 KDA 混合架构使 scheduler_block_size 达 3145728，GB200/GB300 AgentX 的 32768 取值导致引擎 初始化直接失败（必须为其整数倍）。此前运行证明默认留存策略可正常服务， 故不再覆盖。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…fload Variant of the direct-vllm Kimi-K3 B200 agg TP8xPP2 agentic bring-up (#2359) with KV offload to CPU DRAM: SimpleCPUOffloadConnector (kv_role kv_both, lazy_offload false, enable_cross_layers_blocks true), cpu_bytes_to_use_per_rank 299,875,000,000 (the framework's 0.80-utilization budget for cluster:b200-dgxc: 2399 GB/node / 8 ranks), PYTHONHASHSEED=42 for cross-rank block-key hashing, explicit enable-prefix-caching, and dram/vllm-simple kv-offload labeling in the master entry. 中文：直接 vllm serve 的 Kimi-K3 B200 聚合式 TP8xPP2 智能体实验变体 E （基于 #2359）：通过 SimpleCPUOffloadConnector 将 KV 卸载至 CPU 内存 （kv_role kv_both、lazy_offload false、enable_cross_layers_blocks true）， cpu_bytes_to_use_per_rank 为 299,875,000,000（框架按 0.80 利用率对 cluster:b200-dgxc 的预算：每节点 2399 GB / 8 个 rank），设置 PYTHONHASHSEED=42 保证各 rank 前缀块键一致，显式启用前缀缓存，并在主配置 中以 dram/vllm-simple 标注 KV 卸载。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2370。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Approving — the only finding is a minor doc-attribution nit (Kimi-K3 date-added link should stay on #2359, not repoint to this PR), which doesn't affect functionality and is easily fixed on rebase.
Extended reasoning...
Overview
This PR is a follow-on agentic benchmark experiment (Variant E) that layers SimpleCPUOffloadConnector CPU-DRAM KV offloading on top of the already-merged Variant D (#2359) Kimi-K3 B200 TP8xPP2 config. It touches: two doc tables (MODELS.md/MODELS_zh.md), a new container-setup patch script, a new srt-slurm multinode patch (vendored, already reviewed presumably in #2359), a new recipe YAML, a master-config entry, a changelog entry, and launcher routing in launch_b200-dgxc.sh.
Security risks
None. This is benchmark infrastructure — no auth, crypto, or user-facing surfaces. The in-container Python patch script (kimi-k3-container-deps.sh) only patches a vendored vLLM file inside the benchmark container via regex substitution, guarded by an idempotency check and a hard failure if the expected pattern count doesn't match.
Level of scrutiny
Low-to-moderate. This is a benchmark experiment config, not production-serving code, and it closely mirrors the already-vetted #2359 base (same serving stack, same patch, same container). The new surface area specific to this PR is essentially the KV-offload connector config block and associated labeling, which the bug-hunting pass already verified (cpu_bytes_to_use_per_rank math, string vs. bool field, health-check max_attempts, kv-offload-backend version field — all ruled out as non-issues).
Other factors
The only substantive finding was a documentation nit: this PR is based on a stale pre-#2359 file state, so its diff incorrectly repoints the Kimi-K3 "Date added" link from #2359 to #2370. This is a one-line, easily-fixed-on-rebase issue that has no functional impact on the benchmark itself and does not warrant blocking merge — a human can trivially resolve the same-line conflict when rebasing. No outstanding reviewer comments exist on this PR, and I have no prior review recorded here.
| |---|---|---|---|---| | ||
| | Qwen3.8 2.4T | `qwen3.8` | TBD | Agentic coding | | | ||
| | Kimi-K3 | `kimik3` | 2026-07-27 | Agentic coding | | | ||
| | Kimi-K3 | `kimik3` | 2026-07-27 ([#2370](https://github.com/SemiAnalysisAI/InferenceX/pull/2370)) | Agentic coding | | |
There was a problem hiding this comment.
🟡 MODELS.md and MODELS_zh.md are edited from a stale pre-#2359 base, so this PR's diff repoints the Kimi-K3 'Date added' link from #2359 (the PR that actually onboarded the model) to #2370 (this PR, which is just a follow-on offloading experiment on the already-added model). This misattributes when/where Kimi-K3 entered the benchmark suite in both files; the link should stay on #2359.
Extended reasoning...
What the bug is: Both MODELS.md line 21 and MODELS_zh.md line 21 change the Kimi-K3 row's 'Date added' link. The diff's before context is a bare, unlinked 2026-07-27, and the after is 2026-07-27 ([#2370](...)). But on current main (verified via git show HEAD:MODELS.md/HEAD:MODELS_zh.md), that row already reads 2026-07-27 ([#2359](...)) — this was set by an earlier commit that landed after this PR's branch point. Because this PR's working tree is stale relative to main, its diff is computed against the old bare-date version, and applying it as-is would overwrite the correct #2359 link with an incorrect #2370 link.\n\nWhy #2359 vs #2370 matters: the PR's own description spells out the lineage explicitly: '#2359' is 'Variant D (base): direct vllm serve, GPU-resident KV', and this PR (#2370) is 'Variant E — Variant D + SimpleCPUOffloadConnector CPU DRAM offload'. In other words, #2359 is the PR that actually onboarded Kimi-K3 to the benchmark suite (added the model, container, recipes, master-config entry for the first time), while #2370 is a follow-on experiment layered on top of an already-added model — just a KV-offloading variant of an existing config. The 'Date added' column's established convention (see GLM-5.2 → #2268, MiniMax-M3 → #1724 in the same table) is to link the PR that onboarded the model, not every subsequent experiment PR that touches its config.\n\nWhy nothing else in the PR catches this: the change lands as an incidental side effect of stale-branch diffing, not a deliberate edit — there is no test or lint that checks documentation attribution links, so it would merge silently (or, more likely, surface as a same-line merge conflict against main's #2359 edit, since both sides changed the same line from the same bare-date ancestor — but a human resolving that conflict could easily default to 'ours'/#2370 without noticing the semantic error).\n\nStep-by-step proof:\n1. Run git show HEAD:MODELS.md | sed -n '"21p"' on current main → outputs | Kimi-K3 | \kimik3` | 2026-07-27 (#2359) | Agentic coding | |.\n2. This PR's diff hunk for that same line shows a -side of2026-07-27 | Agentic coding(no link at all) and a+side of2026-07-27 (#2370)— proving the diff was authored against a version of the file that predates the #2359 attribution commit.\n3. Read this PR's own description: '#2359' is labeled 'Variant D (base)' and described as the PR onboarding Kimi-K3 with direct vllm serve; this PR (#2370) is labeled 'Variant E', an experiment adding CPU DRAM KV offload on top of the Variant D config — i.e., #2370 doesn't onboard the model, it experiments on an already-onboarded one.\n4. Therefore replacing #2359 with #2370 in the Date-added column is factually wrong regardless of merge mechanics, and merging this PR as authored would regress the currently-correct attribution.\n\n**Fix:** keep (or restore) the link to #2359 in bothMODELS.mdandMODELS_zh.md` line 21 — i.e., drop this PR's edit to that line entirely, or rebase onto current main so the line is left untouched.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30323102172 |
With enable_cross_layers_blocks true, KV-cache initialization crashes on Kimi-K3's KDA hybrid: "shape '[64980, 64, 576]' is invalid for input of size 199618560" — a 12x element mismatch equal to the MLA layers per PP stage (24 gated-MLA layers / PP2), i.e. the cross-layer block folding mishandles the hybrid-model geometry. Run with cross-layer blocks off, matching the GB200 connector configs. 中文：enable_cross_layers_blocks 为 true 时，Kimi-K3 的 KDA 混合架构在 KV 缓存初始化阶段崩溃（元素数量差 12 倍，恰为每个 PP 阶段的 MLA 层数 24/2）——跨层块折叠未正确处理混合架构几何。改为关闭跨层块，与 GB200 连接器配置一致。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30324017964 |
Evidence from this variant's own conc16 run: a 2.36 GiB MLA long-context prefill transient OOM'd while 2.63 GiB sat reserved-but-unallocated — allocator fragmentation. The GPU-resident variant D runs green without this, but the offload connector's staging shifts the allocation layout enough to fragment. Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True; NCCL_CUMEM_ENABLE stays unchanged. 中文：本变体自身 conc16 运行的证据：MLA 长上下文预填充的 2.36 GiB 瞬时分配 OOM，而 2.63 GiB 处于已保留未分配状态——分配器碎片化。GPU 常驻 KV 的变体 D 无需此项即可通过，但卸载连接器的暂存缓冲改变了分配布局。设置 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True；NCCL_CUMEM_ENABLE 保持不变。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Closing in favor of Variant F (#2372), which carries the same SimpleCPUOffloadConnector config forward (with the cross-layer-blocks and allocator-fragmentation findings from this PR's runs) plus the prefix-cache retention experiment. In-flight sweep cancelled. 中文：关闭本 PR，转向变体 F（#2372）——其延续了相同的 SimpleCPUOffloadConnector 配置（并继承本 PR 运行中发现的跨层块与分配器碎片化修复），另加前缀缓存留存实验。进行中的扫描已取消。 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30325856407 |
Set VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0: any positive value must be a multiple of Kimi-K3's KDA-hybrid scheduler_block_size (3145728), so 0 is the only setting below one 3.1M-token scheduler block. Also apply the expandable_segments allocator fix proven on the identical SimpleCPU connector config in #2370 (2.36 GiB MLA prefill transient OOM'd while 2.63 GiB sat reserved-but-unallocated); NCCL_CUMEM_ENABLE unchanged. 中文：将 VLLM_PREFIX_CACHE_RETENTION_INTERVAL 设为 0（任何正值都必须是 Kimi-K3 KDA 混合架构 scheduler_block_size 3145728 的整数倍，0 是唯一低于一个 3.1M token 调度块的取值），并应用已在 #2370 相同 SimpleCPU 连接器配置上验证 的 expandable_segments 分配器修复（2.36 GiB MLA 预填充瞬时分配 OOM，而 2.63 GiB 处于已保留未分配状态）；NCCL_CUMEM_ENABLE 保持不变。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
Summary
Agentic experiment (Variant E) — Variant D (#2359) plus SimpleCPUOffloadConnector KV offload to CPU DRAM.
Identical serving stack to #2359 (direct
vllm servevia srt-slurm PR #278 + multinode patch,vllm/vllm-openai:kimi-k3, TP8×PP2 over 2 B200 nodes, kimi_k3 parsers, mamba_hybrid container patch, conc 1/8/16/32), with CPU DRAM KV offloading enabled:--kv-transfer-configwithSimpleCPUOffloadConnector(kv_role: kv_both,lazy_offload: false,enable_cross_layers_blocks: true)cpu_bytes_to_use_per_rank: 299,875,000,000— exactly the framework's own budget forcluster:b200-dgxc: DRAM min(3,095,781 MiB, 2,861,022 MiB cap) × 0.80 utilization = 2399 GB/node (matrixtotal-cpu-dram-gb), ÷ 8 ranksPYTHONHASHSEED=42(top-level recipe environment) so identical prefixes hash to identical block keys across ranksenable-prefix-caching: true— offloaded blocks are only useful if prefix caching reuses them across agentic turnskv-offloading: dram/kv-offload-backend: vllm-simplewithdram-utilization: 0.80, so results ingest as an offloading variant (exp names:kimik3_p1x8_d0x8_conc*_kvdram-vllm-simple)Related experiments
vllm serve, GPU-resident KV — [Klaud Cold][agentic experiment][Variant D] Kimi-K3 B200 agg TP8xPP2 agentic — direct vllm serve (srt-slurm PR 278) / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——直接 vllm serve（srt-slurm PR 278） #2359中文说明
智能体实验变体 E——在变体 D（#2359）基础上启用 SimpleCPUOffloadConnector 将 KV 卸载至 CPU 内存。
服务栈与 #2359 完全一致（直接 vllm serve、srt-slurm PR #278 + 多节点补丁、TP8×PP2 跨 2 个 B200 节点、kimi_k3 解析器、mamba_hybrid 容器补丁、并发 1/8/16/32），并启用 CPU 内存 KV 卸载：
--kv-transfer-config使用SimpleCPUOffloadConnector（kv_role: kv_both、lazy_offload: false、enable_cross_layers_blocks: true）cpu_bytes_to_use_per_rank: 299,875,000,000——严格按框架对cluster:b200-dgxc的预算计算：内存 min(3,095,781 MiB, 2,861,022 MiB 上限) × 0.80 利用率 = 每节点 2399 GB，÷ 8 个 rankPYTHONHASHSEED=42，保证各 rank 对相同前缀生成一致的块键enable-prefix-caching: true——卸载的 KV 块只有在前缀缓存跨回合复用时才有价值kv-offloading: dram/kv-offload-backend: vllm-simple（dram-utilization: 0.80）标注，结果按卸载变体入库相关实验
变体 D（基础，GPU 常驻 KV）#2359；变体 E（本 PR，+ SimpleCPUOffloadConnector CPU 内存卸载）；变体 A/B/C（Dynamo 前端）#2355/#2357/#2358 均已关闭。
🤖 Generated with Claude Code