[Klaud Cold][agentic experiment][Variant D] Kimi-K3 B200 agg TP8xPP2 agentic — direct vllm serve (srt-slurm PR 278) / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——直接 vllm serve（srt-slurm PR 278） - #2359
Conversation
…ecipe Aggregated TP8 x PP2 across 2 B200 nodes (16 GPUs), plain TP (no expert parallelism) for the agentic-coding trace replay. Dedicated bring-up image vllm/vllm-openai:kimi-k3 with VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1, fastsafetensors load format, kimi_k3 tool-call/reasoning parsers. Model pre-staged at /lustre/fsw/models/Kimi-K3; launch_b200-dgxc.sh gains the kimik3/fp4 model-path mapping, the agentic recipe overlay, and the agentic cache default_mounts used by the GB200/GB300 agentic paths. 中文：新增 Kimi-K3 MXFP4 B200 聚合式 TP8xPP2 Dynamo-vLLM 智能体编码基准测试配方 （2 节点 / 16 GPU，纯 TP，不启用专家并行(EP)）。使用专用 bring-up 镜像 vllm/vllm-openai:kimi-k3（VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1、 fastsafetensors 加载格式、kimi_k3 工具调用/推理解析器）。模型已预置于 /lustre/fsw/models/Kimi-K3；启动器 launch_b200-dgxc.sh 增加 kimik3/fp4 模型路径映射、智能体配方覆盖及智能体缓存挂载。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
中文：在更新日志条目与 MODELS 表格行中补充 PR #2355 链接。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oke test The cquil11/srt-slurm-nv cam/sa-submission-q2-2026 fork rejected the recipe (benchmark.aiperf_server_metrics: Unknown field). Switch the b200-dgxc agentic clone to upstream NVIDIA/srt-slurm v1.0.36 (validated in #2302/#2341), drop the aiperf_server_metrics field, pin dynamo wheel/router to 1.2.1 (the combination validated with v1.0.36), and reduce the bring-up to a single conc-8 smoke test. 中文：cquil11/srt-slurm-nv 分支的 srtctl 校验拒绝了配方字段 benchmark.aiperf_server_metrics（Unknown field）。将 b200-dgxc 智能体路径改用 上游 NVIDIA/srt-slurm v1.0.36（已在 #2302/#2341 验证），移除该字段，dynamo wheel/router 固定为 1.2.1，并将 bring-up 缩减为单并发（conc 8）冒烟测试。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dynamo-vllm worker entrypoint rejected --enable-auto-tool-choice --tool-call-parser kimi_k3 (unrecognized arguments; different arg parser than vllm serve). Chat parsing happens at the dynamo frontend — same convention as the DSv4 GB300 agentic recipes. Keep --reasoning-parser kimi_k3 (accepted by the worker). Also drop the explicit max-model-len and let vLLM derive the native 1M window from the model config, mirroring the agentic recipe convention. 中文：dynamo-vllm worker 入口不接受 --enable-auto-tool-choice 与 --tool-call-parser kimi_k3（unrecognized arguments，与 vllm serve 的参数解析器 不同），聊天解析由 dynamo 前端处理，与 DSv4 GB300 智能体配方约定一致；保留 worker 可接受的 --reasoning-parser kimi_k3。同时移除显式 max-model-len， 由 vLLM 从模型配置推导原生 1M 上下文窗口。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Third sweep attempt: the engine loaded and served (TP8xPP2 healthy in ~14 min), but dynamo 1.2.1's rust frontend tokenizer rejects Kimi-K3's tiktoken model_type 'kimi_k3' (supported: kimi, kimi_k2, kimi_k25, deepseek_v3), so the model never registered and all chat completions returned 404, aborting the AgentX warmup. Switch to the 1.2.0.dev20260426 wheel used by the DSv4 GB300/B200 Dynamo-vLLM recipes. Upstream published v1.4.0-kimi-k3-dev.1 (2026-07-27) as the day-zero K3 build if this wheel also lacks support. 中文：第三次扫描中引擎已成功加载并提供服务（TP8xPP2 约 14 分钟就绪），但 dynamo 1.2.1 的 rust 前端分词器不支持 Kimi-K3 的 tiktoken model_type 'kimi_k3'，模型未能注册，所有请求返回 404，AgentX 预热中止。改用 DSv4 GB300/B200 Dynamo-vLLM 配方所用的 1.2.0.dev20260426 wheel；如仍不支持， 上游已于 2026-07-27 发布 day-zero 构建 v1.4.0-kimi-k3-dev.1。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin dynamo to ba83080ecd31c1ce918559e576d3c5bc9e092ff1 ("feat: Added
support for Kimi-K3", tag v1.4.0-kimi-k3-dev.1) via srt-slurm's
hash-cached source install: it adds the kimi_k3 tiktoken tokenizer to the
rust frontend (dynamo <=1.2.1 404s every request because the model never
registers) and accepts the kimi_k3 tool-call/reasoning parser worker args,
so restore --enable-auto-tool-choice --tool-call-parser kimi_k3
--reasoning-parser kimi_k3.
中文：将 dynamo 固定到 day-zero Kimi-K3 提交 ba83080（"feat: Added support
for Kimi-K3"，标签 v1.4.0-kimi-k3-dev.1），通过 srt-slurm 的哈希缓存源码
安装：该提交为 rust 前端新增 kimi_k3 tiktoken 分词器（dynamo <=1.2.1 因模型
无法注册而全部返回 404），worker 亦支持 kimi_k3 解析器参数，故恢复
--enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3。
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the vLLM OpenAI-frontend spellings (--enable-auto-tool-choice / --tool-call-parser) with dynamo's namespaced worker args: --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3. 中文：将 vLLM OpenAI 前端风格参数（--enable-auto-tool-choice / --tool-call-parser）替换为 dynamo 命名空间的 worker 参数： --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fifth sweep attempt: the day-zero dynamo registered the kimi_k3 tiktoken tokenizer and the engine served, but all warmup requests got 400 — aiperf's conv-aware routing emits nvext.session_control, a removed POC field this dynamo build rejects (schema moved to router/routing_constraints/ agent_hints). Opt out via AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0, matching the GB300 aggregate AgentX recipes; a single aggregate worker has no P/D routing to bind anyway. 中文：第五次扫描中 day-zero dynamo 已成功注册 kimi_k3 tiktoken 分词器并正常 服务，但全部预热请求返回 400——aiperf 的会话感知路由会发送 nvext.session_control（已被移除的 POC 字段，schema 已迁移至 router/routing_constraints/agent_hints）。通过 AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0 关闭，与 GB300 聚合式 AgentX 配方 一致；单聚合 worker 本无需 P/D 路由绑定。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sixth sweep attempt (both A and C variants): warmup requests 500 then the model 503s — the image's first decode step crashes in the KDA hybrid-state postprocess (mamba_hybrid.py postprocess_state, IndexError: index_fill_(): Expected dtype int64 for index; torch requires an int64 index but the runner passes the int32 idx_mapping). Ship an in-container patch through srt-slurm's setup_script hook (same pattern as configs/patches/ vllm_numa_bind_hash_fix.py): coerce the index with .long(), idempotent, refuses to run if the image layout changed. 中文：第六次扫描（A、C 两个变体一致）：预热请求先 500、随后模型 503——镜像 首个解码步在 KDA 混合状态后处理中崩溃（mamba_hybrid.py postprocess_state， IndexError: index_fill_() 需要 int64 索引，但 runner 传入 int32 idx_mapping）。 通过 srt-slurm 的 setup_script 钩子在容器内打补丁：将索引用 .long() 转换， 幂等，且镜像布局变化时拒绝执行。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Serve Kimi-K3 directly with vllm serve (srt-slurm PR #278 frontend.type: vllm, branch kylliang/direct-aggregate-vllm): no dynamo frontend/worker/ router, which removes the dynamo tokenizer/schema gaps entirely, and the OpenAI-frontend flags --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 become legitimate. PR #278 validates single-node only, so ship patches/srt-slurm-pr278-direct-vllm-multinode.patch extending it to vLLM-native multi-node serve (--master-addr/--nnodes/--node-rank, headless non-leader ranks) for the 2-node TP8xPP2 topology. Keeps the mamba_hybrid index-dtype container patch (engine bug is frontend-agnostic). 中文：智能体实验变体 D——通过 srt-slurm PR #278（frontend.type: vllm）直接以 vllm serve 提供服务：去除 dynamo 前端/worker/router，从根本上规避 dynamo 的 分词器与 schema 兼容问题，OpenAI 前端参数 --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 因此可用。PR #278 仅 支持单节点，故新增补丁将其扩展为 vLLM 原生多节点 serve（--master-addr/ --nnodes/--node-rank，非主节点 headless），以运行 2 节点 TP8xPP2 拓扑。保留 mamba_hybrid 索引类型容器补丁（引擎缺陷与前端无关）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2359。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same engine-level OOM as the dynamo-frontend variants: the flashinfer trtllm MXFP4 MoE kernel allocates a ~1.6 GiB runtime workspace outside vLLM's memory pool on the first forward; at 0.95 a 178 GiB B200 has only ~1.35 GiB free. 中文：与 dynamo 前端变体相同的引擎级 OOM：flashinfer trtllm MXFP4 MoE 内核在 首个前向时于 vLLM 显存池外分配约 1.6 GiB 工作区，0.95 下仅剩约 1.35 GiB。 改为 0.90。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30305597893 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| IS_MULTINODE: "true" | ||
| # aiperf's conv-aware routing emits nvext.session_control, a removed POC | ||
| # field this dynamo build 400-rejects at warmup (schema moved to | ||
| # router/routing_constraints/agent_hints). Same opt-out as the GB300 | ||
| # aggregate AgentX recipes — and with a single aggregate worker there is | ||
| # no P/D routing to bind anyway. | ||
| AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0" | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache" | ||
| HF_HUB_CACHE: "/hf_hub_cache" | ||
| WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126" |
There was a problem hiding this comment.
🟡 This recipe is the first to pair the shared agentic_srt.sh inter-concurrency drain wait (which polls for the Dynamo-only metric dynamo_frontend_active_requests) with a no-Dynamo, direct-vllm serve frontend that never emits it, so the idle check is vacuously true from the first poll. It's dormant today because conc-list is a single point [8] (the drain wait only runs between points), but the recipe's own comment says the curve will be widened later — at that point later concurrency points would start against a still-draining server with no error surfaced. Worth wiring AIPERF_SERVER_METRICS_URLS or a vllm:* frontend check before widening the curve.
Extended reasoning...
benchmarks/multi_node/agentic_srt.sh's wait_for_agentic_servers_idle (used between concurrency points, per benchmark.command in this recipe) determines drain by polling the frontend's /metrics for dynamo_frontend_active_requests and, as a fallback, worker URLs from AIPERF_SERVER_METRICS_URLS for vllm:num_requests_running/vllm:num_requests_waiting. This recipe sets dynamo.install: false and frontend.type: vllm, which (via srt-slurm PR #278 + the multinode patch in this PR) runs a direct vllm serve OpenAI server. That server exposes vllm:* metrics, never dynamo_frontend_active_requests — a Dynamo-frontend-only metric name that a repo-wide grep finds nowhere else. metric_sum() (agentic_srt.sh:59-68) returns 0.0 for an absent metric name rather than raising, so frontend_active is silently always 0.\n\nThe fallback path is also dead here: AIPERF_SERVER_METRICS_URLS is exported only by the AMD path (benchmarks/multi_node/amd_utils/server_sglang.sh) and is never set anywhere in this recipe, its benchmark.env, or runners/launch_b200-dgxc.sh. So worker_urls is empty and worker_active also stays 0.0. The idle predicate frontend_active == 0 and worker_active == 0 (agentic_srt.sh:85) is therefore vacuously true from the very first successful poll; after three consecutive polls (~20-30s at the default 10s cadence) it declares the servers idle and returns — regardless of whether the server actually still has requests in flight. This fails silently: no exception, no warning surfaced to the sweep log beyond the routine per-poll status line.\n\nStep-by-step proof of the gap:\n1. Concurrency point N finishes its aiperf profile run and the driver calls wait_for_agentic_servers_idle (agentic_srt.sh:118-119, only when index < len(CONCURRENCIES)-1).\n2. The Python poller fetches http://localhost:/metrics from the direct vllm serve frontend and searches for a line matching dynamo_frontend_active_requests — none exists, so metric_sum returns 0.0.\n3. worker_urls is empty (parsed from the unset AIPERF_SERVER_METRICS_URLS), so the worker-metric loop never executes and worker_active stays 0.0.\n4. frontend_active == 0 and worker_active == 0 is true on poll 1 (and every poll), so idle_polls reaches 3 after ~30s and the function returns via SystemExit(0) — regardless of real server load.\n5. The driver immediately proceeds to build and launch concurrency point N+1's replay against a server that, absent this check, could still be finishing point N's in-flight requests.\n\nThis is provably inert in the PR as shipped: configs/nvidia-master.yaml's new kimik3-fp4-b200-dynamo-vllm-agentic entry sets conc-list: [8], a single-element list, so index < len-1 is 0 < 0 = false and wait_for_agentic_servers_idle is never invoked at all in this run — the refutation on this point is correct and I'm not disputing it. But the recipe's own comment explicitly states intent to 'widen the conc curve once the topology is proven green,' and this PR is what first pairs the Dynamo-oriented drain logic in agentic_srt.sh with a direct-vLLM frontend that can never satisfy it. The moment a second concurrency point is added — likely the very next iteration on this recipe — every point after the first would start against a server the drain wait incorrectly reports as idle, silently corrupting per-concurrency latency/throughput measurements with no error to flag the run as suspect.\n\nFix: before widening conc-list beyond one point, either set AIPERF_SERVER_METRICS_URLS to the frontend's own /metrics endpoint (so the existing vllm:num_requests_running/waiting check picks up real load), or add a frontend-side check for those same vllm:* metric names alongside dynamo_frontend_active_requests in wait_for_agentic_servers_idle. Since the shipped single-point config never executes the buggy path, this doesn't block merge of the bring-up PR — it's a heads-up to fix before the planned curve-widening follow-up.
| health_check: | ||
| interval_seconds: 10 | ||
| max_attempts: 1440 |
There was a problem hiding this comment.
🟡 The launcher's pre-existing sed at runners/launch_b200-dgxc.sh:300 (sed -i 's/^ max_attempts: [0-9]*/ max_attempts: 720/') unconditionally rewrites this recipe's health_check.max_attempts: 1440 (4h) down to 720 (2h) before every srtctl apply, since the recipe line matches the 2-space-indent regex. This is pre-existing launcher behavior surfaced by this PR's recipe, not a bug the PR introduces — and in this specific recipe it's inert since VLLM_ENGINE_READY_TIMEOUT_S: 3600 (1h) already caps engine startup well below even the clobbered 2h health-check window, so no premature abort is possible today. Still worth a follow-up so a future recipe that actually needs >2h isn't silently bitten (e.g. bump the sed's floor or make it a max()-style clamp).
Extended reasoning...
What happens: runners/launch_b200-dgxc.sh line ~300 runs, unconditionally on every multinode srtctl apply on this launcher:
sed -i 's/^ max_attempts: [0-9]*/ max_attempts: 720/' "${CONFIG_FILE%%:*}"The new recipe agg-b200-tp8pp2-agentic.yaml sets health_check.max_attempts: 1440 with exactly 2-space indentation. The regex ^ max_attempts: [0-9]* matches that line (correct indent, [0-9]* matches 1440) and rewrites it to 720. So the recipe author's deliberately-sized 4-hour health-check budget (documented in the recipe's own comments as sized for the ~1.4TB MXFP4 checkpoint loading off shared Lustre across 2 nodes) silently becomes 2 hours at apply time. This is confirmed by three independent verifiers reading the same launcher code and recipe file — the clobber mechanism itself is not in question.
Why this doesn't actually cause the described harm here: the reported failure mode was 'health check declares the job dead and aborts mid weight-load on a cold cache.' But this recipe also sets VLLM_ENGINE_READY_TIMEOUT_S: \"3600\" (1 hour) in backend.aggregated_environment, with a comment stating this env var is the effective weight-load budget ('keep the engine-ready window generous'). Walking through the cases:
- If weight loading finishes within the 1h engine-ready window, the vLLM
/healthendpoint comes up well inside even the clobbered 720-attempt (2h) ceiling — 720 vs. 1440 never matters. - If loading exceeds 1h,
VLLM_ENGINE_READY_TIMEOUT_S=3600fires first and the vLLM engine itself aborts at the 1-hour mark — again independent of whether the health-check ceiling is 2h or 4h, since 1h < 2h < 4h in both cases.
So the binding constraint on this recipe's cold-cache load time is the 1-hour engine-ready timeout, which sits comfortably below even the clobbered 2-hour health-check budget. The 1440→720 rewrite has no observable effect on this recipe's behavior today, and 720 attempts is also the value this same launcher already uses successfully for DSR1-FP8's ~680GB checkpoint.
Why it's still worth flagging (nit, not blocking): the recipe author explicitly set 1440 believing it would take effect, and it silently doesn't — that's a genuine, misleading gotcha for whoever revisits this recipe later (e.g. to widen the concurrency curve or bump VLLM_ENGINE_READY_TIMEOUT_S past 2h for a future larger checkpoint). At that point the same sed would silently reintroduce a real spurious-abort risk with no error or warning. The fix is cheap and low-risk: either raise the sed's forced floor (e.g. to 1440) or change it to a max()-style clamp (only bump up, never down) so it can never silently shrink a value a recipe author intentionally set higher.
Step-by-step proof of the clobber (not of harm):
- Recipe file contains the line
health_check:\n max_attempts: 1440(2-space indent underhealth_check:). - Launcher clones/applies the recipe, then unconditionally runs
sed -i 's/^ max_attempts: [0-9]*/ max_attempts: 720/' "${CONFIG_FILE%%:*}"on the resolved config path beforesrtctl apply. - The regex anchors on exactly 2 leading spaces, which matches this recipe's indentation;
[0-9]*greedily matches1440. - Post-sed, the file on disk contains
max_attempts: 720, andsrtctl applyreads that rewritten value — the 1440 the author wrote in source never reaches the running job. - Given
VLLM_ENGINE_READY_TIMEOUT_S=3600in the same recipe, the 720-attempt (7200s) window is never the limiting factor in practice, so no user-visible regression results from this specific PR.
Inherited from the closed dynamo-frontend variants (#2355/#2358): at gpu-mem-util 0.90 the first long-context MLA prefill OOM'd on a 2.92 GiB transient while 3.39 GiB sat reserved-but-unallocated (fragmentation). Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and drop NCCL_CUMEM_ENABLE. 中文：继承自已关闭的 dynamo 前端变体（#2355/#2358）：0.90 显存利用率下首个 长上下文 MLA 预填充因 2.92 GiB 瞬时分配 OOM，而 3.39 GiB 处于已保留未分配 状态（碎片化）。设置 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True 并 移除 NCCL_CUMEM_ENABLE。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30306520077 |
…refill OOM)" This reverts commit 4370988. The superseded direct-vllm run served the agentic benchmark for 24 minutes on the original env (NCCL_CUMEM_ENABLE=1, no expandable_segments) without any OOM — the allocator change was precautionary carryover from the closed dynamo-frontend variants and was never justified by evidence from this serving path. Restore the env that was demonstrably running. 中文：回滚 4370988。被中断的 direct-vllm 运行在原始环境 （NCCL_CUMEM_ENABLE=1、未设 expandable_segments）下已稳定运行智能体基准测试 24 分钟且无 OOM——该分配器改动只是从已关闭的 dynamo 前端变体沿袭的预防性 措施，并无本服务路径上的证据支持。恢复已被验证可运行的环境。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
0a20fdf to
0c5fe11
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30309340932 |
Keep prefix-cache blocks alive across agentic turn gaps, matching the GB200/GB300 AgentX recipes. 中文：新增 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768，使前缀缓存块在智能体 回合间隔内保持留存，与 GB200/GB300 AgentX 配方一致。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30309851256 |
中文：将智能体并发列表从单点 8 扩展为 1/8/16/32。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30312665286 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30313128047 |
Engine init hard-fails on Kimi-K3 with the GB200/GB300 AgentX value: "VLLM_PREFIX_CACHE_RETENTION_INTERVAL (32768) must be non-negative and a multiple of scheduler_block_size (3145728)" — the KDA hybrid architecture gives K3 a 3.1M-token scheduler block. Default retention served fine in the earlier runs, so drop the override. 中文：移除 VLLM_PREFIX_CACHE_RETENTION_INTERVAL——Kimi-K3 的 KDA 混合架构使 scheduler_block_size 达 3145728，GB200/GB300 AgentX 的 32768 取值导致引擎 初始化直接失败（必须为其整数倍）。此前运行证明默认留存策略可正常服务， 故不再覆盖。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30314948116 |
|
Closing per the author's decision in favor of Variant G (#2374), which is this exact configuration plus VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0 with a wider concurrency curve (1/2/4/8/16/32) — and is also fully green. Everything proven here (direct multinode vllm serve via srt-slurm PR #278 + multinode patch, the mamba_hybrid container patch, gpu-memory-utilization 0.90, the aiperf conv-aware-routing opt-out, launcher/model-path plumbing) is inherited by #2374. This PR's green run (all four concurrencies) remains the baseline reference for the retention comparison. 中文：按作者决定关闭本 PR，转向变体 G（#2374）——其为本配置加 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0，并发曲线更宽（1/2/4/8/16/32），且同样全绿。本 PR 验证的全部内容（直接多节点 vllm serve、mamba_hybrid 容器补丁、0.90 显存利用率、aiperf 路由关闭、启动器接线）均由 #2374 继承。本 PR 的全绿运行（四个并发点）保留为留存对比的基线参考。 |
…n't work with Pipeline yet, offloading & TP16 and DEP8PP2 to be done in followup PR) (#2391) * feat: add Kimi-K3 MXFP4 B200 aggregated TP8xPP2 Dynamo-vLLM agentic recipe Aggregated TP8 x PP2 across 2 B200 nodes (16 GPUs), plain TP (no expert parallelism) for the agentic-coding trace replay. Dedicated bring-up image vllm/vllm-openai:kimi-k3 with VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1, fastsafetensors load format, kimi_k3 tool-call/reasoning parsers. Model pre-staged at /lustre/fsw/models/Kimi-K3; launch_b200-dgxc.sh gains the kimik3/fp4 model-path mapping, the agentic recipe overlay, and the agentic cache default_mounts used by the GB200/GB300 agentic paths. 中文：新增 Kimi-K3 MXFP4 B200 聚合式 TP8xPP2 Dynamo-vLLM 智能体编码基准测试配方 （2 节点 / 16 GPU，纯 TP，不启用专家并行(EP)）。使用专用 bring-up 镜像 vllm/vllm-openai:kimi-k3（VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1、 fastsafetensors 加载格式、kimi_k3 工具调用/推理解析器）。模型已预置于 /lustre/fsw/models/Kimi-K3；启动器 launch_b200-dgxc.sh 增加 kimik3/fp4 模型路径映射、智能体配方覆盖及智能体缓存挂载。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: link PR #2355 in changelog entry and MODELS rows 中文：在更新日志条目与 MODELS 表格行中补充 PR #2355 链接。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: pin agentic srt-slurm to NVIDIA v1.0.36, dynamo 1.2.1, conc-8 smoke test The cquil11/srt-slurm-nv cam/sa-submission-q2-2026 fork rejected the recipe (benchmark.aiperf_server_metrics: Unknown field). Switch the b200-dgxc agentic clone to upstream NVIDIA/srt-slurm v1.0.36 (validated in #2302/#2341), drop the aiperf_server_metrics field, pin dynamo wheel/router to 1.2.1 (the combination validated with v1.0.36), and reduce the bring-up to a single conc-8 smoke test. 中文：cquil11/srt-slurm-nv 分支的 srtctl 校验拒绝了配方字段 benchmark.aiperf_server_metrics（Unknown field）。将 b200-dgxc 智能体路径改用 上游 NVIDIA/srt-slurm v1.0.36（已在 #2302/#2341 验证），移除该字段，dynamo wheel/router 固定为 1.2.1，并将 bring-up 缩减为单并发（conc 8）冒烟测试。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: drop OpenAI-frontend tool-choice flags from dynamo-vllm worker args The dynamo-vllm worker entrypoint rejected --enable-auto-tool-choice --tool-call-parser kimi_k3 (unrecognized arguments; different arg parser than vllm serve). Chat parsing happens at the dynamo frontend — same convention as the DSv4 GB300 agentic recipes. Keep --reasoning-parser kimi_k3 (accepted by the worker). Also drop the explicit max-model-len and let vLLM derive the native 1M window from the model config, mirroring the agentic recipe convention. 中文：dynamo-vllm worker 入口不接受 --enable-auto-tool-choice 与 --tool-call-parser kimi_k3（unrecognized arguments，与 vllm serve 的参数解析器 不同），聊天解析由 dynamo 前端处理，与 DSv4 GB300 智能体配方约定一致；保留 worker 可接受的 --reasoning-parser kimi_k3。同时移除显式 max-model-len， 由 vLLM 从模型配置推导原生 1M 上下文窗口。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: try dynamo wheel 1.2.0.dev20260426 for Kimi-K3 frontend tokenizer Third sweep attempt: the engine loaded and served (TP8xPP2 healthy in ~14 min), but dynamo 1.2.1's rust frontend tokenizer rejects Kimi-K3's tiktoken model_type 'kimi_k3' (supported: kimi, kimi_k2, kimi_k25, deepseek_v3), so the model never registered and all chat completions returned 404, aborting the AgentX warmup. Switch to the 1.2.0.dev20260426 wheel used by the DSv4 GB300/B200 Dynamo-vLLM recipes. Upstream published v1.4.0-kimi-k3-dev.1 (2026-07-27) as the day-zero K3 build if this wheel also lacks support. 中文：第三次扫描中引擎已成功加载并提供服务（TP8xPP2 约 14 分钟就绪），但 dynamo 1.2.1 的 rust 前端分词器不支持 Kimi-K3 的 tiktoken model_type 'kimi_k3'，模型未能注册，所有请求返回 404，AgentX 预热中止。改用 DSv4 GB300/B200 Dynamo-vLLM 配方所用的 1.2.0.dev20260426 wheel；如仍不支持， 上游已于 2026-07-27 发布 day-zero 构建 v1.4.0-kimi-k3-dev.1。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: pin dynamo to day-zero Kimi-K3 commit, restore kimi_k3 parser flags Pin dynamo to ba83080ecd31c1ce918559e576d3c5bc9e092ff1 ("feat: Added support for Kimi-K3", tag v1.4.0-kimi-k3-dev.1) via srt-slurm's hash-cached source install: it adds the kimi_k3 tiktoken tokenizer to the rust frontend (dynamo <=1.2.1 404s every request because the model never registers) and accepts the kimi_k3 tool-call/reasoning parser worker args, so restore --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3. 中文：将 dynamo 固定到 day-zero Kimi-K3 提交 ba83080（"feat: Added support for Kimi-K3"，标签 v1.4.0-kimi-k3-dev.1），通过 srt-slurm 的哈希缓存源码 安装：该提交为 rust 前端新增 kimi_k3 tiktoken 分词器（dynamo <=1.2.1 因模型 无法注册而全部返回 404），worker 亦支持 kimi_k3 解析器参数，故恢复 --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: use dynamo namespaced --dyn-* kimi_k3 parser args on the worker Replace the vLLM OpenAI-frontend spellings (--enable-auto-tool-choice / --tool-call-parser) with dynamo's namespaced worker args: --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3. 中文：将 vLLM OpenAI 前端风格参数（--enable-auto-tool-choice / --tool-call-parser）替换为 dynamo 命名空间的 worker 参数： --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable aiperf conv-aware routing (session_control 400-rejected) Fifth sweep attempt: the day-zero dynamo registered the kimi_k3 tiktoken tokenizer and the engine served, but all warmup requests got 400 — aiperf's conv-aware routing emits nvext.session_control, a removed POC field this dynamo build rejects (schema moved to router/routing_constraints/ agent_hints). Opt out via AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0, matching the GB300 aggregate AgentX recipes; a single aggregate worker has no P/D routing to bind anyway. 中文：第五次扫描中 day-zero dynamo 已成功注册 kimi_k3 tiktoken 分词器并正常 服务，但全部预热请求返回 400——aiperf 的会话感知路由会发送 nvext.session_control（已被移除的 POC 字段，schema 已迁移至 router/routing_constraints/agent_hints）。通过 AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0 关闭，与 GB300 聚合式 AgentX 配方 一致；单聚合 worker 本无需 P/D 路由绑定。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: patch kimi-k3 image mamba_hybrid index_fill_ dtype via setup_script Sixth sweep attempt (both A and C variants): warmup requests 500 then the model 503s — the image's first decode step crashes in the KDA hybrid-state postprocess (mamba_hybrid.py postprocess_state, IndexError: index_fill_(): Expected dtype int64 for index; torch requires an int64 index but the runner passes the int32 idx_mapping). Ship an in-container patch through srt-slurm's setup_script hook (same pattern as configs/patches/ vllm_numa_bind_hash_fix.py): coerce the index with .long(), idempotent, refuses to run if the image layout changed. 中文：第六次扫描（A、C 两个变体一致）：预热请求先 500、随后模型 503——镜像 首个解码步在 KDA 混合状态后处理中崩溃（mamba_hybrid.py postprocess_state， IndexError: index_fill_() 需要 int64 索引，但 runner 传入 int32 idx_mapping）。 通过 srt-slurm 的 setup_script 钩子在容器内打补丁：将索引用 .long() 转换， 幂等，且镜像布局变化时拒绝执行。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: agentic experiment D — direct vllm serve via srt-slurm PR #278 Serve Kimi-K3 directly with vllm serve (srt-slurm PR #278 frontend.type: vllm, branch kylliang/direct-aggregate-vllm): no dynamo frontend/worker/ router, which removes the dynamo tokenizer/schema gaps entirely, and the OpenAI-frontend flags --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 become legitimate. PR #278 validates single-node only, so ship patches/srt-slurm-pr278-direct-vllm-multinode.patch extending it to vLLM-native multi-node serve (--master-addr/--nnodes/--node-rank, headless non-leader ranks) for the 2-node TP8xPP2 topology. Keeps the mamba_hybrid index-dtype container patch (engine bug is frontend-agnostic). 中文：智能体实验变体 D——通过 srt-slurm PR #278（frontend.type: vllm）直接以 vllm serve 提供服务：去除 dynamo 前端/worker/router，从根本上规避 dynamo 的 分词器与 schema 兼容问题，OpenAI 前端参数 --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 因此可用。PR #278 仅 支持单节点，故新增补丁将其扩展为 vLLM 原生多节点 serve（--master-addr/ --nnodes/--node-rank，非主节点 headless），以运行 2 节点 TP8xPP2 拓扑。保留 mamba_hybrid 索引类型容器补丁（引擎缺陷与前端无关）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: point changelog and MODELS rows at experiment PR #2359 中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2359。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: drop gpu-memory-utilization to 0.90 (flashinfer MoE workspace OOM) Same engine-level OOM as the dynamo-frontend variants: the flashinfer trtllm MXFP4 MoE kernel allocates a ~1.6 GiB runtime workspace outside vLLM's memory pool on the first forward; at 0.95 a 178 GiB B200 has only ~1.35 GiB free. 中文：与 dynamo 前端变体相同的引擎级 OOM：flashinfer trtllm MXFP4 MoE 内核在 首个前向时于 vLLM 显存池外分配约 1.6 GiB 工作区，0.95 下仅剩约 1.35 GiB。 改为 0.90。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: expandable_segments allocator, drop NCCL_CUMEM_ENABLE (prefill OOM) Inherited from the closed dynamo-frontend variants (#2355/#2358): at gpu-mem-util 0.90 the first long-context MLA prefill OOM'd on a 2.92 GiB transient while 3.39 GiB sat reserved-but-unallocated (fragmentation). Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and drop NCCL_CUMEM_ENABLE. 中文：继承自已关闭的 dynamo 前端变体（#2355/#2358）：0.90 显存利用率下首个 长上下文 MLA 预填充因 2.92 GiB 瞬时分配 OOM，而 3.39 GiB 处于已保留未分配 状态（碎片化）。设置 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True 并 移除 NCCL_CUMEM_ENABLE。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Revert "fix: expandable_segments allocator, drop NCCL_CUMEM_ENABLE (prefill OOM)" This reverts commit 4370988. The superseded direct-vllm run served the agentic benchmark for 24 minutes on the original env (NCCL_CUMEM_ENABLE=1, no expandable_segments) without any OOM — the allocator change was precautionary carryover from the closed dynamo-frontend variants and was never justified by evidence from this serving path. Restore the env that was demonstrably running. 中文：回滚 4370988。被中断的 direct-vllm 运行在原始环境 （NCCL_CUMEM_ENABLE=1、未设 expandable_segments）下已稳定运行智能体基准测试 24 分钟且无 OOM——该分配器改动只是从已关闭的 dynamo 前端变体沿袭的预防性 措施，并无本服务路径上的证据支持。恢复已被验证可运行的环境。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768 Keep prefix-cache blocks alive across agentic turn gaps, matching the GB200/GB300 AgentX recipes. 中文：新增 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768，使前缀缓存块在智能体 回合间隔内保持留存，与 GB200/GB300 AgentX 配方一致。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: widen agentic conc list to 1/8/16/32 中文：将智能体并发列表从单点 8 扩展为 1/8/16/32。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: remove VLLM_PREFIX_CACHE_RETENTION_INTERVAL (K3 scheduler block) Engine init hard-fails on Kimi-K3 with the GB200/GB300 AgentX value: "VLLM_PREFIX_CACHE_RETENTION_INTERVAL (32768) must be non-negative and a multiple of scheduler_block_size (3145728)" — the KDA hybrid architecture gives K3 a 3.1M-token scheduler block. Default retention served fine in the earlier runs, so drop the override. 中文：移除 VLLM_PREFIX_CACHE_RETENTION_INTERVAL——Kimi-K3 的 KDA 混合架构使 scheduler_block_size 达 3145728，GB200/GB300 AgentX 的 32768 取值导致引擎 初始化直接失败（必须为其整数倍）。此前运行证明默认留存策略可正常服务， 故不再覆盖。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: agentic experiment G — variant D + prefix-cache retention 0 Identical GPU-resident direct-vllm config to variant D (#2359) plus VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0. Any positive value must be a multiple of Kimi-K3's KDA-hybrid scheduler_block_size (3145728), so 0 is the only setting below one 3.1M-token scheduler block. 中文：智能体实验变体 G——与变体 D（#2359）完全相同的 GPU 常驻直接 vllm serve 配置，另加 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0（任何正值都必须是 Kimi-K3 KDA 混合架构 scheduler_block_size 3145728 的整数倍，0 是唯一低于 一个 3.1M token 调度块的取值）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: point changelog and MODELS rows at experiment PR #2374 中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2374。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: variant G conc curve 1/2/4/8/16/32 (add 2 and 4) 中文：变体 G 并发曲线扩展为 1/2/4/8/16/32（新增 2 与 4）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * probe: variant K — drop the kimi-k3 in-container patch script Variant G (#2374, fully green) minus the kimi-k3-container-deps.sh in-container patch (setup_script reference, script file, and launcher copy), to verify whether the mamba_hybrid index_fill_ dtype patch is still required by the current vllm/vllm-openai:kimi-k3 image. Expected to fail at the first decode step if it is; results will be commented on the PR. 中文：探针变体 K——在全绿的变体 G（#2374）基础上移除 kimi-k3 容器内补丁 脚本（setup_script 引用、脚本文件与启动器复制），验证当前 vllm/vllm-openai:kimi-k3 镜像是否仍需 mamba_hybrid index_fill_ 类型补丁。 如仍需要，预计在首个解码步失败；结果将评论在 PR 中。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: point changelog and MODELS rows at experiment PR #2391 中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2391。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: clone fork branch with multinode support, drop git-apply patch The srt-slurm PR #278 multi-node extension now lives as commits on functionstackx/srt-slurm-nv branch klaud/direct-vllm-multinode (head df5baa93), so the launcher clones that branch directly instead of applying srt-slurm-pr278-direct-vllm-multinode.patch onto the upstream kylliang/direct-aggregate-vllm branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 participant
Summary
Agentic experiment (Variant D) for the Kimi-K3 MXFP4 B200 aggregated TP8×PP2 bring-up — direct
vllm serve, no Dynamo.Serves Kimi-K3 directly with
vllm servevia srt-slurm PR NVIDIA/srt-slurm#278 (frontend.type: vllm), cloned from the fork branch functionstackx/srt-slurm-nvklaud/direct-vllm-multinodewhich carries PR #278 plus the multi-node extension:dynamo: install: false) — removes the dynamo frontend tokenizer gap (kimi_k3tiktoken unsupported ≤1.2.1 → 404s) and the dynamo request-schema gaps (nvext.session_control400s) by construction.--enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3go straight tovllm serve.--master-addr/--nnodes/--node-rankcoordination, non-leader ranks--headless— mirroring upstream vLLM's supported multi-node PP/TP serve topology. (Previously shipped aspatches/srt-slurm-pr278-direct-vllm-multinode.patchapplied by the launcher; now baked into the fork branch, no git-apply step.)mamba_hybridindex-dtype container patch from the base PR (the engine bug is frontend-agnostic) and the conc-8 smoke test.Related experiments
This is now the sole surviving experiment — the Dynamo-frontend variants are closed in favor of this PR:
--dyn-*parser args — [Klaud Cold][agentic experiment][Variant A] feat: add Kimi-K3 MXFP4 B200 aggregated TP8xPP2 Dynamo-vLLM agentic bring-up / 新增 Kimi-K3 MXFP4 B200 聚合式 TP8xPP2 Dynamo-vLLM 智能体编码基准测试（bring-up） #2355 (closed in favor of this PR)--tool-call-parser)vllm serve(srt-slurm PR fix: label validation approve gate #278 + multinode fork branch) + native vLLM parser flagsAll fixes proven on the closed variants are inherited here: the
mamba_hybridindex-dtype container patch, the aiperf conv-aware-routing opt-out,gpu-memory-utilization: 0.90(flashinfer MoE workspace OOM at 0.95), andPYTORCH_CUDA_ALLOC_CONF=expandable_segments:True+NCCL_CUMEM_ENABLEdropped (MLA long-context prefill fragmentation OOM).中文说明
Kimi-K3 MXFP4 B200 聚合式 TP8×PP2 bring-up 的智能体实验（变体 D）——直接
vllm serve，不经 Dynamo。通过 srt-slurm PR NVIDIA/srt-slurm#278（
frontend.type: vllm）直接以vllm serve提供服务：完全去除 Dynamo 层（dynamo: install: false），从根本上规避 dynamo 前端分词器缺口（kimi_k3 tiktoken 不受支持导致 404）与请求 schema 缺口（nvext.session_control 导致 400）；OpenAI 前端参数--enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3因此可直接使用。PR #278 仅支持单节点聚合，而 Kimi-K3（约 1.4TB MXFP4 权重）无法装入单个 8×B200 节点，故 launcher 直接克隆已包含多节点扩展提交的 fork 分支 functionstackx/srt-slurm-nvklaud/direct-vllm-multinode（此前以补丁patches/srt-slurm-pr278-direct-vllm-multinode.patch形式由 launcher 应用，现已并入 fork 分支，无需 git-apply），将其扩展为 vLLM 原生多节点 serve（主节点持有 OpenAI 端口，--master-addr/--nnodes/--node-rank协调，非主节点--headless）。保留基础 PR 的 mamba_hybrid 索引类型容器补丁（引擎缺陷与前端无关）及并发 8 冒烟测试。相关实验
本 PR 现为唯一存续实验——Dynamo 前端变体均已关闭并转向本 PR：变体 A #2355（已关闭）；变体 B #2357（已关闭：dynamo worker 拒绝 --tool-call-parser）；变体 C #2358（已关闭）。已关闭变体上验证的全部修复均已继承：mamba_hybrid 索引类型容器补丁、aiperf 会话感知路由关闭、显存利用率 0.90（0.95 下 flashinfer MoE 工作区 OOM）、PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True 及移除 NCCL_CUMEM_ENABLE（MLA 长上下文预填充碎片化 OOM）。
🤖 Generated with Claude Code