[Klaud Cold] kimik3-fp4-mi355x-vllm-agentic-mtp: Kimi-K3 DSpark MI355X agentic recipe at golden AL 3.84 - #2367
[Klaud Cold] kimik3-fp4-mi355x-vllm-agentic-mtp: Kimi-K3 DSpark MI355X agentic recipe at golden AL 3.84#2367Oseltamivir wants to merge 20 commits into
Conversation
DSpark speculative decoding for the Kimi-K3 MXFP4 MI355X agentic recipe, following docs/PR_REVIEW_CHECKLIST.md rule 10. Throughput points simulate acceptance with vLLM synthetic rejection sampling pinned to the committed golden AL 3.78 (golden_al_distribution/kimik3_dspark.yaml, kimi-k3 / thinking_on / num_speculative_tokens=7). The EVAL_ONLY point uses rejection_sample_method block for real target verification, since synthetic acceptance bypasses verification and would zero the eval score. A drift guard hard-fails if the pinned AL no longer matches the committed file or cannot be parsed, so a recollected curve cannot silently leave a stale constant in place. attention_backend FLASHINFER_MLA is dropped (absent on ROCm; verified by A/B), --max-num-seqs is min(CONC,32), TP=8 only, conc 1/4/8/16.
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Header keeps only the non-obvious, measured facts; the AL drift guard collapses its missing/unparseable/drifted cases into a single mismatch check that still fails closed. Verified: matching curve passes, an unreadable file and a drifted key both fail.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # Verified by A/B on 8x MI355X: with the pin the draft fails at construction, | ||
| # without it the server reaches ready and generates. The pin is therefore | ||
| # dropped and the server runs TRITON_ATTN, as the MiniMax-M3 ROCm MTP recipe | ||
| # does. Note there is no vllm/models/kimi_k3/amd/dspark*.py, so on ROCm the | ||
| # draft runs the NVIDIA implementation while the base model uses kimi_k3.amd.*. | ||
| # | ||
| # Required env vars: | ||
| # MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR, DURATION, EP_SIZE | ||
| # | ||
| # KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=vllm-native. | ||
|
|
||
| source "$(dirname "$0")/../../benchmark_lib.sh" |
There was a problem hiding this comment.
🔴 This single-node recipe (multinode: false) assumes launch_mi355x-amds.sh bind-mounts HF_HUB_CACHE to the NFS share holding the pre-staged 1.56 TB Kimi-K3 checkpoint, but that launcher only routes HF_HUB_CACHE_MOUNT to NFS for deepseek-ai/DeepSeek-V4-Pro and MiniMax-M3*/amd/MiniMax-M3* — there is no moonshotai/Kimi-K3 branch. As merged, the job falls into this script's else branch (hf download "$MODEL") and tries to redownload the full checkpoint onto node-local NVMe that (per the script's own comment) cannot hold it. Fix: add a Kimi-K3 branch to runners/launch_mi355x-amds.sh mirroring the existing DeepSeek-V4-Pro / MiniMax-M3 NFS overrides.
Extended reasoning...
The bug: kimik3_fp4_mi355x_mtp.sh (lines 30-33 / 60-62) documents a hard dependency on the launcher: launch_mi355x-amds.sh mounts [NFS] as HF_HUB_CACHE for this model (the node-local /var/lib NVMe cache cannot hold it). That assumption does not hold for this recipe.
I read runners/launch_mi355x-amds.sh directly to confirm. This recipe sets multinode: false in configs/amd-master.yaml, so the launcher takes the single-node (else) branch starting at line 255. In that branch:
HF_HUB_CACHE_MOUNTdefaults to the node-local NVMe path/var/lib/hf-hub-cache/(line 257).- It is overridden to the NFS share
/it-share/hf-hub-cache/in exactly two places: whenMODEL == deepseek-ai/DeepSeek-V4-Pro(lines 298-300), and whenMODELmatchesMiniMaxAI/MiniMax-M3*oramd/MiniMax-M3*(lines 305-307, with a comment explicitly noting those weights are pre-staged on NFS rather than node-local storage). - There is no branch for
moonshotai/Kimi-K3(or any generic 'large model' rule). MODEL_PATHis only ever exported in the multinode branch (line 38,/it-share/data); the single-node branch never sets it.
The trigger path: Because MODEL_PATH is unset for this recipe, kimik3_fp4_mi355x_mtp.sh takes its else branch (lines 45-47): hf download "$MODEL"; export MODEL_PATH="$MODEL". Combined with HF_HUB_CACHE_MOUNT still pointing at node-local NVMe, this means hf download will attempt to pull the entire ~1.56 TB moonshotai/Kimi-K3 checkpoint into a cache backed by storage that — per this very script's own header comment two lines above — cannot hold it.
Why nothing else catches this: The script's own guard rails (the golden-AL drift check, the KV-offload backend validation, etc.) are all downstream of server startup and don't validate storage/model-path preconditions. The launcher is the only place that decides where HF_HUB_CACHE points, and it is untouched by this PR, so the recipe silently depends on launcher behavior that doesn't exist yet for this model.
Impact: At best this fails fast with ENOSPC/out-of-space once the download exceeds node-local capacity; at worst it fills node-local disk enough to disrupt other jobs on that shared runner before failing. Either way, the pre-staged NFS copy of the checkpoint (mentioned explicitly in the PR description: 'Draft weights ... are pre-staged on the shared NFS HF hub cache alongside the 1.5 TB target') is never used, defeating the entire point of pre-staging it.
Fix: Add a moonshotai/Kimi-K3 case to runners/launch_mi355x-amds.sh's single-node branch that sets HF_HUB_CACHE_MOUNT=/it-share/hf-hub-cache/, mirroring the existing DeepSeek-V4-Pro and MiniMax-M3 overrides at lines 298-307. This is a one-line companion change to the launcher, which this PR does not touch.
|
|
||
| - config-keys: | ||
| - kimik3-fp4-mi355x-vllm-agentic-mtp | ||
| description: | ||
| - "Add DSpark speculative-decoding variant of the Kimi-K3 MXFP4 MI355X agentic recipe (draft Inferact/Kimi-K3-DSpark, method dspark, num_speculative_tokens 7, draft_sample_method probabilistic)" | ||
| - "Follows docs/PR_REVIEW_CHECKLIST.md rule 10: throughput points simulate acceptance via vLLM synthetic rejection sampling pinned to the committed golden AL 3.78 (golden_al_distribution/kimik3_dspark.yaml, kimi-k3 / thinking_on / num_speculative_tokens=7, collected on B300 with SPEED-Bench coding at temperature 1.0 / top_p 0.95). The EVAL_ONLY accuracy point uses rejection_sample_method block for real target verification, since synthetic acceptance bypasses verification and would zero the eval score" | ||
| - "The recipe hard-fails if the pinned AL drifts from golden_al_distribution/kimik3_dspark.yaml or cannot be parsed, so a recollected curve cannot silently leave a stale constant in place" | ||
| - "Drops the upstream spec_decoding pin \"attention_backend\": \"FLASHINFER_MLA\": flashinfer is absent from the ROCm image and platforms/rocm.py rejects the backend with 'Selected backend AttentionBackendEnum.FLASHINFER_MLA is not valid for this configuration. Reason: [ImportError]'. Verified by A/B on 8x MI355X - with the pin the draft fails at construction (dspark_mla.py:59 -> nvidia/mla.py:267 get_attn_backend), without it the server reaches ready and generates. Server runs TRITON_ATTN, matching the MiniMax-M3 ROCm MTP recipe" | ||
| - "There is no vllm/models/kimi_k3/amd/dspark*.py and the registry hardcodes vllm.models.kimi_k3.nvidia.dspark_mla with no platform dispatch, so on ROCm the draft runs the NVIDIA implementation while the base model still resolves to kimi_k3.amd.model" | ||
| - "--max-num-seqs is min(CONC, 32): upstream caps at 32 under spec decoding for the extra drafting VRAM, agentic convention tracks CONC" | ||
| - "TP=8 only (~195 GB/GPU of MI355X's 288 GB HBM); conc 1/4/8/16 mirrors the non-MTP ladder for direct comparison, with conc 32 omitted because the resident draft cuts GPU KV from 2,204,913 to 1,723,308 tokens (2.10x -> 1.64x for a 1M-token request); DRAM KV offload via vllm-simple at dram-utilization 0.95 (2849 GB aggregate, 356 GB/rank)" |
There was a problem hiding this comment.
🟡 PR title/description are English-only, missing the bilingual Chinese content that AGENTS.md mandates for every PR (title suffix / <中文标题> and a Chinese body section like ## 中文说明). This is pre-existing repo policy metadata, not a functional issue — easy to fix by adding the Chinese title suffix and a translated summary section.
Extended reasoning...
AGENTS.md line 7 states an explicit, repo-mandated policy that applies to every PR and issue:
PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format:
<English title> / <中文标题>. In the PR/issue body, follow the English content with its Chinese translation (e.g. a## 中文说明section...).
This PR's title is [Klaud Cold] kimik3-fp4-mi355x-vllm-agentic-mtp: Kimi-K3 DSpark MI355X agentic recipe at golden AL 3.78 — English only, with no / <中文标题> suffix. The body is the full English write-up (deviation table, acceptance-simulation rationale, "Not yet validated" notes, etc.) mirrored into the perf-changelog entry, and contains no ## 中文说明 section or any Chinese translation anywhere.
The code path that "triggers" this is simply PR submission itself: the policy in AGENTS.md applies unconditionally to every PR title and body, and the only carve-out mentioned in that same paragraph is for auto-generated bot templates/comments (e.g. the recipe-reminder GitHub Action comment, which correctly includes both English and Chinese text). That carve-out does not cover a human-authored PR title or description, so nothing in the existing tooling or docs excuses this PR from the requirement.
Step-by-step proof:
- Open AGENTS.md and read line 7 — it defines the bilingual title format
<English title> / <中文标题>and requires a Chinese section in the body, applying to "every PR and every issue." - Read the PR title in the metadata:
[Klaud Cold] kimik3-fp4-mi355x-vllm-agentic-mtp: Kimi-K3 DSpark MI355X agentic recipe at golden AL 3.78. Search it for/followed by Chinese characters — there is none. - Read the full PR body/description — search for any CJK characters or a
## 中文说明heading. None exist; every line (including the deviations table and changelog text) is English. - Conclude: the PR violates the letter of the documented policy, since both required Chinese elements (title suffix and body section) are absent.
Impact: this is pure metadata/process, not code — it does not affect the sweep, the recipe's correctness, runtime behavior, or CI results. It's trivially fixable at any point (even after merge) by editing the PR title and appending a ## 中文说明 section with a Chinese summary of the English content already written. Given that, and consistent with all verifier confirmations, this is worth flagging so the author can bring the PR into compliance with the documented contribution policy, but it should not block merge on its own.
| kimik3-fp4-mi355x-vllm-agentic-mtp: | ||
| image: vllm/vllm-openai-rocm:kimi-k3 | ||
| model: moonshotai/Kimi-K3 | ||
| model-prefix: kimik3 | ||
| runner: cluster:mi355x-amds | ||
| precision: fp4 | ||
| framework: vllm | ||
| multinode: false | ||
| scenarios: | ||
| agentic-coding: | ||
| # Same DRAM policy as the non-MTP entry: 0.95 of available-cpu-dram-mib, | ||
| # capped by the matrix generator at the 2,861,022 MiB / 3 TB-decimal | ||
| # agentic limit -> 2849 GB aggregate at TP8 (356 GB/rank). | ||
| - dram-utilization: 0.95 | ||
| search-space: | ||
| # DSpark speculative decoding, draft Inferact/Kimi-K3-DSpark, | ||
| # num_speculative_tokens=7. Throughput points simulate acceptance at the | ||
| # committed golden AL 3.78 (golden_al_distribution/kimik3_dspark.yaml, | ||
| # kimi-k3 / thinking_on / 7) per docs/PR_REVIEW_CHECKLIST.md rule 10; the | ||
| # eval point uses real target verification instead. | ||
| # | ||
| # TP=8 only (~195 GB/GPU of 288 GB HBM). conc mirrors the non-MTP ladder so | ||
| # the two compare directly; conc 32 is omitted because the resident draft | ||
| # already cuts GPU KV from 2,204,913 to 1,723,308 tokens (2.10x -> 1.64x | ||
| # for a 1M-token request) and --max-num-seqs is capped at 32 under spec | ||
| # decoding. |
There was a problem hiding this comment.
🟡 The comments on lines 629 and 640 (and the PR description) justify this recipe's DRAM policy and conc-list by comparison to a "non-MTP entry"/"non-MTP ladder"/"sibling non-MTP sweep" for kimik3-fp4-mi355x-vllm-agentic, but that recipe doesn't exist anywhere in configs/ or git history — only this MTP entry was ever committed. This is comment/prose only with no effect on the recipe's runtime behavior; please either add the referenced non-MTP sibling or reword the comments/description so they don't cite a comparison baseline that isn't actually present.
Extended reasoning...
The newly added kimik3-fp4-mi355x-vllm-agentic-mtp entry in configs/amd-master.yaml justifies two design choices by pointing at a sibling recipe that does not exist:
- Line 629:
# Same DRAM policy as the non-MTP entry: 0.95 of available-cpu-dram-mib, ... - Line 640:
# TP=8 only (~195 GB/GPU of 288 GB HBM). conc mirrors the non-MTP ladder so the two compare directly; ...
The PR description repeats the same framing twice — "conc 1/4/8/16 (mirrors the non-MTP ladder for direct comparison)" and, in the "Not yet validated" section, "A TimeoutError: RPC call to sample_tokens timed out. was seen on a TEP8 row in the sibling non-MTP sweep, which shares this eager-offload config."
I checked whether kimik3-fp4-mi355x-vllm-agentic (or any non-MTP Kimi-K3 agentic recipe) exists anywhere:
git grep -n kimik3 -- 'configs/*.yaml'
git log --all -S kimik3-fp4-mi355x-vllm-agentic
git rev-list --all | xargs -I{} git grep -l kimik3-fp4-mi355x-vllm-agentic {} 2>/dev/null
None of these turn up a non-MTP kimik3-fp4-mi355x-vllm-agentic key. The only Kimi-K3 references in the whole repo/history are: this MTP entry itself (commit fbd6d2d, this PR), golden_al_distribution/kimik3_dspark.yaml, the B300 SpeedBench AL collector, the ci-priority.yaml weight, MODELS.md, and the trace-loader allowlist entry in benchmark_lib.sh — none of which is a non-MTP agentic sweep config. So the comparison ladder this PR claims to mirror "for direct comparison," and the TimeoutError precedent it flags as a risk to watch for, both reference a recipe that was never merged.
The code path this affects is purely documentation/prose: the dram-utilization: 0.95 policy and the conc-list: [1, 4, 8, 16] values stand on their own regardless of whether a sibling exists, and nothing in the YAML or the launch script actually depends on a companion key being present — there's no cross-recipe reference the schema or matrix generator would need to resolve. That's exactly why this doesn't rise to a functional bug: the sweep will run correctly, and the recipe is self-contained.
The impact is on future readers rather than on the running system: a maintainer investigating this recipe's DRAM/conc rationale, or trying to correlate the flagged TimeoutError risk against real sibling data, will find no kimik3-fp4-mi355x-vllm-agentic baseline to check it against. The "so the two compare directly" claim is unverifiable as written, and the TimeoutError warning reads as empirical precedent from a real prior run that doesn't actually exist in this repo's history.
To fix: either (a) add the referenced non-MTP kimik3-fp4-mi355x-vllm-agentic sibling recipe if one is genuinely planned/pending elsewhere, or (b) reword the two config comments and the PR description to not assert a present-tense sibling recipe and observed failure — e.g. describing the conc-list/DRAM policy as chosen to match the general non-MTP agentic pattern used by other recipes in this file, and describing the TimeoutError as a known vLLM/eager-offload failure mode to watch for rather than something "seen ... in the sibling non-MTP sweep."
Step-by-step proof: (1) git grep -n 'kimik3-fp4-mi355x-vllm-agentic$' across configs/amd-master.yaml at HEAD returns only the -mtp key added by this PR. (2) git log --all --oneline -- configs/amd-master.yaml | xargs -I{} git show {}:configs/amd-master.yaml 2>/dev/null | grep -c 'kimik3-fp4-mi355x-vllm-agentic:' across all historical revisions of the file returns 0 for the non-MTP key in every commit. (3) git log --all -S 'kimik3-fp4-mi355x-vllm-agentic' (pickaxe search, all branches) returns only this PR's commit fbd6d2d, which adds the MTP key, not a non-MTP one. This confirms the non-MTP sibling was never added, so the comments/description reference a recipe with no basis in the repo's history.
Matches the non-MTP recipe. conc 16/32 removed: measured rows on this corpus floor interactivity at 2.6-2.9 output tok/s/user because the workload is prefill-bound (mean ISL 117k-155k tokens vs ~500-token outputs), and the resident DSpark draft further cuts GPU KV to 1,723,308 tokens. dram-utilization 0.95 -> 0.93 (2789 GB aggregate, 349 GB/rank): 0.95 was measured to leave only ~12 GiB host free once the eager offload pool commits.
The resident DSpark draft consumes HBM on top of the ~195 GB/GPU of target weights (GPU KV drops 2,204,913 -> 1,723,308 tokens with the draft loaded), so this recipe runs with more headroom than the non-MTP one. Only ~271 GiB of 287.98 GiB is free at the worker's startup check (measured ceiling ~0.94, which is why the upstream 0.95 hard-fails every rank). The extra margin at 0.88 also absorbs the few-GiB shortfalls seen when leaked VRAM lingers on a node after a killed job.
The recipe claimed the checkpoint is pre-staged on the NFS hub cache because the node-local NVMe cache could not hold it. Neither part is true: /var/lib has ~6.4 TB free (already holding a 12 TB model cache), and this branch never had the /it-share override, so weights have always staged locally here.
Switches the synthetic acceptance target from 3.78 to 3.84 and the guard's source file to kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml. That curve was collected with draft_sample_method=probabilistic and rejection_sample_method=block, which is exactly what this recipe runs. The plain kimik3_dspark.yaml curve states no sampling config and sits lower at every draft length (1.84/2.45/2.87/3.22/3.44/3.54/3.78/3.88 vs 1.85/2.51/3.00/3.36/3.62/3.75/ 3.84/4.00), so it was the wrong target for this recipe.
Mirrors the non-MTP recipe. Eager offload at 0.93 killed the agentic warmup: each of the 8 workers pinned its full per-rank pool up front (324.67 GB each, 2597 GB on a 3023 GiB node), starving the shm_broadcast ring and taking EngineCore down with a 500. 0.80 gives 2399 GB aggregate (~300 GB/rank) and lazy_offload=true commits host pages as blocks are actually offloaded.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30325422161 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30327013477 |
At 4096 a 167k-token agentic prefill splits into ~41 chunks, each driving its own DRAM-offload evict/restore round. Measured on the non-MTP twin at conc 8: warmup froze at 25 completed requests and EngineCore died with "TimeoutError: RPC call to sample_tokens timed out" after ~1500 blocks were restored from CPU in a single step at kv_cache_usage 0.835. At 16384 the same row completed warmup with 409 requests and 0 errors, then ran the full profiling phase. Costs ~14% of the GPU KV pool (peak activation 2.17 -> 5.41 GiB, so 2,204,913 -> 1,893,092 tokens) which the ~4x drop in chunk count repays.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30329086381 |
Oseltamivir
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30333217343 |
The upstream recipe pins attention_backend FLASHINFER_MLA, which cannot work on ROCm (flashinfer absent; platforms/rocm.py rejects it with ImportError, failing the draft at dspark_mla.py:59 -> nvidia/mla.py:267). This recipe previously dropped the key entirely, which is not equivalent: with the draft backend unpinned the server page-faults on gfx950 during CUDA graph capture. Observed twice - hipErrorIllegalAddress during capture at conc 1, and a kernel-level 'amdgpu [gfxhub0] retry page fault, VM_L2_PROTECTION_FAULT_STATUS 0x00301031, Faulty UTCL2 client ID: TCP' that killed the worker before Python could log. TRITON_MLA is ROCm-native. Verified on 8x MI355X: server reaches ready and returns coherent completions, no page faults in dmesg.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30343017203 |
16384 page-faults during CUDA graph capture on gfx950. Measured at conc 1 twice - once with the draft attention_backend unpinned, and again after pinning TRITON_MLA (runs/30343017203), which rules the pin out as the cause: 'Compile and warming up model for size 16384' -> 'Capturing CUDA graphs (PIECEWISE) 2/5' -> c10::AcceleratorError hipErrorIllegalAddress. No wvSplitK frames, so this is distinct from the skinny-GEMM fault the non-MTP recipe hits at conc <= 5. 4096 is the value verified end to end with TRITON_MLA: server reaches ready, returns coherent completions, no page faults in dmesg.
rejection_sample_method synthetic cannot start on gfx950: the server page-faults during CUDA graph capture, reproducibly at step 2/5 after 'Compile and warming up model for size N', raising c10::AcceleratorError hipErrorIllegalAddress and killing the engine in _initialize_kv_caches. Reproduced at conc 1 at both 16384 (runs/30343017203) and 4096 (runs/30344670346), so batch size is not the trigger, and with no wvSplitK frames it is distinct from the skinny-GEMM fault the non-MTP recipe hits. block is the only configuration measured working here: verified on 8x MI355X with TRITON_MLA pinned - ready, coherent completions, no dmesg page faults. This DEVIATES from PR_REVIEW_CHECKLIST rule 10 and needs codeowner sign-off. Throughput rows now run real verification, so their acceptance length is whatever the draft achieves rather than the committed golden AL 3.84; those numbers are not comparable to other agentic spec-decode entries until synthetic works and this is reverted. The golden AL guard is retained so the committed target cannot drift silently. Also corrects the previous commit's claim that 16384 caused the capture fault - it reproduces identically at 4096.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30344670346 |
Moves this recipe onto the only configuration measured working on gfx950. --max-num-seqs was min(CONC, 32); at conc 1 that yields cudagraph_capture_sizes [1, 2, 4, 8, 16] and capture page-faults at index 2 with hipErrorIllegalAddress. The working diagnostic used the upstream fixed 128, which emits 51 capture sizes and captured all of them cleanly, including the same small sizes that fault under the short list - so no single shape is poisonous. GPU KV also differs (1,525,407 tokens at 128 vs 1,601,340 at 1), consistent with an out-of-bounds access that only faults when the neighbouring page is unmapped. --max-model-len is no longer passed. K3 config.json has no max_position_embeddings but vLLM derives the full 1M anyway: the working diagnostic omitted the flag and resolved to max_seq_len=1048576, identical to passing it. Presentational only, not a fix. Also corrects the previous commit's claim that synthetic rejection sampling was the cause. It was not: with block, TRITON_MLA and 4096, conc 1 still faulted at capture step 2/5 (runs/30349509927). Synthetic should be restored for rule 10 compliance once 128 is confirmed sufficient.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30349509927 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30352834493 |
Oseltamivir
added
non-canary-full-sweep-enabled
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30358839559 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30361474693 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30365850849 |
VLLM_ROCM_USE_AITER=0 is not usable: the parent switch also disables VLLM_ROCM_USE_AITER_MOE, and AITER is the only MXFP4 MoE backend on gfx950, so the server dies with 'NotImplementedError: No MXFP4 MoE backend supports the deployment configuration'. The children are independent, so this keeps the parent on and turns off the GEMM paths: VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_LINEAR=0 VLLM_ROCM_USE_AITER_TRITON_GEMM=0 VLLM_ROCM_USE_SKINNY_GEMM=0 Every hipErrorIllegalAddress here was immediately preceded by AITER reporting an untuned shape in bf16_tuned_gemm.csv, at four different M values across four stages, and the one fault with a full Python stack landed in wvSplitK from the KDA in_proj_qkvgfab projection - a bf16 linear projection, not an expert matmul. Config-shape changes (batch size, rejection method, draft attention backend, --language-model-only) each moved or kept the fault rather than removing it. Costs: linear GEMMs now use untuned torch/Triton paths, so throughput will be lower, and this deviates from the upstream extra_env block. VLLM_ROCM_USE_SKINNY_GEMM=0 previously traded the fault for HBM exhaustion at 16384; this recipe runs 4096, but that pairing is unmeasured. Also corrects the header's prefix-caching claim: the asserted tokens_per_block/tokens_per_hash failure does not exist. Measured without the flag the server is ready and generates correctly with zero assertions (GPU KV 1,898,006 off vs 1,893,092 on). The flag is kept for measurement validity - 98.63% theoretical prefix cache hit.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30367040157 |
1 participant
Adds the DSpark speculative-decoding variant of the Kimi-K3 MXFP4 MI355X agentic
recipe:
kimik3-fp4-mi355x-vllm-agentic-mtp.4 points, TP8, DRAM KV offload via
vllm-simple, conc 1/4/8/16 (mirrors thenon-MTP ladder for direct comparison), plus one eval point.
Acceptance follows the committed golden AL
Per
docs/PR_REVIEW_CHECKLIST.mdrule 10, throughput points simulateacceptance rather than measuring it, pinned to the committed curve in
golden_al_distribution/kimik3_dspark.yaml(kimi-k3/thinking_on/num_speculative_tokens=7→ AL 3.78):The eval point deliberately uses real verification — synthetic acceptance
bypasses verification and would zero the eval score.
The recipe also hard-fails if the pinned AL drifts from the committed file or
cannot be parsed, so a recollected curve cannot silently leave a stale constant
behind.
Deviations from the upstream recipe, and why
"attention_backend": "FLASHINFER_MLA"platforms/rocm.pyrejects it (Reason: ['ImportError']). Verified by A/B on 8×MI355X: with the pin the draft fails at construction (dspark_mla.py:59→nvidia/mla.py:267 get_attn_backend), without it the server reaches ready and generates. Server runsTRITON_ATTN, as the MiniMax-M3 ROCm MTP recipe does.--gpu-memory-utilization 0.95--max-num-seqs 32min(CONC, 32)"lazy_offload": "false"falsebool("false")isTruein Python, so the string silently selects lazy. The boolean gives the intended eager offload.--enable-prefix-cachingtokens_per_block=1048576 not divisible by tokens_per_hash=3145728. Hybrid models (e.g. Mamba+Attention) need --enable-prefix-caching. K3 is hybrid (69 KDA + 24 gated MLA).Notes
vllm/models/kimi_k3/amd/dspark*.pyand the registry hardcodesvllm.models.kimi_k3.nvidia.dspark_mlawith no platform dispatch. The basemodel still resolves to
kimi_k3.amd.model.Inferact/Kimi-K3-DSpark, 7.1 GB) are pre-staged on the sharedNFS HF hub cache alongside the 1.5 TB target.
gfx950 before this was written.
Not yet validated
dram-utilization: 0.95leaves only ~12–21 GB host freeonce the pool commits (measured). Lowering it is a one-line change if the
sweep shows host pressure.
TimeoutError: RPC call to sample_tokens timed out.was seen on a TEP8 rowin the sibling non-MTP sweep, which shares this eager-offload config. If the
same hang appears here, the offload memory setting is the first thing to look at.