[AMD][AgentX] Add DeepSeek-V4-Pro FP4 MI355X ATOM + MTP agentic-coding recipe #2346
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,120 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| set -x | ||
|
|
||
| # Agentic trace replay benchmark for DeepSeek-V4-Pro FP4 on MI355X using ATOM. | ||
| # | ||
| # Serving flags follow the validated MI355X recipe from | ||
| # https://github.com/ROCm/ATOM/blob/main/recipes/DeepSeek-V4-Agentic-Benchmark.md | ||
| # Image is configured in amd-master.yaml. | ||
| # | ||
| # Required env vars: | ||
| # MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR | ||
| # | ||
| # KV_OFFLOADING=dram is not enabled. | ||
|
|
||
| source "$(dirname "$0")/../../benchmark_lib.sh" | ||
|
|
||
| check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION | ||
|
|
||
| if [[ -n "${SLURM_JOB_ID:-}" ]]; then | ||
| echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}" | ||
| fi | ||
|
|
||
| # `hf download` creates the target dir if missing and is itself idempotent. | ||
| # When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE | ||
| # Either way, MODEL_PATH is what the server is launched with. | ||
| if [[ -n "${MODEL_PATH:-}" ]]; then | ||
| if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then | ||
| hf download "$MODEL" --local-dir "$MODEL_PATH" | ||
| fi | ||
| else | ||
| hf download "$MODEL" | ||
| export MODEL_PATH="$MODEL" | ||
| fi | ||
|
|
||
| if [ -n "${ROCR_VISIBLE_DEVICES:-}" ]; then | ||
| export HIP_VISIBLE_DEVICES="$ROCR_VISIBLE_DEVICES" | ||
| fi | ||
|
|
||
| # ---- Resolve traces and install deps ---------------------------------------- | ||
| resolve_trace_source | ||
| install_agentic_deps | ||
|
|
||
| # Nightly ROCm image may be missing runtime deps; ensure they are present. | ||
| agentic_pip_install --quiet Pillow fastapi uvicorn | ||
|
|
||
| export AIPERF_HTTP_TCP_USER_TIMEOUT=900000 | ||
|
|
||
| # DeepSeek-V4-Pro weights are large; engine startup can exceed default 600s. | ||
| export VLLM_ENGINE_READY_TIMEOUT_S=3600 | ||
|
|
||
| # vllm-project/vllm#43447 keeps local SWA prefix-cache tails sparsely, while | ||
| # vllm-project/vllm#44774 applies the same reachability policy to Mooncake's | ||
| # store mask. 32k matches the trace-replay tuning validated for this workload. | ||
| export VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768 | ||
|
|
||
| # VLLM_PREFIX_CACHE_RETENTION_INTERVAL only applies to sliding-window/Mamba | ||
| # models; this vLLM build raises ValueError if it is set for DSv4. | ||
|
|
||
| # ---- LLM server config ---------------------------------------------------------- | ||
| SERVER_LOG="$RESULT_DIR/server.log" | ||
| VLLM_BACKEND_PORT="$PORT" | ||
| mkdir -p "$RESULT_DIR" | ||
|
|
||
| SERVER_PID="" | ||
| ROUTER_PID="" | ||
| MOONCAKE_MASTER_PID="" | ||
|
|
||
| PARALLEL_ARGS=(-tp "$TP") | ||
| SPEC_ARGS=(--method mtp --num-speculative-tokens 3 ) | ||
| OFFLOAD_ARGS=() | ||
| if [ "$DP_ATTENTION" = "true" ]; then | ||
| PARALLEL_ARGS=(-tp "$TP" --enable-dp-attention) | ||
| fi | ||
| if [ "$EP_SIZE" -gt 1 ]; then | ||
| PARALLEL_ARGS+=(--enable-expert-parallel) | ||
| fi | ||
|
|
||
| # AgentX concurrency counts live session trees, not individual requests. | ||
| # Subagent fan-out can push instantaneous request concurrency above CONC, so | ||
| # leave 2x headroom rather than clipping those bursts at the scheduler. | ||
| MAX_NUM_SEQS=$((2 * CONC)) | ||
|
|
||
| echo "Starting atom server..." | ||
| set -x | ||
| export AITER_BF16_FP8_MOE_BOUND=0 | ||
| export ATOM_MOE_GU_ITLV=1 | ||
| export AITER_LOG_LEVEL=WARNING | ||
| export ATOM_DISABLE_MMAP=true | ||
| export MEM_FRAC_STATIC=0.9 | ||
|
|
||
| { set +x; } 2>/dev/null | ||
| ATOM_CMD=( | ||
| python3 -m atom.entrypoints.openai_server | ||
| --model "$MODEL_PATH" | ||
| --server-port "$VLLM_BACKEND_PORT" | ||
| "${PARALLEL_ARGS[@]}" | ||
| "${SPEC_ARGS[@]}" | ||
| --kv_cache_dtype fp8 | ||
| --trust-remote-code | ||
| --enable_prefix_caching | ||
| --max-num-seqs "$MAX_NUM_SEQS" | ||
| --gpu-memory-utilization "$MEM_FRAC_STATIC" | ||
| "${OFFLOAD_ARGS[@]}" | ||
| ) | ||
|
|
||
| printf '%q ' "${ATOM_CMD[@]}" | tee "$RESULT_DIR/atom_command.txt" | ||
| printf '\n' | tee -a "$RESULT_DIR/atom_command.txt" | ||
| "${ATOM_CMD[@]}" > "$SERVER_LOG" 2>&1 & | ||
| SERVER_PID=$! | ||
| echo "Server PID: $SERVER_PID" | ||
|
|
||
| wait_for_server_ready --port "$VLLM_BACKEND_PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID" | ||
|
|
||
| if [ "${EVAL_ONLY}" = "true" ]; then | ||
| run_eval --port "$PORT" | ||
| else | ||
| build_replay_cmd "$RESULT_DIR" | ||
| run_agentic_replay_and_write_outputs "$RESULT_DIR" | ||
| fi | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1356,6 +1356,20 @@ dsv4-fp4-mi355x-atom-mtp: | |
| search-space: | ||
| - { tp: 8, ep: 1, conc-start: 1, conc-end: 1024, spec-decoding: mtp } | ||
|
|
||
| dsv4-fp4-mi355x-atom-agentic-mtp: | ||
| image: rocm/atom-dev:nightly_202607211523 | ||
| model: deepseek-ai/DeepSeek-V4-Pro | ||
| model-prefix: dsv4 | ||
| runner: cluster:mi355x-amds | ||
| precision: fp4 | ||
| framework: atom | ||
| multinode: false | ||
| scenarios: | ||
| agentic-coding: | ||
| - dram-utilization: 0.60 | ||
| search-space: | ||
|
claude[bot] marked this conversation as resolved.
Comment on lines
+1359
to
+1370
Contributor
There was a problem hiding this comment. 🟡 This PR's title and description (Summary/Test plan) are English-only, but AGENTS.md mandates that every PR title and body include a Simplified Chinese counterpart (title format Extended reasoning...AGENTS.md (linked from the repo's root This PR's title, The convention is not aspirational — it is actively followed in this repository's recent history. Step-by-step proof:
Fix: Rename the PR title to append a Chinese translation (e.g. This is a metadata/process issue only — it does not affect the benchmark config, the launch script, or CI behavior, so it should not block merge on its own, but it should be corrected to keep the repository's PR history consistent with its own documented convention. |
||
| - { tp: 8, ep: 1, dp-attn: false, kv-offloading: none, conc-list: [1, 2, 4, 8, 16, 32, 48], spec-decoding: mtp } | ||
|
|
||
| qwen3.5-bf16-mi325x-sglang-mtp: | ||
| image: lmsysorg/sglang:v0.5.12-rocm720-mi30x | ||
| model: Qwen/Qwen3.5-397B-A17B | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 The ATOM_CMD server launch omits
--trust-remote-code, which every other DeepSeek-V4-Pro launch script in the repo (both siblingfixed_seq_len/dsv4_fp4_mi355x_atom_mtp.shanddsv4_fp4_mi355x_atom.shon the sameatom.entrypoints.openai_serverentrypoint, plus the agentic vLLM/SGLang/H200 DSv4 scripts) passes. DeepSeek-V4-Pro requires custom remote modeling code, so without this flag the ATOM server will fail to load the model/tokenizer at startup, causingwait_for_server_readyto time out and the entire recipe to fail.Extended reasoning...
The bug:
benchmarks/single_node/agentic/dsv4_fp4_mi355x_atom_mtp.shbuildsATOM_CMD(lines 68-79 in the diff, thepython3 -m atom.entrypoints.openai_server ...array) without the--trust-remote-codeflag. This is the exact command that gets exec'd to start the model server, and its readiness is polled bywait_for_server_readya few lines later.Why this breaks the recipe: DeepSeek-V4-Pro is a custom architecture that ships remote modeling/tokenizer code on the Hugging Face Hub. Loaders (transformers/AITER/ATOM) require
trust_remote_code=True(surfaced here via the--trust-remote-codeCLI flag) to load such checkpoints; omitting it causes the loader to refuse to proceed at startup rather than silently falling back. This isn't speculative — every other launch script in the repo serving the exact same model through the exact same entrypoint passes the flag:benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_atom_mtp.sh(sameatom.entrypoints.openai_server, same model, same--method mtp) — passes--trust-remote-codeat line 49 (and again at line 74 in a second invocation).benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_atom.sh— passes it at line 63 (and line 85).I verified both fixed_seq_len sibling scripts directly and confirmed the flag is present at those exact lines, and it is conspicuously the only DSv4-Pro launch script in the repo missing it.
Why nothing else catches this: There's no schema/lint validation on these ATOM_CMD argument arrays, and
set -euo pipefailonly fails on non-zero exit of the foreground shell — the server is launched in the background ("${ATOM_CMD[@]}" > "$SERVER_LOG" 2>&1 &), so a crash-on-load failure surfaces only viawait_for_server_readytiming out after polling the port, not as an immediate script error. That makes the omission easy to introduce (e.g. by adapting a different framework's agentic launch script as a template) and easy to miss in review, since the script is otherwise syntactically valid and passesbash -n.Step-by-step proof of impact:
MODEL=deepseek-ai/DeepSeek-V4-Pro.ATOM_CMDwithout--trust-remote-code."${ATOM_CMD[@]}" > "$SERVER_LOG" 2>&1 &launches the ATOMopenai_serverprocess in the background.trust_remote_code=Trueand the process exits, logging the failure to$SERVER_LOG.wait_for_server_ready --port "$VLLM_BACKEND_PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"polls the port, finds it never opens (and/or detects the dead PID), and eventually times out/fails.dsv4-fp4-mi355x-atom-agentic-mtp's search-space (all 7 concurrency levels inconfigs/amd-master.yaml) fails at the server-start step — no benchmark data is ever collected.Fix: Add
--trust-remote-codeto theATOM_CMDarray, matching the siblingfixed_seq_len/dsv4_fp4_mi355x_atom_mtp.shscript, e.g. inserting it alongside--kv_cache_dtype fp8or near--served-model-name "$MODEL".